We’re alive and kicking

Escrito por Mark Forrester on abril 27, 2012 Blog, News.

It’s been an incredibly tough few days at WooThemes, for ourselves and a lot of our users trying to use our services. We were hacked, and badly. Our lifeblood, the WooThemes database, and all the content on our server was deleted. Not only that, but the backups were deleted and all traces of how they got onto our server too. Leaving us with scraps to work from in re-building the mothership.

We panicked, we yelped, we formulated a game plan and then we grafted like never before (the picture above showing the team going loco on very little sleep & lots of caffeine) – acting as quickly as humanly ninjaly possible. We responded transparently to our loyal community, keeping everyone informed of our recovery progress on a temporary P2 installation.

Where we’re at

Long story short, as we’ll save the juicy details for another blog post, we’re now back up online, with a few small glitches and missing functionality that we’ll be adding back over the next few days. We’ve also moved host from VPS to the market leaders in hosting large WordPress specific sites – WPEngine where we now have a monster dedicated server with backups upon backups of backups, and super tight security. A special thank you to WPEngine’s Jason Cohen, who spent most his Birthday setting up our new server, and his team who have really re-assured us that our website is now finally very safe and secure.

Users should be able to login to their accounts and access their themes. A select few, unfortunately some of our newest users, might be prompted for some personal details to verify their account, and have the facility from their dashboard to select their themes again.

Likewise, some of our recent club subscribers might need to send us some personal details so that we can link your recurring subscriptions back to your club membership. We promise a very quick and efficient fix.

Important note: None of your private information was stolen, and as your card details and personal data were kept safe somewhere else we need to re-link some transactions to user accounts that were deleted during the malicious hack.

Thank you, thank you, thank you

We were absolutely blown away by your support, you could have cursed us for the downtime and how it effected you and your clients, but instead you sent us pictures of beer, bikini girls and pizza to lift the team spirits, you tweeted us and emailed us words of encouragement, and one user even went as far as writing us a song, entitled “Injustice – WooThemes Comeback” . Thank you. It really unified the team and helped us get through this tough time.

Without sounding like an Oscar acceptance speech I need to say a massive thank you to the whole team for really pulling together and demonstrating the awesomeness that WooThemes is capable of. A very large and special thank you to Jeff and Warren who worked tirelessly through many a night fixing the unfixable mothership!

Woo fuel that powered Warren and Jeff through a good few sleepless nights at WooHQ.
Please do contact us if your account is acting funny and you haven’t been prompted to send us information. The team are ready to help out.
cta-banner-10-product-page-v2_2x

84 Responses

  1. Laurent
    abril 28, 2012 at 8:14 pm #

    Welcome back ! 😛

    • AJ Clarke
      abril 29, 2012 at 5:45 pm #

      Yep. Welcome back!

  2. Wil
    abril 28, 2012 at 8:22 pm #

    Good to see you back! That first descriptive paragraph is down-right scary!!

    • Praveen Gowda I V
      abril 29, 2012 at 8:11 am #

      Nice to have you guys back !!

  3. Russ
    abril 28, 2012 at 8:26 pm #

    Great work guys, you kept everyone informed and kicked ass on getting back online… Great to see you back.
    From the team at Number8Media NZ

  4. James
    abril 28, 2012 at 8:37 pm #

    Welcome back online WooThemes!

    After dealing with this, there is nothing too big for the ninjas to handle! Although you all probably deserve a proper nights sleep and a day off first 🙂

    Having followed your blog and tweets for the last few days, I’m now in no doubt that WooThemes is the most professional development team out there! You did a fantastic job of keeping the community informed.

    Nice to have you back.

  5. Regiosfera
    abril 28, 2012 at 8:42 pm #

    Congratulations, It was a fantastic job. I hope never repeat this.

  6. Kulturchaot
    abril 28, 2012 at 8:44 pm #

    Willkommen zurück 😉

  7. Ostheimer Webdesign
    abril 28, 2012 at 8:46 pm #

    Welcome back, I am glad you made it!

  8. shawn
    abril 28, 2012 at 9:18 pm #

    I look forward to the upcoming article on ‘how to properly secure’ a WP server and some of the tricks you may have learned along the way. This is just yet another reminder that it can happen to anyone and I’m hoping there will be some awesome insights to share with the community.

    • Mark Forrester
      abril 28, 2012 at 9:22 pm #

      Definitely Shawn. We’ve learnt heaps through our own mistakes and misfortunes. Watch this space!

      • DeepTitanic
        abril 28, 2012 at 9:45 pm #

        I also look forward to any insight/tips that you guys can provide (for WC too). I just don’t think that I could take being digitally violated that!

        I would also like to say that you shouldn’t be in a rush to go back into production. I’d be quite happy for you to do what ever it takes to get secure.

        Good Luck!

        • Mark Forrester
          abril 28, 2012 at 9:59 pm #

          Thanks for the support. Product development has gone on temporary hold, but don’t worry we’ve got loads of goodness waiting to be released soon.

          • DeepTitanic
            abril 28, 2012 at 10:04 pm #

            No Rush .. So long as I can get to themes and the forum I’m happy.

      • Martin
        abril 29, 2012 at 2:24 am #

        Positive from a negative
        Online hacks are now day to day life for web developers, and how to try to best protect against these, giving the customer the best web experience while keeping their data secure is an impossible task
        I look forward to the next blog post from the ninjas with tips regarding securing wp / hosting
        Props to the updates

    • jeffikus
      abril 28, 2012 at 9:23 pm #

      Oh yes, we’ve definitely learnt a lot!

  9. deeptitanic
    abril 28, 2012 at 9:20 pm #

    Welcome Back!

  10. Grant Griffiths
    abril 28, 2012 at 9:23 pm #

    Hey guys. Glad to see you are back up and running. Heard good things about WPEngine too.

  11. jamie
    abril 28, 2012 at 9:24 pm #

    Great job guys , lovely to have you back 🙂

  12. Dre
    abril 28, 2012 at 9:27 pm #

    Great to see you all back at it. Well handled, folks! Well handled!

    Dre

  13. Alyssa
    abril 28, 2012 at 9:51 pm #

    So glad to have you back! It really can happen to anyone, and I along with everyone else REALLY appreciate the updates along the way!!

  14. Cyrus
    abril 28, 2012 at 10:33 pm #

    Glad to see you guys are back. Could you guys share more info on your experience with VPS.net. We moved with those guys because we saw you guys were with them … a big name client in the WP community. Now we are scared we might get hacked too 🙂

    I am glad to see you guys back though. Woo rocks 🙂

  15. Kevin Gilbert
    abril 28, 2012 at 10:34 pm #

    You guys have demonstrated, once again, why I choose to do business with you. You handle things professionally at all times and in the face of adversity, you pull together and prove what can be accomplished when a group of folks work together toward a common goal. Well played, WooTeam!

  16. Stenio Ribeiro
    abril 28, 2012 at 10:44 pm #

    wOOO!

  17. Marek
    abril 28, 2012 at 11:13 pm #

    Glad you guys are back, hope you’ll give us WP users insight into how/what happened, and how we can prevent. We were hacked as well, about 3 weeks ago and two of our clients. Sigh. Keep up the good Woork! 😛

  18. Johnny
    abril 28, 2012 at 11:47 pm #

    Glad to have you back! Or almost back – my membership is listed as “Expired” when I lock in..

    I presume its a general problem?

    • djpechi
      abril 29, 2012 at 1:23 am #

      Same problem here :S any ideas?

      • Joe Watts
        abril 29, 2012 at 4:07 pm #

        Same problem for me.

    • Peter Ricci
      abril 30, 2012 at 4:12 am #

      Same issues here, despite trying clearing cache as explained etc. http://woo.com/woothemes-server-status/

  19. Tie
    abril 29, 2012 at 1:21 am #

    You guys ROCK! (From our whole team at euphoric MEDIA)

    I stay tuned into all of your updates on the progress to getting back up and I have to say you guys have kicked some serious being hacked and getting back online ass!

    I feel so much more connected to the whole WO0 Themes company and team now too after seeing the pictures of everyone working hard at headquarters and how straight forward and personal the team was with all of us and how awesome the support was during the whole thing.

    I had a client’s theme I needed a backup for and the theme got sent to me within 2 hours of requesting it!!

    Thank you and I am sold for life! WOO Themes are the best!

    Peace In,

    Tie Love

  20. Rick Hubbard
    abril 29, 2012 at 1:28 am #

    Commendations to the WooThemes Ninjas…everyone is thrilled that such a great company is back on the air.

    Double commendations for keeping WooFans up-to-date with your progress and status…it was like hearing that a best friend’s site was hacked–which, in a way, is what happened.

    Ninjas Rock!

  21. David
    abril 29, 2012 at 1:42 am #

    Glad to see you back online, hope things are getting back to normal for you. Don’t think there is anything that can quite prepare you for a bad hack. I’ve only had one really bad experience and I wouldn’t wish it on anyone. Glad you got things back up and running.

  22. Mairead
    abril 29, 2012 at 1:56 am #

    Just wanted to say well done. You all very obviously worked flat out to get this back up and running. Very impressed 🙂

  23. Douglas Karr
    abril 29, 2012 at 2:02 am #

    We’ve been converting all of our clients over to WPEngine and it’s been a great experience all around! Welcome! And kudos to WPEngine for reaching out.

  24. Wordpress Developer
    abril 29, 2012 at 2:29 am #

    Woo Hoo, Great to see you guys are back! I was very concerned if hacking was due to the poor software or scripts recently developed. Particularly WooCommerce which we are using for many of our clients. However, despite of the fact Woo Ninjas were busy getting woo back up they responded to my concerns and I thank them for Heroic Support! They also confirmed that WooCommerce is well built and there is nothing to worry about! Thanks guys and again glad to have you guys back!

  25. Derek
    abril 29, 2012 at 2:47 am #

    The blog looks great. Is the affiliate program going to be fixed? My old affiliate link turns out isn’t working now.

  26. Cain
    abril 29, 2012 at 3:36 am #

    Not that you would have thrown in the towel, but I truly appreciate the outstanding effort Woo put forth to get everything right side up. Thanx so much for being open and honest and still providing assistance throughout this ordeal. Congrats on your new server home at WPEngine. From my handful of dealings with them, they’re a decent lot; I think you’re in good hands.

    Welcome back!

  27. Anton Zuiker
    abril 29, 2012 at 4:04 am #

    Impressive. So glad you came through this. Didn’t know what I could do in my little corner of the world. so I sat and hoped for the best.

  28. lowell
    abril 29, 2012 at 5:04 am #

    Just passing along an exploit to your WooFramework I caught elsewhere on the interwebz:

    https://gist.github.com/2523147

    Looks like you guys just recovered from something major; sorry to have to be the bearer of more bad news.

    • Matt Stigall
      abril 29, 2012 at 5:16 am #

      Eeeek!

      Hopefully after the Wooteam gets some sleep they can fix this ASAP.

    • Mark Forrester
      abril 29, 2012 at 8:05 am #

      That has already patched in the latest version of the framework in all our themes.

  29. Matt Stigall
    abril 29, 2012 at 5:14 am #

    Do you have any information on the attack that you can share. I’ve seen quite a few forums and other websites being hit this weekend and don’t know if it’s a coordinated attack or not.

    • Mark Forrester
      abril 29, 2012 at 4:13 pm #

      I think it was an isolated incident, but we are still investigating the cause/culprit.

  30. Eric Zentner
    abril 29, 2012 at 5:40 am #

    Welcome back! You don’t know how much you’re gonna miss someone til they’re gone!

  31. Nguyen
    abril 29, 2012 at 5:45 am #

    Congratulations WooThemes returned. It seems that information about the affiliate program has not been restored, all previous income is lost and no signs of recovery 🙁

  32. DrewAPicture
    abril 29, 2012 at 8:26 am #

    Glad you guys managed to get back online. Looking forward to reading any details you’re willing to share on the server compromise in a future post. Keep on truckin!

  33. Schalk
    abril 29, 2012 at 11:21 am #

    Hi

    Excited to have you back, but I still can’t access any pages, beside this one…

    I cleared and ( force refreshed ) my browser cache, and also visited from another computer ( same internet connection ).

    I stil get a 403 Frobidden error: nginx/0.7.65

    However, when visiting from my iphone ( Vodacom cennection ) it works!

    Do I simply have to wait for my ISP (telkom) to get up to date, or can i do something about it.

    Regards

  34. Jack Sternfeld
    abril 29, 2012 at 12:04 pm #

    Welcome back guys. I suppose you’re working on the broken affiliate links? Please. Thanks.

  35. Trung Nguyen
    abril 29, 2012 at 1:43 pm #

    Welcome back, Woo – we waited for this time so long

  36. Frank McClung
    abril 29, 2012 at 2:41 pm #

    Fantastic comeback Woo! I immediately noticed a speed increase on the new site. Well done.

  37. Moses
    abril 29, 2012 at 4:29 pm #

    Good advert for Coke 😉 welcome back guys.

  38. Nick
    abril 29, 2012 at 5:00 pm #

    Congrats guys, glad to see you moving over to WP Engine. VPS.net should not be in business; after 30 days with them I had 30 support tickets open. And with over 20 complaints at the BBB, they should not be in business. You’re in good hands with WP Engine.

    I’m curious though, do you guys not store off-site backups and why couldn’t you recover with those?

    Nick

  39. kim
    maio 1, 2012 at 1:03 am #

    Happy to see you managed to pull it off. A reminder for everybody at the same time. Ouch!

  40. vrob
    maio 1, 2012 at 1:59 am #

    I think you’re crazy if you don’t check out CloudFlare. They prevent DDOS attacks like this and it actually makes me nervous that you’re trying to do it on your own.

  41. Al Johnson
    maio 1, 2012 at 2:15 pm #

    Why would someone do this?? Massive well done to you guys for sorting it out. Sounds like a mammoth effort!
    Rock On Ninja Styleee

  42. Morten Ross
    maio 1, 2012 at 2:18 pm #

    Hi,

    I just received what seems to be a newsletter from Adii with some dodgy URIs:

    http://woothemes.createsend3.com/t/y-l-jlnyhd-dhhhjjdjh-p/

    Is this legit or phish?

    Morten

  43. Calzo
    maio 1, 2012 at 2:47 pm #

    That’s all well and good but where’s Thursday’s drop? haha, just teasing. Glad you are back online!

  44. Phil
    maio 1, 2012 at 3:02 pm #

    Glad to hear you guys made it through. Feels good to come out the other end of something like this I bet.

  45. Joseph
    maio 1, 2012 at 3:35 pm #

    Great to see you all safe and sound !

  46. Companion
    maio 1, 2012 at 4:09 pm #

    Welcome back guys.

    I just wanted to say that I check Woothemes out a lot. The speed, transparency, and professionalism that you guys put into this recovery is really inspiring and in my opinion it will generate more memberships and interest for you.

    I certainly will be using only Woothemes for my future projects, because I know I can trust you not to let me down.

    Thank you.

  47. Chris
    maio 1, 2012 at 4:13 pm #

    offsite backups of your database are essential.

  48. Ronald
    maio 1, 2012 at 4:55 pm #

    Is updating within the theme menu not possible? I see

    Framework Update
    You have the latest version of WooFramework

    → Your version: 4.4.3

    That does not look good…

  49. Tho Huynh
    maio 1, 2012 at 5:22 pm #

    Glad to hear you’re back 🙂
    Love you all

  50. Jatin
    maio 1, 2012 at 5:31 pm #

    Welcome back.!!! I love your themes 😀

  51. machbio
    maio 1, 2012 at 7:21 pm #

    so how secure is your product WOO-COmmerce..

  52. Cameron
    maio 1, 2012 at 7:40 pm #

    No offsite Backup? Ouch. Even Google does tape backups of everything. -Yes tape. How did you manage to piece it all back together? 🙂

    • Ryan Ray
      maio 2, 2012 at 2:08 am #

      We did have offsite backup I believe, rsynced between locations. They found that too. 🙁

  53. joa
    maio 1, 2012 at 7:43 pm #

    Bravo to all the team !

    Take care against new attack and go straight 😉

    I hope your woo-commerce theme will improve against attack soon and kept safe ?

  54. phillip
    maio 1, 2012 at 9:16 pm #

    He got me on April 20th. Changed my wp-config on 20+ sites and put a photo of a dead baby-saying why do you kill children…

    Solution: find him and attached vice grips to all his appendages.

  55. Karl Steinmann
    maio 2, 2012 at 1:04 am #

    It’s constantly shocking and appalling that this sort of thing happens.

    But what’s even more shocking (and AMAZING, I should add) is how fast you got it back together considering the scale of devastation that was wrought!

    I’m truly astounded and really glad, both for selfish reasons (I bought a new package just days before Woo went down (no causal relationship, I assure you!)), and in general, because I love Woo and I know it to be a great company composed of really hard working and passionate people.

    Having seen what happened to a pretty cutting edge “tech” firm (you… Woo), there’s a lesson to be learned here by all of us.

    I’ve been looking at WPEngine myself. One thing everybody can do to protect their investment in time and energy, even if they can’t afford a premium host like WPEngine, is to make absolutely sure everything is backed up offsite everyday (preferably automatically). It’s invaluable insurance! I’m using blogVault.net and really like it (they have a 30 day free trial, too), but there are quite a number of options available insofar as WP back goes. Even WP itself is offering a plan these days…

    Anyway, best of luck on the rest of the resurrection, Woo ninjas. My kudos to you all. It’s truly amazing what you have done in such a short period of time. Many lesser firms would have just been… DONE! Fini. Stick a sword in it.

    Good job!

    🙂

  56. Anthony Vyner
    maio 2, 2012 at 1:09 am #

    You guys are amazing!
    Have a much better week!

  57. Valerie
    maio 2, 2012 at 2:35 am #

    fyi protection against ddos:

    http://blog.cloudflare.com/cloudflarewebops-for-everyone

  58. dave
    maio 2, 2012 at 7:27 am #

    alas my site and three others that I know of are still down as other hosts attempt to get it sorted

  59. 212
    maio 3, 2012 at 9:30 am #

    THANK YOU GUYS! YOU ARE AWESOME!

  60. alvin
    maio 4, 2012 at 8:13 am #

    Congratulation, Woothemes…
    what doesn’t kill you makes you stronger…

    🙂

    anyway…

    1) Do you know who is responsible in doing this to your team ?

    2) Why do this person does this bad thing to you ?

  61. Schalk
    maio 4, 2012 at 8:22 am #

    Hi,

    While I am also very impressed with the way you handles things, I also like to urge you to respond to my “expired subscription”. I have emailed the support@w and used the panic button contact form, but still no response.

    Even if you can add a few themes to my Dropbox, please private message me and I will let you know which ones.

    Hope to be fully alive and kicking soon!

    Regards
    s

  62. Joel
    maio 9, 2012 at 4:54 pm #

    I would like to purchase a theme, but your website is not working correctly.
    Nothing happens when I click the Buy It Now button and your ‘test a theme before you by it’ leads me to a blank page. When will you be accepting new accounts?

    • Magnus
      maio 9, 2012 at 11:27 pm #

      Which theme are you trying to purchase? Send us a mail through our contact form and we will help you out!

  63. Chris@Apple Roof Cleaning
    maio 10, 2012 at 7:07 am #

    Wow, that sucks being hacked, it was probably a competitor!
    But welcome back!

  64. Κατασκευή ιστοσελίδων
    maio 15, 2012 at 10:08 am #

    You are awasome. Good job !

  65. Jann
    maio 25, 2012 at 11:09 am #

    I have move all my client sites to WP Engine almost a year ago and I have to confirm that the experience on all levels has been absolutely amazing.

  66. Luiza Prigenzi
    maio 27, 2012 at 3:33 pm #

    Good to konw about you. You guys are really awesome!

  67. scott.b
    junho 4, 2012 at 4:36 pm #

    I really need to get myself a plush doll!

Trackbacks/Pingbacks

  1. Fantastic Customer Service: Grace Under Destruction – Woothemes | Schnikisms