Migrating Yoast.com’s webshop from EDD to WooCommerce

Written by Anton Timmermans on February 27, 2018 Blog, Customer Stories.

At Yoast, we optimize every aspect of websites’ performance. Our goal is to make the web a better place by making websites more usable, easier to navigate, faster, and more reliable.

In September 2017 we migrated our webshop from Easy Digital Downloads (EDD) to WooCommerce — it’s where we sell all the tools, products, and content available on yoast.com. In this post I’ll explain why we did it, what we built, and what the benefits will be for us and for our users in the future. WooCommerce has provided us the technical foundation to build on for years to come.

Why the change?

Yoast.com is experiencing incredible growth driven by two main components — the popularity of our plugin, and the expanding audience of our SEO blog. The combination of these two elements, however, makes yoast.com a challenging website to manage.

Historically, the site has evolved from a simple WordPress installation. Easy Digital Downloads (EDD) was added by Yoast founder Joost de Valk to facilitate selling premium plugins. We also used EDD to serve updates for all the Premium installations out there in the world.

Over the years we’ve made more and more tweaks to make sure EDD could still serve our needs. Making it possible for our customers to pay in either euros or dollars, for example, was an enormous effort. Since we wanted to add other currencies in the future, we needed to find a different solution.

When we looked into other options, we considered several needs:

  • Being able to support multi-currency purchases.
  • Being able to support recurring payments.
  • The cost of building integration and the maintainability of the platform.
  • Being able to support user accounts.
  • Creating an SEO platform.
  • Taking into account future compatibility, since we want to have a site that will still work in 5-10 years.

To achieve all of the above, we had to come up with a plan.

Ideation

In January 2017 we scheduled a meeting with our team of architects: Joost, Omar, Jip, and myself. We discussed what our needs were and how we could best meet them.

Our first conclusion was that we didn’t want to move away from WordPress. Even though we now have plugins for other platforms, we are WordPress fans and deeply care about its mission. WordPress is where our roots are, and it has worked very well for us as a CMS.

Why WooCommerce?

We then considered which eCommerce platform we wanted to use. We asked ourselves what the best eCommerce solution for WordPress is, and came to the conclusion that it’s WooCommerce. But, like EDD, WooCommerce didn’t support multiple currencies. So we needed a solution.

I’m a big fan of the idea that everything is a remix. We were inspired by the idea of using a different site for each language, an idea that’s been executed well by the MultilingualPress plugin. So we remixed it: to avoid having to write a ton of custom code, we decided to use a multisite installation.

As a result, WooCommerce doesn’t need to manage different currencies. On the dollar site, WooCommerce does everything in dollars. On the euro site, WooCommerce does everything in euros. This also makes it relatively easy to add a new currency. We’d just add a new site and copy the settings. In combination with MultilingualPress, in the future this will also make it possible for us to support different languages.

yoast.com USD and EU
yoast.com offers users a choice between U.S. dollars and euro, via switcher.

WooCommerce ended up being a great fit for yoast.com for several reasons:

  • A good data model.
  • A large ecosystem.
  • Built-in REST API and Webhook support.
  • The possibility to dogfood our plugin in combination with WooCommerce.
  • WooCommerce solves the eCommerce domain. We didn’t want to spend much time working on the eCommerce domain. SEO is our expertise, so we want to focus on SEO.
  • WooCommerce has a vibrant community in which we can participate.

eCommerce: SKUs, historical records, and refunds

I want to focus for a moment on the eCommerce domain. One aspect to consider here is the SKU. Every product has an SKU — a unique identifier. An SKU might sound like something superfluous. Why do I need an SKU when I have an ID in the database? But then you realize that every financial department of every company already has this system in place. SKUs are not auto-incremented — you need a way to track products in a way that is unambiguous. Names are not suited for this. So on yoast.com the SKU is now the same number our finance department uses to track products.

Another requirement was to have a historical record of every purchase. WooCommerce does this by default. Once an order is complete, that’s it. All the data is immutable. If a customer changes their name, address, or email, the new information is only used for new orders. For a developer this may sound strange, but it is actually a strength. You wouldn’t want your bank to change transactions after the fact either — once they are done, they are done. Any mutation is a new transaction.

Having a historical record makes the biggest difference when it comes to refunds. In the old setup, refunded orders would disappear from our exports, which would change that month’s revenue. In WooCommerce, every refund has its own date and amount. So once a month is done, it is really done.

In WooCommerce, every refund has its own date and amount. So once a month is done, it is really done.

Technical details of migrating Yoast.com to WooCommerce

Migration

We have a lot of historical data, which meant we had to migrate a large number of orders. We chose a tool called pandas, a tool designed to handle large datasets, and a good choice for migrating it all at once. If we planned another migration we would probably go in a different direction — we would slowly format historical data over months. That would drastically reduce the amount of data to migrate on the day of the actual move.

There was one main pain point that made the process more complex than it needed to be. Because all the data was saved in one meta value in the database, it needed to be unserialized. Python doesn’t have an efficient and correct way to do PHP unserialization, so we ended up shelling out to PHP to be able to unserialize this data.

Aggregating the data

You might have noticed that we now have two sites. But we still want to have an overview of all the data in one location. This is one of the reasons we build MyYoast. All orders we receive are synched to MyYoast. This means that MyYoast knows everything.

WooCommerce has a system to sync all orders: webhooks. The biggest downside of webhooks, in general, is that if the receiving system is down, the webhook is not received. When looking at webhooks’ code, you find the following snippet:

“`
/*
* Let other plugins intercept deliver for some messages queue like rabbit/zeromq
*/
return apply_filters( ‘woocommerce_webhook_should_deliver’, $should_deliver, $this, $arg );
“`

We considered using a message queue as suggested in the comment, but we went with a simpler approach.

There is a delayed_job project from the Ruby on Rails community. We found a PHP port which suited our needs. Delayed job does what its name suggests: it schedules a job to be executed later. This means a job can never be lost. And if the job fails it will be retried four more times. After that, the job can be used for debugging what went wrong. This gives us a really robust setup.

WebhookJobs

We configured WooCommerce to try to send webhooks to our custom dummy URL: `http://my-yoast-job.url`. This ensures that we can catch these requests from the code. We have one `Scheduler` class which is responsible to schedule jobs. It catches the webhook requests and turns them into jobs instead.

JobClasses

We have several classes that handle jobs, one for each of the fundamental building blocks of yoast.com. The jobs are then handled by a worker in the background. This process is quick, so a customer will instantly see their products in MyYoast.

WordPress as an application

If you have never read The Twelve-Factor App, I would highly recommend you do so. It suggests a very robust framework for web-application development. How does this translate to a WordPress context? Better than you might think.

WordPress has a few unique features that make it harder to set up. But you can work around those to get a robust WordPress installation. If you only do one of these, it should be the dependencies factor. Start using Composer! It is here, and it works. Rarst created a good overview of how to use Composer for WordPress development. Auto-updating is not an option for a complex web application. So you need a different solution, and Composer does a very good job.

Performance

We started out without using object cache. That turned out to be a bad idea. If you handle a substantial volume of sales using WooCommerce you need object caching. Without it, an order model needs to be completely reinstantiated on every request concerning that order. It’s very costly, because the order model is required on a many pages. At the very minimum, all checkout pages need to access the order.

We also created a few pull requests on WooCommerce itself to improve performance without caching. One of these was a modification to cache the currencies. Before the modification, all currencies would be translated every time any piece of code needed just one currency. That is very inefficient considering that each shop only runs in one currency.

One thing we could not simply fix inside WooCommerce itself was searching for orders and subscriptions. Doing this would result in a query that could take down our whole website — this query contains many searches for post meta, and those take a long time. We worked around this issue by building our own search functionality on top of MyYoast.

The meta search issue would also be solved if WooCommerce used custom tables, which is luckily already on the WooCommerce technical roadmap. This issue also informed our thinking about our own plugin. Can we reduce the amount of meta keys that Yoast SEO creates? As a result, we’ve added the creation of a custom table to our own technical roadmap, too.

Current list of WordPress plugins and WooCommerce extensions

To give a comprehensive idea of where we landed, here is the list of plugins we currently use that are relevant to our site’s function as a store. The full list contains 55 active plugins.

Current list of WordPress plugins and WooCommerce extensions

Yoast.com now runs WooCommerce
Take a look for yourself over at yoast.com, now powered by WooCommerce.

Takeaways and results

We launched this new platform on August 29th, 2017. In the weeks that followed we have greatly improved the site’s performance. We’ve achieved all of the goals we wanted to hit. Since the migration we have also started to build a lot of other things. Our support team now has a much easier time refunding and transferring accounts. We have migrated our Yoast academy to the multisite. And we have a bunch of exciting plans for 2018, including:

  • Adding a shop where we sell in British pounds. Thanks to our past choices and our move to WooCommerce, we can do this in a day, it will require almost no additional code. Adding a sales dashboard for our sales team.
  • Adding a sales dashboard for our sales team.
  • Adding Composer support to our premium plugins.

WooCommerce has given Yoast the technical foundation on which to build and improve our platform into the next decade. Are you considering migrating an existing webshop to another platform? Have you done this already? Share your experience in the comments.

Unlock recurring revenue with WooCommerce Subscriptions

12 Responses

  1. Danny van Kooten
    February 27, 2018 at 4:11 pm #

    Nice job, Yoast team!

    I actually went through a very similar process last year, switching from EDD to our custom built shop (in Golang) which also accepts multiple currencies. It was a pain to extract the (maybe?) serialized data out of the various meta entries but it was absolutely worth it in the end. Having all data in a sane and normalized data structure now is the bomb.

    Interesting to read that you used Pandas for the data migration, we’re using it to spit out quarterly reports.

    Anyway, keep up the good work and enjoy your new system. 🙂

    • Omar Reiss
      February 27, 2018 at 4:41 pm #

      ❤️

    • Anton Timmermans
      February 27, 2018 at 6:18 pm #

      Thanks for the kind words.

      Yeah, I really like pandas.

  2. Álvaro Góis
    February 28, 2018 at 12:22 pm #

    I remember talking to the Inpsyde crew about MultilingualPress compatibility with WooCommerce, back in WordCamp Europe in Sevilla. They were hoping for a solution that would work (and so did I …). But then, in WordCamp Europe in Paris, we talked again and they seemed to have given up trying. What has changed in the meantime? Or is it your custom code that it’s making it work?

    Multisite still seems to me the best bet on a multilingual WordPress website, and I think the UI/UX is also better than the usual multilingual plugins, specially than WPML.

    Thanks.

    • Matty Cohen
      March 1, 2018 at 1:53 pm #

      Hi Álvaro,

      Multilingualpress uses WordPress Multisite to make multiple languages work. It has a great feature for copying the content of an existing site, as well, which is a great time-saver. WooCommerce and Multilingualpress work well together, because it’s possible to transact in multiple currencies across multisite stores, offer different prices according to currency, and different gateways at checkout.

  3. seo şefi
    February 28, 2018 at 4:07 pm #

    Do you have support for all languages? plus pdf document for settings made with yoast

  4. Megri Digitizing UK
    March 1, 2018 at 1:34 pm #

    Ahhh, great work. I personally find it to be very helpful, hope that you will keep on sharing insightful ideas with us.

    • Anton Timmermans
      March 2, 2018 at 11:33 am #

      Thanks! We will definitely write more posts about our progress on our dev blog.

  5. Joshua
    March 1, 2018 at 6:23 pm #

    I just did a conversion from Shopify to Woo. We used Pandas too! Wish I’d seen this post before we started.

    There is very little information out there about multi-site woocommerce, technical aspects of managing different countries with woo (setup, SEO, performance, day to day management for admins), and migrations to woocommerce from other carts.

    Any chance you guys write something more in-depth and technical?

    • Anton Timmermans
      March 2, 2018 at 11:36 am #

      Thank you for your feedback. I definitely want to write some more posts that go in depth. I also want to write about MyYoast because that has a lot of cool stuff too.

      • Michael Rimbach
        March 9, 2018 at 1:23 pm #

        I can’t wait for it!

  6. Ajansad Murat
    March 20, 2018 at 2:13 pm #

    Thanks for the woocommerce details. we have also been using the woocommerce for the last 1 year. SEO performance, design performance and conversion optimization are super easy in woocommerce platform.