Migrating Yoast.com’s webshop from EDD to WooCommerce

At Yoast, we optimize every aspect of websites’ performance. Our goal is to make the web a better place by making websites more usable, easier to navigate, faster, and more reliable.

In September 2017 we migrated our webshop from Easy Digital Downloads (EDD) to WooCommerce — it’s where we sell all the tools, products, and content available on yoast.com. In this post I’ll explain why we did it, what we built, and what the benefits will be for us and for our users in the future. WooCommerce has provided us the technical foundation to build on for years to come.

Why the change?

Yoast.com is experiencing incredible growth driven by two main components — the popularity of our plugin, and the expanding audience of our SEO blog. The combination of these two elements, however, makes yoast.com a challenging website to manage.

Historically, the site has evolved from a simple WordPress installation. Easy Digital Downloads (EDD) was added by Yoast founder Joost de Valk to facilitate selling premium plugins. We also used EDD to serve updates for all the Premium installations out there in the world.

Over the years we’ve made more and more tweaks to make sure EDD could still serve our needs. Making it possible for our customers to pay in either euros or dollars, for example, was an enormous effort. Since we wanted to add other currencies in the future, we needed to find a different solution.

When we looked into other options, we considered several needs:

• Being able to support multi-currency purchases.
• Being able to support recurring payments.
• The cost of building integration and the maintainability of the platform.
• Being able to support user accounts.
• Creating an SEO platform.
• Taking into account future compatibility, since we want to have a site that will still work in 5-10 years.

To achieve all of the above, we had to come up with a plan.

Ideation

In January 2017 we scheduled a meeting with our team of architects: Joost, Omar, Jip, and myself. We discussed what our needs were and how we could best meet them.

Our first conclusion was that we didn’t want to move away from WordPress. Even though we now have plugins for other platforms, we are WordPress fans and deeply care about its mission. WordPress is where our roots are, and it has worked very well for us as a CMS.

Why WooCommerce?

We then considered which eCommerce platform we wanted to use. We asked ourselves what the best eCommerce solution for WordPress is, and came to the conclusion that it’s WooCommerce. But, like EDD, WooCommerce didn’t support multiple currencies. So we needed a solution.

I’m a big fan of the idea that everything is a remix. We were inspired by the idea of using a different site for each language, an idea that’s been executed well by the MultilingualPress plugin. So we remixed it: to avoid having to write a ton of custom code, we decided to use a multisite installation.

As a result, WooCommerce doesn’t need to manage different currencies. On the dollar site, WooCommerce does everything in dollars. On the euro site, WooCommerce does everything in euros. This also makes it relatively easy to add a new currency. We’d just add a new site and copy the settings. In combination with MultilingualPress, in the future this will also make it possible for us to support different languages.

WooCommerce ended up being a great fit for yoast.com for several reasons:

• A good data model.
• A large ecosystem.
• Built-in REST API and Webhook support.
• The possibility to dogfood our plugin in combination with WooCommerce.
• WooCommerce solves the eCommerce domain. We didn’t want to spend much time working on the eCommerce domain. SEO is our expertise, so we want to focus on SEO.
• WooCommerce has a vibrant community in which we can participate.

eCommerce: SKUs, historical records, and refunds

I want to focus for a moment on the eCommerce domain. One aspect to consider here is the SKU. Every product has an SKU — a unique identifier. An SKU might sound like something superfluous. Why do I need an SKU when I have an ID in the database? But then you realize that every financial department of every company already has this system in place. SKUs are not auto-incremented — you need a way to track products in a way that is unambiguous. Names are not suited for this. So on yoast.com the SKU is now the same number our finance department uses to track products.

Another requirement was to have a historical record of every purchase. WooCommerce does this by default. Once an order is complete, that’s it. All the data is immutable. If a customer changes their name, address, or email, the new information is only used for new orders. For a developer this may sound strange, but it is actually a strength. You wouldn’t want your bank to change transactions after the fact either — once they are done, they are done. Any mutation is a new transaction.

Having a historical record makes the biggest difference when it comes to refunds. In the old setup, refunded orders would disappear from our exports, which would change that month’s revenue. In WooCommerce, every refund has its own date and amount. So once a month is done, it is really done.

In WooCommerce, every refund has its own date and amount. So once a month is done, it is really done.

Technical details of migrating Yoast.com to WooCommerce

Migration

We have a lot of historical data, which meant we had to migrate a large number of orders. We chose a tool called pandas, a tool designed to handle large datasets, and a good choice for migrating it all at once. If we planned another migration we would probably go in a different direction — we would slowly format historical data over months. That would drastically reduce the amount of data to migrate on the day of the actual move.

There was one main pain point that made the process more complex than it needed to be. Because all the data was saved in one meta value in the database, it needed to be unserialized. Python doesn’t have an efficient and correct way to do PHP unserialization, so we ended up shelling out to PHP to be able to unserialize this data.

Aggregating the data

You might have noticed that we now have two sites. But we still want to have an overview of all the data in one location. This is one of the reasons we build MyYoast. All orders we receive are synched to MyYoast. This means that MyYoast knows everything.

WooCommerce has a system to sync all orders: webhooks. The biggest downside of webhooks, in general, is that if the receiving system is down, the webhook is not received. When looking at webhooks’ code, you find the following snippet:

“`
/*
* Let other plugins intercept deliver for some messages queue like rabbit/zeromq
*/
return apply_filters( ‘woocommerce_webhook_should_deliver’, $should_deliver, $this, $arg );
“`

We considered using a message queue as suggested in the comment, but we went with a simpler approach.

There is a delayed_job project from the Ruby on Rails community. We found a PHP port which suited our needs. Delayed job does what its name suggests: it schedules a job to be executed later. This means a job can never be lost. And if the job fails it will be retried four more times. After that, the job can be used for debugging what went wrong. This gives us a really robust setup.

We configured WooCommerce to try to send webhooks to our custom dummy URL: `http://my-yoast-job.url`. This ensures that we can catch these requests from the code. We have one `Scheduler` class which is responsible to schedule jobs. It catches the webhook requests and turns them into jobs instead.

We have several classes that handle jobs, one for each of the fundamental building blocks of yoast.com. The jobs are then handled by a worker in the background. This process is quick, so a customer will instantly see their products in MyYoast.

WordPress as an application

If you have never read The Twelve-Factor App, I would highly recommend you do so. It suggests a very robust framework for web-application development. How does this translate to a WordPress context? Better than you might think.

WordPress has a few unique features that make it harder to set up. But you can work around those to get a robust WordPress installation. If you only do one of these, it should be the dependencies factor. Start using Composer! It is here, and it works. Rarst created a good overview of how to use Composer for WordPress development. Auto-updating is not an option for a complex web application. So you need a different solution, and Composer does a very good job.

Performance

We started out without using object cache. That turned out to be a bad idea. If you handle a substantial volume of sales using WooCommerce you need object caching. Without it, an order model needs to be completely reinstantiated on every request concerning that order. It’s very costly, because the order model is required on a many pages. At the very minimum, all checkout pages need to access the order.

We also created a few pull requests on WooCommerce itself to improve performance without caching. One of these was a modification to cache the currencies. Before the modification, all currencies would be translated every time any piece of code needed just one currency. That is very inefficient considering that each shop only runs in one currency.

One thing we could not simply fix inside WooCommerce itself was searching for orders and subscriptions. Doing this would result in a query that could take down our whole website — this query contains many searches for post meta, and those take a long time. We worked around this issue by building our own search functionality on top of MyYoast.

The meta search issue would also be solved if WooCommerce used custom tables, which is luckily already on the WooCommerce technical roadmap. This issue also informed our thinking about our own plugin. Can we reduce the amount of meta keys that Yoast SEO creates? As a result, we’ve added the creation of a custom table to our own technical roadmap, too.

Current list of WordPress plugins and WooCommerce extensions

To give a comprehensive idea of where we landed, here is the list of plugins we currently use that are relevant to our site’s function as a store. The full list contains 55 active plugins.

• CMB2: Used for adding more custom fields to post types.
• CMB2 Field Type: Used to attach mutiple posts as a meta value.
• CMB2 Post Search field: Used to search for a single post to attach as a meta value.
• Google Authenticator: Adding 2FA to our login.
• Google Authenticator, Per User Prompt: This way all Yoast employees can use 2FA, but customers don’t have to.
• MailChimp for WordPress and MailChimp for WordPress – Premium: Used to connect orders to MailChimp.
• Members: For managing permissions to different roles, such as support engineers.
• MultilingualPress: Used to manage the different sites in the multisite.
• MyYoast Custom Login: Styles the login screen.
• New Relic Reporting for WordPress: We use NewRelic to track our performance.
• Postmark: For sending emails reliably.
• Romance Admin Color Scheme: Everyone needs a bit of pink in their lives!
• WooCommerce: As you might guess, to process orders on yoast.com!
• Yoast SEO: WooCommerce: Adding compatibility between WooCommerce and Yoast SEO.
• Yoast SEO Premium: Used for redirects, internal link suggestions, and multiple keywords.

Current list of WordPress plugins and WooCommerce extensions

• WooCommerce – Country Based Payments: Used to show different payment methods to customers from different countries.
• WooCommerce AdyenCw: Used to add the Adyen payment gateways to WooCommerce.
• WooCommerce Coupon Links: Used to create links that automatically apply a certain coupon.
• WooCommerce Dynamic Pricing: Used to offer discounts on bulk purchases.
• WooCommerce Email Customizer: Used to style the emails WooCommerce sends out.
• WooCommerce EU VAT Number: Used to collect the VAT number of European customers and validate the number using the VIES service.
• WooCommerce Give Products: Used to give products away.
• WooCommerce PayPal Express Checkout Gateway: Used to add a PayPal payment gateway to WooCommerce with support for WooCommerce Subscriptions.
• WooCommerce Product Bundles: Used to create the bundles in our store.
• WooCommerce Subscriptions: Used to keep track of the time remaining on plugin support and updates.

Takeaways and results

We launched this new platform on August 29th, 2017. In the weeks that followed we have greatly improved the site’s performance. We’ve achieved all of the goals we wanted to hit. Since the migration we have also started to build a lot of other things. Our support team now has a much easier time refunding and transferring accounts. We have migrated our Yoast academy to the multisite. And we have a bunch of exciting plans for 2018, including:

• Adding a shop where we sell in British pounds. Thanks to our past choices and our move to WooCommerce, we can do this in a day, it will require almost no additional code. Adding a sales dashboard for our sales team.
• Adding a sales dashboard for our sales team.
• Adding Composer support to our premium plugins.

WooCommerce has given Yoast the technical foundation on which to build and improve our platform into the next decade. Are you considering migrating an existing webshop to another platform? Have you done this already? Share your experience in the comments.

About

Anton Timmermans