An AI’s answer is only as up-to-date as the documentation it uses

Earlier this year, a merchant taking in-person payments asked our AI support assistant how to quickly accept a payment with their card reader. It checked our documentation and confidently replied: “tap ‘Simple Payment.’”

But there was no Simple Payment button. We had renamed it to “Add Custom Amount” months before, but the documentation hadn’t been updated. The merchant couldn’t find what the AI described and eventually gave up.

Support reported the issue, we updated the quick start guide and the product page, and the documentation was fixed. But this incident stuck with me because the assistant did exactly what it was supposed to do. It read the documentation and shared what it found. The problem was that the documentation described a product that had already changed.

This is the main point I want you to remember from this: when you ask an AI how to do something in WooCommerce — like changing a setting, finding a feature, or fixing an error — it answers based on the documentation, not on real-time knowledge of the product. The product changes with every release, but the docs are written by people. Any gap between them can lead to a wrong answer that sounds completely confident.

So we created something to close that gap.

It’s a WordPress plugin, built as an internal Automattic project, and it works like a documentation writer who is always watching for changes.

It keeps track of WooCommerce releases and any incoming requests to fix documentation. When something changes, like a renamed button, a moved setting, or a new feature, an AI model drafts an update. The draft then has to meet certain standards before it gets published:

1. We score it from 1 to 5 using our quality rubric, which looks at accuracy, voice and tone, structure, grammar and style, findability, and visuals.
2. It also has to pass automated safety checks. If a draft deletes more than about 30% of a page, breaks the page’s formatting, leaves a placeholder, or adds a new image link, it gets flagged for a human to review.
3. A writer from our team reviews the draft in a preview and approves it before it goes live on woocommerce.com.

My favorite part of the rubric is in the findability section: the first paragraph of every doc must work as a standalone answer that an AI can quote. We’re not just writing documentation for people scrolling through a page anymore. We’re creating the source material for every AI answer about WooCommerce, and our rubric reflects that.

We don’t just score new drafts. We also score the live documentation using the same rubric.

By late June, 39 out of the 117 WooCommerce docs we reviewed were below our quality standards. That’s why we built this system. WooCommerce updates all the time, and even the best team of writers can’t re-read every page after each release. But the plugin can. It checks pages as a writer would with unlimited time, and it flags the ones that need attention.

One part of the pipeline automatically applies our terminology list to keep product names consistent, like using “WooCommerce” instead of “woocommerce.” It does this with a direct find-and-replace. 

Some entries in the list were written as writing advice instead of exact word swaps. The system couldn’t tell the difference, so it pasted the advice into live pages. For a short time, a published doc told readers “(descriptive link text describing the destination)” where a link should have been.

I noticed this while working on something related. We audited all 258 WooCommerce docs, found 9 that were affected, fixed each one, then removed the problematic entries and added a safeguard to prevent this kind of mistake from happening again.

I’m sharing this because it’s the same lesson from a different angle. Automation follows rules exactly as they’re written, quickly and without judgment. That’s what makes it both helpful and risky. Our system is built with this in mind: strict checks that fail safely, a human reviewing anything important, and audits for the automated steps.

What this means when you’re troubleshooting

If you ask an AI assistant a WooCommerce question and take the answer at face value — and most people do, since that’s the point — the quality of the answer was set before you even typed your question. It depends on whether the documentation matches the product you’re using.

Our job is to keep the documentation accurate, and the system above is how we do that. There’s a smaller part for you, but it’s still important:

• If an AI’s instructions don’t match what you see on your screen, the documentation might be outdated. Check the doc it’s using before thinking you’ve made a mistake.
• Let us know if you spot an issue. When using our AI assistant, rate a reply thumbs-down or tap ‘Contact Support’ to tell us what’s wrong. If you’re on a document that doesn’t seem current, tell our team via the ‘Was this guide helpful?’ thumb at the bottom of the page.

The merchant with the card reader didn’t do anything wrong. Neither did the AI. The responsibility was ours, and building a system that makes fixes quickly — and catches changes before you ever notice them — was the best way we could learn from that experience.

About

David Wilson

David is an AI Engineer at Automattic who specializes in shipping production LLM infrastructure for customer support and documentation systems across WordPress.com and WooCommerce.