The kernel was green. Every unit test passed, the end-to-end suite passed, the numerical kernels matched their reference outputs to the last bit. Then we pointed the same kernel at our own homepage and asked it to run the code we'd been showing prospective users, and it lit up red.

None of the failures were bugs in Equana the product. They were bugs in Equana the marketing — snippets that had been true once, or had never been true, sitting in the hero section, the docs, the comparison pages, promising things the real system didn't do. We fixed them, then we did something more useful: we made it impossible for them to come back. Every code sample on the site now runs against the real kernel in CI, and if the output in the copy doesn't match the output the kernel produces, the build fails.

This is the story of that sprint and the rule it left behind.

The premise: a code sample is a claim

Marketing copy is usually judged by taste — is it clear, is it compelling. But a code sample is a different kind of sentence. It's a falsifiable claim: "type this, get that." When we write

x = mean([1, 2, 3, 4]) println(x) 2.5

we're not being persuasive, we're asserting a fact about the system. And facts about a running system can be checked mechanically. The moment we saw code samples as claims rather than decoration, the rest followed: claims get tested, and there was no reason ours shouldn't be.

The mechanism is straightforward. Every example page on the site is a real notebook, so the deploy pipeline drives them like a user would: a Playwright suite opens every example as an anonymous visitor, waits for the kernel, clicks "Run all cells", and fails the deployment if any cell reports an interpreter error. No hand-maintained fixtures: the source of truth is the kernel, and the published code is checked against it on every deploy. This is doctest discipline aimed at a website instead of a docstring — executable documentation, where the docs fail the build when they lie.

What the first run caught

The first full run was educational. Here's the taxonomy of what was wrong.

Output drift. A hero snippet advertised a computed result of 0.0980. The real kernel returned 0.1083. Nobody had lied — the number had been correct against some earlier version of a function and then quietly diverged when the implementation improved, and the copy was never updated. This is the most insidious failure mode because it's invisible to a human proofreader: 0.0980 looks exactly as plausible as 0.1083. Only the kernel knows which one it actually produces. This single class of bug is the entire argument for the pipeline. Numbers in marketing rot silently, and no amount of careful reading catches it.

Syntax that never parsed. Several snippets carried trailing semicolons — x = 3; — a reflex from MATLAB and C. Equana doesn't use them, and the parser rejects them. The code had been written by hand, eyeballed, and shipped without anyone ever pressing enter on it. It looked like Equana. It wasn't.

A function overload that didn't exist. One example called plot with an argument shape the real plot has no method for. It read naturally and matched what a reader would expect the API to offer — but expectation isn't dispatch. The kernel resolves calls against an actual method table, and there was no method there. Running the sample turned a plausible-looking call into a hard error.

Documented but unregistered. arange had a working implementation in the numerical library and a docstring in the registry, but it hadn't been wired into the set of names the interpreter actually recognizes as builtins. So it was documented, it was implemented, and it was uncallable — the worst possible state, because every surface said it worked. Executing a sample that used it surfaced the gap immediately. (It's registered now, which is why you'll see it doing real work in our FFT tutorial.)

An advertised feature that had been removed. A comparison page still described a dual-indexing capability that had been cut during a redesign of the array semantics. The feature was gone from the kernel; the boast was still on the page. A reader would have tried it, watched it fail, and reasonably concluded the whole product was vaporware.

Five categories, one root cause: the copy and the code were maintained by different processes, on different timelines, with no connection between them. The code had tests. The copy had a proofreader. Only one of those catches 0.0980 drifting to 0.1083.

Marketing copy is code that rots

The generalizable lesson isn't about numerics or about our particular bugs. It's this: any assertion about a running system will drift out of sync with that system unless something mechanical holds them together.

We already know this about code. It's why we have regression tests — not because we're bad engineers, but because we're honest about entropy. Systems change, and yesterday's true statement becomes today's false one unless a test pins it in place. A code sample on a website is subject to exactly the same entropy. The function it calls gets refactored, an output format changes, an overload is added or removed, a feature is cut — and the sample, sitting in a Markdown file nobody re-runs, keeps making its now-false promise to every visitor.

The failure isn't writing a wrong number once. It's having no process that would ever notice. A code sample without an execution check is untested code that happens to live in your marketing directory instead of your source tree. Treating it as prose is the mistake. It's code, and code rots unless you run it.

The rule we shipped

The discipline is one sentence: no code sample ships anywhere — hero, docs, comparison pages, blog — unless it executes against the real kernel in CI.

A few things fall out of taking that seriously.

The real kernel, not a mock. The whole value is that the sample runs through the same interpreter and the same WASM numerical kernels a user hits at equana.dev. A mock would just be a second thing to keep in sync — a new source of drift. The reason the check is trustworthy is that there's nothing between the sample and production behavior.

Outputs are asserted, not just "does it run." Catching the plot overload only needed execution to not throw. Catching 0.0980 vs 0.1083 needed the output to be checked against the copy. We assert the printed result, so both classes get caught.

It applies uniformly. Hero copy is the most-read code on the site and historically the least tested, because it lived in a design file, not a source file. The same page that gets the most trust deserves the most scrutiny. There's no tier of copy that's exempt; the marketing homepage runs through the identical gate as the docs.

Green is now a property of the whole site, not just the repo. Before, "tests pass" meant the code was consistent with itself. Now it also means the code is consistent with everything we say about the code. That's a stronger and more honest definition of green, and it's the one worth having.

The deeper shift is cultural. Writing marketing used to be a place where an engineer could relax the standards they'd never relax in a pull request — round a number, sketch a call, describe a feature aspirationally. That gap is closed. The copy is under the same version control, the same review, and the same CI as the product, because the copy makes the same kind of claims the product does. When your marketing is executable, honesty stops being a matter of diligence and becomes a matter of the build passing.

If you want to see the kind of code these samples verify, the FFT walkthrough runs a full spectral analysis end-to-end in a browser tab, and the piece on designing a language that reads like English covers the syntax those samples are written in. Every equana block in both — and in this post — runs against the real kernel before it reaches you. That's the point.