No-flicker A/B testing.

Watch a storefront running a typical A/B test and you can usually catch it in the act. The page loads, and then the headline swaps, or the price, or the whole hero image. That flash is the testing tool doing the only thing it can: wait for the real page to show up, then rewrite it in the browser. The industry even has a name for the flash, “flicker,” and the standard fix is a snippet that hides your page until the rewrite finishes. Either way you’re hiding or slowing the page to run the test, which taxes the exact thing the test is meant to improve.

When I went and measured that tax on real stores, it ran anywhere from 100 to 400 milliseconds: sometimes the whole page held back until the swap finished, sometimes the content visibly rewriting inside that window. And that was on stores whose testing tool was doing everything right, caching its responses and all. It isn’t one tool being sloppy. It’s the unavoidable cost of patching a page after it has already arrived. On a storefront, where a few hundred milliseconds is the difference between a sale and a bounce, that is not a rounding error.

This month I got the better approach working end to end: the version is chosen before the page is ever sent, and the shopper just receives a normal, fast page. Nothing gets rewritten, so there’s nothing to flicker.

Why we can test this way

Most testing tools don’t have much choice. They don’t build your pages; Shopify does. So all they can do is let Shopify’s page arrive and then edit it in the shopper’s browser, racing their eyes.

We’re in a different spot, and it’s the whole reason this works. LayerKick already sits in front of the store, caching the finished pages and serving them fast. So a test version isn’t some special thing we have to build, it’s just another version of the page we already cache, and we hand the right shopper the right one. The choice happens on our side, before the page goes out, so by the time it reaches the shopper there’s nothing left to change. The usual trade-off is “accept the flicker, or replatform your entire store.” Serving the store’s own cached pages gets you the clean result of the second option without touching how the store is built.

Worth saying plainly how unusual that is. I can’t be sure we’re the only ones, but a rudimentary search didn’t turn up another server-side A/B testing tool for Shopify storefronts. The one exception I know of is Shopify’s own server-side price experiments, in beta right now. Everything else I could find works client-side, and somewhere in the shopper’s journey it pays that 100 to 400 millisecond tax.

Right now this is proven out on staging, run through every scenario in our test suite. The revenue half, tying each finished order back to the version its buyer saw, rides on tracking plumbing that lands in the next few weeks. It goes live on a real store once it has earned that, and not before. This entry is the “here’s what we built and why” one; the “here’s what it did on a real store” one comes later.

The same shopper sees the same version

A test is only trustworthy if a shopper who comes back tomorrow sees the same version they saw today, and if a 50/50 split really is 50/50. We get both by working out a shopper’s version from their ID every time, instead of deciding once and storing the answer. Same ID, same version, every visit, with nothing to keep in sync behind the scenes.

The ID itself is the honest part. Plenty of tools quietly set a brand-new tracking cookie to pull this off, which is one more privacy prompt and one more thing to look after. We don’t. We only key off an identifier when the shopper already carries Shopify’s own analytics cookie. If Shopify is allowed to remember this shopper, so are we, under the exact same consent they already gave. If they haven’t consented, we set nothing: they see the store’s normal page and simply sit out the test.

The metrics that matter

Our priority with the results view is simple: cover the metrics that actually matter in a shopper’s journey, and cover them well. What we care about most is conversion and the steps leading up to it, whether people are getting further into the store on a given version, and whether more of them check out at the end of it.

Inside conversion there’s a split that a single “conversion rate” can hide, and it’s the one we most want a merchant to see: average order value (AOV) versus total revenue. A version can lift the share of visitors who buy while quietly lowering what each of them spends. So the numbers sit side by side, per version: sessions, orders, conversion rate, revenue per visitor, and average order value. If version B wins on conversions but drags AOV down, the whole trade-off is in one place and you make the real call, instead of celebrating half a result.

The order-to-version connection is the piece still in flight, as I said up top. So what’s solid today, and proven on staging, is the half that has to be right before any of those numbers is worth trusting: picking the version, serving it clean, and deciding honestly when a result is real. That last part, knowing when a result is real, is the subtlest of the three, so it’s worth explaining in plain English.

Knowing when you’ve actually got a winner

Here’s the trap in the usual way of testing. The classic method makes you fix your sample size up front and forbids you from looking until you hit it, because if you peek early and stop the moment things look good, you’ll crown winners that aren’t real. On a live store that’s a rotten deal: you’re either waiting longer than you need, or fooling yourself by peeking. And nobody actually resists checking a test running on their own store.

So we use a different approach (the technical name is Bayesian, but the idea is simple). Instead of one yes/no verdict at a finish line, the system keeps a running, honest estimate of how likely each version is to genuinely be the better one. The important part: the probability itself stays honest no matter when or how often you look. Peeking, the thing that quietly wrecks the classic method’s math, doesn’t wreck this number. And it answers in plain language, “version B is 98% likely to be beating the original, most likely by around 8%,” an estimate that starts loose and firms up as visitors arrive, not a figure you’d need a statistician to read.

That same honesty is what makes it fast. The moment it’s confident enough, either way, it calls the test, so you’re not waiting on an arbitrary calendar. On a realistic single-digit lift, that verdict lands in a fraction of the visitors a fixed-count test is locked into; the chart below runs the whole race. Whether that’s days or weeks depends entirely on a store’s traffic, which is why everything below is counted in visitors, not time. And it flags a loser just as fast as a winner, so you stop paying for a version that’s costing you sales instead of only hearing the good news. The one thing it won’t do is jump early: below a minimum number of visitors it stays quiet rather than read too much into a handful of shoppers. It rechecks every running test on a schedule, and the first time one crosses the line, a notification is triggered, once. No dashboard to babysit. It speaks up the moment the evidence is real, and not a shopper sooner.

The picture is two estimates, one per version, that pull apart as visitors arrive until you’re sure which is ahead. The honest version is subtler than you’d sketch it, and honestly cooler: real lifts are small, a few percent, so the curves never fly apart; they inch together as visitors pile in, and the view has to zoom to keep the gap visible. Every number on screen is computed live from those two true rates, so it holds up if you recompute it. Notice where we can call it versus where a fixed-count test is still committed to keep counting.

That “sure B wins” percentage, the probability B actually beats the original, is a precise quantity, and the chart above is exactly it. Take those two curves, subtract one from the other, and you get a single curve, our estimate of the gap between the two versions. Zero means no difference. The probability B is really better is just the shaded part of that curve past zero, added up. That shaded area doesn’t stand in for the number; it is the number.

What’s next in the log

Building a tester that reports revenue this honestly sets a bar LayerKick then has to clear itself. So a later entry turns the same scrutiny on us: is the whole platform actually making stores faster and selling more, measured the same unforgiving way. Building the tool is the easy part. Proving it in public is the point.