What is an A/B test?

An A/B test is a randomized controlled experiment applied to a business decision: a population of users is split into two groups assigned at random, one sees the current version, the control, and the other the variant, the treatment, and the two are compared on a metric decided before looking at a single result. Randomization is the core of the method, because it is the mechanism that makes the attribution valid by construction, with no assumptions about hidden factors separating the two groups. The reference paper on the topic, Kohavi and colleagues at KDD 2013, describes how Bing ran over 200 concurrent experiments a day, and groups the practical challenges into three areas: cultural and organizational, engineering, and the statistical reliability of results. The method only works when there is enough traffic to make randomization meaningful, and that is exactly where most small and mid-sized businesses fall outside its scope.

See also: Feature flag · A switch in the code that turns a feature on or off without a new deploy, separating deploy from release.

What it takes for an A/B test to actually mean something

A badly run A/B test is worse than no test at all: it produces false certainty. Three conditions separate an experiment that looks like one from one that actually is one. First, the success metric has to be fixed before launch, with guardrail metrics that must not get worse, or the team ends up choosing after the fact which number tells the best story. Second, sample size is calculated up front; a test on too little traffic cannot carry the effect you hope to measure: most often it concludes nothing, and when it does conclude it overstates the effect, as I explain under statistical significance. Third, the planned duration is honored unless a sequential design was set up in advance or a guardrail metric degrades; peeking at results and stopping once they look good, plus a novelty effect that fades after the first week, are the most common ways an A/B test lies. The rollout can go through a feature flag, which is only the plumbing.

An enterprise example

An e-commerce site with millions of monthly visitors can test a new checkout on half its traffic for a week: the volume makes a large sample possible in a short time. A B2B distributor selling industrial machinery does not have that option: few monthly visitors, a sales cycle measured in weeks with several decision makers, a site that generates leads, not transactions. Randomly testing two product pages there does not reach a sufficient sample within the time the decision has to be made, short of very large effects. What I set up in a comparable case was an observed staged rollout instead: the new page went first to 20% of organic traffic for a month, tracking quote requests and time on page, then to 100% only after a stable signal across several weeks. That is not an A/B test: it is a before/after comparison with its declared limits, including the chance that seasonality or a competitor's campaign gets confused with the effect of the change.

Why it matters for decision makers

Whoever decides has to first ask whether the available traffic or population can support a randomized experiment, before commissioning one already condemned to conclude nothing. For one-time decisions like choosing a vendor or a platform migration, there is no second group to withhold the choice from in the first place: the honest alternative is a pilot on a narrow segment, measured carefully, without dressing it up as a controlled experiment that cannot exist in that context. And when one number improves while a collateral indicator gets worse, the same caution applies that governs every isolated metric: see Goodhart's law. An A/B test forces a discipline: deciding in advance what would count as proof, and accepting a negative result just as readily as a positive one.