How results are calculated
An experiment's results page answers one question in plain words: did a variation change your primary goal? Here is how it gets there: who counts, how each goal is tested, and when a verdict is firm. The statistics are named for those who want them.
Who counts
- A visitor counts from the first time a page of the test loads for them, in whichever variation they get. The control counts too.
- They stay in the variation they saw first, for the whole test, even if they're later moved to another one.
- A conversion counts only if it happens at or after that moment, by the visitor's own clock, and between the test's start and its completion.
- A visitor is a browser, recognized by a cookie. The same person on their phone and their laptop counts twice: identifying them doesn't merge the two in results.
- Orders count once per order ID, however often the confirmation page loads. Orders sent without an ID are each counted.
- Amounts are added up as sent: currencies aren't converted. With one currency, results show it; with several, plain numbers.
- Visits through QA and preview links are kept apart (see QA and preview). Known bots and crawlers never count.
- Paused days count as days running: the clock runs from the first start to completion. While paused nobody sees the test, but a visitor already in it who converts then can still count for their variation.
- If your plan's tracked visits run out, visitors still see the variations but aren't counted; the results page names the missing days. See Plans and usage.
How each goal is tested
Each variation is compared with the control, one goal at a time, by the number the goal is judged by (see Goals and audiences). The tests are frequentist, two-sided and sequential, at the test's confidence level: 95% (α = 0.05) unless you chose 90% or 99% when you set it up:
| Judged by | Compared | Test |
|---|---|---|
| Conversion rate (Purchase rate) | The share of visitors who converted at least once | Two-proportion z-test |
| Conversions per visitor (Orders per visitor) | Each visitor's number of conversions | Welch's t-test |
| Total (Revenue) | The amount per visitor, so an uneven split can't skew a total | Welch's t-test |
| Per visitor (Revenue per visitor) | Each visitor's amount, 0 for those without one | Welch's t-test |
| Per conversion (Average order value) | Each conversion's amount | Welch's t-test |
- A difference is significant when its p-value is below α: at 95%, if the variation changed nothing, a call like that would come less than 1 time in 20, however often you look (see How sure the calls are).
- "Likely somewhere between −3% and +12%" is the confidence interval of the difference at the test's level, as a share of the original's value, valid each time you look; it leaves out 0 exactly when the difference is significant. The z-test uses unpooled variance; averages are compared on the normal approximation.
- With several variations, p-values are Holm-adjusted and the decision uses the adjusted one, so trying many variations doesn't make a false winner more likely. Intervals are Bonferroni-width then: 97.5% each when two variations face the control, for example.
- The primary goal decides the verdict, and is tested on its own. Secondary goals show side effects, and are judged together: their p-values are Holm-adjusted across all of them, so however many you add, a side effect that isn't there is flagged in at most 1 test in 20 at 95%. A small real one takes more to show, so watch the goals that matter rather than every goal you have.
- Every goal is judged by how far along the test is: the share of the primary goal's planned sample that's in.
- In a multivariate test every combination is a variation, adjusted as above. What each change does pools every combination that shows an option and compares it with its section's original, adjusted within each section.
Outliers
For the significance test only, amounts and each visitor's number of conversions are capped at the 99th percentile: anything above the top 1% (of the values above 0, from all variations together) counts as that much. One huge order, or one visitor reloading a page 500 times, can't decide a test. Rates aren't capped. Totals and the numbers shown are not capped either, but the uplift and its range come from the capped values, so with a big outlier they can differ from the plain numbers beside them. Details shows each goal's cap.
How sure the numbers are
Each goal's line in the overview carries a label, from the adjusted p-value when there are several variations:
| Label | When |
|---|---|
| Very likely real | Significant, with a p-value below a fifth of α: 0.01 at 95% confidence. |
| Likely real | Significant: a p-value below α, 0.05 at 95% confidence. |
| Might be real | Not significant, but a p-value below 0.2. |
| Could be chance | A p-value of 0.2 or more. |
| Not enough data | Nothing to test yet, for example when nobody has converted. |
While the test is early (under 7 days or under 30 conversions, and not completed), the first three end in "so far".
The verdict
The verdict is about the primary goal and the most promising variation: a clear winner first, then the biggest uplift. B stands for that variation's name, and Purchase rate for your primary goal:
| Verdict | When |
|---|---|
| Something's off with the traffic split | The traffic-split check failed (below). It comes before everything else. |
| No primary goal | The test has no primary goal to decide on. |
| Waiting for visitors, or for conversions | No variation can be compared with the control yet. It says why, from the table: no visitors yet, a version without visitors, or no conversions in the control. Once a test has ended, it says so (Ended without enough data). |
| Ship “B”: it's a winner | B is significantly better on the primary goal, no secondary goal is significantly worse, and the test has 7 days and 30 conversions. |
| “B” is winning, but it's early | The same, before 7 days or 30 conversions. Early leads often shrink. |
| “B” wins on Purchase rate, but costs you elsewhere | B is better, or leaning better, on the primary goal and significantly worse on a secondary goal. |
| Leaning towards “B”, not proven | Not significant, but B's p-value is below 0.2 and in its favour, and the change isn't yet too small to matter. |
| Keep the original | Every variation is significantly worse. Before 7 days or 30 conversions it adds that it's early, but the drop is clear. |
| No real difference | The change is too small to matter, or the test reached its planned sample with nothing significant; and it has 7 days and 30 conversions. |
| Too early to tell | Anything else, including a variation leaning the wrong way. |
- 7 days and 30 conversions don't block a verdict, they soften it: until the test has both, Ship becomes “B” is winning, but it's early, and No real difference becomes Too early to tell. A full week covers weekdays and weekends. The 30 is the smaller of the control's count and the variation's: visitors who converted for a conversion-rate goal, every conversion otherwise.
- Too small to matter means the whole confidence interval lies within ±10%. The 10% is fixed: it's the smallest change the verdict counts as worth acting on.
- The planned sample is what each variation needs for a 10% change to be found 80% of the time by the end of the test, at its confidence level, worked out from the control's numbers so far: visitors, or conversions for averages per conversion; less what visitor history takes out (below). How far along shows it, with the days left at the current pace.
How sure the calls are
A test compared once, at a sample size decided in advance, is wrong at most 1 time in 20 at 95% confidence. Checked every day and stopped at the first winner, the same test calls winners that aren't there far more often: each look is another chance for luck to pass for a result. People check their tests every day, so Peekbird's tests are made for it.
- They're sequential. Early on, a call needs overwhelming evidence; by the planned sample, a little more than a test looked at only once. However often you look, a call is as sure as the confidence level says: a winner called on day 9 is as reliable as one called at the end.
- A test ends at its planned sample. From then on its results count the visitors up to that moment, and its call stands: Ship, Keep the original, or No real difference. How far along shows the date. Visitors who come after aren't counted; to test again, complete it and start a new one.
- The planned sample is about a fifth larger than a once-only test would need at 95%: that's the price of being able to look whenever you like.
- Side effects are held to the same promise. In the simulation, with five secondary goals the change didn't move, 4.1% of tests flagged a side effect on some day (18% when each goal was tested on its own); a real 10% drop in one of them was caught in 71%.
- The confidence level is chosen when you set up a test, in its Goals step: 95% suits most tests, 90% calls sooner, for changes that are easy to undo, and 99% is for changes that are costly to get wrong. It can't change once the test has run, so it can't be picked to suit the results.
Measured by simulation: 2,000 experiments for each case, judged each day for four weeks by the results page's own code. With no real difference, how many got a wrong call (Ship, or Keep the original) on some day; with a real one, how many it found by the planned sample:
| Confidence | Promised: wrong at most | No real difference: a wrong call | A real 10% change found | A real 20% change found |
|---|---|---|---|---|
| 90% | 1 in 10 | 7.7% | 81% | 99.9% |
| 95% (the usual) | 1 in 20 | 3.8–4.3% | 81% | 99.8% |
| 99% | 1 in 100 | 0.7–1.0% | 82% | 99.9% |
Until October 2026, results were judged the classic way, as if looked at once: checked daily, 23% of tests with no real difference got a wrong call on some day. Tests that were running then are judged the new way too, so an early call may have become a softer one until more visitors come.
The sample size calculator plans the way Peekbird does. The significance calculator checks finished numbers once, with the classic test.
The chance to be better
Some testing tools give each variation a chance to beat the original, such as "87% chance B is better". Turn on Show each variation's chance to be better in the site's Testing settings and Peekbird's results show it too, beside each goal: the chance given the data so far, with no prior assumptions (a Bayesian reading, with flat priors: Beta distributions for rates, normal ones for averages; for goals better lower, like the bounce rate, the chance of being lower).
It answers a different question from the call. It's how things stand now, and it moves as visitors come. Checked every day, a 95% chance for one version or the other turned up on some day in 41% of simulated tests where nothing had changed; the call, from the sequential test, was wrong in 4% of the same tests. Read the chance to see how things stand, and act on the call.
Visitor history
Results take account of what each visitor did before they entered the test, which makes some tests shorter. A visitor who has bought before is more likely to buy again whatever they're shown: if by chance more of them landed in B, B looks better than it is. Peekbird adjusts each version's numbers by how its visitors' history differs from everyone's, so chance mixes like that stop adding noise. (The method is known as CUPED, a regression adjustment.)
- What's read: from the 90 days before a visitor entered the test, their visits, how often they'd done the goal, and what they'd spent; and of the visit they entered it in, whether it was on a phone, and where it came from (direct or email, an ad, social media).
- Only what came before they saw a version, so the version can't have changed it. Visitors are split at random, so history is balanced on average: the adjustment takes out noise, never a real effect. How much it adjusts by is learned from all visitors together.
- For conversion rates, conversions per visitor, totals and amounts per visitor. Not for averages per conversion (order value), the bounce and engagement rates, or QA traffic.
- How much it helps depends on the site. How far along says how much shorter it made each test, and a goal's numbers say "adjusted for visitor history" when they were. Where it would take out next to nothing (mostly new visitors), results are worked out without it.
In the simulation, a shop with many repeat buyers reached its planned sample around day 15 instead of 18; a shop of mostly new visitors, no sooner. Either way, tests where nothing had changed still got a wrong call in 3–4% of cases, as without it. Because the uplift and its range come from the adjusted numbers, they can differ a little from the plain rates beside them.
The traffic-split check
Every test checks that visitors split the way you set, uneven weights included, with a chi-square test (a sample ratio mismatch check). Variations at 0% are left out. When its p-value is below 0.001, it fails: the verdict becomes Something's off with the traffic split, since results can't be trusted. The usual causes are redirects losing visitors, errors in variation code, and bots.
Until each variation should have had at least 5 visitors, the split isn't judged: it shows as Too few visitors to tell.
There's no restart: fix the cause, complete the test and start a new one, since the visitors counted so far stay skewed. The check compares with the weights as they are now, so changing them partway through a test can set it off.
Watching the visits
Below the numbers, Watch the visits opens each version's recordings from the days the test ran: all of them, those that completed the primary goal, and those that didn't. Heatmaps shows where that version's visitors clicked and how far they scrolled. The numbers say which version wins; the visits often show why.
When results update
Results are worked out afresh each time they load. While a test runs or is in QA, the page reloads them every minute, and Refresh does it at once. A test in QA shows its QA traffic. To see the QA traffic of a running test, turn on Show QA traffic in results in the site's Testing settings.
