Skip to content
Peekbird

Experiments

An experiment shows different versions of your pages to different visitors, and measures which does better. Create one under your site's Experiments with New experiment. It's saved as a draft after the first step, so you can finish it later.

Four kinds of test

TypeWhat it does
A/B testChanges the page itself, in the visual editor or with JavaScript and CSS. 2–10 variations, the first being the control: your page as it is.
Split URLSends visitors to another page, keeping their query string and #hash. The control stays on the original page. 2–10 variations.
MultivariateChanges several parts of a page (sections) and tests every combination: 2–3 sections, 2–4 options each counting the original, up to 16 combinations.
In your codeYour own code asks which variation a visitor gets, in the page or on your server, and does what it says: prices, search results, a checkout flow. 2–10 variations, and no pages to choose.
  • Variations change the page with the visual editor or code: see JavaScript API for what code can use.
  • An "In your code" test is decided where your code asks: peekbird.variation('key') in the page, or a server SDK (see Server-side SDKs). Asking counts the visitor in the test, so ask where the variations start to differ. Its Preview & QA tab has the code for each language.
  • Each split-URL variation needs the address it sends visitors to (Send visitors to: a full URL or a path like /landing-b) before the test can start, even in QA. Parameters in that address win over the visitor's own. A visitor already on that page stays put.
  • In a multivariate test, the first option of each section is the page as it is, and every combination is a variation of its own, with the traffic split evenly. Comparing 16 combinations one by one takes about 8 times as long as an A/B test; each section's effect comes sooner, since it pools the other sections (see How results are calculated).

Which pages it runs on

Under Pages, list where the test runs (Run on pages where the URL…) and anything to leave out (Except where the URL…). A page must match one of the first and none of the second. Type an address into the tester to see Runs here or Doesn't run: it matches exactly as your site does.

The URL…Matches when
is exactlyThe same address. Its query string is ignored, unless the rule has one: then the page needs those parameters, and others (like utm_source) are fine.
starts withThe address, query string included, begins with the text. Ending it with / means a section: /products/ matches /products, /products/red-shoe and /products?page=2, but not /products-sale (which /products matches).
containsThe text appears anywhere in the address, query string included: sale also matches /shoes?ref=sale.
matches pattern* stands for anything, and the whole address must match: /products/* matches /products/red-shoe, not /products. The query string counts only if the pattern has a ?, and then all of it must match.
matches regexA regular expression, ignoring letter case, tested against the full address as the browser has it: https://, www., query string and #hash included.

Except for regex, matching ignores letter case, http:// or https://, a leading www., a trailing slash and the #hash. A rule starting with / matches the path on any domain the snippet runs on, so one test works on staging and production. Any other rule starts with the domain: write shop.com/products or /products, since starts with products never matches.

Who takes part

  • Audiences narrow the test to some visitors, such as mobile visitors or new ones; with several, visitors must match any or all of them. They're checked when a visitor enters: see Goals and audiences.
  • How many of them take part: 0.01–100%. The rest see your page as it is and aren't counted. Each variation's Weight % sets its share of those who take part; weights add up to 100% and can be uneven.
  • A visitor's variation is worked out from their visitor ID and the experiment, so it stays the same on every page and every visit. Their browser also keeps it for the Visitor cookie lifetime (days) in Site settings (365 by default). To keep it across subdomains, set the Cookie domain there, like .shop.com.
  • A visitor can be in several experiments at once: each one splits visitors on its own, unless they share a mutually exclusive group.

Changing the split of a paused test:

  • Raising the share that takes part only adds visitors.
  • Lowering it, or changing weights, keeps most visitors where they are, since their browser remembers their variation. Some can move: a visitor who came back during the pause, while another test was live on the site, is split again.
  • A variation set to 0% sends its visitors to the others the next time they load a page of the test.
  • Results keep counting every visitor in the variation they saw first.
  • The traffic-split check compares visitors with the weights as they are now, so changing them partway through can set it off.

Mutually exclusive groups

Two tests that could sway each other, like a new checkout button and a free-shipping banner on the same checkout, mix their results when visitors see both: neither test can tell which change made the difference. Put them in one group, and no visitor is in both.

  • Choose the group in a test's Audience & traffic step, or make one there with New group. Use Manage groups, or the site's Testing settings (under A/B testing), to make, rename or delete them; a deleted group's tests carry on without one.
  • The group's running tests split its visitors evenly: with two, each can reach half of them; with three, a third. The setting How many of them take part then applies within its share, and its audiences still decide who enters.
  • A visitor in one of the group's tests stays in it, and out of the others, even when tests start or stop. Paused tests count too: their visitors stay out of the rest.
  • When a test in the group starts or stops, the shares are worked out again for visitors who haven't entered one yet.
  • A test in QA, or opened with a preview link, ignores its group, so you can check it whatever share you're in.
  • Tests decided in your code follow their group too, on your server and in the page (see Server-side SDKs).

Statuses

StatusWhat it meansFrom here
DraftBeing set up. It never reaches your site.Start in QA or Start test (on Review), or delete it.
QARuns only in browsers that opened a QA or preview link; their visits are QA traffic.Back to draft, or Start.
RunningLive for every visitor who matches, and counting.Pause, or Complete.
PausedShown to nobody: everyone sees your page as it is.Resume, or Complete.
CompletedStopped for good. Its results are final.Archive.
ArchivedKept for reference.Delete it.
  • Starting, in QA or for everyone, needs a primary goal.
  • You can edit a test in Draft, QA or Paused. A running test is locked, so pause it first: changing it mid-run mixes two versions into one result. Edits made while it's paused still go into the same results, which count from its first start.
  • While paused, the days still count as days running. On Resume, visitors get the same variation as before, unless you changed the split.
  • Complete stops the test for good, and everyone sees your page as it is again: to keep a winner, make the change on your site. Its results stop counting at that moment.
  • There's no restart and no copy: to test a new version, or to count from zero, create a new experiment.
  • Only drafts and archived tests can be deleted: complete and archive a test first, or move one in QA back to draft. Deleting can't be undone.

Checking a test

Before you start, run it in QA and preview each variation: see QA and preview. Once it's live, the test's page warns you when a variation's code fails on visitors' pages, or a visual change finds nothing to change. For how variations load without a flash of the original page, and on single-page apps, see Speed, CSP and single-page apps.

See what your visitors see.

Start free in a few minutes: one snippet, every feature. Move to a bigger plan when you need more tracked visits.