2d:null 2e:["$","title",null,{"children":"A/B Testing - Deterministic Bucketing | Kaleb Fulgham | Observable"}] 33:T857,md` There are several techniques to determine whether a certain change improves a particular observable metric. One such technique which we'll explore today is A/B testing or Multivariate testing. > A/B testing is a way to compare two versions of a single variable, typically by testing a subject's response to variant A against variant B, and determining which of the two variants is more effective. *[via Wikipedia](https://en.wikipedia.org/wiki/A/B_testing)* > A/B testing is essentially an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal. *via Optimizely* ## Basic Functionality * A/B & Multivariate testing * Coin toss bucketing * Total allocation of traffic * Allocation for each variation (50/50, 90/10, 33,33,34) * Audience segmentation * Experiment impression tracking In a simple given A/B test experiment, *A* usually represents the existing control experience and *B* is the changed treatment variation of the experience. The changes could be relatively small or large. For example: a single word or phrase, a banner or button color, an image displayed, sorting alphabetically vs most popular, or a design layout change. However, it's important to remember the goal of this method of experimentation. We desire to learn how the change in *B* impacts the *metric / goal* versus the existing *A* experience. Is *B* a winner, loser, or neutral? Do we run the experiment **only once** with the first person & we're done? **No.** We need to continue running the experiment across as many people as possible to remove random chance and bias within the population. How long should the experiment run? A few days, weeks, or a few hours depending on how many people visit. Use [Evan Miller's Sample Size Calculator](http://www.evanmiller.org/ab-testing/sample-size.html). When creating an experiment, it is important to define a hypothesis including which metric(s) will be impacted. Here is a helpful hypothesis template: **IF** ...
**BY** ...
**WILL** ...
**BECAUSE** ... `34:T5ba,// https://github.com/perezd/node-murmurhash/blob/master/murmurhash.js#L115 murmurhashv3 = (e, t) => { var n, i, r, a, o, s, u, c; for (n = 3 & e.length, i = e.length - n, r = t, o = 3432918353, s = 461845907, c = 0; c < i; ) u = 255 & e.charCodeAt(c) | (255 & e.charCodeAt(++c)) << 8 | (255 & e.charCodeAt(++c)) << 16 | (255 & e.charCodeAt(++c)) << 24, ++c, u = (65535 & u) * o + (((u >>> 16) * o & 65535) << 16) & 4294967295, u = u << 15 | u >>> 17, u = (65535 & u) * s + (((u >>> 16) * s & 65535) << 16) & 4294967295, r ^= u, r = r << 13 | r >>> 19, a = 5 * (65535 & r) + ((5 * (r >>> 16) & 65535) << 16) & 4294967295, r = (65535 & a) + 27492 + (((a >>> 16) + 58964 & 65535) << 16); switch (u = 0, n) { case 3: u ^= (255 & e.charCodeAt(c + 2)) << 16; case 2: u ^= (255 & e.charCodeAt(c + 1)) << 8; case 1: u ^= 255 & e.charCodeAt(c), u = (65535 & u) * o + (((u >>> 16) * o & 65535) << 16) & 4294967295, u = u << 15 | u >>> 17, u = (65535 & u) * s + (((u >>> 16) * s & 65535) << 16) & 4294967295, r ^= u } return r ^= e.length, r ^= r >>> 16, r = 2246822507 * (65535 & r) + ((2246822507 * (r >>> 16) & 65535) << 16) & 4294967295, r ^= r >>> 13, r = 3266489909 * (65535 & r) + ((3266489909 * (r >>> 16) & 65535) << 16) & 4294967295, r ^= r >>> 16, r >>> 0 }35:[] 31:{"id":"32f3d53d7b0e913d","slug":"a-b-testing-deterministic-bucketing","title":"A/B Testing - Deterministic Bucketing","thumbnail":"af0a45b17e8bc321c33c6641bf51cde47f9935ce1db5f1986895a4d46db59171","access_level":"public","create_time":"$D2018-07-22T19:54:21.835Z","update_time":"$D2023-05-10T05:38:57.371Z","view_time":null,"first_public_time":"$D2018-07-23T04:33:25.788Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"457d56e422a65092","login":"kalebdf","name":"Kaleb Fulgham","avatar_url":"https://avatars-next.observableusercontent.com/b7f649dc0cdab822ff5adbd305c5e843a04773003c3f46b7cba52a79ec92cf8a","tier":"starter_2024"},"version":548,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"8c388caae427fc24","login":"kalebdf","name":"Kaleb Fulgham","avatar_url":"https://avatars-next.observableusercontent.com/b7f649dc0cdab822ff5adbd305c5e843a04773003c3f46b7cba52a79ec92cf8a","tier":null}],"members":[{"type":"individual","id":"8c388caae427fc24","login":"kalebdf","name":"Kaleb Fulgham","avatar_url":"https://avatars-next.observableusercontent.com/b7f649dc0cdab822ff5adbd305c5e843a04773003c3f46b7cba52a79ec92cf8a","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"A/B Testing - Deterministic Bucketing","cells":[{"id":0,"value":"md`# A/B Testing - Deterministic Bucketing`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":538,"value":"md`\n_UPDATE: We are hiring! Contact Kaleb at kalebdf@hey.com_\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":53,"value":"md`## A/B & Multivariate Testing\n\nLet's create an experiment, **New Membership Button**.\n\n**IF** the new membership form's action button changes
\n**BY** using the phrase \"Join Now\" we
\n**WILL** increase new memberships
\n**BECAUSE** people desire to belong.\n\nThe metric is *new memberships* with a desired effect of increasing it.\n\nControl is *A* / * \"Sign Up\" * / *0*. Treatment is *B* / *\"Join Now\"* / *1*.\n\nScenario: A new potential customer **Pat** enters. We flip a coin which can randomly land, 50/50, heads / 0 or tails / 1.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":49,"value":"variation = Math.floor(Math.random() * 2) // Could be 0 or 1 (click the ▶ button to flip again)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":422,"value":"variationLabel = (variation === 1) ? 'Join Now' : 'Sign Up'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":95,"value":"html`