28:null 29:["$","title",null,{"children":"Habit, working memory, and reward learning | Personal | Observable"}] 2e:T48d,# Habit, working memory, and reward learning In "[A habit and working memory model as an alternative account of human reward-based learning](https://rdcu.be/e5tII)", Collins argues that vanilla reinforcement learning models do not provide the best account of how human behaviour evolves during reward learning. Instead, she argues that fast but somewhat forgetful loading of reward associations into working memory coupled with a slower tendency to habitually repeat past actions provides a better fit. An appealing feature of Collins' work is that the individual model components are fairly simple, making them easy to grasp and reproduce. To see just _how_ easy to reproduce, this week I challenged myself to reproduce as much of her model as I could in half an hour before a lab journal club to discuss the paper. In that short window, I managed to put together a small simulation of how each of the three low-level learning rules in the paper respond to a step change in reward. You can find the fruits of that half hour [here](https://observablehq.com/d/9086f2a1bf98c67a) and a slightly tidier version below. If only more models were this easy to play with!2f:T400,

Working memory ${Inputs.bind( Inputs.form({ alphaPos: Inputs.range([0, 1], {label: tex`\alpha_\text{WM}(1)`}), alphaNeg: Inputs.range([0, 1], {label: tex`\alpha_\text{WM}(0)`}), phi: Inputs.range([0, 1], {label: tex`\phi_\text{WM}`}) }), viewof wmParams )}

RL ${Inputs.bind( Inputs.form({ alphaPos: Inputs.range([0, 1], {label: tex`\alpha_\text{RL}(1)`}), alphaBias: Inputs.range([0, 1], {label: tex`\text{bias}_\text{RL}`}) }), viewof rlParams )}

Habit ${Inputs.bind( Inputs.form({ alphaPos: Inputs.range([0, 1], {label: tex`\alpha_\text{H}(1)`}), alphaBias: Inputs.range([0, 1], {label: tex`\text{bias}_\text{H}`}), r0: Inputs.range([0, 1], {label: tex`r_0`, value: 1}) }), viewof hParams )}

30:[] 2c:{"id":"95e1ff16808e6b19","slug":null,"title":"Habit, working memory, and reward learning","thumbnail":"983bc92c734e20f833884c2e9d6be717502f2a3894b7106ac11b0d2c86b83511","access_level":"unlisted","create_time":"$D2026-02-24T13:15:32.798Z","update_time":"$D2026-02-24T17:45:38.574Z","view_time":null,"first_public_time":null,"has_unpublished_edits":null,"fork":true,"stars":0,"starred":false,"owner":{"type":"team","id":"d088261d55cee556","login":"personal409","name":"Personal","avatar_url":"https://avatars-next.observableusercontent.com/78247113ca3143dcc04a3ed672f5656583f2500b1e6b17f5ce0a9522922567c1","tier":"starter_2024"},"version":153,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"8cd17d30e36ce759","login":"emerson-harkin","name":"Emerson Harkin","avatar_url":"https://avatars-next.observableusercontent.com/5b5d60731dade04ba6abae02eba47a5f5c217278f06ecb70122e46396867fdb3","tier":null}],"members":[{"type":"individual","id":"8cd17d30e36ce759","login":"emerson-harkin","name":"Emerson Harkin","avatar_url":"https://avatars-next.observableusercontent.com/5b5d60731dade04ba6abae02eba47a5f5c217278f06ecb70122e46396867fdb3","tier":null,"role":"editor"},{"type":"team","id":"d088261d55cee556","login":"personal409","name":"Personal","avatar_url":"https://avatars-next.observableusercontent.com/78247113ca3143dcc04a3ed672f5656583f2500b1e6b17f5ce0a9522922567c1","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Habit, working memory, and reward learning","cells":[{"id":0,"value":"$2e","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":67,"value":"$2f","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":20,"value":"{\n const trajectories = [\n ...simulateWM(reward, wmParams),\n ...simulateRL(reward, rlParams),\n ...simulateHabit(reward, hParams),\n ];\n return html`\n ${Plot.plot({\n marks: [Plot.line(reward, {x: \"trialNumber\", y: \"y\", stroke: \"black\"})],\n y: {label: \"Reward\"},\n x: {label: \"Trial number\"},\n height: 100\n })}\n ${Plot.plot({\n marks: [Plot.line(trajectories, {x: \"trialNumber\", y: \"y\", stroke: \"variable\"})],\n color: {legend: true},\n y: {domain: [0, 1], label: null},\n x: {label: \"Trial number\"}\n })}`\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":51,"value":"## Modules\n\nAll three model components follow the general template of a sum of exponentially-weighted moving averages, each of which has the form\n${tex.block`\\hat{x}_{t+1} = \\hat{x}_t + \\alpha (x_t - \\hat{x}_t)`}\nwhere ${tex`\\hat{x}_t`} is the value of the average on trial ${tex`t`}, ${tex`x`} is the quantity being averaged, and ${tex`\\alpha`} is a learning rate parameter. An important example of this type of model in the context of reward learning is the Rescorla–Wagner learning rule, in which ${tex`x`} is taken to be the reward associated with a conditioned stimulus, ${tex`\\hat{x}`} is the learned value of the conditioned stimulus, and ${tex`\\alpha`} is the product of stimulus intensity and associability. Since the Rescorla–Wagner rule will be familiar to many people, interpreting Collins' model components as embellishments of this basic rule might be helpful to build further intuition for how they work.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":118,"value":"### Working memory\n\n_Rescorla–Wagner plus..._\n\n- separate learning rates for rewards and reward omissions, parameterized as a pair of learning rates\n- forgetting","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":10,"value":"simulateWM = (reward, params) => {\n const init = 0.5;\n const response = [{trialNumber: reward[0].trialNumber, y: init, variable: \"WM\"}];\n for (const rew of reward.slice(1)) {\n const last = response[response.length - 1];\n let delta;\n if (rew.y > 0.5) {\n delta = params.alphaPos * (rew.y - last.y) + params.phi * (init - last.y);\n } else {\n delta = params.alphaNeg * (rew.y - last.y) + params.phi * (init - last.y);\n }\n response.push({\n trialNumber: rew.trialNumber,\n y: last.y + delta,\n variable: \"WM\"\n });\n }\n return response;\n}","pinned":true,"hidden":true,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":122,"value":"### RL\n\n_Rescorla–Wagner plus..._\n\n- separate learning rates for rewards and reward omissions, parameterized as a base learning rate for rewards and a bias factor for reward omissions","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":12,"value":"simulateRL = (reward, params) => {\n const init = 0.5;\n const response = [{trialNumber: reward[0].trialNumber, y: init, variable: \"RL\"}];\n for (const rew of reward.slice(1)) {\n const last = response[response.length - 1];\n let delta;\n if (rew.y > 0.5) {\n delta = params.alphaPos * (rew.y - last.y);\n } else {\n delta = params.alphaPos * params.alphaBias * (rew.y - last.y);\n }\n response.push({\n trialNumber: rew.trialNumber,\n y: last.y + delta,\n variable: \"RL\"\n });\n }\n return response;\n}","pinned":true,"hidden":true,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":127,"value":"### Habit\n\n_Rescorla–Wagner plus..._\n\n- separate learning rates for rewards and reward omissions, parameterized as a base learning rate for rewards and a bias factor for reward omissions\n- reward omissions can have non-zero value","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":15,"value":"simulateHabit = (reward, params) => {\n const init = 0.5;\n const response = [{trialNumber: reward[0].trialNumber, y: init, variable: \"Habit\"}];\n for (const rew of reward.slice(1)) {\n const last = response[response.length - 1];\n let delta;\n if (rew.y > 0.5) {\n delta = params.alphaPos * (rew.y - last.y);\n } else {\n delta = params.alphaPos * params.alphaBias * (params.r0 - last.y);\n }\n response.push({\n trialNumber: rew.trialNumber,\n y: last.y + delta,\n variable: \"Habit\"\n });\n }\n return response;\n}","pinned":true,"hidden":true,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":129,"value":"## Nuts and bolts\n\nThis block sets up the reward function.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"reward = {\n const trialNumber = d3.range(-50, 200);\n const rew = [];\n for (const t of trialNumber) {\n if (t < 0) {\n rew.push({trialNumber: t, y: 0, variable: \"reward\"});\n } else if (t < 100) {\n rew.push({trialNumber: t, y: 1, variable: \"reward\"});\n } else {\n rew.push({trialNumber: t, y: 0, variable: \"reward\"});\n }\n }\n return rew;\n}","pinned":true,"hidden":true,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":136,"value":"The following blocks set parameter defaults.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":56,"value":"viewof wmParams = Inputs.input({\n alphaPos: 0.1,\n alphaNeg: 0.1,\n phi: 0.1\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":73,"value":"viewof rlParams = Inputs.input({\n alphaPos: 0.1,\n alphaBias: 1\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":82,"value":"viewof hParams = Inputs.input({\n alphaPos: 0.1,\n alphaBias: 0.1,\n r0: 1\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W30","stdlib":"1"}} 2d:"$2c:body"