27:null 28:["$","title",null,{"children":"Don't Use Moving Averages | unconed | Observable"}] 2d:T458,You pick a _window_, for example 10 days, and replace every sample with the local average. This creates the smoother blue curve. Because you don't have data for the future, this window usually points backwards in time: the data for day N is the average of the 10 days before, which causes a visual lag of ~5 days. For the sake of convenience, this lag is shifted out, so you can compare the two curves one-to-one. The moving average seems perfectly fine as a solution. In this post I will show that this is subtly wrong, and that this can have significant effects on how you perceive trends in the data. And it has nothing to do with lag. In short: moving averages look more interesting and rich than they actually are. The problem is particularly bad when the noise is irregular and artificial. This often means human or computer error (which causes outliers), but can also be inherent in the process that generates the data. The noise above is therefor carefully tuned to highlight this: it's a uniform random walk, but with additional random outliers added on top, mildly biased in the positive direction.2e:T4fc,Plot.plot({ width: 800, height: 250, grid: true, fx: {label: null}, color: { domain: ["Exponential Moving Average", "Exponential Moving Average^2", "Exponential Moving Average^3", "Exponential Moving Average^4"], range: [Color.teal, Color.teal, Color.teal, Color.teal], }, marks: [ Plot.line(toCurve(freqResponse(exponentialAverage(dirac(256, 0), lambda1)), "Exponential Moving Average", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 1, opacity: 0.5}), Plot.line(toCurve(freqResponse(exponentialAverage(exponentialAverage(dirac(256, 0), lambda1), lambda1)), "Exponential Moving Average^2", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 1}), Plot.line(toCurve(freqResponse(exponentialAverage(exponentialAverage(exponentialAverage(dirac(256, 0), lambda1), lambda1), lambda1)), "Exponential Moving Average^3", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 2}), Plot.line(toCurve(freqResponse(exponentialAverage(exponentialAverage(exponentialAverage(exponentialAverage(dirac(256, 0), lambda1), lambda1), lambda1), lambda1)), "Exponential Moving Average^4", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 3}), ] })2f:T4fc,Plot.plot({ width: 800, height: 250, grid: true, fx: {label: null}, color: { domain: ["Exponential Moving Average", "Exponential Moving Average^2", "Exponential Moving Average^3", "Exponential Moving Average^4"], range: [Color.teal, Color.teal, Color.teal, Color.teal], }, marks: [ Plot.line(toCurve(freqResponse(exponentialAverage(dirac(256, 0), lambda1)), "Exponential Moving Average", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 1, opacity: 0.5}), Plot.line(toCurve(freqResponse(exponentialAverage(exponentialAverage(dirac(256, 0), lambda2), lambda2)), "Exponential Moving Average^2", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 1}), Plot.line(toCurve(freqResponse(exponentialAverage(exponentialAverage(exponentialAverage(dirac(256, 0), lambda3), lambda3), lambda3)), "Exponential Moving Average^3", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 2}), Plot.line(toCurve(freqResponse(exponentialAverage(exponentialAverage(exponentialAverage(exponentialAverage(dirac(256, 0), lambda4), lambda4), lambda4), lambda4)), "Exponential Moving Average^4", {offset: -128}), {x: "x", y: "y", ffx: 'label', stroke: "label", strokeWidth: 3}), ] })30:T59a,## It's Correlated, Jim There is only one exception to all of this, and it's dubious: the 7-day moving average, and related divisions like a 24-hour moving average. Many data sets will have 7-day (and 24-hour) cycles in them. When you apply a 7-day moving average to this data, what you are really doing is constructing a _synthetic week_. You are making an ensemble with 1 Monday, 1 Tuesday, 1 Wednesday, and so on. When the window advances, you remove e.g. one Tuesday, and replace it with the next Tuesday. If it's hourly, you remove one 2 AM and add another 2 AM. In that situation, it's often not justified to use a smooth window, because that will _emphasize_ a different side of the week or day as you advance, giving it the highest weight under the peak. You would see a lot of up and down, but those trends might disappear if you sampled e.g. only Sundays. The same applies if you use any <7-day moving average: you are smearing out the difference between weekdays and weekends in a way that is not actually meaningful. You already know what the actual pattern is, it's every 7 days. In fact, if there are weekday effects, then even a 7-day moving average is a poor choice. You only have 1 sample of any given weekday in your ensemble, which is bad sampling practice by itself. Additionally 6 out of 7 samples in each subsequent ensemble are _identical_. This is just trying to bake too much result from too little data.31:T413,Plot.plot({ width: 1200, height: 800, grid: true, x: {label: null}, y: {label: null}, fx: {label: null, axis: null}, fy: {label: null, axis: null}, color: { domain: variants.flatMap((_, i) => [`v${i}`, `m${i}`, `b${i}`]), range: variants.flatMap(() => [Color.black, Color.blue, Color.green]), }, marks: [ ...variants.map((values, i) => Plot.line( toCurve(values, `v${i}`, {}), {x: "x", y: "y", fx: 0, fy: (d) => +d.label.slice(1), stroke: "label", strokeWidth: 1}), ), ...variants.map((values, i) => Plot.line( toCurve(binomialAverage(values, 30), `b${i}`, {offset: -17.5, clip: true}), {x: "x", y: "y", fx: 1, fy: (d) => +d.label.slice(1), stroke: "label", strokeWidth: 2}), ), ...variants.map((values, i) => Plot.line( toCurve(movingAverage(values, 10), `m${i}`, {offset: -5, clip: true}), {x: "x", y: "y", fx: 2, fy: (d) => +d.label.slice(1), stroke: "label", strokeWidth: 2}), ), ] })32:[{"id":"715db1223e067f00500780077febc6cebbdd90c151d3d78317c802732252052ab0e367039872ab9c77d6ef99e5f55a0724b35ddc898a1c99cb14c31a379af80a","href":"https://static.observableusercontent.com/files/715db1223e067f00500780077febc6cebbdd90c151d3d78317c802732252052ab0e367039872ab9c77d6ef99e5f55a0724b35ddc898a1c99cb14c31a379af80a","name":"penguins.csv","type":"text/csv","lastModified":1706362827915.882,"size":13506}] 2b:{"id":"a51954c61a72e1ef","slug":null,"title":"Don't Use Moving Averages","thumbnail":"fe86c768c6dbf0d8cc81e48071ae693d3856f30c355c5efe750e4bc9abf7d44a","access_level":"public","create_time":"$D2024-01-27T13:40:27.915Z","update_time":"$D2024-01-28T14:13:56.545Z","view_time":null,"first_public_time":"$D2024-01-27T13:40:27.915Z","has_unpublished_edits":null,"fork":{"id":"7dcef0ebff4028d9","slug":"input-chart","title":"Input + Chart","owner":{"id":"f35c755083683fe5","login":"observablehq","name":"Observable"}},"stars":29,"starred":false,"owner":{"type":"team","id":"80a61746f6a22aaa","login":"unconed","name":"","avatar_url":"https://avatars-next.observableusercontent.com/0e7f3c06495f9792cdb6e24c2e0228e26b88eb3591d874c51b0e56339b278e8d","tier":"starter_2024"},"version":1229,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"1a2aae7ef5a46fe1","login":"unconed","name":"","avatar_url":"https://avatars-next.observableusercontent.com/0e7f3c06495f9792cdb6e24c2e0228e26b88eb3591d874c51b0e56339b278e8d","tier":null}],"members":[{"type":"individual","id":"1a2aae7ef5a46fe1","login":"unconed","name":"","avatar_url":"https://avatars-next.observableusercontent.com/0e7f3c06495f9792cdb6e24c2e0228e26b88eb3591d874c51b0e56339b278e8d","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":3,"collection":null,"type":"notebook","body":{"title":"Don't Use Moving Averages","cells":[{"id":4,"value":"# Don't Use Moving Averages\n\nIn data science and journalism, the moving average is a common tool to smooth out noisy data.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":7,"value":"Plot.plot({\n height: 300,\n grid: true,\n x: {label: null},\n y: {label: null},\n color: {\n domain: [\"Input\", \"Moving Average\"],\n range: [Color.black, Color.blue],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(values, \"Input\"), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 1}),\n Plot.line(toCurve(movingAverage(values, 10), \"Moving Average\", {offset: -5, clip: true}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":51,"value":"$2d","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":313,"value":"----","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":304,"value":"## Smooth vs Smooth\n\nThere are of course many smoothing and interpolation methods out there, and it might seem like there is no right answer. But as your basic fallback option, both math and I say you should use a __binomial average__ instead.\n\nThis is where you _weigh_ each sample in your moving average according to a curve that starts and stops at zero, approximating a gaussian:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":346,"value":"Plot.plot({\n width: 400,\n height: 150,\n grid: true,\n x: {label: null},\n y: {label: null},\n color: {\n domain: [\"Binomial Window\"],\n range: [Color.green],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(binomialAverage(dirac(35, 0), 35), \"Binomial Window\", {offset: -17.5}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":498,"value":"Binomial windows are easy to make. For a given length N, you just take the N'th row of Pascal's triangle, and divide by ${tex`2^N`}:\n\n${tex`\\begin{matrix}\n [ & & & & & 1 & & & & & ] & / & 1 \\\\\n [ & & & & 1 & & 1 & & & & ] & / & 2 \\\\\n [ & & & 1 & & 2 & & 1 & & & ] & / & 4 \\\\\n [ & & 1 & & 3 & & 3 & & 1 & & ] & / & 8 \\\\\n [ & 1 & & 4 & & 6 & & 4 & & 1 & ] & / & 16 \\\\\n & & ... & & ...& & ... & & ...\n\\end{matrix}`}\n\nThese weights always sum to 1, resulting in this:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":267,"value":"Plot.plot({\n height: 300,\n grid: true,\n x: {label: null},\n y: {label: null},\n color: {\n domain: [\"Input\", \"Binomial Average\"],\n range: [Color.black, Color.green],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(values, \"Input\"), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 1}),\n Plot.line(toCurve(binomialAverage(values, 35), \"Binomial Average\", {offset: -17.5, clip: true}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":319,"value":"The difference with the moving average is small in absolute terms:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":297,"value":"Plot.plot({\n height: 300,\n grid: true,\n x: {label: null},\n y: {label: null},\n color: {\n domain: [\"Moving Average\", \"Binomial Average\"],\n range: [Color.blue, Color.green],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(movingAverage(values, 10), \"Moving Average\", {offset: -5, clip: true}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 2, strokeDasharray: [2, 4]}),\n Plot.line(toCurve(binomialAverage(values, 35), \"Binomial Average\", {offset: -17.5, clip: true}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 2}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":385,"value":"But when viewed side by side, it's clear the binomial is much smoother:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":369,"value":"Plot.plot({\n height: 600,\n grid: true,\n x: {label: null},\n y: {label: null},\n fy: {label: null, axis: null},\n color: {\n domain: [\"Moving Average\", \"Binomial Average\"],\n range: [Color.blue, Color.green],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(movingAverage(values, 10), \"Moving Average\", {offset: -5, clip: true}), {x: \"x\", y: \"y\", fy: 'label', stroke: \"label\", strokeWidth: 2}),\n Plot.line(toCurve(binomialAverage(values, 35), \"Binomial Average\", {offset: -17.5, clip: true}), {x: \"x\", y: \"y\", fy: 'label', stroke: \"label\", strokeWidth: 2}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":324,"value":"Here's the crucial thing: this binomial average is not over 10 days, but over _35_. That's why it ends 12.5 days sooner than the blue curve (= (35 - 10) / 2). In order to get a curve that is roughly equivalent to a given moving average, you need to use a binomial window that is 3.5x longer.\n\nThis might seem like a downside of binomial windows, but the opposite is true.\n\nThe green curve is perceptually clear: there is no noise to be seen, only a blobby path. This intuitively communicates the level of detail being captured. The blue curve is deceptive: it's only about as informative as the green curve, but it contains little fractal 'decorations' that make it look more _data-ish_. They hint at a subtle and elusive underlying pattern that isn't there.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":661,"value":"----","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":659,"value":"## Impulse Response","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":564,"value":"To understand why this happens, consider what a moving average actually is. It's a series of sums:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":566,"value":"\\begin{matrix}\na &+& b &+& c &+& d \\\\\n& & b &+& c &+& d &+& e \\\\\n& & & & c &+& d &+& e &+& f \\\\ \n& & & & & & d &+& e &+& f &+& g \\\\\n& & & & & & & ...\n\\end{matrix}","pinned":false,"hidden":false,"mode":"tex","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":575,"value":"The difference between two subsequent values is the removal of one term (e.g. ${tex`a`}) and the addition of another (e.g. ${tex`e`}). If your input is perfectly flat, except for a single bad outlier, then you get this:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":580,"value":"Plot.plot({\n height: 300,\n grid: true,\n color: {\n domain: [\"Input\", \"Moving Average\"],\n range: [Color.black, Color.blue],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(dirac(128, 64), \"Input\"), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 1}),\n Plot.line(toCurve(movingAverage(dirac(128, 64), 10), \"Moving Average\", {offset: -4.5, clip: true}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":587,"value":"It creates a _plateau_ when the bad value enters the window, and which persists until it exits the window again. It doesn't actually become smooth at all. The moving average is just a sum of these sharp-edged plateaus, with their height proportional to ${tex`1/window`}. If there are severe outliers, the only way to make it 'smooth' is to use absurdly long windows, so that each edge is vertically very small, which nobody does.\n\nThe binomial average instead creates a smooth bump regardless of window length, which is both qualitatively and quantitatively better:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":589,"value":"Plot.plot({\n height: 300,\n grid: true,\n color: {\n domain: [\"Input\", \"Binomial Average\"],\n range: [Color.black, Color.green],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(dirac(128, 64), \"Input\"), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 1}),\n Plot.line(toCurve(binomialAverage(dirac(128, 64), 10), \"Binomial Average\", {offset: -5, clip: true}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":419,"value":"This isn't just an opinion. In signal processing terms, both averages are low-pass filters. Their frequency response is also approximately window-like, preserving all the low frequencies near 0, and dampening all the high frequencies:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":404,"value":"Plot.plot({\n width: 800,\n height: 150,\n grid: true,\n fx: {label: null},\n color: {\n domain: [\"Moving Average\", \"Binomial Average\"],\n range: [Color.blue, Color.green],\n },\n marks: [\n Plot.line(toCurve(freqResponse(movingAverage(dirac(256, 0), 10)), \"Moving Average\", {offset: -128}), {x: \"x\", y: \"y\", fx: 'label', stroke: \"label\", strokeWidth: 3}),\n Plot.line(toCurve(freqResponse(binomialAverage(dirac(256, 0), 35)), \"Binomial Average\", {offset: -128}), {x: \"x\", y: \"y\", fx: 'label', stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":531,"value":"If you overlay them, you can see they agree tightly in the middle:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":528,"value":"Plot.plot({\n width: 800,\n height: 250,\n grid: true,\n fx: {label: null},\n color: {\n domain: [\"Moving Average\", \"Binomial Average\"],\n range: [Color.blue, Color.green],\n },\n marks: [\n Plot.line(toCurve(freqResponse(movingAverage(dirac(256, 0), 10)), \"Moving Average\", {offset: -128}), {x: \"x\", y: \"y\", ffx: 'label', stroke: \"label\", strokeWidth: 3, strokeDasharray: [2, 4]}),\n Plot.line(toCurve(freqResponse(binomialAverage(dirac(256, 0), 35)), \"Binomial Average\", {offset: -128}), {x: \"x\", y: \"y\", ffx: 'label', stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":360,"value":"But the binomial average _eliminates_ all the high frequencies, whereas the moving average does not. It has the classic \"peeking over the fence\" shape of:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":545,"value":"f(x) = \\left| \\frac{\\sin x}{x} \\right|","pinned":false,"hidden":false,"mode":"tex","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":561,"value":"In a 35-sample binomial window, the first and last ~7 samples barely contribute anything, so it's more fair to say it's about 2x longer, as the tails can be safely truncated up to a point:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":776,"value":"Window truncation: ${viewof logThreshold}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":355,"value":"Plot.plot({\n width: 400,\n height: 150,\n color: {\n domain: [\"Binomial Window\", \"Truncated Window\"],\n range: [Color.green, Color.green],\n },\n marks: [\n Plot.line(toCurve(binomialAverage(dirac(35, 0), 35), \"Binomial Window\", {offset: -17.5}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 2, strokeDasharray: [2, 4]}),\n Plot.line(toCurve(truncatedWindow, \"Truncated Window\", {offset: -17.5}), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":694,"value":"Plot.plot({\n width: 800,\n height: 250,\n grid: true,\n fx: {label: null},\n color: {\n domain: [\"Binomial Average\", \"Truncated Binomial Average\"],\n range: [Color.green],\n },\n marks: [\n Plot.line(toCurve(freqResponse(binomialAverage(dirac(256, 0), 35)), \"Binomial Average\", {offset: -128}), {x: \"x\", y: \"y\", ffx: 'label', stroke: \"label\", strokeWidth: 2, strokeDasharray: [2, 4]}),\n Plot.line(toCurve(freqResponse(truncate(binomialAverage(dirac(256, 0), 35), threshold)), \"Truncated Binomial Average\", {offset: -128}), {x: \"x\", y: \"y\", ffx: 'label', stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":489,"value":"That is, a 10 day moving average is actually a _mix_ of two signals:\n- The data, nicely smoothed over ~20 days\n- Additional random noise with a period of less than 20 days","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":613,"value":"It creates the illusion that you are tracking a trend tightly, when in fact, you are constantly deciding far _too early_ what the average actually is, adding jitter.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1112,"value":"It might seem strange that a flat 10-day average can somehow be equivalent to a smooth 20-day average—after all, isn't half the data missing? But the key thing to remember is that most time-series data is highly _correlated_.\n\nAs such, the difference between two subsequent values in a moving average, e.g ${tex`(e - a)`} is _not_ a sum of two independent variables. This allows for 'spooky action at a distance' in the frequency domain, and is the essence of the field of stochastic signal processing.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":668,"value":"----","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":666,"value":"## All the Way Down","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":617,"value":"Using binomial averages isn't a matter of taste, but of fourier transforms and the central limit theorem. A binomial average over N days is in fact equivalent to repeatedly doing a 2-day moving average N times:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1075,"value":"$${tex`\\begin{matrix}\n [ & & & & & 1 & & & & & ] & / & 1 \\\\\n [ & & & & 1 & & 1 & & & & ] & / & 2 \\\\\n [ & & & 1 & & 2 & & 1 & & & ] & / & 4 \\\\\n [ & & 1 & & 3 & & 3 & & 1 & & ] & / & 8 \\\\\n [ & 1 & & 4 & & 6 & & 4 & & 1 & ] & / & 16 \\\\\n & & ... & & ...& & ... & & ...\n\\end{matrix}`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1081,"value":"That's literally what the pattern in Pascal's triangle is: add the two values above, and repeat. That is, when you repeatedly smear out a flat box-window over itself, you get a binomial window. In the limit, this must become gaussian.\n\nThe same applies to dice. If you throw a single die, every number is equally likely: it's a flat window 1-6. If you throw two dice, it's triangular, as 7 is most likely. At three dice and more, it starts to look more and more gaussian. These aren't strictly binomial, they're _splines_, but the same rule applies.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1140,"value":"Plot.plot({\n width: 800,\n height: 150,\n axis: null,\n marks: [\n Plot.barY(dice(1).map((v, i) => ({v, i: i + 1})), {x: \"i\", y: \"v\", fx: 1}),\n Plot.barY(dice(2).map((v, i) => ({v, i: i + 2})), {x: \"i\", y: \"v\", fx: 2}),\n Plot.barY(dice(3).map((v, i) => ({v, i: i + 3})), {x: \"i\", y: \"v\", fx: 3}),\n Plot.barY(dice(4).map((v, i) => ({v, i: i + 4})), {x: \"i\", y: \"v\", fx: 4}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":943,"value":"In addition to the regular moving average, people commonly use the _exponential moving average_ (EMA). It's extremely simple in code, and requires virtually no memory. This is applied left-to-right, which creates an exponential tail over time. It has a sharp peak in its impulse response, which is visible in the output.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":944,"value":"Plot.plot({\n height: 300,\n grid: true,\n color: {\n domain: [\"Input\", \"Exponential Moving Average\"],\n range: [Color.black, Color.teal],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(dirac(64, 32), \"Input\"), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 1}),\n Plot.line(toCurve(exponentialAverage(dirac(64, 32), lambda1), \"Exponential Moving Average\"), {x: \"x\", y: \"y\", stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1085,"value":"Plot.plot({\n height: 600,\n grid: true,\n x: {label: null},\n y: {label: null},\n fy: {label: null, axis: null},\n color: {\n domain: [\"Exponential Moving Average\", \"Binomial Average\"],\n range: [Color.teal, Color.green],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(exponentialAverage(values, lambda1), \"Exponential Moving Average\", {offset: 0, clip: true}), {x: \"x\", y: \"y\", fy: 'label', stroke: \"label\", strokeWidth: 2}),\n Plot.line(toCurve(binomialAverage(values, 35), \"Binomial Average\", {offset: -17.5, clip: true}), {x: \"x\", y: \"y\", fy: 'label', stroke: \"label\", strokeWidth: 2}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":978,"value":"Its frequency response is much 'fatter' than either the moving or binomial average, for the same center-lobe:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":934,"value":"Plot.plot({\n width: 800,\n height: 150,\n grid: true,\n fx: {label: null},\n color: {\n domain: [\"Exponential Average\", \"Binomial Average\"],\n range: [Color.teal, Color.green],\n },\n marks: [\n Plot.line(toCurve(freqResponse(exponentialAverage(dirac(256, 0), lambda1)), \"Exponential Average\", {offset: -128}), {x: \"x\", y: \"y\", fx: 'label', stroke: \"label\", strokeWidth: 3}),\n Plot.line(toCurve(freqResponse(binomialAverage(dirac(256, 0), 35)), \"Binomial Average\", {offset: -128}), {x: \"x\", y: \"y\", fx: 'label', stroke: \"label\", strokeWidth: 3}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":954,"value":"Plot.plot({\n width: 800,\n height: 250,\n grid: true,\n fx: {label: null},\n color: {\n domain: [\"Exponential Moving Average\", \"Moving Average\", \"Binomial Average\"],\n range: [Color.teal, Color.blue, Color.green],\n },\n marks: [\n Plot.line(toCurve(freqResponse(exponentialAverage(dirac(256, 0), lambda1)), \"Exponential Moving Average\", {offset: -128}), {x: \"x\", y: \"y\", ffx: 'label', stroke: \"label\", strokeWidth: 3}),\n Plot.line(toCurve(freqResponse(movingAverage(dirac(256, 0), 10)), \"Moving Average\", {offset: -128}), {x: \"x\", y: \"y\", ffx: 'label', stroke: \"label\", strokeWidth: 2, strokeDasharray: [2, 4]}),\n Plot.line(toCurve(freqResponse(binomialAverage(dirac(256, 0), 35)), \"Binomial Average\", {offset: -128}), {x: \"x\", y: \"y\", ffx: 'label', stroke: \"label\", strokeWidth: 3, strokeDasharray: [2, 4]}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":999,"value":"This makes EMAs even worse at smoothing.\n\nHowever, the solution is very easy: just apply multiple EMAs in series. This pushes the tails down and makes it more gaussian:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1007,"value":"$2e","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1038,"value":"You can adjust the smoothing factor of your EMA to compensate for the narrowing:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1019,"value":"$2f","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1095,"value":"This creates something much closer to binomial, e.g. repeated 6 times:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1092,"value":"Plot.plot({\n height: 600,\n grid: true,\n x: {label: null},\n y: {label: null},\n fy: {label: null, axis: null},\n color: {\n domain: [\"Exponential Moving Average^6\", \"Binomial Average\"],\n range: [Color.teal, Color.green],\n legend: true,\n },\n marks: [\n Plot.line(toCurve(exponentialAverage(exponentialAverage(exponentialAverage(exponentialAverage(exponentialAverage(exponentialAverage(values, lambda6), lambda6), lambda6), lambda6), lambda6), lambda6), \"Exponential Moving Average^6\", {offset: -4, clip: true}), {x: \"x\", y: \"y\", fy: 'label', stroke: \"label\", strokeWidth: 2}),\n Plot.line(toCurve(binomialAverage(values, 35), \"Binomial Average\", {offset: -17.5, clip: true}), {x: \"x\", y: \"y\", fy: 'label', stroke: \"label\", strokeWidth: 2}),\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1132,"value":"\nIf you use vanilla moving averages of either kind, you're ignoring the entire collective edifice of Laplace, Fourier and Gauss—_they_ are the E, M and A in EMA. It's vestigial dark age tech, and the fix is easy. Apply them repeatedly, instead of once.\n\nRegardless of the technique, you are choosing a parameter called \"main frequency lobe\" and it is 100% a choice.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1124,"value":"----","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":647,"value":"$30","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":867,"value":"See, I can tune a knob, and produce pathologically noisy input which the moving average can never smooth out, looking even worse than the EMA. It's very simple, you anti-correlate e.g. every 10th sample. The result will still have sharp peaks and dips everywhere after a 10-day window:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":830,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":894,"value":"It makes for pretty, interesting, suggestive graphs though, that look like little landscapes.\n\nUnfortunately, that part of the output is, effectively, procedurally generated. Even when the data is real, it's essentially a hash function of the correlations inside, mangled by the gaps in the \"fence\".","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":791,"value":"`* * *`","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":615,"value":"In a world where people care about not lying with graphs, moving averages ought to be avoided. If you want to smooth things out, smooth them out. Don't half-ass it. You can do so using basic 17th century math.\n\nIn practice, I suspect this is the kind of thing many data jugglers will actively be averse to: it reveals exactly how vague a trend actually is. Like turning on the fluorescent lights in the club at 4am closing time, it removes any and all mystique.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1203,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":69,"value":"----","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":290,"value":"_In the interest of actually having nothing up my sleeve, you can re-run this notebook with any random seed:_","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":38,"value":"viewof seed = htl.html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":88,"value":"----","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1200,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":114,"value":"n = 128","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"values = {\n const rnd = Random.number(Seed(1), -1, 1);\n const walk = integrate(seq(n).map(i => rnd()));\n return walk.map(w => w + rnd()*rnd() + 4*Math.max(0,rnd()*rnd()*rnd()));\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":832,"value":"variants = {\n return seq(3).map(i => {\n const rnd = Random.number(Seed(2 + i), -1, 1);\n const pos = Random.number(Seed(3 + i), 0, 118);\n const walk = integrate(seq(n).map(i => rnd()));\n const jittered = walk.map(w => (w + rnd()*rnd() + 6*Math.max(0,rnd()*rnd()*rnd())));\n const w = 2;\n for (let j = 0; j < 64; ++j) {\n const k = pos()|0;\n jittered[k] += w;\n jittered[k+10] -= w;\n }\n return jittered;\n });\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":728,"value":"truncatedWindow = truncate(binomialAverage(dirac(35, 0), 35), threshold)\n ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":737,"value":"threshold = Math.pow(10, logThreshold)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":952,"value":"lambda1 = 0.28","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1022,"value":"lambda2 = Math.pow(lambda1, 0.5) / 1.414","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1029,"value":"lambda3 = Math.pow(lambda1, 0.33333) / 1.414","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1031,"value":"lambda4 = Math.pow(lambda1, 0.25) / 1.414","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1102,"value":"lambda5 = Math.pow(lambda1, 0.2) / 1.414","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1106,"value":"lambda6 = Math.pow(lambda1, 0.16666) / 1.414","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":193,"value":"----","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":90,"value":"Color = ({blue: \"#3090ff\", green: '#10af50', teal: '#30b0d0', black: '#404040'})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":17,"value":"Hash = {\n /** Murmur3 32-bit hash */\n\nconst C1 = 0xcc9e2d51;\nconst C2 = 0x1b873593;\nconst C3 = 0xe6546b64;\nconst C4 = 0x85ebca6b;\nconst C5 = 0xc2b2ae35;\n\nconst add = (a, b) => ((a|0) + (b|0)) >>> 0;\nconst rot = (a, b) => ((a << b) | (a >>> (32 - b))) >>> 0;\nconst mul = Math.imul;\n\nconst mixBits = (x, d) => {\n d = mul(d, C1);\n d = rot(d, 15);\n d = mul(d, C2);\n\n x ^= d;\n x = rot(x, 13);\n x = add(mul(x, 5), C3);\n\n return x;\n};\n\nconst scrambleBits = (x, n = 0) => {\n x ^= n;\n\n x ^= x >>> 16;\n x = mul(x, C4);\n x ^= x >>> 13;\n x = mul(x, C5);\n x ^= x >>> 16;\n\n return x;\n};\n\n return {scrambleBits, mixBits};\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"Random = {\n const uint32 = (seed = 1) => () => {\n seed = Hash.mixBits(seed, 1);\n return Hash.scrambleBits(seed, 1) >>> 0;\n };\n const float = (seed = 1) => {\n const rnd = uint32(seed);\n return () => rnd() / 0x100000000;\n };\n const number = (seed = 1, min = 0, max = 0xFFFFFFFF) => {\n const rnd = float(seed);\n return () => rnd() * (max - min) + min;\n };\n return {uint32, float, number};\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":200,"value":"Seed = (i) => Hash.mixBits(seed, i);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":27,"value":"seq = (n, s = 0, d = 1) => Array.from({length: n}).map((_, i) => s + d * i);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":95,"value":"dirac = (n, i) => seq(n).map(j => +(i === j));","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":33,"value":"integrate = (values) => values.reduce((a, b, i) => (a[i] = (a[i - 1]||0) + b, a), [])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":59,"value":"convolve = (values, filter) => values.map((_, i) => filter.reduce((a, b, j) => a + b * (values[i - j] || 0), 0));","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":696,"value":"truncate = (filter, threshold) => filter.map(v => v < threshold ? 0 : v);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":710,"value":"countNonZero = (filter) => filter.map(v => +!!v).reduce((a, b) => a + b, 0);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":55,"value":"movingAverage = (values, n) => {\n const filter = seq(n).map(i => 1/n);\n return convolve(values, filter);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":232,"value":"binomialAverage = (values, n) => {\n let filter = [1];\n for (let i = 0; i < n; ++i) filter = seq(filter.length + 1).map(i => ((filter[i - 1] || 0) + (filter[i] || 0)) / 2);\n return convolve(values, filter);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":937,"value":"exponentialAverage = (values, lambda) => {\n const out = [];\n const n = values.length;\n for (let i = 0; i < n; ++i) out[i] = (out[i-1]||0) * (1 - lambda) + values[i] * lambda;\n return out;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":398,"value":"freqResponse = (filter) => {\n const n = filter.length;\n return seq(n).map(i => {\n let band = (i + n/2) % n;\n let re = 0;\n let im = 0;\n for (let j = 0; j < n; ++j) {\n let th = j * band * Math.PI * 2 / n;\n let r = filter[j];\n re += Math.cos(th) * r;\n im += Math.sin(th) * r;\n }\n return Math.sqrt(re*re + im*im);\n });\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1143,"value":"dice = (n) => {\n const die = [1/6, 1/6, 1/6, 1/6, 1/6, 1/6];\n let out = die;\n for (let i = 1; i < n; ++i) out = convolve([...out, 0, 0, 0, 0, 0], die);\n return out;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":80,"value":"toCurve = (values, label, {offset = 0, clip = false} = {}) => values.map((y, x) => ({x: x + offset, y, label})).filter(({x}) => clip ? x >= 0 : true);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":713,"value":"viewof logThreshold = htl.html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W32","stdlib":"1","titling":"auto"}} 2c:"$2b:body"