3f:null 40:["$","title",null,{"children":"Benford Distribution | geo-theo | Observable"}] 46:T8e2, ### Explanation of Newcomb-Benford Law Newcomb-Benford's (NB) Law states the first digit of numbers in real-life data sets is governed by ${tex`p_i = \log_{10}\left (1 + \frac{1}{i}\right)`} where ${tex`p_i`} is the probability of the digit ${tex`i \in [1...9]`}. The following table shows the expected probability for each digit. | ${tex`i`} | ${tex`p_i`} | |---|----------| | 1 | 0.30103 | | 2 | 0.17609 | | 3 | 0.12494 | | 4 | 0.09691 | | 5 | 0.079181 | | 6 | 0.066947 | | 7 | 0.057992 | | 8 | 0.051153 | | 9 | 0.045757 | This "Law" in data occurs frequently enough that it has been used to test for accounting fraud, government corruption, and the validity of scientific results, such as COVID-19 data. Not all data have this distribution, for NB's law to hold data should: - **have sufficient span** either a complete or several orders of magnitude in the data's span. - **have large enough sample size** random chance could make a few first digits non-NB. If the data set is small enough (say 25 or less), this could have a significant impact on statistical tests for NB distributions. - **be right skewed** often this is associated with some combinatory or self-organized aspect of the data. Right skew generally increases the span, returning us to the first expectation. - **not be human assigned** serial numbers or telephone numbers would not work. Likewise, numbers that have been manipulated by humans to serve some purpose will not work. Identifying such numbers is the point of this exercise. - **not be bound** the data should not have a hard maximum or minimum imposed by the physical setting. For example, daily temperatures are not Benford distributed. One what to think of data having these characteristics is that the data have some underlying, exponential process operating generate it. This would mean that random samples from the data would stretch across orders of magnitude, but be the samples would be evenly distributed across several orders of magnitude. As such, we see why NB's Law emerges from data sets having the above characteristics, by considering logarithmic axes drawn on top of linear axes. Can you see the red lines on the log scale are at exactly the points give by ${tex`p_i = \log_{10}\left (1 + \frac{1}{i}\right)`}?47:T6c3,(() => { const counts = d3.rollup( purchasesDigit1, rows => rows.length, d => d[4] ); const data = d3.range(1, 10).map(digit => { const count = counts.get(digit) ?? 0; const share = count / (purchasesDigit1.length || 1); const expected = Benford_distribution.find(d => d.digit === digit)?.p_i ?? 0; return { digit, count, share, expected }; }); return Plot.plot({ title: "Distribution of 2023 credit card purchase first digits", subtitle: "Observed purchases compared with Benford's expected distribution", width, height: 360, marginLeft: 60, x: { label: "First digit", domain: d3.range(1, 10) }, y: { label: "Share of purchases", tickFormat: ".0%", grid: true }, color: { domain: ["Purchases", "Benford"], range: ["lightblue", "#1f4e79"], legend: true }, marks: [ // Actual purchase distribution Plot.barY(data, { x: "digit", y: "share", fill: "lightblue", inset: 1, tip: true, title: d => `First digit: ${d.digit}\n` + `Samples: ${d3.format(",")(d.count)}\n` + `Purchase share: ${d3.format(".2%")(d.share)}\n` + `Benford expected: ${d3.format(".2%")(d.expected)}` }), // Benford expectation Plot.lineY(data, { x: "digit", y: "expected", stroke: "#1f4e79", strokeWidth: 3 }), // Points make the expected values easy to identify Plot.dot(data, { x: "digit", y: "expected", fill: "#1f4e79", r: 4 }), Plot.ruleY([0]) ] }); })() 48:T4c8,### Testing for Benford Distributions Give a data set that may be Bedford, the first digits can be evaluated to estimate the probability of each digit's value. Call that data set ${tex`p^\dagger`}. In addition to visual inspection, the ${tex`d^*`} statistic is frequently used ${tex`d^* = \frac{1}{ 1.03606 }\sqrt{ \sum_{i=1}^9 (p_i-p^\dagger_i)^2} `}. Where the denominator is just the maximum difference between Benford distribution of first digits and some data set. This makes ${tex`d^* \in`} [0,1] where ${tex`d^* =`} 0 is maximally Benford and ${tex`d^* =`} 1 is maximally non-Benford. Provided there are sufficient samples, values of ${tex`d^* > `} 0.25 might be considered fraudulent, and values over 0.19 should attract additional scrutiny. However, questions about the span of the data and the number of samples make firm guidelines difficult to state. ### Further reading [An introduction](https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1740-9713.2016.00919.x) ## Something to try Below I've uploaded a CSV file. It consists of 6 columns of different length. They are all drawn from data you may find interesting. In your groups, develop a method of determining if each column is Bedford distributed. 49:T121b,(() => { const columns = ["A", "B", "C", "D", "E", "F"]; const digits = d3.range(1, 10); // Calculate first-digit frequencies for each column. const data = columns.flatMap(column => { const values = mystery .map(d => Math.abs(Number(d[column]))) .filter(value => Number.isFinite(value) && value > 0); const counts = d3.rollup( values, rows => rows.length, value => Number(value.toExponential()[0]) ); return digits.map(digit => ({ column, digit, count: counts.get(digit) ?? 0, share: (counts.get(digit) ?? 0) / (values.length || 1) })); }); const width = 928; const height = 500; const marginTop = 85; const marginRight = 20; const marginBottom = 65; const marginLeft = 65; const fx = d3.scaleBand() .domain(columns) .range([marginLeft, width - marginRight]) .paddingInner(0.2) .paddingOuter(0.05); const x = d3.scaleBand() .domain(digits) .range([0, fx.bandwidth()]) .padding(0.08); const color = d3.scaleOrdinal() .domain(columns) .range(d3.schemeSpectral[6]); const y = d3.scaleLinear() .domain([0, d3.max(data, d => d.share) || 1]) .nice() .range([height - marginBottom, marginTop]); const svg = d3.create("svg") .attr("width", width) .attr("height", height) .attr("viewBox", [0, 0, width, height]) .attr( "style", "max-width: 100%; height: auto; font: 12px sans-serif;" ); // Title. svg.append("text") .attr("x", marginLeft) .attr("y", 22) .attr("font-size", 17) .attr("font-weight", 600) .text("Mystery distributions by first digit"); // Separate legend boxes for columns A–F. const legend = svg.append("g") .attr("transform", `translate(${marginLeft}, 42)`); legend.append("text") .attr("y", 11) .text("Column"); const items = legend.selectAll("g") .data(columns) .join("g") .attr("transform", (d, i) => `translate(${65 + i * 60}, 0)`); items.append("rect") .attr("width", 14) .attr("height", 14) .attr("fill", d => color(d)) .attr("stroke", "#888") .attr("stroke-width", 0.5); items.append("text") .attr("x", 21) .attr("y", 11) .text(d => d); // Horizontal gridlines. svg.append("g") .attr("transform", `translate(${marginLeft}, 0)`) .call( d3.axisLeft(y) .ticks(6) .tickSize(-(width - marginLeft - marginRight)) .tickFormat(() => "") ) .call(g => g.select(".domain").remove()) .call(g => g.selectAll("line").attr("stroke", "#e5e7eb")); // Nine bars per column. const groups = svg.append("g") .selectAll("g") .data(d3.group(data, d => d.column)) .join("g") .attr("transform", ([column]) => `translate(${fx(column)}, 0)`); const bars = groups.selectAll("rect") .data(([, rows]) => rows) .join("rect") .attr("x", d => x(d.digit)) .attr("y", d => y(d.share)) .attr("width", x.bandwidth()) .attr("height", d => y(0) - y(d.share)) .attr("fill", d => color(d.column)) .attr("stroke", "#888") .attr("stroke-width", 0.4); // Hover tooltips. bars.append("title") .text(d => `Column: ${d.column}\n` + `First digit: ${d.digit}\n` + `Samples: ${d3.format(",")(d.count)}\n` + `Share: ${d3.format(".2%")(d.share)}` ); bars .on("pointerenter", function() { d3.select(this) .attr("stroke", "#111") .attr("stroke-width", 1.5); }) .on("pointerleave", function() { d3.select(this) .attr("stroke", "#888") .attr("stroke-width", 0.4); }); // Digit labels beneath each group. groups.append("line") .attr("x1", 0) .attr("x2", fx.bandwidth()) .attr("y1", y(0)) .attr("y2", y(0)) .attr("stroke", "currentColor") .attr("stroke-width", 1.5); // Column labels beneath the digit labels. svg.append("g") .attr("transform", `translate(0, ${height - marginBottom + 23})`) .call(d3.axisBottom(fx).tickSize(0).tickPadding(7)) .call(g => g.select(".domain").remove()) .call(g => g.selectAll("text") .attr("font-size", 13) .attr("font-weight", 600)); // Percentage axis. svg.append("g") .attr("transform", `translate(${marginLeft}, 0)`) .call(d3.axisLeft(y).ticks(6, ".0%").tickSize(0)) .call(g => g.select(".domain").remove()); svg.append("text") .attr("transform", "rotate(-90)") .attr("x", -(marginTop + height - marginBottom) / 2) .attr("y", 17) .attr("text-anchor", "middle") .text("Share of samples"); return Object.assign(svg.node(), {scales: {color}}); })()4a:T1ff0,(() => { const columns = ["A", "B", "C", "D", "E", "F"]; const digits = d3.range(1, 10); // Calculate first-digit proportions for each Mystery dataset. const emptyColumns = []; const data = columns.flatMap(column => { const values = mystery .map(d => Math.abs(Number(d[column]))) .filter(value => Number.isFinite(value) && value > 0); if (!values.length) emptyColumns.push(column); const counts = d3.rollup( values, rows => rows.length, value => Number(value.toExponential()[0]) ); return digits.map(digit => ({ column, digit, count: counts.get(digit) ?? 0, total: values.length, share: values.length ? (counts.get(digit) ?? 0) / values.length : 0 })); }); const benford = Benford_distribution .map(d => ({ digit: Number(d.digit), share: Number(d.p_i) })) .filter(d => digits.includes(d.digit)) .sort((a, b) => a.digit - b.digit); // Dimensions. const width = 1000; const height = 560; const marginTop = 95; const marginRight = 25; const marginBottom = 55; const marginLeft = 65; // Main groups: first digits 1–9. const x = d3.scaleBand() .domain(digits) .range([marginLeft, width - marginRight]) .paddingInner(0.22) .paddingOuter(0.08); // Six bars per group, ordered A–F. const xColumn = d3.scaleBand() .domain(columns) .range([0, x.bandwidth()]) .padding(0.06); const color = d3.scaleOrdinal() .domain(columns) .range(d3.schemeSpectral[6]); const maxShare = Math.max( d3.max(data, d => d.share) ?? 0, d3.max(benford, d => d.share) ?? 0 ); const y = d3.scaleLinear() .domain([0, maxShare ? maxShare * 1.15 : 1]) .nice() .range([height - marginBottom, marginTop]); const digitCenter = digit => x(digit) + x.bandwidth() / 2; const svg = d3.create("svg") .attr("width", width) .attr("height", height) .attr("viewBox", [0, 0, width, height]) .attr( "style", "max-width: 100%; height: auto; font: 12px sans-serif;" ); // Title. svg.append("text") .attr("x", marginLeft) .attr("y", 24) .attr("font-size", 18) .attr("font-weight", 600) .text("Mystery first-digit distributions vs. Benford"); // Dataset legend. const legend = svg.append("g") .attr("transform", `translate(${marginLeft}, 45)`); legend.append("text") .attr("y", 11) .text("Dataset"); const items = legend.selectAll("g") .data(columns) .join("g") .attr("transform", (d, i) => `translate(${65 + i * 55}, 0)`); items.append("rect") .attr("width", 14) .attr("height", 14) .attr("fill", d => color(d)) .attr("stroke", "#888") .attr("stroke-width", 0.5); items.append("text") .attr("x", 20) .attr("y", 11) .text(d => d); // Benford legend entry. const referenceLegend = legend.append("g") .attr("transform", "translate(415, 7)"); referenceLegend.append("line") .attr("x1", 0) .attr("x2", 40) .attr("stroke", "#666") .attr("stroke-width", 2.5) .attr("stroke-dasharray", "1,6") .attr("stroke-linecap", "round"); referenceLegend.append("circle") .attr("cx", 20) .attr("r", 4.5) .attr("fill", "#111"); referenceLegend.append("text") .attr("x", 49) .attr("dy", "0.35em") .text("Benford"); // Horizontal gridlines. svg.append("g") .attr("transform", `translate(${marginLeft}, 0)`) .call( d3.axisLeft(y) .ticks(7) .tickSize(-(width - marginLeft - marginRight)) .tickFormat(() => "") ) .call(g => g.select(".domain").remove()) .call(g => g.selectAll("line") .attr("stroke", "#e5e7eb")); // Grouped Mystery bars. const groups = svg.append("g") .selectAll("g") .data(d3.group(data, d => d.digit)) .join("g") .attr("transform", ([digit]) => `translate(${x(digit)}, 0)`); const bars = groups.selectAll("rect") .data(([, rows]) => rows) .join("rect") .attr("x", d => xColumn(d.column)) .attr("y", d => y(d.share)) .attr("width", xColumn.bandwidth()) .attr("height", d => y(0) - y(d.share)) .attr("fill", d => color(d.column)) .attr("stroke", "#888") .attr("stroke-width", 0.3); bars.append("title") .text(d => `Dataset: ${d.column}\n` + `First digit: ${d.digit}\n` + `Samples: ${d3.format(",")(d.count)}\n` + `Share: ${d3.format(".2%")(d.share)}` ); bars .on("pointerenter", function() { d3.select(this) .attr("stroke", "#111") .attr("stroke-width", 1.5); }) .on("pointerleave", function() { d3.select(this) .attr("stroke", "#888") .attr("stroke-width", 0.3); }); // Slightly bold baseline beneath each digit group. groups.append("line") .attr("x1", 0) .attr("x2", x.bandwidth()) .attr("y1", y(0)) .attr("y2", y(0)) .attr("stroke", "#333") .attr("stroke-width", 1.5); // Smooth dotted curve through Benford's expected proportions. const line = d3.line() .x(d => digitCenter(d.digit)) .y(d => y(d.share)) .curve(d3.curveMonotoneX); svg.append("path") .datum(benford) .attr("d", line) .attr("fill", "none") .attr("stroke", "#666") .attr("stroke-width", 2.5) .attr("stroke-dasharray", "1,7") .attr("stroke-linecap", "round") .attr("pointer-events", "none"); // 50%-opacity horizontal reference lines across each group. svg.append("g") .attr("stroke", "#111") .attr("stroke-opacity", 0.5) .attr("stroke-width", 1.5) .attr("pointer-events", "none") .selectAll("line") .data(benford) .join("line") .attr("x1", d => x(d.digit)) .attr("x2", d => x(d.digit) + x.bandwidth()) .attr("y1", d => y(d.share)) .attr("y2", d => y(d.share)); // Benford points. const points = svg.append("g") .selectAll("circle") .data(benford) .join("circle") .attr("cx", d => digitCenter(d.digit)) .attr("cy", d => y(d.share)) .attr("r", 5.5) .attr("fill", "#111") .attr("stroke", "white") .attr("stroke-width", 1.5); points.append("title") .text(d => `Benford\nFirst digit: ${d.digit}\n` + `Expected share: ${d3.format(".2%")(d.share)}` ); points .on("pointerenter", function() { d3.select(this).attr("r", 7); }) .on("pointerleave", function() { d3.select(this).attr("r", 5.5); }); // Benford percentage labels above the points. // A white outline keeps labels readable over colored bars. svg.append("g") .attr("text-anchor", "middle") .attr("font-size", 12) .attr("font-weight", 700) .attr("fill", "#111") .attr("stroke", "white") .attr("stroke-width", 4) .attr("stroke-linejoin", "round") .attr("paint-order", "stroke") .attr("pointer-events", "none") .selectAll("text") .data(benford) .join("text") .attr("x", d => digitCenter(d.digit)) .attr("y", d => y(d.share) - 13) .text(d => d3.format(".2%")(d.share)); // Axes. svg.append("g") .attr("transform", `translate(0, ${y(0)})`) .call(d3.axisBottom(x).tickSize(0).tickPadding(10)) .call(g => g.select(".domain").remove()) .call(g => g.selectAll("text") .attr("font-size", 13) .attr("font-weight", 600)); svg.append("g") .attr("transform", `translate(${marginLeft}, 0)`) .call(d3.axisLeft(y).ticks(7, ".0%").tickSize(0).tickPadding(8)) .call(g => g.select(".domain").remove()); svg.append("text") .attr("x", (marginLeft + width - marginRight) / 2) .attr("y", height - 10) .attr("text-anchor", "middle") .text("First digit"); svg.append("text") .attr("transform", "rotate(-90)") .attr("x", -(marginTop + height - marginBottom) / 2) .attr("y", 17) .attr("text-anchor", "middle") .text("Share of samples"); if (emptyColumns.length) { svg.append("text") .attr("x", marginLeft) .attr("y", 80) .attr("fill", "#666") .text(`No valid nonzero samples: ${emptyColumns.join(", ")}`); } return Object.assign(svg.node(), {scales: {color}}); })()4b:T343f,(() => { const columns = ["A", "B", "C", "D", "E", "F"]; const digits = d3.range(1, 10); const color = d3.scaleOrdinal() .domain(columns) .range(d3.schemeSpectral[6]); const benford = Benford_distribution .map(d => ({ digit: Number(d.digit), share: Number(d.p_i) })) .filter(d => digits.includes(d.digit)) .sort((a, b) => a.digit - b.digit); // Validate all nine probabilities and normalize small rounding differences. const probabilitySum = d3.sum(benford, d => d.share); if (benford.length !== 9 || new Set(benford.map(d => d.digit)).size !== 9 || benford.some(d => !Number.isFinite(d.share) || d.share <= 0) || Math.abs(probabilitySum - 1) > 0.001) { throw new Error("Benford_distribution must contain digits 1–9 once each, with positive p_i probabilities summing to 1 (use proportions, not percentages)."); } benford.forEach(d => { d.share /= probabilitySum; }); const probability = new Map(benford.map(d => [d.digit, d.share])); const alpha = 0.05; const df = 8; // Nine categories minus one; no fitted parameters. // Upper-tail probability for chi-square with exactly 8 degrees of freedom. // P(X >= statistic) = exp(-t) * (1 + t + t²/2 + t³/6), t = statistic/2. // Log-space evaluation avoids overflow in the polynomial for large statistics. function chiSquarePValue8(statistic) { if (statistic === 0) return 1; if (statistic === Infinity) return 0; const t = statistic / 2; const logT = Math.log(t); const terms = [0, logT, 2 * logT - Math.log(2), 3 * logT - Math.log(6)]; const largest = Math.max(...terms); const logSum = largest + Math.log(d3.sum(terms, v => Math.exp(v - largest))); return Math.min(1, Math.exp(-t + logSum)); } const formatP = p => p == null ? "—" : p < 0.0001 ? "<0.0001" : d3.format(".4f")(p); // Prepare all Mystery datasets. const datasets = new Map(columns.map(column => { const values = mystery .map(d => Math.abs(Number(d[column]))) .filter(value => Number.isFinite(value) && value > 0); const counts = d3.rollup( values, rows => rows.length, value => Number(value.toExponential()[0]) ); return [column, { total: values.length, rows: digits.map(digit => ({ digit, count: counts.get(digit) ?? 0, share: values.length ? (counts.get(digit) ?? 0) / values.length : 0 })) }]; })); // Pearson's statistic uses counts, not the plotted proportions. for (const dataset of datasets.values()) { dataset.rows.forEach(row => { row.expected = dataset.total * probability.get(row.digit); row.contribution = dataset.total ? (row.count - row.expected) ** 2 / row.expected : null; }); dataset.chiSquared = dataset.total ? d3.sum(dataset.rows, d => d.contribution) : null; dataset.pValue = dataset.total ? chiSquarePValue8(dataset.chiSquared) : null; dataset.minExpected = d3.min(dataset.rows, d => d.expected); dataset.smallExpected = dataset.total > 0 && dataset.minExpected < 5; dataset.conclusion = !dataset.total ? "No valid samples" : dataset.smallExpected ? "Small expected counts; approximation may be unreliable" : dataset.pValue < alpha ? "Reject Benford at α = 0.05" : "Do not reject Benford at α = 0.05"; } // Keep the axis fixed across A–F. const maxShare = Math.max( d3.max(benford, d => d.share) ?? 0, d3.max( [...datasets.values()].flatMap(d => d.rows), d => d.share ) ?? 0 ); const width = 1000; const height = 500; const marginTop = 35; const marginRight = 45; const marginBottom = 55; const marginLeft = 65; const x = d3.scaleLinear() .domain([1, 9]) .range([marginLeft, width - marginRight]); const y = d3.scaleLinear() .domain([0, maxShare ? maxShare * 1.15 : 1]) .nice() .range([height - marginBottom, marginTop]); const line = d3.line() .x(d => x(d.digit)) .y(d => y(d.share)) .curve(d3.curveMonotoneX); const container = d3.create("div") .style("max-width", `${width}px`) .style("font", "13px sans-serif"); container.append("div") .style("font-size", "18px") .style("font-weight", "600") .style("margin-bottom", "14px") .text("First-digit distributions: Mystery vs. Benford"); // Dropdown and legend. const toolbar = container.append("div") .style("display", "flex") .style("align-items", "center") .style("flex-wrap", "wrap") .style("gap", "20px") .style("margin-bottom", "8px"); const control = toolbar.append("label") .style("display", "inline-flex") .style("align-items", "center") .style("gap", "8px"); control.append("span").text("Mystery dataset:"); const dropdown = control.append("select") .style("font", "inherit") .style("padding", "5px 10px") .style("border", "1px solid #bbb") .style("border-radius", "4px"); dropdown.selectAll("option") .data(columns) .join("option") .attr("value", d => d) .text(d => d); const mysteryLegend = toolbar.append("span") .style("display", "inline-flex") .style("align-items", "center") .style("gap", "7px"); const swatch = mysteryLegend.append("span") .style("width", "14px") .style("height", "14px") .style("border-radius", "50%") .style("border", "1px solid #888"); const legendText = mysteryLegend.append("span"); const benfordLegend = toolbar.append("span") .style("display", "inline-flex") .style("align-items", "center") .style("gap", "7px"); benfordLegend.append("span") .style("width", "11px") .style("height", "11px") .style("border-radius", "50%") .style("background", "#111"); benfordLegend.append("span").text("Benford"); const testSummary = container.append("div") .attr("aria-live", "polite") .style("padding", "10px 12px") .style("margin", "12px 0 4px") .style("background", "#f3f4f6") .style("border-radius", "5px") .style("line-height", "1.6"); const svg = container.append("svg") .attr("width", width) .attr("height", height) .attr("viewBox", [0, 0, width, height]) .style("display", "block") .style("max-width", "100%") .style("height", "auto"); // Horizontal gridlines. svg.append("g") .attr("transform", `translate(${marginLeft}, 0)`) .call( d3.axisLeft(y) .ticks(7) .tickSize(-(width - marginLeft - marginRight)) .tickFormat(() => "") ) .call(g => g.select(".domain").remove()) .call(g => g.selectAll("line").attr("stroke", "#e5e7eb")); // Axes. svg.append("g") .attr("transform", `translate(0, ${y(0)})`) .call( d3.axisBottom(x) .tickValues(digits) .tickFormat(d3.format("d")) .tickSizeOuter(0) .tickPadding(8) ); svg.append("g") .attr("transform", `translate(${marginLeft}, 0)`) .call(d3.axisLeft(y).ticks(7, ".0%").tickSize(0).tickPadding(8)) .call(g => g.select(".domain").remove()); svg.append("text") .attr("x", (marginLeft + width - marginRight) / 2) .attr("y", height - 10) .attr("text-anchor", "middle") .text("First digit"); svg.append("text") .attr("transform", "rotate(-90)") .attr("x", -(marginTop + height - marginBottom) / 2) .attr("y", 17) .attr("text-anchor", "middle") .text("Share of samples"); // Selected Mystery curve and dots. const mysteryPath = svg.append("path") .attr("fill", "none") .attr("stroke-width", 3) .attr("stroke-linecap", "round") .attr("pointer-events", "none"); const mysteryDots = svg.append("g") .selectAll("circle") .data(datasets.get("A").rows) .join("circle") .attr("cx", d => x(d.digit)) .attr("r", 6.5) .attr("stroke", "#555") .attr("stroke-width", 0.7); mysteryDots.append("title"); mysteryDots .on("pointerenter", function() { d3.select(this).attr("r", 8); }) .on("pointerleave", function() { d3.select(this).attr("r", 6.5); }); // Benford remains above the Mystery curve. svg.append("path") .datum(benford) .attr("d", line) .attr("fill", "none") .attr("stroke", "#666") .attr("stroke-width", 2.5) .attr("stroke-dasharray", "1,7") .attr("stroke-linecap", "round") .attr("pointer-events", "none"); const benfordDots = svg.append("g") .selectAll("circle") .data(benford) .join("circle") .attr("cx", d => x(d.digit)) .attr("cy", d => y(d.share)) .attr("r", 4.5) .attr("fill", "#111") .attr("stroke", "white") .attr("stroke-width", 1); benfordDots.append("title") .text(d => `Benford\nFirst digit: ${d.digit}\n` + `Expected share: ${d3.format(".2%")(d.share)}` ); // Benford percentage labels. svg.append("g") .attr("font-size", 12) .attr("font-weight", 700) .attr("text-anchor", "middle") .attr("fill", "#111") .attr("stroke", "white") .attr("stroke-width", 4) .attr("stroke-linejoin", "round") .attr("paint-order", "stroke") .attr("pointer-events", "none") .selectAll("text") .data(benford) .join("text") .attr("x", d => x(d.digit)) .attr("y", d => y(d.share) - 14) .text(d => d3.format(".2%")(d.share)); const status = container.append("div") .style("color", "#666") .attr("aria-live", "polite"); // Display all six tests together, with the selected dataset highlighted. container.append("div") .style("font-weight", "600") .style("margin-top", "18px") .text("Chi-squared goodness of fit to Benford"); const table = container.append("div") .style("overflow-x", "auto") .append("table") .style("width", "100%") .style("border-collapse", "collapse") .style("margin-top", "8px") .style("font-variant-numeric", "tabular-nums"); table.append("thead").append("tr").selectAll("th") .data(["Dataset", "n", "χ²", "df", "p-value", "Min. expected", "Result"]) .join("th") .attr("scope", "col") .style("text-align", "left") .style("padding", "8px") .style("border-bottom", "2px solid #bbb") .text(d => d); const resultRows = table.append("tbody").selectAll("tr") .data(columns) .join("tr"); resultRows.each(function(column) { const dataset = datasets.get(column); const values = [column, d3.format(",")(dataset.total), dataset.total ? d3.format(".3f")(dataset.chiSquared) : "—", dataset.total ? df : "—", formatP(dataset.pValue), dataset.total ? d3.format(".2f")(dataset.minExpected) : "—", dataset.conclusion]; d3.select(this).selectAll("td").data(values).join("td") .style("padding", "8px") .style("border-bottom", "1px solid #ddd") .text(d => d); }); container.append("p") .style("color", "#666") .style("line-height", "1.5") .text("χ² = Σ (observed − expected)² / expected; expected = n × Benford probability. " + "df = 8. Probabilities are normalized to sum to 1 to allow for small rounding differences. " + "P-values use the chi-square approximation and assume independent observations. " + "Expected counts below 5 are flagged. Results use α = 0.05 separately for each dataset, " + "without adjustment for six comparisons. A non-significant result does not prove Benford fit."); function update(animate = true) { const column = dropdown.property("value"); const dataset = datasets.get(column); const hasData = dataset.total > 0; testSummary.text(hasData ? column + ": χ² = " + d3.format(".3f")(dataset.chiSquared) + " · df = " + df + " · p " + (dataset.pValue < 0.0001 ? "" : "= ") + formatP(dataset.pValue) + " · n = " + d3.format(",")(dataset.total) + " — " + dataset.conclusion : column + ": no valid nonzero samples; the test cannot be calculated."); resultRows.style("background", d => d === column ? "#eef2ff" : null); swatch.style("background", color(column)); legendText.text(`Mystery ${column}`); mysteryPath .datum(dataset.rows) .attr("display", hasData ? null : "none") .attr("stroke", color(column)); mysteryDots .data(dataset.rows) .attr("display", hasData ? null : "none") .attr("fill", color(column)); mysteryDots.select("title") .text(d => `Dataset: ${column}\n` + `First digit: ${d.digit}\n` + `Samples: ${d3.format(",")(d.count)}\n` + `Share: ${d3.format(".2%")(d.share)}\n` + `Expected count: ${d3.format(".2f")(d.expected)}\n` + `χ² contribution: ${d.contribution == null ? "—" : d3.format(".3f")(d.contribution)}` ); function move(selection) { selection.interrupt("dataset"); return animate ? selection.transition("dataset").duration(450) : selection; } move(mysteryPath).attr("d", line); move(mysteryDots).attr("cy", d => y(d.share)); status.text(hasData ? `${column}: ${d3.format(",")(dataset.total)} valid nonzero samples. Hover over dots for details.` : `${column}: no valid nonzero samples. Only Benford is displayed.` ); } dropdown.on("change", () => update()); update(false); return Object.assign(container.node(), {scales: {color}}); })()4c:T2c08,(() => { const digits = d3.range(1, 10); const columns = ["A", "B", "C", "D", "E", "F"]; let selectedDigit = null; // Prepare the datasets. const data = [{ dataset: "Benford", values: new Map( Benford_distribution.map(d => [Number(d.digit), Number(d.p_i)]) ), theoretical: true }]; for (const column of columns) { const values = mystery .map(d => Math.abs(Number(d[column]))) .filter(value => Number.isFinite(value) && value > 0); data.push({ dataset: column, values: d3.rollup( values, rows => rows.length, value => Number(value.toExponential()[0]) ), theoretical: false }); } const series = d3.stack() .keys(digits) .value((d, digit) => d.values.get(digit) ?? 0) .offset(d3.stackOffsetExpand)(data); const segments = series.flatMap(s => s.map(segment => ({ digit: s.key, row: segment.data, start: segment[0], end: segment[1], share: segment[1] - segment[0] })) ); const benford = segments.filter(d => d.row.theoretical); // Dimensions and scales. const width = 928; const marginTop = 35; const marginRight = 24; const marginBottom = 15; const marginLeft = 85; const height = data.length * 52 + marginTop + marginBottom; const x = d3.scaleLinear() .domain([0, 1]) .range([marginLeft, width - marginRight]); const y = d3.scaleBand() .domain(data.map(d => d.dataset)) .range([marginTop, height - marginBottom]) .padding(0.28); const color = d3.scaleOrdinal() .domain(digits) .range(d3.quantize(d3.interpolateViridis, 9)); // HTML wrapper allows native buttons and a dropdown. const container = d3.create("div") .style("max-width", `${width}px`) .style("font", "12px sans-serif"); container.append("div") .style("font-size", "17px") .style("font-weight", "600") .style("margin", "8px 0 16px") .text("First-digit distributions of Benford vs Mystery datasets"); const toolbar = container.append("div") .style("display", "flex") .style("align-items", "center") .style("flex-wrap", "wrap") .style("gap", "10px") .style("margin-bottom", "14px"); // Clickable legend. const legend = toolbar.append("div") .style("display", "flex") .style("align-items", "center") .style("gap", "3px"); legend.append("span") .style("margin-right", "5px") .text("First digit"); const legendButtons = legend.selectAll("button") .data(digits) .join("button") .attr("type", "button") .attr("aria-label", d => `Isolate digit ${d}`) .style("display", "inline-flex") .style("align-items", "center") .style("gap", "5px") .style("padding", "5px") .style("border", "1px solid transparent") .style("border-radius", "4px") .style("background", "transparent") .style("color", "inherit") .style("font", "inherit") .style("cursor", "pointer"); const swatches = legendButtons.append("span") .style("display", "inline-block") .style("width", "14px") .style("height", "14px") .style("background", d => color(d)) .style("border", "1px solid #999"); legendButtons.append("span").text(d => d); // Keep reset and dropdown adjacent. const controls = toolbar.append("div") .style("display", "flex") .style("align-items", "center") .style("flex-wrap", "wrap") .style("gap", "10px"); const reset = controls.append("button") .attr("type", "button") .style("padding", "6px 10px") .style("border", "1px solid #bbb") .style("border-radius", "4px") .style("background", "#f3f4f6") .style("color", "#222") .style("font", "inherit") .style("cursor", "pointer") .text("Show all"); const dropdownLabel = controls.append("label") .style("display", "inline-flex") .style("align-items", "center") .style("gap", "6px"); dropdownLabel.append("span").text("Isolate First Digit:"); const dropdown = dropdownLabel.append("select") .style("padding", "5px 8px") .style("border", "1px solid #bbb") .style("border-radius", "4px") .style("background", "white") .style("color", "#222") .style("font", "inherit") .style("cursor", "pointer"); dropdown.selectAll("option") .data([ {value: "", label: "All"}, ...digits.map(d => ({value: String(d), label: String(d)})) ]) .join("option") .attr("value", d => d.value) .text(d => d.label); // Chart. const svg = container.append("svg") .attr("width", width) .attr("height", height) .attr("viewBox", [0, 0, width, height]) .style("display", "block") .style("max-width", "100%") .style("height", "auto") .style("overflow", "visible"); // Neutral tracks. svg.append("g") .selectAll("rect") .data(data) .join("rect") .attr("x", x(0)) .attr("y", d => y(d.dataset)) .attr("width", x(1) - x(0)) .attr("height", y.bandwidth()) .attr("fill", "#f3f4f6"); const bars = svg.append("g") .selectAll("rect") .data(segments) .join("rect") .attr("x", d => x(d.start)) .attr("y", d => y(d.row.dataset)) .attr("width", d => x(d.share) - x(0)) .attr("height", y.bandwidth()) .attr("fill", d => color(d.digit)) .attr("stroke", "white") .attr("stroke-width", 0.7); bars.append("title") .text(d => `Dataset: ${d.row.dataset}\n` + `First digit: ${d.digit}\n` + `${d.row.theoretical ? "Expected share" : "Share"}: ` + `${d3.format(".2%")(d.share)}` + (d.row.theoretical ? "" : `\nSamples: ${d3.format(",")( d.row.values.get(d.digit) ?? 0 )}`) ); bars .on("pointerenter", function() { d3.select(this).attr("fill-opacity", 0.75); }) .on("pointerleave", function() { d3.select(this).attr("fill-opacity", 1); }); // One guide per Benford endpoint. const guides = svg.append("g") .attr("stroke", "#777") .attr("stroke-width", 1.3) .attr("stroke-dasharray", "2,5") .attr("stroke-linecap", "round") .attr("pointer-events", "none") .selectAll("line") .data(benford) .join("line") .attr("x1", d => x(d.end)) .attr("x2", d => x(d.end)) .attr("y1", y("Benford") + y.bandwidth() + 3) .attr("y2", y("F") + y.bandwidth()); // Outline follows the Benford bar when isolated. const frame = svg.append("rect") .attr("x", x(0) - 3) .attr("y", y("Benford") - 3) .attr("width", x(1) - x(0) + 6) .attr("height", y.bandwidth() + 6) .attr("rx", 4) .attr("fill", "none") .attr("stroke", "#222") .attr("stroke-width", 2.5) .attr("pointer-events", "none"); const labels = svg.append("g") .attr("font-size", 11) .attr("font-weight", 700) .attr("pointer-events", "none") .selectAll("text") .data(segments) .join("text") .attr("x", d => x((d.start + d.end) / 2)) .attr("y", d => y(d.row.dataset) + y.bandwidth() / 2) .attr("dy", "0.35em") .text(d => d3.format(".1%")(d.share)); // Axes. svg.append("g") .attr("transform", `translate(0, ${marginTop - 8})`) .call(d3.axisTop(x).ticks(10, ".0%").tickSizeOuter(0)) .call(g => g.select(".domain").remove()); svg.append("g") .attr("transform", `translate(${marginLeft}, 0)`) .call(d3.axisLeft(y).tickSize(0).tickPadding(12)) .call(g => g.select(".domain").remove()) .call(g => g.selectAll("text") .attr("font-size", d => d === "Benford" ? 14 : 13) .attr("font-weight", d => d === "Benford" ? 800 : 600)); const emptyRows = new Set( data .filter(d => d3.sum(digits, digit => d.values.get(digit) ?? 0) === 0 ) .map(d => d.dataset) ); svg.append("g") .selectAll("text") .data(data.filter(d => emptyRows.has(d.dataset))) .join("text") .attr("x", marginLeft + 8) .attr("y", d => y(d.dataset) + y.bandwidth() / 2) .attr("dy", "0.35em") .attr("fill", "#666") .text("No valid nonzero values"); const status = container.append("div") .style("margin-top", "4px") .style("color", "#666") .attr("aria-live", "polite"); // Update the chart and synchronize both controls. function update(animate = true) { const isolated = selectedDigit !== null; const selected = benford.find(d => d.digit === selectedDigit); const visible = d => !isolated || d.digit === selectedDigit; const segmentStart = d => isolated ? 0 : d.start; const narrow = d => x(d.share) - x(0) < 34; // Interrupt any earlier animation before starting another. function move(selection) { selection.interrupt("isolate"); return animate ? selection.transition("isolate") .duration(450) .ease(d3.easeCubicInOut) : selection; } bars .attr("display", d => visible(d) ? null : "none") .attr("fill-opacity", 1); move(bars) .attr("x", d => x(segmentStart(d))); labels .attr("display", d => { if (emptyRows.has(d.row.dataset)) return "none"; const show = isolated ? d.digit === selectedDigit : d.row.theoretical && !narrow(d); return show ? null : "none"; }) .attr("text-anchor", d => isolated && narrow(d) ? "start" : "middle" ) .attr("fill", d => isolated && narrow(d) ? "#222" : d.digit <= 5 ? "white" : "#222" ); move(labels) .attr("x", d => { if (isolated && narrow(d)) return x(d.share) + 7; return x(segmentStart(d) + d.share / 2); }); guides.attr("display", d => visible(d) ? null : "none"); move(guides) .attr("x1", d => x(isolated ? d.share : d.end)) .attr("x2", d => x(isolated ? d.share : d.end)); move(frame) .attr("width", x(selected ? selected.share : 1) - x(0) + 6); legendButtons .attr("aria-pressed", d => String(d === selectedDigit)) .style("border-color", d => d === selectedDigit ? "#222" : "transparent" ) .style("font-weight", d => d === selectedDigit ? "700" : "400" ); swatches.style("opacity", d => !isolated || d === selectedDigit ? 1 : 0.25 ); dropdown.property("value", isolated ? String(selectedDigit) : ""); status.text(isolated ? `Digit ${selectedDigit}: bars start at 0%; the dotted line marks Benford’s expected share.` : "Click a legend digit or use the dropdown to isolate it. Dotted lines mark Benford’s cumulative boundaries." ); } legendButtons.on("click", (event, digit) => { selectedDigit = selectedDigit === digit ? null : digit; update(); }); dropdown.on("change", event => { selectedDigit = event.target.value === "" ? null : Number(event.target.value); update(); }); reset.on("click", () => { selectedDigit = null; update(); }); // Click any bar to isolate its digit; click again to show all. bars .style("cursor", "pointer") .on("click", (event, d) => { selectedDigit = selectedDigit === d.digit ? null : d.digit; update(); }); update(false); return Object.assign(container.node(), {scales: {color}}); })()4d:T2182,(() => { const columns = ["A", "B", "C", "D", "E", "F"]; const digits = d3.range(1, 10); const mysteryColor = "#dc2626"; const benfordColor = "#2563eb"; const expected = new Map( Benford_distribution.map(d => [ Number(d.digit), Number(d.p_i) ]) ); // Prepare all datasets to keep the axis fixed when switching. const datasets = new Map(columns.map(column => { const values = mystery .map(d => Math.abs(Number(d[column]))) .filter(value => Number.isFinite(value) && value > 0); const counts = d3.rollup( values, rows => rows.length, value => Number(value.toExponential()[0]) ); return [column, { total: values.length, rows: digits.map(digit => ({ digit, count: counts.get(digit) ?? 0, share: values.length ? (counts.get(digit) ?? 0) / values.length : null, expected: expected.get(digit) ?? 0 })) }]; })); const allRows = [...datasets.values()].flatMap(d => d.rows); const maxShare = d3.max( allRows, d => Math.max(d.share ?? 0, d.expected) ) || 1; const width = 928; const height = 510; const marginTop = 60; const marginBottom = 35; const marginLeft = 240; const marginRight = 125; const x = d3.scaleLinear() .domain([0, Math.min(1, maxShare * 1.12)]) .nice() .range([marginLeft, width - marginRight]); const y = d3.scalePoint() .domain(digits) .range([marginTop, height - marginBottom]) .padding(0.5); const container = d3.create("div") .style("max-width", `${width}px`) .style("font", "13px sans-serif"); container.append("div") .style("font-size", "18px") .style("font-weight", "600") .style("margin-bottom", "14px") .text("First-digit proportions: mystery dataset vs. Benford"); // Dataset dropdown. const toolbar = container.append("div") .style("display", "flex") .style("align-items", "center") .style("flex-wrap", "wrap") .style("gap", "20px"); const control = toolbar.append("label") .style("display", "inline-flex") .style("align-items", "center") .style("gap", "8px"); control.append("span").text("Mystery dataset:"); const dropdown = control.append("select") .style("font", "inherit") .style("padding", "5px 10px") .style("border", "1px solid #bbb") .style("border-radius", "4px"); dropdown.selectAll("option") .data(columns) .join("option") .attr("value", d => d) .text(d => d); // Legend. const legend = toolbar.append("div") .style("display", "flex") .style("gap", "16px"); for (const [label, color, hollow] of [ ["Mystery", mysteryColor, false], ["Benford", benfordColor, true] ]) { const item = legend.append("span") .style("display", "inline-flex") .style("align-items", "center") .style("gap", "6px"); item.append("span") .style("width", "10px") .style("height", "10px") .style("border-radius", "50%") .style("background", hollow ? "white" : color) .style("border", `2px solid ${color}`); item.append("span").text(label); } const svg = container.append("svg") .attr("width", width) .attr("height", height) .attr("viewBox", [0, 0, width, height]) .style("display", "block") .style("max-width", "100%") .style("height", "auto"); // Percentage axis and vertical gridlines. svg.append("g") .attr("transform", `translate(0, ${marginTop - 15})`) .call(d3.axisTop(x).ticks(7, ".0%").tickSizeOuter(0)) .call(g => g.select(".domain").remove()) .call(g => g.selectAll(".tick line") .clone() .attr("y2", height - marginBottom - marginTop + 15) .attr("stroke-opacity", 0.1)); // Column headings. svg.append("text") .attr("x", 20) .attr("y", 24) .attr("font-weight", 600) .text("Digit"); const observedHeader = svg.append("text") .attr("x", marginLeft - 20) .attr("y", 24) .attr("text-anchor", "end") .attr("fill", mysteryColor) .attr("font-weight", 600); svg.append("text") .attr("x", (marginLeft + width - marginRight) / 2) .attr("y", 24) .attr("text-anchor", "middle") .attr("font-weight", 600) .text("Proportion →"); svg.append("text") .attr("x", width - 15) .attr("y", 24) .attr("text-anchor", "end") .attr("fill", benfordColor) .attr("font-weight", 600) .text("Benford %"); // Fixed rows: digits 1–9. const rows = svg.append("g") .selectAll("g") .data(digits) .join("g") .attr("transform", digit => `translate(0, ${y(digit)})`); rows.append("line") .attr("x1", 20) .attr("x2", width - 15) .attr("stroke", "#eee"); rows.append("text") .attr("x", 25) .attr("dy", "0.35em") .attr("font-size", 14) .attr("font-weight", 600) .text(d => d); // Mystery percentages and counts on the left. const observedLabels = rows.append("text") .attr("x", marginLeft - 20) .attr("dy", "0.35em") .attr("text-anchor", "end") .attr("fill", mysteryColor) .attr("font-weight", 600); const connectors = rows.append("line") .attr("stroke", "#aaa") .attr("stroke-width", 2); const deviationLabels = rows.append("text") .attr("dy", "-0.75em") .attr("text-anchor", "middle") .attr("fill", "#555") .attr("font-size", 11) .attr("font-weight", 600); const observedDots = rows.append("circle") .attr("r", 6.5) .attr("fill", mysteryColor); const observedTips = observedDots.append("title"); // Hollow blue dots remain identifiable when the values overlap. const expectedDots = rows.append("circle") .attr("cx", digit => x(expected.get(digit) ?? 0)) .attr("r", 4.5) .attr("fill", "white") .attr("stroke", benfordColor) .attr("stroke-width", 2.5); expectedDots.append("title") .text(digit => `Benford\nFirst digit: ${digit}\n` + `Expected share: ${d3.format(".2%")( expected.get(digit) ?? 0 )}` ); // Benford percentages in a fixed column on the right. rows.append("text") .attr("x", width - 15) .attr("dy", "0.35em") .attr("text-anchor", "end") .attr("fill", benfordColor) .attr("font-weight", 600) .text(digit => d3.format(".2%")(expected.get(digit) ?? 0) ); const status = container.append("div") .style("color", "#666") .attr("aria-live", "polite"); function update(animate = true) { const column = dropdown.property("value"); const dataset = datasets.get(column); const byDigit = new Map( dataset.rows.map(d => [d.digit, d]) ); const hasData = dataset.total > 0; observedHeader.text(`${column}: share (count)`); observedLabels.text(digit => { const d = byDigit.get(digit); return hasData ? `${d3.format(".2%")(d.share)} (${d3.format(",")(d.count)})` : "No data"; }); observedTips.text(digit => { const d = byDigit.get(digit); return ( `Dataset: ${column}\nFirst digit: ${digit}\n` + `Count: ${d3.format(",")(d.count)}\n` + `Share: ${hasData ? d3.format(".2%")(d.share) : "No data"}` ); }); move(deviationLabels) .attr("x", digit => { const d = byDigit.get(digit); const observed = d.share ?? 0; const expectedValue = d.expected; return (x(observed) + x(expectedValue)) / 2; }) .text(digit => { const d = byDigit.get(digit); if (!hasData) return ""; const deviation = (d.share - d.expected) * 100; return `${deviation >= 0 ? "+" : ""}${deviation.toFixed(1)}%`; }); observedDots.attr("display", hasData ? null : "none"); connectors.attr("display", hasData ? null : "none"); deviationLabels.attr("display", hasData ? null: "none") function move(selection) { selection.interrupt("dataset"); return animate ? selection.transition("dataset").duration(450) : selection; } move(observedDots) .attr("cx", digit => x(byDigit.get(digit).share ?? 0)); move(connectors) .attr("x1", digit => x(byDigit.get(digit).share ?? 0)) .attr("x2", digit => x(byDigit.get(digit).expected)); status.text( `${column}: ${d3.format(",")(dataset.total)} valid nonzero samples. ` + "Red shows Mystery; blue shows Benford. " + "Left labels show share (count); right labels show expected share." ); } dropdown.on("change", () => update()); update(false); return container.node(); })()4e:T480,Plot.plot({ title: "D* Statistic for each Mystery Values Column", width:800, height:400, marginBottom: 50, style: {fontSize:"14px"}, x: { label: "d* statistic", labelOffset:35, labelAnchor: "center", domain: [0, .4] }, y: { label: "Mystery Column" , padding:0.6}, color: { domain: ["A","B","C","D","E","F","P"], range: ["#19189A","#562E10","#DAC53E","#613B91","#39935F","#5486AB","#DE7373"], }, marks: [ Plot.rect([{ x1: 0, x2: 0.19 }], { x1: "x1", x2: "x2", fill: "#8fd19e", opacity: 0.4 }), Plot.rect([{ x1: 0.19, x2: 0.25 }], { x1: "x1", x2: "x2", fill: "#f5e08a", opacity: 0.4 }), Plot.rect([{ x1: 0.25, x2: 1 }], { x1: "x1", x2: "x2", fill: "#e88b8b", opacity: 0.4 }), Plot.barX(dStarData, { x: "dstar", y: "column", r: 3, fill: "column" }), Plot.ruleX([0.19, 0.25], { stroke: "black", strokeDasharray: "2,2" }), Plot.text( [{ x: 0.19, label: "0.19" }, { x: 0.25, label: "0.25" }], { x: "x", text: "label", frameAnchor: "top", dy:-12, fontSize: 11 } ), Plot.tip(dStarData, Plot.pointer({ x: "dstar", y: "column", title: d => `${d.column} = ${d.dstar.toFixed(3)}` })) ] })4f:[{"id":"0786be72140ead175fb2d952e668c209b605c71350d9dd66b9f0af6ca796b6f7ac0bfde12394fa9e3d1f576ccd606e1772e07e001d2ab1878f452522a16e9784","href":"https://static.observableusercontent.com/files/0786be72140ead175fb2d952e668c209b605c71350d9dd66b9f0af6ca796b6f7ac0bfde12394fa9e3d1f576ccd606e1772e07e001d2ab1878f452522a16e9784","name":"2023 Purchases - digit1.csv","type":"text/csv","lastModified":1729629855746.169,"size":4230},{"id":"82bcda9673e15d725fa2d123ee12ae14f6d4255ce997e832654edcc7f2314c44715b2da3c80115c4158f6eec3f5c9b6321bc23a552a06c5d0fc5f9c39fa2b6e6","href":"https://static.observableusercontent.com/files/82bcda9673e15d725fa2d123ee12ae14f6d4255ce997e832654edcc7f2314c44715b2da3c80115c4158f6eec3f5c9b6321bc23a552a06c5d0fc5f9c39fa2b6e6","name":"Mystery@1.csv","type":"text/csv","lastModified":1729697438749.061,"size":5297},{"id":"1ea3becf1fc8c04b5f8112e519f0fd4c02303756a02034878e7db7d4438e52a3f00666482b7a9f637670b130dbbf383f3cd0ccb7a6d83e4fb00c4c8b2ebb8669","href":"https://static.observableusercontent.com/files/1ea3becf1fc8c04b5f8112e519f0fd4c02303756a02034878e7db7d4438e52a3f00666482b7a9f637670b130dbbf383f3cd0ccb7a6d83e4fb00c4c8b2ebb8669","name":"Mystery.csv","type":"text/csv","lastModified":1729696759901.689,"size":26226}] 44:{"id":"7d7cc49e302d6e88","slug":"benford-distribution","title":"Benford Distribution","thumbnail":"b1ad467c7a4bf77cdcb0cf229b31bdc52beb646543f0f37052e69e74c09b7565","access_level":"public","create_time":"$D2026-09-14T17:27:37.906Z","update_time":"$D2026-09-16T16:19:35.534Z","view_time":null,"first_public_time":"$D2026-09-14T17:27:58.173Z","has_unpublished_edits":null,"fork":{"id":"dbd2bc5cf33a1bd1","slug":null,"title":"Benford Distribution","owner":{"id":"518a19f15bc65c30","login":"jesse-johnson-ws","name":"Jesse Johnson's Workspace"}},"stars":0,"starred":false,"owner":{"type":"team","id":"7c4453bfba6d243c","login":"geo-theo","name":"geo-theo","avatar_url":"https://avatars-next.observableusercontent.com/12c9c666edcaa0723c7004924decd51156fdaf73f4de24c96fed6ee187078b9f","tier":"starter_2024","bio":"","following":false},"version":601,"publish_version":601,"role":null,"authored":false,"authors":[{"type":"individual","id":"0ebe4fb56285b831","login":"geo-theo","name":"geo-theo","avatar_url":"https://avatars-next.observableusercontent.com/12c9c666edcaa0723c7004924decd51156fdaf73f4de24c96fed6ee187078b9f","tier":null,"bio":"","following":false}],"members":[{"type":"team","id":"7c4453bfba6d243c","login":"geo-theo","name":"geo-theo","avatar_url":"https://avatars-next.observableusercontent.com/12c9c666edcaa0723c7004924decd51156fdaf73f4de24c96fed6ee187078b9f","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Benford Distribution","cells":[{"id":0,"value":"# Benford Distribution\nThe objective of this in-class exercise is to visually compare the first digit frequencies of a set of randomly generated data to the Benford distribution. In groups, you will work to achieve the most informative and easily understood visualization possible.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":272,"value":"$46","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":274,"value":"Plot.plot({\n x: { type: \"log\", domain: [1, 10], grid: true }\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":278,"value":"Plot.plot({\n x: {\n type: \"linear\",\n domain: [0, 1],\n grid: true\n },\n marks: [\n Plot.ruleX(\n d3\n .range(1, 10)\n .map((x) => Math.log10(1 + 1 / x))\n .map(\n (\n (sum) => (value) =>\n (sum += value)\n )(0)\n ),\n { stroke: \"red\" }\n )\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":301,"value":"## Assignment\nBased on your understanding of the Newcomb-Benford law develop quantitative and graphical methods for testing a set of numbers for NB. Your graphs should do two things: 1) show the distribution of first digits in relation to the values anticipated by NB. 2) provide a single, scalar valued statistical summary of the differences.\n\nSee below for a formula for the statistical summary.\n\nI've provided some conveniences variables below, as javascript.\n* `Benford_distribution` provides the first digit probabilities for the distribution.\n* `Random_distribution` provides a random distribution of first digits.\n* `bins` counts how many of those randomly sampled first digits are each of the nine digits.\n\nThe visual your group creates should show the Benford predictions and the results of the random sampled numbers. Fork this page and get going!\n\n*** You are on your honor to not look at the other forks! I want to see original ideas for how to do this. ***","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":194,"value":"// Place code to create your plot here!\nPlot.plot({\n title: \"Benford’s law: expected distribution\",\n width,\n height: 360,\n marginLeft: 60,\n x: {\n label: \"First digit\",\n domain: d3.range(1, 10)\n },\n y: {\n label: \"Expected share\",\n tickFormat: \".0%\",\n grid: true\n },\n marks: [\n Plot.barY(Benford_distribution, {\n x: \"digit\",\n y: \"p_i\",\n fill: \"steelblue\",\n inset: 1,\n tip: true,\n title: d =>\n `First digit: ${d.digit}\\n` +\n `Expected share: ${d3.format(\".2%\")(d.p_i)}`\n }),\n Plot.ruleY([0])\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"Benford_distribution = d3\n .range(1, 10)\n .map((x) => ({ digit: x, p_i: Math.log10(1 + 1 / x) }))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":290,"value":"// This is the number of samples that will be used to compute randomly generated Benford distributions see how many samples you need to get really good vs somewhat noisy distributions.\nnumber_of_samples = 10000","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":292,"value":"// This is the number of orders of magnitude the random_distribution is over. See how 'Benford' this is for shorter spans vs longer ones\nspan = 3","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":47,"value":"random_distribution = d3.range(number_of_samples).map((x) => ({\n sample: x,\n value: parseInt(Math.pow(10, d3.randomUniform().call() * span).toString()[0])\n}))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":51,"value":"function getFirstDigit(x) {\n // Note this wont work if x < 1\n return parseInt(x.toString[0]);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":137,"value":"bins = d3\n .bin()\n .thresholds(9)\n .value((d) => d.value)(random_distribution)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":306,"value":"display(Inputs.table(random_distribution));","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":317,"value":"(() => {\n const counts = d3.rollup(\n random_distribution,\n rows => rows.length,\n d => d.value\n );\n\n const data = d3.range(1, 10).map(digit => ({\n digit,\n count: counts.get(digit) ?? 0,\n share: (counts.get(digit) ?? 0) /\n (random_distribution.length || 1)\n }));\n\n return Plot.plot({\n title: \"Distribution of random samples\",\n width,\n height: 360,\n marginLeft: 60,\n x: {\n label: \"First digit\",\n domain: d3.range(1, 10)\n },\n y: {\n label: \"Share of samples\",\n tickFormat: \".0%\",\n grid: true\n },\n marks: [\n Plot.barY(data, {\n x: \"digit\",\n y: \"share\",\n fill: \"lightblue\",\n inset: 1,\n tip: true,\n title: d =>\n `First digit: ${d.digit}\\n` +\n `Samples: ${d3.format(\",\")(d.count)}\\n` +\n `Share: ${d3.format(\".2%\")(d.share)}`\n }),\n Plot.ruleY([0])\n ]\n });\n})()","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":264,"value":"## Test Data\nBelow I've uploaded the first digit of all of my 2023 credit card purchases. What do you think? Benford? Try it out! Use the plot technique you developed above.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":261,"value":"purchasesDigit1 = await FileAttachment(\"2023 Purchases - digit1.csv\").csv({typed: \"auto\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"purchasesDigit1","database":"$undefined"},{"id":298,"value":"purchasesDigit1[0].digit_1","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":339,"value":"$47","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":287,"value":"$48","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":296,"value":"mystery = await FileAttachment(\"Mystery@1.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"mystery","database":"$undefined"},{"id":304,"value":"display(Inputs.table(mystery));","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":325,"value":"$49","pinned":false,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":302,"value":"

\n\n# VISUALIZATIONS","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":331,"value":"## Superimposed Benford Line","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":333,"value":"$4a","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":337,"value":"$4b","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":335,"value":"

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":348,"value":"##### Please note, P column is purchase data from above brought down for comparison sake.\n##### Additionally, Column B from the mystery data has a somewhat small sample size, which may cause it to seem erroneously not benford when, with more samples, it might be benford.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":341,"value":"const benfordExpected = Benford_distribution.sort((a, b) => a.digit - b.digit).map((d) => d.p_i)\n// sorts and maps ben dist proportions to their num 1-9","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":342,"value":"const total = purchasesDigit1.length //counts up total entries\nconst counts = new Array(9).fill(0) //creates empty array\npurchasesDigit1.forEach(d=>counts[d[4]-1]++) //loops through every entry in purchase digit, converts each to 0 based index, then increments the count for that digit. eg. purchaseDigit1[0]=5 give counts[4] a +1 to its value\nconst actualProportions=counts.map(c=>c/total) //converts the above counts into their corresponding proportion over the entire dataset","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":343,"value":"const sumSquaredDiffs = d3.sum(d3.range(9).map(i => (actualProportions[i] - benfordExpected[i]) ** 2))//maps 0-8 to i, then uses that to calculate the difference between our data's proportions and the benford proportions, then squares it, THEN d3.sum add all of those squared differences up.\nconst dstarPurchases=Math.sqrt(sumSquaredDiffs)/1.03606 //takes the above sum, square roots it and then normalizes(rescales) to be between 0-1","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":344,"value":"function getFD(x) {\n // Note this wont work if x < 1\n return parseInt(x.toString()[0]);\n}","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":345,"value":"function getDStar(column) {\n const digitsData = mystery.map((row) => ({ digits: getFD(row[column]) })); //makes array of first digits for myserty csv column whatever\n const total = digitsData.length; // gets total for that column\n const counts = new Array(9).fill(0); //makes new array to count up the occurances\n digitsData.forEach((d) => counts[d.digits - 1]++); //counts it\n const actualProportions = counts.map((c) => c / total); //makes a proportion out of each count\n\n const sumSquaredDiffs = d3.sum(actualProportions.map((p, i) => (p - benfordExpected[i]) ** 2)); // first half of calculation for dstar\n\n return Math.sqrt(sumSquaredDiffs) / 1.03606; //second half of dstar\n} // basically same thing as previous d.star calc for purchasedigit1, just in a function so it can be reused","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":346,"value":"const dStarData = [\n { column: \"A\", dstar: getDStar(\"A\") },\n { column: \"B\", dstar: getDStar(\"B\") },\n { column: \"C\", dstar: getDStar(\"C\") },\n { column: \"D\", dstar: getDStar(\"D\") },\n { column: \"E\", dstar: getDStar(\"E\") },\n { column: \"F\", dstar: getDStar(\"F\") },\n { column: \"P\", dstar: dstarPurchases}\n];","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":347,"value":"$4e","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":307,"value":"