2e:null 2f:["$","title",null,{"children":"Classification: Quantile, Quantize and Threshold scales | Caglar Koylu | Observable"}] 34:T844,md`--- ## Conclusion A discrete color scale groups data points into classes, which help summarize the observations, and build visualizations where colors can be distinguished and looked-up on a legend. However, the result is highly dependent on the way the classes are built. To analyse a distribution, *quantize* scales will slice the domain’s extent into intervals of roughly equal lengths. But if the distribution is skewed, they might end up grouping almost all the data points in one class, and the outliers in their own specific classes. On the other hand, *quantile* scales will slice the domain’s distribution into intervals of roughly equal absolute frequencies (*i.e.* number of individual data points). They will systematically allow finding the "higher", "lower", "middle" classes (and, given enough data, even tranches of 10%, 5%, 1%…), which make them extremely useful in combination with other variables (such as geographic position, for maps). But they do so only by ranking the values, which might make us lose track of the distribution of the absolute values. *Threshold* scales allow the specification of precise cut values, which can emerge from personal observation of the data (especially if it is “naturally clustered”, or if special thresholds are deemed interesting for external reasons (in our example, if a law applies above a certain revenue). Various clustering algorithms can provide reasonable thresholds — the most common in geography is known as “Jenks natural breaks”, and implemented as _ckmeans_ in [simple-statistics](https://simplestatistics.org/docs/#ckmeans). We used it to [select](#simple) the cut values [53440, 135096] below. (More about this topic in the article and code by David Schnurr: “[Using clustering to create a new D3.js color scale](https://medium.com/@dschnr/using-clustering-to-create-a-new-d3-js-color-scale-dec4ccd639d2)”, Medium, oct. 2016.) Each scale tells a different aspect of the same story; in an exploratory phase, it is worth switching from one approach to the other. In a restitution phase, it must be chosen wisely. `35:T5a4,function showScaleGrouping(data, scales) { const margins = { left: 130 }; const x = d3 .scaleLinear() .domain(d3.extent(data)) .range([margins.left, width - 5]); const r = 3; const rectHeight = 10; const chart = svg` `; return chart; }36:[] 32:{"id":"b09e6ce8148526e8","slug":"classification-quantile-quantize-and-threshold-scales","title":"Classification: Quantile, Quantize and Threshold scales","thumbnail":"06eaf72a4ed439fac72353b8ce859a122f9a21540cc783715ee04231a6a50012","access_level":"public","create_time":"$D2020-02-12T14:05:41.679Z","update_time":"$D2021-02-22T15:57:14.912Z","view_time":null,"first_public_time":"$D2020-02-12T14:13:27.019Z","has_unpublished_edits":null,"fork":{"id":"44e9e378887c1a4d","slug":"quantile-quantize-and-threshold-scales","title":"Quantile, Quantize and Threshold scales","owner":{"id":"863951e3ebe4c0ae","login":"d3","name":"D3"}},"stars":6,"starred":false,"owner":{"type":"team","id":"1522f7962f5f1185","login":"caglarkoylu","name":"Caglar Koylu","avatar_url":"https://avatars-next.observableusercontent.com/94b0c40bf8a6b04057088ea171864451b3e3b27b7346d866d2a7be21d077be76","tier":"starter_2024"},"version":677,"publish_version":677,"role":null,"authored":false,"authors":[{"type":"individual","id":"60e9af384dfce94c","login":"caglarkoylu","name":"Caglar Koylu","avatar_url":"https://avatars-next.observableusercontent.com/94b0c40bf8a6b04057088ea171864451b3e3b27b7346d866d2a7be21d077be76","tier":null}],"members":[{"type":"individual","id":"60e9af384dfce94c","login":"caglarkoylu","name":"Caglar Koylu","avatar_url":"https://avatars-next.observableusercontent.com/94b0c40bf8a6b04057088ea171864451b3e3b27b7346d866d2a7be21d077be76","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":40,"collection":null,"type":"notebook","body":{"title":"Classification: Quantile, Quantize and Threshold scales","cells":[{"id":407,"value":"md`# Classification: Quantile, Quantize and Threshold scales`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":0,"value":"md`Quantile and quantize scales have very similar names, confusingly so. Here’s how to remember their meanings:\n\n*Quantiles* are defined, in statistics, as separating a population into intervals of similar sizes (the 10% poorest, the 50% tallest, the 1% richest…); a quantile scale is essentially defined by its domain, a fixed set of values — when applied to a new value, it will then compute its ranking with respect to the initial distribution, and map that ranking to the output range.\n\nTo *Quantize* means to group values with discrete increments — like expressing a list of floating point numbers with 1 decimal digit, or rounding time to the closest minute. It is the output *range* that is discretized, and such a scale allows to transform an initial continuous range into a discrete set of classes. \n\n*Threshold* scales will allow us to directly specify the cut values that separate the classes.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":42,"value":"md`Though they can be applied almost interchangeably to many real-world distributions, these scales answer slightly different questions. Let’s consider for example the following list of 100 numbers, each representing the annual pre-tax income of a resident of the United States, in 2018. (This distribution is supposedly representative of the whole population.)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":30,"value":"data = [\n 11002,29017,45793,7000,120040,30138,21699,47058,24001,6000,69007,40000,55001,30001,61150,12000,85530,\n 83000,23100,96225,45003,34300,43000,63131,52001,36000,10001,225786,0,75000,195100,33010,5000,31213,\n 79050,40010,37002,50000,60000,66529,39048,27276,28007,153420,44500,145443,89550,16024,50,25001,300577,\n 102035,20581,170240,126101,18001,15000,4000,0,100003,35000,14001,72046,30000,0,65006,56000,42000,17158,\n 135096,70040,114068,22216,60020,2742,35030,25000,76005,40600,48335,58000,900,8000,19002,92000,13000,\n 50008,20000,15100,108023,50600,26483,38002,53440,32007,25654,80130,20000,9500,1968\n]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":132,"value":"md`We’ll represent each of these 100 individuals by a tiny square:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":184,"value":"chart(data)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":81,"value":"md`The following will display these dollar values in a readable [format](https://github.com/d3/d3-format):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":79,"value":"format = d3.format(\"$,.0f\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":56,"value":"md`> _The data’s domain extended from ${format(d3.min(data))}\n to USD${format(d3.max(data))}\n with an average of ${format(d3.mean(data))}.\n The median income was ${format(d3.median(data))}._`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":199,"value":"md`A simple way to color these values is with a linear scale ranging from white to red:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":223,"value":"linear = d3.scaleLinear()\n .domain([0, d3.max(data)])\n .range([\"white\", \"red\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":188,"value":"chart(data, linear) // chart is a helper function defined in the Annex","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":219,"value":"md`This chart doesn’t tell us much, and you’d need to have a perfect eye (and perfect screen as well as perfect lighting conditions) to be able to tell those colors apart.\n\nThis is why the astute practitioner will usually switch to discrete scales, grouping the values into classes. \n\n[d3.scaleThreshold](https://github.com/d3/d3-scale/blob/master/README.md#scaleThreshold) creates such a scale. Suppose we decide to separate the data points in three subsets, by cutting at the threshold values ${threshold.domain().map(format).join(\" and \")}.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":298,"value":"threshold = d3.scaleThreshold()\n .domain([10000, 100000])\n .range([\"white\", \"pink\", \"red\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":305,"value":"md`Paraphrasing the above line of code, we could say: “incomes below\n${format(threshold.domain()[0])} are represented in white, from ${format(threshold.domain()[0])} to\n${format(threshold.domain()[1])} in pink, and above that, in red”.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":302,"value":"chart(data, threshold)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":658,"value":"naturalbreaks3 = simple.ckmeans(data, 3).map(v => v.pop())","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":651,"value":"//Using natural breaks with threshold\nthresholdNaturalBreaks = d3.scaleThreshold()\n .domain(naturalbreaks3)\n .range([\"white\", \"pink\", \"red\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":656,"value":"chart(numericSort(data),thresholdNaturalBreaks)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":669,"value":"naturalbreaks5 = simple.ckmeans(data, 5).map(v => v.pop())","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":664,"value":"thresholdNaturalBreaks5 = d3.scaleThreshold()\n .domain(naturalbreaks5)\n .range([\"blue\", \"white\", \"pink\", \"red\", \"black\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":676,"value":"chart(numericSort(data),thresholdNaturalBreaks5)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":307,"value":"md`If we sort the values in ascending order, we can make sense of the distribution, and the thresholded colors, while simplifying the data, help a lot visually:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":207,"value":"chart(numericSort(data), linear) // numericSort is defined in the Annex below","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":311,"value":"chart(numericSort(data), threshold)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":335,"value":"md`---\n[d3.scaleQuantize](https://github.com/d3/d3-scale/blob/master/README.md#scaleQuantize) will take care of choosing the thresholds from the extent of the dataset, subdividing it into as many regular intervals as there are classes (here, the 3 elements in the range):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":323,"value":"quantize = d3.scaleQuantize()\n .domain(d3.extent(data)) // pass only the extreme values to a scaleQuantize’s domain\n .range([\"white\", \"pink\", \"red\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":325,"value":"chart(data, quantize)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":332,"value":"md`The thresholds chosen by the quantize scale:\n${quantize.thresholds().map(format).join(\", \")}.\n\nThis scale shows important inequalities, but does not let us see much of the distribution, except for the few top earners. If the top income had been even higher, say $1,000,000, the thresholds would have been set at $333,333 and $666,667, showing only the top-1% in one class, and the remaining 99% in the other.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":515,"value":"md`*quantize*.nice will “nice” the domain before applying the quantize strategy (this method exists only for quantize scales, not for threshold and quantile scales).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":518,"value":"md`Note that we have applied *scale*.nice to a copy of the scale (obtained with *scale*.copy), so that we don’t change qz in the examples above.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":483,"value":"md`> _For our dataset the niced domain becomes [${quantize.copy().nice().domain().map(format).join(\", \")}], and the quantize thresholds are [${quantize.copy().nice().thresholds().map(format).join(\", \")}]._`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":348,"value":"md`---\n\n[d3.scaleQuantile](https://github.com/d3/d3-scale/blob/master/README.md#scaleQuantile) adopts a different method to evaluate the thresholds, by slicing the *domain* into intervals of (roughly) equal absolute frequency (*i.e.* number of individuals):\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":236,"value":"quantile = d3.scaleQuantile()\n .domain(data) // pass the whole dataset to a scaleQuantile’s domain\n .range([\"white\", \"pink\", \"red\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":238,"value":"chart(data, quantile)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":248,"value":"md`Remember the definition of quantiles? This chart shows us in red the 33 individuals that made the most, in white the 33 that made the least, and in pink, the 33 in-between (give or take 1).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":264,"value":"md`Of course, once sorted, nothing remains except “the top class is richer than the middle-class, which is richer than the poorer class” — a truism, totally independent of the data’s structure.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":256,"value":"chart(numericSort(data), quantile)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":290,"value":"md`In other words, the quantile scale is, on its own, *absolutely not* informative about the distribution. However, it is very good at giving us classes with a balanced number of elements — this is useful on a map, to show where high incomes and low incomes are concentrated spatially. See [Quantile Choropleth](https://observablehq.com/@d3/quantile-choropleth) for a full-blown example.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":114,"value":"md`By doing a bit more statistics with [d3.quantile](https://observablehq.com/@d3/d3-mean-d3-median-and-friends), we can learn that people earning more than\n ${format(d3.quantile(data.slice().sort(d3.ascending), 0.9))}\n made more money than 90% of their compatriots.\nPeople earning less than\n ${format(d3.quantile(data.slice().sort(d3.ascending), 0.1))}\n made less than 90% of their compatriots.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":364,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":434,"value":"jenks = d3\n .scaleThreshold()\n .domain([53440, 135096])\n .range([\"white\", \"pink\", \"red\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":411,"value":"showScaleGrouping(data, {\n scaleQuantile: quantile,\n scaleThreshold: threshold,\n scaleJenks: jenks,\n scaleQuantize: quantize,\n scaleQuantizeNice: quantize.copy().nice()\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":49,"value":"md`---\n## Annex`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":34,"value":"d3 = require(\"d3-array@2\", \"d3-format@1\", \"d3-scale@3\", \"d3-selection@1\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":210,"value":"function numericSort(x) {\n return (\n x\n // ensure the array is not changed in-place\n .slice()\n // comparator function that treats input as numeric\n .sort((a, b) => a - b)\n );\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":134,"value":"function chart(data, scale) {\n const w = 30,\n cols = Math.floor(Math.min(600, width) / w),\n lines = Math.ceil(100 / cols);\n const chart = d3\n .create(\"svg\")\n .attr(\"width\", cols * w)\n .attr(\"height\", lines * w);\n\n chart\n .append(\"g\")\n .attr(\"transform\", \"translate(2,2)\")\n .attr(\"style\", \"stroke:black; fill:white;\")\n .selectAll(\"rect\")\n .data(data)\n .join(\"rect\")\n .attr(\"width\", w - 3)\n .attr(\"height\", w - 3)\n .attr(\"x\", (_, i) => w * (i % cols))\n .attr(\"y\", (_, i) => w * ((i / cols) | 0))\n .style(\"fill\", d => (scale ? scale(d) : \"#ddd\"));\n return chart.node();\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":410,"value":"$35","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":569,"value":"simple = require(\"simple-statistics@7.0.7/dist/simple-statistics.min.js\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":523,"value":"thanks = md`Thanks to [Guillaume Calmettes](/@gcalmettes) and [Sylvain Lesage](/@severo) for proofreading and code suggestions.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":527,"value":"// This documentation is released under the CC BY-SA 4.0 license.\n// https://creativecommons.org/licenses/by-sa/4.0/\nLICENSE = \"cc-by-sa-4.0\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W36","stdlib":"1","titling":"auto"}} 33:"$32:body"