31:T4a4,{ var test = interpretFloat16Bytes(testValue); var position = floatToPosition(test); return plot( graphDiv[0], [ floats.normals, floats.subnormals, floats.nans, floats.infinities, floats.zeros ] .map(data => ({ type: 'scattergl', mode: 'markers', x: data.values, y: data.exponents, hovertext: data.labels, hoverinfo: 'text', name: data.label })) .concat([ { x: [position.x], y: [position.y], name: 'Test value', type: 'scattergl', mode: 'markers', marker: { size: 15, symbol: 'circle-open', color: 'black' } } ]), { dragmode: 'pan', width: width, yaxis: { fixedrange: true, title: 'Exponent (zero-offset removed)' }, xaxis: { range: graphDiv[0].layout && graphDiv[0].layout.xaxis.type === scaleType ? graphDiv[0].layout.xaxis.range : null, type: scaleType, title: scaleType === 'log' ? 'Positive values' : 'Value' }, legend: { y: -0.2 } }, { scrollZoom: true } ); }32:T4ca,md`As long as I've been programming, I've almost completely taken for granted that any difference between floating-point and continuous real numbers can mostly be ignored. Trying to use half-precision floating-point on mobile GPUs taught me there's not always as much precision as we might hope. This notebook doesn't address floating-point *arithmetic* where things really go awry, but it represents an attempt to nail down for myself exactly where the floating point numbers are on the number line. There are about 4.3 billion 32-bit single-precision floating-point numbers (${tex`2^{32} = ${Math.pow( 2, 32 ).toLocaleString()}`}). That's too many to easily plot. Instead we'll consider 16-bit [half-precision floating-point format](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) as defined by [IEEE 754-2008](https://en.wikipedia.org/wiki/IEEE_754-2008_revision). There are 65536 (${tex`2^{16}`}) half-precision floating-point numbers (henceforth "half-floats"). Using [plotly.js](https://github.com/plotly/plotly.js), the plot above displays them all. Interpretation of [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) floating-point numbers falls into five different cases, described below. `33:T435,md` The coloring of the bits represents the sign, exponent, and significand (also called the *mantissa*). You can click to toggle the bits. The interpretation of normalized half-float numbers is given by ${tex.block`(−1)^{\mathrm{signbit}} × 2^{\mathrm{exponent}−15} × 1.\mathrm{significand}_2.`} Since binary numbers with a [radix point](https://en.wikipedia.org/wiki/Radix_point) are probably unfamiliar, this is mathematically equivalent to ${tex.block`(−1)^{\mathrm{signbit}} × 2^{\mathrm{exponent}−15} × \left(1 + \frac{\mathrm{significand}}{1024}\right).`} Note that the exponent is stored in [offset-binary](https://en.wikipedia.org/wiki/Offset_binary) with a zero offset of 15. So for example, if the bits of the exponent are ${tex`10010_2 = 18_{10}`} (binary and decimal representations, respectively), then the mathematical exponent term is ${tex`2^{18 - 15} = 2^{3} = 8`}.`34:T56c,md` ## Subnormal Numbers [Subnormal numbers](https://en.wikipedia.org/wiki/Denormal_number), or *subnormals*, are a special case for which all bits of the exponent are zero. Their value in half-float precision is represented by ${tex.block`(−1)^{\mathrm{signbit}} × 2^{-14} × 0.\mathrm{significand}_2.`} Mathematically, this is equivalent to ${tex.block`(−1)^{\mathrm{signbit}} × 2^{-14} × \left(0 + \frac{\mathrm{significand}}{1024}\right).`} The interpretation is similar to that of normalized numbers, except that the exponent term is ${tex`2^{-14}`} (direct interpretation would suggest ${tex`2^{-15}`}), and the implicit first bit of the significand is zero rather than one. Why do subnormal numbers exist? Subnormal numbers allow *gradual underflow*, ensuring that the difference between two distinct numbers is zero only if the numbers are identical. (Conversely, without gradual underflow, ${tex`a - b = 0`} does not imply ${tex`a = b`}.) The plot below highlights the smallest five exponents. So that we can see the spacing, we use here only two bits of precision in the significand. For normalized numbers, we see that with each decrease in exponent the space between numbers is cut in half and the numbers cover half of the remaining distance to zero. For subnormals though, the spacing is not reduced and the numbers cover the *full* remaining distance to zero. `35:T583,function interpretFloat16Bytes(bytes) { function float16to32(bytes) { // Javascript doesn't support half-float, so we tease apart the float32 bytes // and convert them into float32 bytes instead. var floatBytes = new Uint8Array(4); // Isolate the sign bit: var signBit = (bytes[1] & 0b10000000) >> 7; // Isolate the mantissa var mantissa = ((bytes[1] & 0b00000011) << 8) | bytes[0]; // Isolate the exponent: var exponent16 = (bytes[1] & 0b01111100) >> 2; var exponent32 = exponent16 - 15 + 127; if (exponent16 === 0) { // Subnormal. This one is tricky so we'll just do it mathematically instead of bitwise return (signBit > 0 ? -1 : 1) * Math.pow(2, -14) * (mantissa / 1024); } else if (exponent16 === 31) { // Transfer this special case into the 32-bit exponent exponent32 = 255; } // Combine the sign and exponent into the first byte floatBytes[3] = (signBit << 7) | ((exponent32 & 0b11111110) >> 1); floatBytes[2] = ((exponent32 & 0b00000001) << 7) | ((mantissa & 0b1111111000) >> 3); floatBytes[1] = (mantissa & 0b00000111) << 5; floatBytes[0] = 0; return new Float32Array(floatBytes.buffer)[0]; } return { value: float16to32(bytes), sign: (bytes[1] & 0b10000000) >> 7, exponent: (bytes[1] & 0b01111100) >> 2, significand: ((bytes[1] & 0b00000011) << 8) | bytes[0] }; }36:Tc15,function tabulateFloat16(precision) { function float16to32(sign, exponent, significand) { const value = new Float32Array(1); const bytes = new Uint8Array(value.buffer); if (exponent === 0x1f) { // Infinities and NaNs exponent = 0xff; } else if (exponent === 0x00) { // Subnormals. I'm not going to bother trying to do this via bitwise conversion. return ((sign > 0 ? -1 : 1) * (Math.pow(2, -14) * significand)) / 1024; } else { // Convert to float32's offset-binary for normals: exponent = exponent - 15 + 127; } // Float32 is 23 bits, Float16 is 0. We shift the significand over // so that we can convert to float32. significand <<= 13; bytes[3] = (sign > 0 ? 0x80 : 0x0) | ((exponent >> 1) & 0x7f); bytes[2] = ((exponent & 0x1) << 7) | ((significand & 0x7f0000) >> 16); bytes[1] = (significand & 0xff00) >> 8; bytes[0] = significand & 0xff; return value[0]; } var subnormals = { exponents: [], values: [], labels: [], label: 'Subnormals' }; var normals = { exponents: [], values: [], labels: [], label: 'Normals' }; var infinities = { exponents: [], values: [], labels: [], label: 'Infinities' }; var nans = { exponents: [], values: [], labels: [], label: 'NaNs' }; var zeros = { exponents: [], values: [], labels: [], label: 'Zeros' }; for (let i = 0, sign = 0; sign < 2; sign++) { for (let exponent = 0; exponent < 32; exponent++) { for ( let significand = 0; significand < 1024; significand += Math.pow(2, 10 - precision) ) { let destination = normals; if (exponent === 0 && significand === 0) { destination = zeros; } else if (exponent === 31) { destination = significand === 0 ? infinities : nans; } else if (exponent === 0) { destination = subnormals; } var value = float16to32(sign, exponent, significand); var valueDisplay = exponent < 31 ? value : (sign > 0 ? -1 : 1) * Math.pow(2, 16) * (1 + significand / 1024); var exponentDisplay = Math.max(-14, exponent - 15); if (exponent === 0 && significand === 0) { exponentDisplay = exponentDisplay + (sign > 0 ? -0.2 : 0.2); } var valueLabel = value; if (significand === 0 && exponent === 0) { valueLabel = sign > 0 ? '-0' : '+0'; } destination.exponents.push(exponentDisplay); destination.values.push(valueDisplay); destination.labels.push( `value = ${valueLabel}
sign bit = ${sign.toString( 2 )}₂
exponent = ${exponent .toString(2) .padStart( 5, '0' )}₂
significand = ${significand .toString(2) .padStart(10, '0')}₂` ); i++; } } } return { normals, subnormals, nans, infinities, zeros }; }37:[] 2f:{"title":"Half-Precision Floating-Point, Visualized","cells":[{"id":499,"value":"md`# Half-Precision Floating-Point, Visualized`","pinned":false,"hidden":false,"mode":"ojs"},{"id":343,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":1121,"value":"viewof testValue = binaryInput({\n type: 'float16',\n value: 1,\n outputAsBytes: true,\n displayParts: true,\n fractionalSignificand: true,\n title: 'Test value'\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":401,"value":"viewof scaleType = radio({\n title: 'X-scale',\n options: ['linear', 'log'],\n value: 'linear',\n description: '(log scaling displays only values greater than zero)'\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1033,"value":"viewof precision = slider({\n min: 0,\n max: 10,\n step: 1,\n value: 10,\n title: 'Bits of precision in significand',\n description:\n '(excluding the implicit most significant bit due to (de)normalization)'\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":504,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":914,"value":"md`\n\n## Normalized numbers\n\nMost numbers are \"normalized\". This means the first bit of the significand is implicitly 1 and not stored directly. A number is interpreted as normalized whenever the bits of the exponent are not all zeros and not all ones.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1202,"value":"md`\n\nFor example, the number π in half-float precision is represented as`","pinned":false,"hidden":false,"mode":"ojs"},{"id":918,"value":"binaryInput({\n type: 'float16',\n value: Math.PI,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":740,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":833,"value":"md`By the above definition, the spacing between consecutive floating point numbers doubles as you move from one exponent to the next. The plot below shows doubling of the spacing between exponent ${tex`2^0`} and exponent ${tex`2^1`}.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":813,"value":"plot(\n graphDiv[2],\n [\n floats.normals,\n floats.subnormals,\n floats.nans,\n floats.infinities,\n floats.zeros\n ].map(data => ({\n type: 'scattergl',\n mode: 'markers',\n x: data.values,\n y: data.exponents,\n hovertext: data.labels,\n hoverinfo: 'text',\n name: data.label\n })),\n {\n dragmode: 'pan',\n width: width,\n height: 400,\n yaxis: { fixedrange: true, title: 'Exponent (zero-offset removed)' },\n xaxis: {\n range:\n graphDiv[2].layout && graphDiv[2].layout.xaxis.type === scaleType\n ? graphDiv[2].layout.xaxis.range\n : [0.99 * 2, 1.01 * 2],\n type: scaleType,\n title: scaleType === 'log' ? 'Positive values' : 'Value'\n },\n legend: { y: -0.3 }\n },\n {\n scrollZoom: true\n }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":831,"value":"md`\n\n### Examples\n\nRepeating the above example, the number π in half-float precision is`","pinned":false,"hidden":false,"mode":"ojs"},{"id":711,"value":"binaryInput({\n type: 'float16',\n value: Math.PI,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":743,"value":"md`The smallest positive normal number is`","pinned":false,"hidden":false,"mode":"ojs"},{"id":747,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b0000010000000000,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":753,"value":"md`while the largest normal number is`","pinned":false,"hidden":false,"mode":"ojs"},{"id":751,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b00111101111111111,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":700,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":783,"value":"{\n var floats = tabulateFloat16(2);\n return plot(\n graphDiv[1],\n [\n floats.normals,\n floats.subnormals,\n floats.nans,\n floats.infinities,\n floats.zeros\n ].map(data => ({\n type: 'scattergl',\n mode: 'markers',\n x: data.values,\n y: data.exponents,\n hovertext: data.labels,\n hoverinfo: 'text',\n name: data.label\n })),\n {\n dragmode: 'pan',\n width: width,\n height: 300,\n yaxis: {\n fixedrange: true,\n title: 'Exponent (zero-offset removed)',\n range: [-16, -8.5]\n },\n xaxis: {\n range:\n graphDiv[1].layout && graphDiv[1].layout.xaxis.type === scaleType\n ? graphDiv[1].layout.xaxis.range\n : [-1e-4, 8e-4],\n type: scaleType,\n title: scaleType === 'log' ? 'Positive values' : 'Value'\n },\n legend: { y: -0.4 }\n },\n {\n scrollZoom: true\n }\n );\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1213,"value":"md`\nIn order to handle this special case, some computers move computation involving subnormal numbers out of hardware and into software, with the result that they may be [as much as 100 times slower](https://en.wikipedia.org/wiki/Denormal_number#Performance_issues) than computations involving normalized numbers.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":842,"value":"\nmd`\n\n### Examples\n\nThe smallest positive subnormal number is`","pinned":false,"hidden":false,"mode":"ojs"},{"id":723,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b0000000000000001,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":729,"value":"md`while the largest subnormal number is`","pinned":false,"hidden":false,"mode":"ojs"},{"id":726,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b0000001111111111,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":705,"value":"md`\n## Zero\n\nWhen the exponent and significand are both ${tex`0_2`}, the value is interpreted as zero. The sign bit then distinguishes between ${tex`+0`} and ${tex`-0`}. The plot below shows the zeros in purple (with a slight offset in exponent so that positive and negative zeros can be distinguished).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":993,"value":"plot(\n graphDiv[4],\n [\n floats.normals,\n floats.subnormals,\n floats.nans,\n floats.infinities,\n floats.zeros\n ].map(data => ({\n type: 'scattergl',\n mode: 'markers',\n x: data.values,\n y: data.exponents,\n hovertext: data.labels,\n hoverinfo: 'text',\n name: data.label\n })),\n {\n dragmode: 'pan',\n width: width,\n height: 300,\n yaxis: {\n fixedrange: true,\n title: 'Exponent (zero-offset removed)',\n range: [-15, -11.5]\n },\n xaxis: {\n range:\n graphDiv[4].layout && graphDiv[4].layout.xaxis.type === scaleType\n ? graphDiv[4].layout.xaxis.range\n : [-0.5e-3, 0.5e-3],\n type: scaleType,\n title: scaleType === 'log' ? 'Positive values' : 'Value'\n },\n legend: { y: -0.4 }\n },\n {\n scrollZoom: true\n }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1090,"value":"md`Positive and negative zero are considered numerically equal, and although distinctness of the two zeros may be one of the most rarely used features of IEEE 754 floating-point, it can serve a purpose. One example is Kahan's classic [Branch Cuts for Complex Elementary Functions, or Much Ado About Nothing's Sign Bit](https://people.freebsd.org/~das/kahan86branch.pdf), which explains use of the sign bit for consistent treatment of multivalued complex analytic functions as you approach discontinuous [branch cuts](http://mathworld.wolfram.com/BranchCut.html) from one side or the other.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1244,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b0000000000000000,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1248,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b1000000000000000,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":702,"value":"md`\n## Infinities\n\nThere are two infinities, ${tex`+\\infty`} and ${tex`-\\infty`}, represented by the maximum possible exponent ${tex`11111_2 = 32_{10}`} and with a significand equal to zero. The sign bit disambiguates positive and negative infinity.\n\nThe plot below shows positive infinity in red. Interpretation is a special case which does not place infinites on the number line at all, though for the sake of understanding where they fit into the scheme of things, they're positioned below as if interpreted as normalized numbers.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":963,"value":"plot(\n graphDiv[3],\n [\n floats.normals,\n floats.subnormals,\n floats.nans,\n floats.infinities,\n floats.zeros\n ].map(data => ({\n type: 'scattergl',\n mode: 'markers',\n x: data.values,\n y: data.exponents,\n hovertext: data.labels,\n hoverinfo: 'text',\n name: data.label\n })),\n {\n dragmode: 'pan',\n width: width,\n height: 300,\n yaxis: {\n fixedrange: true,\n title: 'Exponent (zero-offset removed)',\n range: [11.5, 17]\n },\n xaxis: {\n range:\n graphDiv[3].layout && graphDiv[3].layout.xaxis.type === scaleType\n ? graphDiv[3].layout.xaxis.range\n : [1e3, 150000],\n type: scaleType,\n title: scaleType === 'log' ? 'Positive values' : 'Value'\n },\n legend: { y: -0.4 }\n },\n {\n scrollZoom: true\n }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":762,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b0111110000000000,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":765,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b1111110000000000,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":699,"value":"md`\n## NaN (Not a Number)\n\nAll numbers for which the exponent is ${tex`11111_2`} and the significand is nonzero are NaN ([Not a Number](https://en.wikipedia.org/wiki/NaN)) values. Since the exponent accounts for five bits, that leaves eleven free bits. Since exactly one combination indicates infinity (significand equal to zero), that leaves ${tex`2^{11} - 1 = ${Math.pow(\n 2,\n 11\n) - 1}`} unique NaN values.\n\nThe plot above shows NaN values in green, positioned similarly to the infinities as if interpreted as normalized numbers.\n\nA single non-zero bit in the significand turns an infinity into a NaN:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":774,"value":"binaryInput({\n type: 'float16',\n literalValue: 0b1111110000001000,\n displayParts: true,\n fractionalSignificand: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1110,"value":"md`## Definitions and Imports`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1126,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":1174,"value":"function floatToPosition(parts) {\n var sign = parts.sign ? -1 : 1;\n var log2Exponent = Math.max(-14, parts.exponent - 15);\n var significand = (parts.exponent === 0 ? 0 : 1) + parts.significand / 1024;\n return { x: sign * Math.pow(2, log2Exponent) * significand, y: log2Exponent };\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":350,"value":"floats = tabulateFloat16(precision)","pinned":false,"hidden":false,"mode":"ojs"},{"id":336,"value":"graphDiv = [0, 1, 2, 3, 4].map(() => document.createElement('div'))","pinned":false,"hidden":false,"mode":"ojs"},{"id":156,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":95,"value":"import { binaryInput } from '@rreusser/binary-input'","pinned":false,"hidden":false,"mode":"ojs"},{"id":399,"value":"import { slider, radio } from '@jashkenas/inputs'","pinned":false,"hidden":false,"mode":"ojs"},{"id":353,"value":"function plot() {\n var gd = arguments[0];\n\n // Create or reuse this plot div\n var hasPlot = !!gd.data;\n\n for (var i = 1, args = []; i < arguments.length; i++) args.push(arguments[i]);\n\n var layout = Array.isArray(args[0]) ? args[1] || {} : args[0].data;\n layout = Object.assign(layout, {\n width: Math.min(width, 800),\n margin: Object.assign(\n { t: 40, r: 20, b: 40, l: 50 },\n (layout || {}).margin || {}\n ),\n legend: Object.assign(\n {\n orientation: 'h'\n },\n (layout || {}).legend || {}\n ),\n hovermode: 'closest'\n });\n\n // Use Plotly.react to update the plot\n gd.value = Plotly.react.bind(null, gd).apply(null, args);\n\n // Return for use with viewof\n return gd;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":330,"value":"Plotly = require('plotly.js@2.35.3/dist/plotly-gl2d.min.js')","pinned":false,"hidden":false,"mode":"ojs"},{"id":732,"value":"html`\n