d3.autoType d3-dsv now supports automatic type inference. When you parse a CSV file, it can optionally detect common data types such as numbers, dates and booleans, and convert values to the corresponding JavaScript type. Why is this useful? If you don’t perform any type conversion, you’ll get strings.
untyped = d3.csv(url)
untyped[0]
Strings are fine for inspection, but before you can start working with the data, you typically need to convert strings to an appropriate type.
"95.35" + "98.84" // concatenation
95.35 + 98.84 // addition
Prior to automatic type inference, you would write a row function. This function is called for each row in the CSV file, taking an object with string values and returning an object with values of the desired types. Thanks to arrow functions and destructuring, these can be written fairly concisely.
typed = d3.csv(url, ({date, close}) => ({date: new Date(date), close: +close}))
typed[0]
While flexible and explicit, row functions can be tedious to write because you have to enumerate the column names and types. To use automatic type inference, pass d3.autoType in place of the row function.
auto = d3.csv(url, d3.autoType)
auto[0]
So automatic type inference is great… right? What’s the catch? 🤔 The problem is that a value can have ambiguous type and d3.autoType might chose the wrong one. In these cases, you’ll still need to implement your own row function to convert types explicitly. And because automatic type inference cannot be guaranteed to be correct, you should be especially careful using it. Unless you carefully review the results, it could corrupt your data and weaken your analysis or result in a misleading visualization! Let’s consider a few more interesting examples by calling d3.autoType directly to inspect how it works.
d3.autoType({date: "2007-04-23", close: "93.24"})
For dates, d3.autoType uses ECMAScript’s date-time string format, which is a small subset of ISO 8601. YYYY is a valid ECMAScript date:
new Date("2007")
However, it’s also coercible to a number:
+"2007"
If the input is both a valid date and a valid number, d3.autoType favors the number. (This is usually what you want, because otherwise a four-digit number would always be converted to a date!)
d3.autoType({date: "2007"})
When a date-only string such as YYYY-MM-DD is specified, the inferred time is midnight UTC; however, if a date-time string such as YYYY-MM-DDTHH:MM is specified without a time zone, it is assumed to be local time. This is a little confusing, but this is the behavior standardized by ECMAScript (and it’s consistent with how dates are displayed in Observable).
d3.autoType({date: "2007-01-01"}).date.toISOString()
d3.autoType({date: "2007-01-01T00:00"}).date.toISOString()
new Date("2007-01-01T00:00:00Z")
new Date("2007-01-01T00:00:00")
Some values are coercible to numbers but should be preserved as strings. For example, FIPS county codes are five-digit codes that identify U.S. counties. If you use d3.autoType, the conversion to number will strip leading zeroes.
d3.autoType({id: "06075"})
There are myriad ways of representing numbers, booleans and dates as strings. d3.autoType only supports a very small and strict subset of formats. For example, none of these values are considered numbers because they are not strictly coercible using JavaScript’s unary + operator:
d3.autoType([ "$1.00", // currency symbol "(123)", // parenthesis "1,234", // comma "12px" // suffix ])
Dates and booleans are likewise strict:
d3.autoType([ "January 1, 2018", // not YYYY-MM-DD "TRUE" // not lowercase ])
Automatic type inference is intentionally limited so that the behavior of d3.autoType is easier to understand and remember, which hopefully means it’s less surprising. For a given input value, the value is trimmed and then re-assigned as follows: If empty, then null. If exactly “true”, then true. If exactly “false”, then false. If exactly “NaN”, then NaN. Otherwise, if coercible to a number, then a number. Otherwise, if a date-only or date-time string, then a Date. Otherwise, a string (the original untrimmed value). To see d3.autoType in practice, see this example line chart.
Appendix
url = "https://gist.githubusercontent.com/mbostock/14613fb82f32f40119009c94f5a46d72/raw/d0d70ffb7b749714e4ba1dece761f6502b2bdea2/aapl.csv"
d3 = require("d3@^5.9")