D3-Annotation with D3 Line Chart Adding Annotations to Charts Based on Mike Bostock's basic Observable line chart
Bitcoin Price (USD) over Time
Some data visualizations aim to help people discover new insights. Others aim to share insights that other people have found. Annotations are important for both kinds, as they are good at highlighting unusual areas in the data or providing background information. Annotations are to information graphics as sparklines ( ) are to text reports - just another way to bring text and graphics closer together. Providing textual context close to data reduces the cognitive burden created when descriptions are far away from the data they refer to. Edward Tufte has long emphasized the advantages of keeping graphics and text in close proximity. Despite the clear benefits, it can be tedious to use annotations in practice, especially with interactive web pieces where there aren't convenient graphical user interfaces to help with placing them. Happily, there are now several JavaScript libraries that can make the life of an annotator (you!) better. This interactive notebook will show you how to use Susie Lu's d3-annotation plugin in Observable when making d3 graphs. I like this library because it simplifies the most frustrating part of adding labels to charts (and the reason that I occasionally take screenshots of interactives and annotated them in PowerPoint): positioning labels through guess-and-checking x-y coordinates is slow! d3-annotate has an "edit" mode that lets you tweak the positions of each annotation using your cursor, making this process considerably easier. See this GIF for an example: Prerequisites / Setup If you're familiar with the basics of d3 and JavaScript, you'll be able to follow this guide. If this is your first Observable notebook, I recommend checking out the 5-minute tutorial, which covers how to run cells and introduces the reactive programming model. In this example, I've combined data from 99bitcoins and the Coindesk API to show how the d3-annotation API work together with Observable's interactive notebook features to make annotating a breeze. I recommend forking this notebook (the fork button is in the top left corner), and updating the annotations in your copy. You can use this notebook as a starting point to mark up d3 visualizations of your own datasets!
Charting Basics 1) Download the data from Coindesk using the fetch api.
// Adjust the dates in the URL based on your time area of interest. Format is YYYY-MM-DD. data = { const startDate = '2017-07-01' // Adjust these dates based on your area of interest. const endDate = '2018-03-07' const query = `start=${startDate}&end=${endDate}` const {bpi} = await (await fetch(`https://api.coindesk.com/v1/bpi/historical/close.json?${query}`)).json(); return Object.entries(bpi).map(([key, value]) => ({date: new Date(key), value})); }
2) Plot the data with D3.js. I've modified the code from this example and placed helper functions in the "General Charting Functions" section at the end. (Since this is a reactive notebook, the order of the cells on the page do not matter). Helper functions are borrowed from Mike Bostock's example, and are hidden at the end of this notebook. The chart should show up immediately below this paragraph.
render_linechart = function(data, xAxis, yAxis, lineFunction, debug) { /* Return a DOM node with containing a line chart. Arguments: data: a list of data objects (dictionaries or iterables) xAxis: a d3.axis yAxis: a d3.axis lineFunction: a function used to extract values from each datapoint and render line segments. debug: variable that forces the chart to redraw every time this step reruns */ const svg = d3.select(DOM.svg(width, height)); svg.append("g") .call(xAxis); svg.append("g") .call(yAxis); svg.append("path") .datum(data) .attr("fill", "none") .attr("stroke", "#3172bc") // Enigma Blue .attr("stroke-width", 1.5) .attr("stroke-linejoin", "round") .attr("stroke-linecap", "round") .attr("d", lineFunction) return svg.node(); }
Rise of Bitcoin
chart = { labelAnnotations // Make this cell dependent on annotations so the chart will redraw on annotation updates return render_linechart(data, xAxis, yAxis, line, DEBUG) }
The chart won't appear on the initial page load. Load it by clicking the checkbox:
viewof CHECKED = checkbox( { options: [ { value: 'toggle', label: 'Toggle Edit Mode' } ], value: '' })
annotations_display = { let collection = labels.collection() if (collection) { return collection.annotations } else { return "Manually rerun this cell to check the annotation marked by annotation_number" } }
DEBUG = { return (CHECKED == "toggle") }
Building the Annotations List
To start the annotation process, define a list of points to comment on. You may select either individual points in time, or regions that you'd like to circle. Below, I define an array of dictionaries. The offset coordinates (dx, dy) used were not lucky guesses- they were tuned through the process described in the next section. If you're starting from scratch, use 0 as default values for both. I've expanded this list for legibility, but you can collapse the cell labelAnnotations when you're done editing. These events were selected from the 99bitcoins Price Timeline.
labelAnnotations = { return [{ id: "bitcoin-cash-fork", // If you don't provide a custom "type" attribute in your options dictionary, , // the default type in the getAnnotations function will be used. note: { label: "Bitcoin splits into Bitcoins and Bitcoin Cash", title: "08-01-2017" }, data: get_date_data("2017-08-01T00:00", data), dx: -15, dy: -57, },{ id: "china-shutdown-exchanges", note: { label: "China shuts down all crypto exchanges", title: "09-15-2017", }, data: get_date_data("2017-09-15T00:00", data), dx: -18, dy: -50, },{ id: "segwit-2x-delayed", type: d3.annotationCallout, // this type of annotation draws a line under your label. note: { label: "Segwit2X delayed", title: "11-08-2017" }, data: get_date_data("2017-11-08T00:00", data), dx: -2, dy: 104, },{ id: "cross-ten-thousand", type: d3.annotationCalloutCurve, // this type of annotation lets you used a curved connector. note: { label: "First time over $10k", title: "11-28-2017" }, connector : { // pass the connector an array of points to define your curve. points: [ [-9, -24], [-30, -44.6] ] }, data: get_date_data("2017-11-28T00:00", data), dx: -53, dy: -47, },{ id: "all-time-high", note: { label: "First time over $20k", title: "12-16-2017" }, data: get_date_data("2017-12-16T00:00", data), dx: -102, dy: 2, },{ id: "segwit-2x-hard-fork", note: { label: "Segwit2X hard fork", title: "12-29-2017" }, data: get_date_data("2017-12-29T00:00", data), dx: -4, dy: 89, }, { id: "valley-of-uncertainty", note: { label: "What caused this rapid drop + rebound?", title: "Jan - Feb '18" }, color: "#ef4837", // Brighter color to make this annotation stand out x: 803, y: 336, dx: -5, dy: 115, subject: { radius: 52, radiusPadding: 5 }, type: d3.annotationCalloutCircle // This annotation refers to an area rather than a point }, ] } // I used 4 of the 8 annotation types, be sure to see the other 4 options when building your own! // http://d3-annotation.susielu.com/#types
This next section is where d3-annotation really shines. Ensure the box below the graph, named CHECKED, is selected. Then, for each of your annotations: Adjust the annotation by dragging the gray circles with your cursor. Different annotation types have different types of circles to adjust. Take those updated values, and paste the updated properties into the matching annotation in the labelAnnotations cell. (The display_annotations cell below the graph displays those updated values each time you rerun it). Run labelAnnotations, and the chart automatically refreshes. Step 2 updates the internal properties of the annotations in the chart. How can we access those updated values? Observable doesn't give scripts access to window.copy for security, so the technique described in the docs won't work (Calling labels.annotations() in the browser console has no effect). However, we can access the raw properties that were edited through our dragging interactions by running the "annotations_display" cell immediately below the chart. You can take the updated values (_dx, _dy, _x, and _y for the basic labelAnnotation) and update their counterparts in the labelAnnotations cell. Every time you rerun the labelAnnotations cell, the chart refreshes (Either click the play button in the cell's top right corner, or hit shift + enter) When you're happy with the layout, toggle the CHECKED box to remove the circles.
getAnnotations = function(annotationList, x, y, parseTime, formatTime, debug){ /* Return a list of d3.annotation objects Arguments annotationList - a list of dictionaries used to configure each annotation x: a d3.scale y: a d3.scale parseTime: function for turning strings into js dates formatTime: function for turning js date into a string debug: a boolean flag */ chart // reactive hack to force annotations to render whenever the chart is redrawn let makeLabelAnnotations = d3.annotation() .editMode(debug) // GLOBAL VARIABLE .type(d3.annotationLabel) // Adjust this arg to adjust the default annotation styling. .accessors({ x: d => x(parseTime(d.date)), y: d => y(d.close) // If you use a new dataset, you may need to update this accessor. }) .accessorsInverse({ date: d => formatTime(x.invert(d.x)), close: d => y.invert(d.y) }) .annotations(annotationList) return makeLabelAnnotations }
// This function is responsible for converting the list of dictionaries into actual d3.annotation objects labels = getAnnotations(labelAnnotations, x, y, parseTime, formatTime, DEBUG)
applyAnnotations = function(annotations, target) { /*Draws d3.annotation objects onto a designated DOM node */ d3.select(target) .append("g") .attr("class", "annotation-group") .call(annotations) return annotations }
// Lastly: actually place these annotations on your chart applyAnnotations(labels, chart)
Helper Functions The following two helper functions are used to help with looking up values for the annotations, but aren't specific to the annotation library. If you try this exercise with a new dataset, you may want to put your data cleaning/filtering/reshaping functions in this section.
days_are_equal = function(day_a, day_b) { /* True if two datetimes occur on the same day */ return (day_a.getDate() === day_b.getDate() & day_a.getMonth() === day_b.getMonth() & day_a.getFullYear() == day_b.getFullYear()) }
// Helper function to lookup where annotations match values get_date_data = function(datestring, data) { /* Returns the first item in a data whose date property matches the provided datestring Args: data: an array of data points to filter on datestring: a string representing a datetime */ const filtered = data.filter(x => days_are_equal(x.date, parseTime(datestring))) return { date: datestring, close: filtered[0].value } }
General Charting Functions The functions below are largely unchanged from their definitions mbostock's original line chart.
parseTime = d3.isoParse
formatTime = d3.isoFormat
height = 600
margin = ({top: 20, right: 30, bottom: 30, left: 40})
x = d3.scaleTime() .domain(d3.extent(data, d => d.date)) .range([margin.left, width - margin.right])
y = d3.scaleLinear() .domain([0, d3.max(data, d => d.value)]).nice() .range([height - margin.bottom, margin.top])
xAxis = g => g .attr("transform", `translate(0,${height - margin.bottom})`) .attr("class", "x-axis") .call(d3.axisBottom(x) .ticks(width / 80) .tickSizeOuter(0))
yAxis = g => g .attr("transform", `translate(${margin.left},0)`) .attr("class", "y-axis") .call(d3.axisLeft(y)) .call(g => g.select(".domain").remove()) .call(g => g.select(".tick:last-of-type text").clone() .attr("x", 3) .attr("text-anchor", "start") .attr("font-weight", "bold") .text(data.y))
// Helper function for creating a line in D3 v4 line = d3.line() .defined(d => !isNaN(d.value)) .x(d => x(d.date)) .y(d => y(d.value))
html`<hr>`
Call to Action: Filling in Context Gaps with Enigma Public The 99bitcoins timeline doesn't have any events listed for activity in 2018, hence my annotation for the "Valley of Uncertainty". An open question for people with financial interest in this space is whether we (in March 2018) are in a cryptocurrency "bubble". While I can't answer that question yet, there are some datasets which may offer clues. This section will briefly introduce a few datasets that could be used to source smaller events that could be worthwhile to annotate in the future. Enigma Public gathers a wealth of datasets from government sources, most of which don't have a convenient API for retrieving data. After searching for blockchain in our search engine, I've decided to examine a dataset that changes almost daily, SEC Form D Filings. Companies must file these forms before they can start selling stock to investors. As corporate registrations are handled state-by-state in different datsets, SEC Form D is convenient way to keep track of emerging companies at the federal level in a single dataset. One naive (but quick!) filter is to search for businesses that mention certain keywords in their filings. While this misses more subtle companies, it gives us a window into new investor activity in this space that may not be captured in the the typical APIs that people consider. I've included some helper functions from the next notebook in this series that simplify interactions with the Enigma Public API.
These form fields are prefilled with terms from SEC Form D 2018, but they can be edited to work with any Enigma dataset! However, if the new datasets have a different schema, you'll need to update the charting code too.
viewof DATASET_ID = html`<input type="text" value="a6c397ff-4e20-4674-b47b-1eaa92d9c8bc">`
viewof QUERY = html`<input type="text" value="bitcoin|blockchain|cryptocurrency">`
ROW_LIMIT = 1000
raw_payload = get_enigma_dataset(DATASET_ID, ROW_LIMIT, QUERY)
columns = raw_payload.current_snapshot.fields
normalized_dataset = normalize_enigma_payload(raw_payload).sort(function(x, y) { return d3.ascending(x.filed_as_of_date, y.filed_as_of_date) })
build_crypto_linechart = function(dataset, year) { return vegalite({ data: { values: dataset }, mark: "bar", width, height: 240, autosize: "fit", encoding: { x: {field: "filed_as_of_date", type: "temporal", timeUnit: "yearmonthday", axis: { format: "%m/%d", title: "Day" }}, y: { aggregate: "count", field: "*", type: "quantitative", scale: { domain: [0, 5]}}, }, title: `Daily Cryptocurrency Related Form D Filings in ${year}` }) }
build_crypto_linechart(normalized_dataset, 2018)
We can compare this chart to the activity in 2017, using the 2017 Form D table:
PREV_YEAR_DATASET = "2635adc1-0638-4bbb-8bcc-59a65bfd7b6f"
prev_payload = get_enigma_dataset(PREV_YEAR_DATASET, ROW_LIMIT, QUERY)
previous_year_data = normalize_enigma_payload(prev_payload)
build_crypto_linechart(previous_year_data, 2017)
Amazingly, despite the turmoil early in 2018, this hasn't stopped new companies from entering the cryptocurrency market, and putting "Bitcoin" and "Blockchain" in their Form D paperwork. We can see that Form D single-day filings climbed together with bitcoin prices, and filing activity has been much more consistent in 2018 than it was in 2017. In fact, there have already been filings in 2018, compared to for all of 2017! I've printed a preview of the 51 columns of data available for each of the 2018 filings, or you can use the dedicated data explorer to view everything at once. I encourage you to modify this notebook to select the datasets and columns most relevant to your work.
Name | Filed as of Date | Revenue Range --- | ---
Next Steps This is just the beginning of what is possible with d3-annotation and adding context to datasets by combining information from multiple sources. Try annotating interesting points on another type of graphic, such as this scatterplot of deleted European Politician Tweets. Add interactive capabilities to this line chart, such as triggering actions when tooltips are clicked or hovered over. See this block for an example, as well as the official d3-annotation gallery of examples. Check out where bitcoin shows up in public datasets with Enigma Public, and look for relationships between bitcoin related entities and the events marked on our timeline. There are numerous hits in the collections for corporate registration lists, US trademarks, and the SEC "Fails to Deliver" logs. In part 2 of this series on using visualization in Observable, we'll use the Vega Lite charting library to explore a public dataset which updates daily. If you're hungry for more public data explorations and technique posts, check out the Enigma Blog or our monthly open data newsletter, Between Two Rows. Questions? Feedback? I'd love to hear from you. I'm @hydrosquall on [Twitter]((https://twitter.com/hydrosquall) and [Github]((https://github.com/hydrosquall). With thanks to Peter Henderson, Susie Lu, and Erick Katzenstein for reviewing earlier drafts of this post.
Sources Significant Bitcoin events: 99bitcoins.com Mike Bostock's basic line chart example Pricing data: coindesk API.
Appendix See Elijah Meek's post introducing Susie Lu's d3-annotation when it first launched last year for more about the value of annotations in custom dataviz. See below for the cells used to (somewhat hacky) way to get d3-svg-annotation coupled with d3 in Observable, without needing to use a custom d3 bundle. Several charting libraries have some form of annotation support built in, such as (plotly.js, semiotic). However, they are full-fledged graphing frameworks, and wouldn't be good fits for adding annotations to existing D3 charts. Check out @jashkenas's inputs notebook for input controls for your notebooks! (This was the source for the "debugging" checkbox). If you'd like to save your chart as a static .png file, use the download option below.
DOM.download(await rasterize(chart), null, "Download Chart as PNG")
d3_base = require("https://d3js.org/d3.v4.min.js")
d3_svg_annotation = require("d3-svg-annotation")
d3 = { // Add const bundle = Object.assign({}, d3_base, d3_svg_annotation); return bundle; }
import { checkbox } from "@jashkenas/inputs"
import { get_enigma_dataset, normalize_enigma_payload } from "@hydrosquall/visually-exploring-nyc-events-from-enigma-public-with-vega-"
vegalite = require("@observablehq/vega-lite@0.1")
import {rasterize} from "@mbostock/saving-svg"
import {sparkline} from "@mbostock/introduction-to-html"
// style fixes html`<style> #r1934 { font-size: 1.3rem;} </style>`