2d:null 2e:["$","title",null,{"children":"Untitled | Jonas Almeida | Observable"}] 33:T51c,html`

Tidying data up at Web scale

Jonas S Almeida, ..., Praful?, Eric?

Abstract

Exploring how the opinionated Tidy data wrangling framework Eric Dawson told us about at the Cloud4Bio hackathon can be approached in the web computing domain. This is a large community (tidyverse.org), with their own opinionated Domain Specific Language to traverse complex data landscapes. This could also be a good excuse to explore literate programming as an Observable Notebook.

Introduction

Because data wrangling is often where data scientists end up spending most of their time, there is a high premium set frameworks that normalize and integrate that process. That is the case in a variety of applications, ranging from interactive visualization, such as D3js to or high end Artificial Inteligence libraries like TensorFlow (links point to the corresponding data wrangling methods).

`34:[] 31:{"id":"1338704a279de086","slug":"tidy","title":"","thumbnail":"fcdcfa62f619eb5f1347b13f8017e2f1badc6c9f427cec078d2562651404dd75","access_level":"public","create_time":"$D2019-03-20T15:16:38.631Z","update_time":"$D2019-03-27T15:51:21.289Z","view_time":null,"first_public_time":"$D2019-03-20T15:16:41.659Z","fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"b96b15d71f10f6b6","login":"jonasalmeida","name":"Jonas Almeida","avatar_url":"https://avatars-next.observableusercontent.com/bd90e360f80f2fda3dd6aa9bdda3474b409710441d06e67f9ac936b3660aba05","tier":"starter_2024"},"version":100,"publish_version":100,"role":null,"authored":false,"authors":[{"type":"individual","id":"2fe10bd213abfed1","login":"jonasalmeida","name":"Jonas Almeida","avatar_url":"https://avatars-next.observableusercontent.com/bd90e360f80f2fda3dd6aa9bdda3474b409710441d06e67f9ac936b3660aba05","tier":null}],"members":[{"type":"individual","id":"2fe10bd213abfed1","login":"jonasalmeida","name":"Jonas Almeida","avatar_url":"https://avatars-next.observableusercontent.com/bd90e360f80f2fda3dd6aa9bdda3474b409710441d06e67f9ac936b3660aba05","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"","cells":[{"id":0,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":15,"value":"html`\n

Methods

Let's get some data from a public source and go about making it tidy, say CDC data on cancer across the US. The first n results are stored in the variable dt which will be used to validade equivalent tidyness in JS.

`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":32,"value":"n=100","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":26,"value":"url=`https://chronicdata.cdc.gov/resource/u9ek-bct3.json?$limit=${n}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8,"value":"dt = (await fetch(url)).json()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":94,"value":"d3 = require('d3')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":97,"value":"tf = require(\"https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@1.0.0/dist/tf.min.js\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":84,"value":"html`\n

Results

D3 and TF approach data wrangling in complementary ways. The former aims at rapid data integration whereas the latter pays particular attention to memeory management - the data is not necessarily preloaded in memory, a very useful feature to analyse large datasets that do not fit whole into memory.

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":88,"value":"html`\n

D3

...

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W34","stdlib":"1","titling":"auto"}} 32:"$31:body"