2c:null 2d:["$","title",null,{"children":"JavaScript Data Wrangling | Andy Wang | Observable"}] 32:T46f,md`If you want to visualize data on the web, you need to be able to interact with your data using JavaScript. While some amount of data preparation can be done in other programs (R, Python), some data manipulation can (and should!) happen in JavaScript. And, it isn't even that bad, I promise. This tutorial is written for people with _some_ JavaScript familiarity. If you need more of the basics, you can see my free [web development coursebook](https://info340.github.io/). For a more in depth tutorial, see [this notebook series](https://observablehq.com/@dakoop/learn-js-data) that is based on the wonderful site [learnjsdata](http://learnjsdata.com/). ### Table of Contents - [Loading data](#loading_data) - [Accessing Data](#accessing_data) - [Summarizing Data](#summarizing_data) - [Working with Dates](#date_data) - [Filtering Data](#filtering_data) - [Aggregating Data](#aggregating_data) - [Sorting Data](#sorting_data) - [Appendix](#appendix) Feel free to read through this tutorial, fork the [starter code](https://observablehq.com/@info474/javascript-data-wrangling-blank), or watch the video walkthroughs below! `33:[{"id":"70a3445122e7911d4ff1e0aaa918594caa7d1c82b6537035445ab146bd3108e4289996dcb246f540d248f5245e908c8f0fe643add3916f0c5c45ccfa3dc39790","href":"https://static.observableusercontent.com/files/70a3445122e7911d4ff1e0aaa918594caa7d1c82b6537035445ab146bd3108e4289996dcb246f540d248f5245e908c8f0fe643add3916f0c5c45ccfa3dc39790","name":"json_diagram@1.png","type":"image/png","lastModified":1612490651830.633,"size":1209515}] 30:{"id":"287e6abe977390bc","slug":"javascript-data-wrangling","title":"JavaScript Data Wrangling","thumbnail":"99685999f208ae37bf2a72c9714c6e5f4a97b5a19e2a220e428a4d72ec029637","access_level":"public","create_time":"$D2021-02-05T02:04:11.830Z","update_time":"$D2021-02-07T00:43:07.985Z","view_time":null,"first_public_time":"$D2021-02-05T15:39:21.403Z","has_unpublished_edits":null,"fork":{"id":"6090a2ccf391c818","slug":"javascript-data-wrangling","title":"JavaScript Data Wrangling","owner":{"id":"82cd20deb9e45d2a","login":"uw-info474","name":"Interactive Info Vis"}},"stars":1,"starred":false,"owner":{"type":"team","id":"761bb01238c592aa","login":"ypwangreg","name":"Andy Wang","avatar_url":"https://avatars-next.observableusercontent.com/f066769b5c435c0b184330c02f552465c6cffc37280bf952717c038b64c390ca","tier":"starter_2024","bio":""},"version":864,"publish_version":864,"role":null,"authored":false,"authors":[{"type":"individual","id":"31d22653369ef02c","login":"ypwangreg","name":"Andy Wang","avatar_url":"https://avatars-next.observableusercontent.com/f066769b5c435c0b184330c02f552465c6cffc37280bf952717c038b64c390ca","tier":null,"bio":""}],"members":[{"type":"individual","id":"31d22653369ef02c","login":"ypwangreg","name":"Andy Wang","avatar_url":"https://avatars-next.observableusercontent.com/f066769b5c435c0b184330c02f552465c6cffc37280bf952717c038b64c390ca","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"JavaScript Data Wrangling","cells":[{"id":0,"value":"md`# JavaScript Data Wrangling\n_Everything you never wanted to know about working with data in JavaScript..._`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":26,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":632,"value":"displayCaution(\n \"These videos were recorded using d3 version 5: this notebook has been updated, but the videos have not be (only the Aggregating Data section has changed)\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":589,"value":"html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":595,"value":"html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":32,"value":"loading_data = md`## Loading Data\nIn this tutorial, we'll be working with this [building permit data](https://data.seattle.gov/Permitting/Building-Permits/76t5-zqzr), which contains all building permits issued in the city of Seattle.\n\nWe'll be using lots of functionality from the [D3.js JavaScript library](https://d3js.org/), so we'll need to \\`require\\` it here:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10,"value":"// Require the d3 library\nd3 = require(\"d3\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":53,"value":"md`With the D3 library loaded, we can use use the \\`d3.csv()\\` function to both _load_ and _parse_ the data from its source (it's a large dataset, this may take a few seconds):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":13,"value":"// Load the Seattle building permit data using `d3.csv()`\ndata = d3.csv(\"https://data.seattle.gov/api/views/76t5-zqzr/rows.csv\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":66,"value":"md`If you were to download that CSV data, you could view it in a table, displaying the following structure:\n- Each **observation** (building permit) is a _row_\n- Each **feature** or **variable** about that observation is a _column_ (e.g., IssuedDate)\n- All observations are stored in a single **table**\n\nThis common structure is considered [Tidy Data](https://vita.had.co.nz/papers/tidy-data.pdf), and you can view a subset of the data here:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":70,"value":"// Display a table showing a subset of the data\nrender_data_table(data)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":34,"value":"accessing_data = md`## Accessing Data`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":97,"value":"md`When this is represented in JavaScript Object Notation (JSON):\n- Each **observation** (building permit) is an _object_\n- Each **feature** or **variable** about that observation is stored in a _key_ of that object\n- All objects are stored in a single **array**\n\nTo explore the building permit data structure, click on the carat (▸\\`Array\\`) in the cell below\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":139,"value":"// Inspect the data\ndata","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":125,"value":"displayImage(json_diagram, 600)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":143,"value":"md`To access a particular observation in the dataset, pass an _index_ (e.g., observation number) into square brackets \\`[]\\` next to your variable name:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":147,"value":"// Get the first (zero-ith) element\ndata[0]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":149,"value":"md`You can access _keys_ of each object using dot notation:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":153,"value":"// Access the PermitClass key of the first element\ndata[0].PermitNum","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":43,"value":"summarizing_data = md`## Summarizing Data`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":159,"value":"md`When you are _getting to know_ your data, you often ask these foundational questions:\n- How many **observations** are in the data?\n- How many **features** are present for each observation?\n- What are the **unique values** of a given feature?\n- What are the summary statistics (\\`min\\`, \\`max\\`, \\`mean\\`, etc.) of each **feature**?\n\nTo answer these questions, you can use a mixture of JavaScript methods and D3 or lodash utilities. Note, as is often the challenge, we need to translate these questions into programming tasks!\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":162,"value":"md`**Counting Observations**:\nCounting the _number_ of observations can be conceptualized as measuring the number of elements in our array. This can be accessed using the \\`Array.length\\` property`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":170,"value":"// How many observations (building permits) are in the dataset?\ndata.length","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":175,"value":"md`**Accessing Features**:\nEach object in our array should have the same _keys_ (e.g. _features_ about our data). We can access these from any object using the \\`Object.keys()\\` method:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":177,"value":"// Get a list of features in our dataset from the first observation\nObject.keys(data[0])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":180,"value":"displayCaution(\n \"Remember: It's important to do the appropriate domain research to understand the meaning of each feature in your dataset.\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":320,"value":"md`**Unique Feature Values:** To determine the unique set of values for a given feature (across your full dataset), it is helpful to use some functions from the [lodash](https://lodash.com/docs/4.17.15) package`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":323,"value":"// Require the lodash package\n_ = require('lodash')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":325,"value":"md`After loading that package (remember, in Observable cell order doesn't matter, so you can load it anywhere), you can leverage the \\`_.uniqBy()\\` method to access the unique values of a particular feature. Similar to the D3 functions described below, this function accepts two arguments:\n- An **array** of data that you want to access unique values from\n- A **function** for accessing a _value_ from each element in the array(_optional_)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":333,"value":"// Get a list of the unique values of the `PermitClass` feature\npermit_classes = _.uniqBy(data, d => d.PermitClass)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":335,"value":"md`Interestingly (oddly?), the \\`_.uniquBy()\\` method returns the entire object from our data. If you want to only return the information of interest (e.g., the \\`PermitClass\\`), you can use the JavaScript \\`.map()\\` iterator.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":338,"value":"// Get a list of the unique values of the `PermitClass` feature (only returning the string values of interest)\npermit_class_list = _.uniqBy(data, d => d.PermitClass).map(d => d.PermitClass)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":186,"value":"md`**Summary Statistics**:\nTo compute summary statistics, you can use JavaScript iteration methods such as \\`Array.reduce()\\` to compute sums, averages, etc. (though these can be quite dense to read and write....):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":189,"value":"// What is the total number of housing units added across the dataset?\ndata.reduce((sum, d) => (sum += +d.HousingUnitsAdded), 0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":196,"value":"md`D3 also provides some helper functions to allow us to do this a bit more expressively (e.g., \\`d3.min()\\`, \\`d3.sum()\\`, \\`d3.max()\\`). Each function accepts two argurments:\n- An **array** of data on which you want to compute your statistic\n- A **function** for accessing a _value_ from each element in the array(_optional_)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":199,"value":"// Get the sum of three values (an example of d3.sum _without_ an accessor function)\nd3.sum([1, 2, 3])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":201,"value":"// Get the sum of the total number of housing units added across the dataset\nd3.sum(data, d => +d.HousingUnitsAdded) // seems like a lot!","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":204,"value":"md`Note, in the cell above the _accessor function_ \\`d => +d.HousingUnitsAdded\\` allows us to access the \\`HousingUnitsAdded\\` property of each element in our data. The \\`+\\` ensures that we are converting the string to a number.\n\nYou can use similar functions to compute other summary statistics about the data:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":211,"value":"// What is the average estimated project cost?\nd3.mean(data, d => +d.EstProjectCost)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":215,"value":"// What is the standard deviation across project cost?\nd3.deviation(data, d => +d.EstProjectCost)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":216,"value":"// What is the most expensive project?\nd3.max(data, d => +d.EstProjectCost)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":226,"value":"date_data = md`### An aside: Working with Dates\nWorking with dates and times in JavaScript can be intimidating (I find it intimidating....), but it really just involves two steps:\n1. **Parsing** your string data to a proper [Date](https://www.w3schools.com/jsref/jsref_obj_date.asp) object \n2. Using a **formatter** to display the Date object in a human-readable way (often back into a string). \n\nLuckily, d3 provides us with the functions \\`d3.timeParse()\\` to convert from a string to a date, and \\`d3.timeFormat()\\` to format the date objects in readable ways. Let's begin by taking a look at the date of the first permit:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":789,"value":"data.find(d => { return d.IssuedDate != \"\"; }) // find first valid IssuedDate. It is in 2017.. not the earliest","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":797,"value":"data.findIndex(d => { return d.IssuedDate != \"\"; })","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":799,"value":"earliest = d3.minIndex(data, d => parser(d.IssuedDate)) //using the min to find the earliest permit. Note that parser is in below and I changed the format to match","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":231,"value":"// First Permit Issue date\ndata[earliest].IssuedDate","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":816,"value":"d3.max(data, d => parser(d.IssuedDate)) // that is 2 days ago.. interesting..","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":233,"value":"md`After checking a few more dates, you can see that the string is written in the format \\`YEAR-MONTH-DAY\\`. You can use that information to create a parser for your dates (see the [documentation](https://github.com/d3/d3-time-format#locale_parse) for mor information on parsing):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":240,"value":"// Create a time parsing function based on the structure of the date-strings in our data\nparser = d3.timeParse(\"%m/%d/%Y\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":648,"value":"parser(data[earliest].IssuedDate).getFullYear()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":244,"value":"md`You can then use your \\`parser\\` to convert your (string) dates into proper date objects (which we can then use to sort, filter, aggregate, etc.):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":246,"value":"// Parse the date of the first permit issued\nparser(data[earliest].IssuedDate)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":249,"value":"md`Once you have properly parsed your date strings, you can use them in your summary functions:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":251,"value":"// Get the extent (min and max) of the dates in the dataset\ndate_range = d3.extent(data, d => parser(d.IssuedDate))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":253,"value":"md`If you want to display those date in a human readable way, you'll need to create a **formattter**:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":255,"value":"// Define a formatting function for *displaying* date objects\nformatter = d3.timeFormat(\"%B %d, %Y\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":256,"value":"md`Then we can write properly formatted sentences like this one:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":260,"value":"// Write out a sentence describing the date range\nmd`The permit data ranges from [ **${formatter(date_range[0])}** - **${formatter(date_range[1])}** ]`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":221,"value":"md`The next step in the data wrangling process is to take these numbers and use them to filter your dataset to ask questions of interest. For example, I want to know more information about that expensive project!`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":35,"value":"filtering_data = md`## Filtering Data\nMany data analysis and visualization questions require reducing your data to a smaller subset. For example, in this section, we'll answer these questions:\n- _How many projects have been completed_?\n- _How many building permits have been issued in 2020_?\n- _What is the description of the most expensive project_?\n\nTo answer these questions, we'll be using the JavaScript \\`Array.filter()\\` method (at times in conjunction with some of the D3 methods above). The \\`filter()\\` method accepts a _function_ which returns a _boolean_ (\\`true\\` or \\`false\\`) indicating if you should keep each value:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":278,"value":"// How many projects have been completed? Only keep the objects where the StatusCurrent is \"Completed\"\nnum_completed_projects = data.filter(d => d.StatusCurrent === \"Completed\")\n .length","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":281,"value":"// How many permits have been issued in 2020? Only keep the objects where year is 2020\n// Note, we need to parse the date of each object, then get the *year* to perform the filter\npermits_2020 = data.filter(d => {\n // Note, some of the IssuedDate values are empty strings, so we need to check that first\n // (here, we only keep the object -- return true -- if the date isn't \"\" and has a year of 2020)\n return d.IssuedDate !== \"\" && parser(d.IssuedDate).getFullYear() === 2020;\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":302,"value":"// What is the description of the most expensive project?\n// Writing this out in a few different lines for clarity\nmost_expensive_2020 = {\n // Determine the *value* of most expensive project in the dataset\n const highest_cost = d3.max(permits_2020 , d => +d.EstProjectCost);\n\n // Filter down the dataset to the observations where the estimated cost is equal to the max\n // This will return an array with 1+ objects (more than one if multiple projects have the same cost)\n const most_expensive = permits_2020.filter(d => +d.EstProjectCost === highest_cost);\n\n // Extract the description from the first element in the array\n const description = most_expensive;//.Description;\n return description;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":44,"value":"aggregating_data = md`## Aggregating Data\nWhile filtering and summarizing data allows us to access single values of interest, we often want to repeat that process for _groups_ of observations. In other words, we want to **group** our data by some feature (e.g., the year, the type of permit, etc.), and produce a **summary** for each group. In this section, we'll answer the questions:\n- _How many permits were issued each year_?\n- _What is the total number of permits issued of each type (\\`PermitClass\\`)_?\n- _What is the average estimated cost associated with each permit type (\\`PermitClass\\`)_?\n\nWe, of course, can accomplish this using straightforward JavaScipt methods, though it's a bit verbose:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":315,"value":"// Calculate permits per year using plain old JavaScript\npermits_per_year = {\n let per_year = []; // this is the variable that we'll return\n\n // Iterate through the permits, adding new objects to the `per_year` array\n data.forEach(permit => {\n // Skip rows where the IssuedDate is not present\n if (permit.IssuedDate === \"\") return;\n\n // Store the year of this permit in a variable (for readability)\n const year = parser(permit.IssuedDate).getFullYear();\n\n // See if this year is present in the dataset\n let this_year = per_year.find(d => d.year === year);\n\n // If this year hasn't been added, add it -- otherwise, increment the number of permits by 1\n if (this_year === undefined) {\n per_year.push({ year: year, permits: 1 });\n } else {\n this_year.permits += 1;\n }\n });\n return per_year;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":363,"value":"md`Luckily, the lodash package makes this _really_ straightforward using the \\`_.countBy()\\` method:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":368,"value":"// Calculate permits per year using lodash -- wow, so much easier!\npermits_per_year_lodash = _.countBy(data, d => {\n // Ok, it's a little tricky because we still have to deal with the missing date\n if (d.IssuedDate === \"\") return \"No date present\";\n return parser(d.IssuedDate).getFullYear();\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":377,"value":"md`Unfortunately, lodash returns an _object_, but we usually want an _array_ for our visualization packages. To convert from an object to an array, we can iterate through the _keys_ of the array using the \\`Array.map()\\` method:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":381,"value":"// Use the Object.keys() method to convert our object into an array of objets\npermits_per_year_lodash_array = Object.keys(permits_per_year_lodash).map(d => {\n return { key: d, value: permits_per_year_lodash[d] };\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":386,"value":"md`D3 provides various methods that return the data in the desired format, but they take a moment to explain. The \\`d3.rollup()\\` [function](https://observablehq.com/@d3/d3-group) will compute summary statistics by group. The \\`d3.rollup()\\` function takes three (or more) arguments:\n\\`d3.rollup(data, reducer_function, key_function)\\`\n- \\`data\\`: The data (an _array of objects_) that you want to summarize\n- \\`reducer_function\\`: A function that computes your summary statistics for each group\n- \\`key_function\\`: A function that gets the _key_ to _group_ your data\n\nFor example, if we want to calculate the **number of permits in each permit class**, we can do the following:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":389,"value":"// Count the number of permits by PermitClass\nby_class = d3.rollup(\n data, // our array of objects\n v => v.length, // the reducer_fuction that calculate a summary statistic for each group\n d => d.PermitClass // the key_function that gruops the data by this key\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":680,"value":"md`The above returns a JavaScript Map object, which you can interact with in a few ways:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":676,"value":"// Retrive the keys using a spread operator\n[...by_class.keys()]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":856,"value":"md`Javascript spread operator just like ***list** or ****dict** in Python. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":688,"value":"// Retrive the values using a spread operator\n[...by_class.values()]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":779,"value":"// Get a value of interest\nby_class.get(\"Multifamily\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":692,"value":"md`Alternatively, you can use \\`d3.rollups()\\` -- note the plural \\`s\\` -- to return an _array of arrays_ with the key-value pairs:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":694,"value":"by_class_array = d3.rollups(data, v => v.length, d => d.PermitClass)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":421,"value":"md`Even more exciting is that we can compute _multiple_ summary statistics in our \\`.rollup()\\` function!`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":425,"value":"// Calculate summary statistics by permit class\nsummary_by_class = d3.rollup(\n data,\n v => ({\n count: v.length, // number of elements with this PermitClass\n mean_cost: d3.mean(v, d => +d.EstProjectCost), // get the average estimated project cost!\n max_cost: d3.max(v, d => +d.EstProjectCost) // get the max estimated project cost!\n }),\n d => d.PermitClass\n)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":744,"value":"// Calculate summary statistics by permit class (using .rollups() to return an array instead of a map)\nsummary_by_class_array = d3.rollups(\n data,\n v => ({\n count: v.length,\n mean_cost: d3.mean(v, d => +d.EstProjectCost), // get the average estimated project cost!\n max_cost: d3.max(v, d => +d.EstProjectCost) // get the max estimated project cost!\n }),\n d => d.PermitClass\n)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":709,"value":"md`You can now ask great questions like, _What is the average cost for Commercial permit classes submitted?_`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":711,"value":"summary_by_class.get(\"Commercial\").mean_cost","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":713,"value":"md`Even more exciting (!!) is that the \\`d3.rollup\\` methods allow you to create multiple levels of grouping by providing additional key functions. For example, if we want to calculate the _average project cost by PermitClass and year_, we can do the following:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":718,"value":"// Calculate summary statistics by permit class and year\nsummary_by_class_and_year = d3.rollup(\n data,\n v => ({\n count: v.length,\n mean_cost: d3.mean(v, d => +d.EstProjectCost), // get the average estimated project cost!\n max_cost: d3.max(v, d => +d.EstProjectCost) // get the max estimated project cost!\n }),\n d => d.PermitClass,\n d => (d.IssuedDate ? parser(d.IssuedDate).getFullYear() : \"\")\n)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":729,"value":"md`You can now ask more nuanced questions like, _How many Industrial Permits were issued in 2015?_`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":735,"value":"summary_by_class_and_year.get(\"Industrial\").get(2015).count","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":433,"value":"md`And just to enjoy some of the fruits of our labor, we can visualize our aggregated data using the \\`vega-lite\\` package`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":749,"value":"// Reformatting into key / value pairs for vegalite\nvl_data = summary_by_class_array.map(d => {\n let obj = {};\n const [key, value] = d;\n obj = { key: key, value: value.count };\n return obj;\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":436,"value":"// Load the vega-lite package\nimport { vl } from \"@vega/vega-lite-api\"","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":439,"value":"// Create a bar chart of the number of permits of each class\nsimpleBar = vl\n .markBar() // Make a bar chart\n .data(vl_data) // Using the summary data by Permit Class\n .encode(\n vl\n .x()\n .fieldQ(\"value\") // Encode the number of permits on the x axis\n .axis({ title: \"Number of Permits\" }), // Set the axis title\n vl\n .y()\n .fieldO(\"key\") // Encode our Key on the y axis (Permit Class),\n .sort(null) // Show the order that they appear in the data (for demonstration)\n .axis({ title: \"Permit Class\" }) // Set the axis title\n )\n .render() // display the chart","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":470,"value":"md`Oh no! The bars aren't in the desired order -- this means that we'll need to take a few more minutes and talk about **sorting**.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":473,"value":"sorting_data = md`### Sorting Data\nIn order to sort the visual elements in our graphics, we often have to sort them in the data. Note, sorting happens **in place** in JavaScript, so for _demonstration purposes only_, we'll clone our \\`vl_data\\` array for working with it here:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":482,"value":"// Clone our summary_by_class data so we can sort it (for demonstration only)\ndata_to_sort = _.clone(vl_data)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":484,"value":"md`To sort our data, we'll pass in a **comparator function** to the \\`Array.sort()\\`. The comparator function describes how to compare any two items in the array (for a cool interactive visualization of different sorting algorithms, see [this project](http://sorting.at/))`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":487,"value":"// Here is our sorting function to sort by the number of permits (recall data structure above)\nsort_by_num_permits = (a, b) => +b.value - +a.value","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":489,"value":"// Sort the data using the function described above (which could have been written in line)\ndata_to_sort.sort(sort_by_num_permits)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":492,"value":"md`To confirm that our data is sorted, you could inspect it above using the ▸, or we could display the sorted data in a chart:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":494,"value":"// Make a bar chart of the number of permits -- same code as above, but using the sorted_data\nsorted_bar = vl\n .markBar() // Make a bar chart\n .data(data_to_sort) // Using the *sorted* data\n .encode(\n vl\n .x()\n .fieldQ(\"value\") // Encode the number of permits on the x axis\n .axis({ title: \"Number of Permits\" }), // Set the axis title\n vl\n .y()\n .fieldO(\"key\") // Encode our Key on the y axis,\n .sort(null) // Show the order that they appear in the data (for demonstration)\n .axis({ title: \"Permit Class\" }) // Set the axis title\n )\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":500,"value":"md`Whew! Thanks for reading! Hopefully that helps guide you through some of the foundations of working with data so you can get onto the _fun stuff_ (making awesome visualizations)!`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":23,"value":"appendix = md`## Appendix`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":63,"value":"import {\n render_data_table,\n table_styles,\n displayImage,\n displayCaution\n} from \"@uw-info474/utilities\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":121,"value":"json_diagram = await FileAttachment(\"json_diagram@1.png\").url()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":80,"value":"table_styles","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W33","stdlib":"1"}} 31:"$30:body"