31:Tb02,### 1c. Practicing with time formatting If we want to work with d3.timeParse() then we need to know how to define something like "%d-%m-%Y" to begin with. How do we do that? The kind of neat thing about D3's time format system is that it uses the same kind of syntax that was first pioneered by C's strtime function (Python also uses this too, actually). Here's a table to help you understand the syntax involved: | Directive | Meaning | Example | | --------- | ----------------------------------------- | ----------------- | | `%Y` | 4-digit year | `2025` | | `%y` | 2-digit year | `25` | | `%m` | Month number (01–12) | `03` | | `%b` | Abbreviated month name | `Jan`, `Feb` | | `%B` | Full month name | `January` | | `%d` | Day of month (01–31) | `09` | | `%e` | Day of month (space-padded) | ` 9` | | `%a` | Abbreviated weekday | `Mon` | | `%A` | Full weekday | `Monday` | | `%H` | Hour (00–23, 24-hour clock) | `14` | | `%I` | Hour (01–12, 12-hour clock) | `02` | | `%p` | AM/PM indicator | `AM`, `PM` | | `%M` | Minute (00–59) | `05` | | `%S` | Second (00–61, includes leap seconds) | `09` | | `%L` | Milliseconds (000–999) | `123` | | `%Z` | Time zone abbreviation | `GMT`, `UTC` | | `%j` | Day of year (001–366) | `032` | | `%U` | Week number (Sunday as first day of week) | `07` | | `%W` | Week number (Monday as first day of week) | `06` | | `%x` | Locale-specific date | depends on locale | | `%X` | Locale-specific time | depends on locale | | `%%` | Literal `%` | `%` | For example, if you have a timestamp such as "01/15/20", you can easily designate the format as "%m/%d/%y"; this tells d3.timeParse() that "I expect timestamps to be in a month/day/year format".32:T7b2,--- ## Part 3: Visualizing Taxi Data In today's lecture, we covered the [NYC Taxi Open Dataset](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page), which gives us a lot of juicy info about taxi pickups across many years in NYC. Let's look at one specific subset of that data: the week of Thanksgiving in November, 2024. We've imported three files into this notebook already: - yellow_tripdata_2024-11_thanksgiving.csv - Google Drive alternative - this Google Colab Pre-Processing - taxi_zones_wgs84.json - Google Drive alternative - Google Colab Pre-Processing ${comment(html`The taxi data was originally formatted in the .parquet format, which JavaScript and D3 cannot read. We've pre-processed the dataset, removed some unnecessary columns, and converted it into a .csv for you already via this Google Colab Notebook. Furthermore, the same data repository provides shapefiles for taxi regions in NYC. However, this geodata's coordinates were localized to the New York State Plane coordinates (EPSG:2263). We used this Google Colab notebook to convert all coordinates to EPSG:4326 (longitude/latitude) as well as transform it into GeoJson.`)}33:T4b2,### 3c. Understanding Zooming and D3 The ability to zoom into geodata visualizations is a useful tool to distinguish finer details. In understand how to incorporate zoom functionality, observe the code sample below: ```js // We assume that you have an `svg` and a `chart` already const zoom = d3.zoom() // Zoom Multiplier: [min, max] .scaleExtent([1, 10]) // Event Listener: on "zoom", transform your visualization container .on("zoom", (event) => chart.attr("transform", event.transform)) ``` In this code example, we are creating a callback function called zoom. This function checks how much you are scrolling with your mouse and restricts the zoom amount up to 10x the original scale. It then creates an event listener such that, upon detecting a mouse scroll, it transforms the chart ${comment(html`We modify chart instead of svg because we want the visualization to be windowed within the dimensions of the svg itself. If we applied the transformation to svg itself, then you'll see some pretty weird behavior.`)}34:[{"id":"75cb7bac41d92045362fe0f2b330cfe9f580491b3135d7ecc045218c9fefd879989415498d1279885f0a761966ae4fe4a1003f18d98ba410c26eacc86021520d","href":"https://static.observableusercontent.com/files/75cb7bac41d92045362fe0f2b330cfe9f580491b3135d7ecc045218c9fefd879989415498d1279885f0a761966ae4fe4a1003f18d98ba410c26eacc86021520d","name":"delhi-climate.csv","type":"text/csv","lastModified":1761909080621.336,"size":78199},{"id":"7eb327eeacfae3653a05935f330ee182a5496ae30259c5a320dbe32abe8562adb723691244160ccb12145757acff986a779c07fc08451128e0c50265310a58ac","href":"https://static.observableusercontent.com/files/7eb327eeacfae3653a05935f330ee182a5496ae30259c5a320dbe32abe8562adb723691244160ccb12145757acff986a779c07fc08451128e0c50265310a58ac","name":"yellow_tripdata_2024-11_thanksgiving@1.csv","type":"text/csv","lastModified":1761918042045.923,"size":41907705},{"id":"a43e63dc436a37c84868d63415fdd135baf79308c801872b5904e4abf54231fb86a84fa5121383a079696d2c00b62020515bd91575c0ff3cdc3df9fe9039f464","href":"https://static.observableusercontent.com/files/a43e63dc436a37c84868d63415fdd135baf79308c801872b5904e4abf54231fb86a84fa5121383a079696d2c00b62020515bd91575c0ff3cdc3df9fe9039f464","name":"taxi_zone_lookup.csv","type":"text/csv","lastModified":1761917574639.568,"size":12331},{"id":"063c94850c24db387c42eb0fbd5134b662112f4e0f97353b7fab3d547bc7238d8ea36d7a4826f64acbd25ad5ecf2bfe68d452c4885890822b7a083757906c4ee","href":"https://static.observableusercontent.com/files/063c94850c24db387c42eb0fbd5134b662112f4e0f97353b7fab3d547bc7238d8ea36d7a4826f64acbd25ad5ecf2bfe68d452c4885890822b7a083757906c4ee","name":"taxi_zones_wgs84.json","type":"application/json","lastModified":1761919360728.063,"size":4450907}] 2f:{"title":"Week 9 Lab: Building interactive spatio-temporal visualizations","cells":[{"id":0,"value":"# Week 9 Lab: Building interactive spatio-temporal visualizations","pinned":false,"hidden":false,"mode":"md"},{"id":36,"value":"---\n\n## Part 1: About Time Handling\n\nWhen handling datasets from real-world environments, you will likely come across timestamps with radically different formats. A crucial skill here is knowing how to handle all the different types of data formats. Let's take some time to really understand how to do this.","pinned":false,"hidden":false,"mode":"md"},{"id":41,"value":"### 1a. The ideal time format: \"ISO-8601\"\n\nConsider that everyone has a different way to encode timestamps. You may see date types from 2020-01-15 to \"January 15, 2020\" to even crazy ones like 2020-01-15T10:30:00. How do you parse this consistently? One of the more common answers to this problem is the ISO-8601\" format.\n\nIn the ISO-8601 format, a date written out like \"September 27, 2022 at 6pm\" is represented as 2022-09-27T18:00:00.000. In fact, JavaScript (and by extension, D3) prefer to encode dates in the ISO-8601 format. However, the methodology to achieve this is different between plain JavaScript and D3.","pinned":false,"hidden":false,"mode":"md"},{"id":61,"value":"### 1b. JavaScript Date() v.s. D3's timeParse()\n\nIn plain JavaScript, you are encouraged to treat timestamps as Date() objects. This object class **expects your timestamp to already be in the ISO-8601 format**. Though this means you will likely have to do some data handling beforehand, the Date() object class gives you some nifty additional functions, such as using .getYear() or .getMonth() to extract the year and month of a timestamp. You can look up the properties and functions of the Date() class [here](https://www.w3schools.com/jsref/jsref_obj_Date.asp).","pinned":false,"hidden":false,"mode":"md"},{"id":256,"value":"Alternatively, D3 really wants you to handle timestamps using d3.timeParse(). Unlike JavaScript's Date() object class, which expects your timestamp to be in a specific format already, D3 lets you **define the expected format of your input string**. This makes D3's implementation a bit more flexible with wonky timestamps. In the end, **the output is the same:** it'll return a Date() object class in the ISO-8601 format.","pinned":false,"hidden":false,"mode":"md"},{"id":70,"value":"Consider the example below, where a sample timestamp \"2020-01-15T10:30:00\" is provided. Note that this timestamp is already in the ISO-8601 format.","pinned":false,"hidden":false,"mode":"md"},{"id":75,"value":"// JavaScript's Date()\n{\n return new Date(\"2020-01-15T10:30:00\");\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":78,"value":"// D3's timeParse()\n{\n let parser = d3.timeParse(\"%Y-%m-%dT%H:%M:%S\")\n return parser(\"2020-01-15T10:30:00\")\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":81,"value":"In these examples, both work because the provided timestamp is already in the ISO-8601 format. However, what if we had a wonky timestamp like \"15-01-2020\"?","pinned":false,"hidden":false,"mode":"md"},{"id":88,"value":"// JavaScript's Date() WON'T WORK: this is an invalid timestamp\n{\n return new Date(\"15-01-2020\");\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":92,"value":"// D3's timeParse() WILL WORK: we can manually define the expected format of the timestamp\n{\n let parser = d3.timeParse(\"%d-%m-%Y\")\n return parser(\"15-01-2020\")\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":99,"value":"When working with D3 and need to work with time-series data, you are **heavily encouraged** to use d3.timeParse() to force timestamps into a Date() object with the ISO-8601 format. You'll know if you've implemented it correctly if d3.timeParse() doesn't return null.","pinned":false,"hidden":false,"mode":"md"},{"id":59,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":113,"value":"#### [TO-DO]\n\nLet's practice with this formatting first. Below are code blocks with a designated _INPUT_TIMESTAMP. Complete each d3.timeParse() code so that the date comes out correctly (i.e. each code block returns an actual timestamp, rather than null).","pinned":false,"hidden":false,"mode":"md"},{"id":115,"value":"// ===== EXAMPLE =====\nex1 = {\n const _INPUT_TIMESTAMP = \"2020/01/15\";\n let parser = d3.timeParse(\"%Y/%m/%d\");\n return parser(_INPUT_TIMESTAMP);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":132,"value":"ex2 = {\n const _INPUT_TIMESTAMP = \"2020-01-15 10:30:00\";\n // TO-DO\n let parser = d3.timeParse();\n return parser(_INPUT_TIMESTAMP);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":135,"value":"ex3 = {\n const _INPUT_TIMESTAMP = \"01-15-2020\";\n // TO-DO\n let parser = d3.timeParse();\n return parser(_INPUT_TIMESTAMP);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":137,"value":"ex4 = {\n const _INPUT_TIMESTAMP = \"20200115\";\n // TO-DO\n let parser = d3.timeParse();\n return parser(_INPUT_TIMESTAMP);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":140,"value":"ex5 = {\n const _INPUT_TIMESTAMP = \"Wed, 15 Jan 2020 10:30:00\";\n // TO-DO\n let parser = d3.timeParse();\n return parser(_INPUT_TIMESTAMP);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":253,"value":"### 1d. Extracting info from Date()\n\nWe've generated ex1 to ex5 above. The function d3.timeParse() helped us convert each of our timestamps into Date() objects. Now, let's say we want to extract the year or month of each of these. The Date() class has some nifty functions you can use to do that, as explained [here](https://www.w3schools.com/jsref/jsref_obj_Date.asp).","pinned":false,"hidden":false,"mode":"md"},{"id":268,"value":"#### [TO-DO]\nFor each of code block below, first _convert the strings into Date() objects using d3.timeParse()_, then _output the day, month, year, etc. of that timestamp (pay attention to each code block's instructions)_. You must use [functions](https://www.w3schools.com/jsref/jsref_obj_Date.asp) provided by the Date() class to achieve this (i.e. don't cheat and use string parsing to extract the date from the input string).","pinned":false,"hidden":false,"mode":"md"},{"id":270,"value":"// ===== EXAMPLE =====\nex6 = {\n // Get the month (i.e. 0-11) of this timestamp.This should return 2. \n const _INPUT_TIMESTAMP = \"2018-03-05\";\n const parser = d3.timeParse(\"%Y-%m-%d\");\n \n const timestamp = parser(_INPUT_TIMESTAMP)\n return timestamp.getMonth();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":275,"value":"ex7 = {\n // Get the full year of this timestamp. This should return 2019.\n \n const _INPUT_TIMESTAMP = \"07/22/2019\";\n // TO-DO\n const parser = d3.timeParse();\n \n const timestamp = parser(_INPUT_TIMESTAMP);\n // TO-DO\n return\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":285,"value":"ex8 = {\n // Get the day of the week (i.e. 0-6) of this timestamp. This should correspond to a Sunday and return 0.\n \n const _INPUT_TIMESTAMP = \"23-Aug-2020\";\n // TO-DO\n const parser = d3.timeParse();\n \n const timestamp = parser(_INPUT_TIMESTAMP);\n // TO-DO\n return\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":291,"value":"ex9 = {\n // Get the hour of this timestamp. This should return 14.\n \n const _INPUT_TIMESTAMP = \"2021-11-10T14:37:05\";\n // TO-DO\n const parser = d3.timeParse();\n \n const timestamp = parser(_INPUT_TIMESTAMP);\n // TO-DO\n return\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":295,"value":"ex10 = {\n // Get the date (i.e. 1-31) of this timestamp. This should return 12.\n \n const _INPUT_TIMESTAMP = \"Fri, 12 Feb 2022 09:15:00\";\n // TO-DO\n const parser = d3.timeParse();\n \n const timestamp = parser(_INPUT_TIMESTAMP);\n // TO-DO\n return\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":225,"value":"---\n\n## Part 2: Implementing Time-Series Charts\n\nLet's try to apply what we've learned in practical application: charting time-series data. The dataset we'll be using is the \"Daily Climate time series data\", which features climate data from Delhi from 2013 to 2017.","pinned":false,"hidden":false,"mode":"md"},{"id":227,"value":"climate_data = FileAttachment(\"delhi-climate.csv\").csv({typed:true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":229,"value":"climate_data","pinned":true,"hidden":false,"mode":"ojs"},{"id":312,"value":"### 2a. Creating new columns from timestamps\n\nLet's say we want to plot this data in a scatter plot. Specifically, we want to plot meantemp across time within a year (i.e. the X-axis just represents Jan 1st to Dec. 31st). Further, let's color each point on the scatter plot by the year that point was recorded.\n\nTo make our lives easier, we need **two** new columns. Firstly, we need a year column that separates the year from the existing timestamps of each row. Secondly, we need a modified day_month column that is a copy of the existing date but with a constant year (e.g. all dates have the year 2020). Why we do this will become obvious later in Part 2b.","pinned":false,"hidden":false,"mode":"md"},{"id":306,"value":"#### [TO-DO]\n\nUsing either Array.map or forEach(), create a new version of the climate data called climate_data_2 that features two new columns: a new year column extracted from the existing date column, and a new day_month column that keeps the day and month of the date but sets the year to some constant (e.g. 2020).\n\n${comment(html`To achieve the date_month column, you will need to create an entirely new Date() value. Pay attention to the Date() constructor and its examples to understand how to do this.`)}","pinned":false,"hidden":false,"mode":"md"},{"id":309,"value":"climate_data_2 = {\n // TO-DO\n return climate_data\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":317,"value":"### 2b. Plotting the scatter plot\n\nWith the year column created, we can proceed in earnest with plotting the scatter data.","pinned":false,"hidden":false,"mode":"md"},{"id":240,"value":"#### [TO-DO]\nProduce a simple scatter plot where:\n\n1. The chart is a 600px wide and 400px tall chart, with margins of 50px on all sides.\n2. Each point represents a single row.\n3. The X-axis represents date.\n4. The Y-axis represents meantemp.\n5. The color represents year.\n\n${comment(html`Bonus points if you can also create a legend to go alongside it, to visualize how color maps to the year. Bonus points too if you can figure out how to make the X-axis ticks labeled to show the month only.`)}","pinned":false,"hidden":false,"mode":"md"},{"id":232,"value":"climate_chart = {\n\n // TO-DO\n \n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":358,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":10,"value":"taxi_data = FileAttachment(\"yellow_tripdata_2024-11_thanksgiving@1.csv\").csv({typed:true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":346,"value":"taxi_data","pinned":false,"hidden":false,"mode":"ojs"},{"id":372,"value":"zone_data = FileAttachment(\"taxi_zones_wgs84.json\").json()","pinned":true,"hidden":false,"mode":"ojs"},{"id":457,"value":"### 3a. Creating counts for each Location ID\n\nOur taxi_data dataset has the weekly taxi pickups and drop-offs for the week of Thanksgiving. Let's say that we want to aggregate, for the entire week, the total number of pickups and drop-offs that occurred per location in our taxi zones.\n\nTo achieve this, we will have to use a special function from D3 called d3.rollup(). It's a special function that lets you create a JavaScript Map that effectively acts as a dictionary. The general syntax is as follows:\n\n```js\nd3.rollup(data,\n v => // the aggregated rows as an array\n d => d.key);\n```\n\nSo if we have a huge dataset and we want to aggregate the data while also creating a dictionary where we want to identify the total number of counts per location, then we can use this kind of logic.","pinned":false,"hidden":false,"mode":"md"},{"id":460,"value":"#### [TO-DO]\n\nCreate two rollups of taxi_data: one that maps the total number of entries per PULocationID, and another that maps the total number of dropoffs per DOLocationID.","pinned":false,"hidden":false,"mode":"md"},{"id":465,"value":"pickup_lookup = {\n // TO-DO\n \n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":468,"value":"dropoff_lookup = {\n // TO-DO\n \n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":470,"value":"### 3b. Creating an dropdown to switch between pickups and dropoffs\n\nLet's create an interaction. In particular, we want to create a dropdown input that lets us select between viewing pickups and dropoffs. This will be pretty cool to visualize in Part 3c, when we visualize the plot itself.","pinned":false,"hidden":false,"mode":"md"},{"id":475,"value":"#### [TO-DO]\n\nUsing the [Select input](https://observablehq.com/@observablehq/input-select), create a 2-item dropdown named `taxi_type` that lets the user select between \"Pickup\" and \"DropOff\".","pinned":false,"hidden":false,"mode":"md"},{"id":477,"value":"// TO-DO\nviewof taxi_type = ","pinned":true,"hidden":false,"mode":"ojs"},{"id":490,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":187,"value":"### 3d. Plotting geodata with zoom and pickup/dropoff\n\nLet's combine the steps from Parts 3a, 3b, and 3c to create an interactive feature, with zooming functionality!\n\nThe order of operations generally can be described as:\n\n1. Use both pickup_lookup or dropoff_lookup to append new properties to each location: pickups and dropoffs.\n1. Create two color scales that maps pickup and dropoff counts to some color scheme\n1. Use the taxi_type input, identify what the color scale D3 should use when mapping the chloropleth map of our geodata.","pinned":false,"hidden":false,"mode":"md"},{"id":192,"value":"taxi_chart = {\n\n // TO-DO\n \n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":46,"value":"---\n\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":48,"value":"import { css, comment, unique } from \"@rk2546/2025-infovis-cse_notebook-gallery\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":204,"value":"unique","pinned":true,"hidden":false,"mode":"ojs"},{"id":52,"value":"css","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W34","stdlib":"1","titling":"auto"}