2e:null 2f:["$","title",null,{"children":"Working with data using Vega and D3 | Caglar Koylu | Observable"}] 34:Tb70,md `## Working with geodata We need to load our geodata in file formats, data formats and coordinate systems that d3 suppports. The file formats natively supported by D3 are [GeoJSON](http://geojson.org/) and TopoJson along with vanilla csv.I'll discuss each and use examples from a GeoJSON file containing all the disbaility parking spots in Vancouver (my hometown) I downloaded from the Vancouver Open Data catalogue.
But geodata are more complicated than just the file format!

Geodata are coded using

a geodetic coordinate system format called a "datum" as a spatial reference. A geodetic datum uniquely defines all locations on Earth with coordinates;
and a projection.

Datum: Let's talk about the "datum" format first (DO NOT mix this up with the convention of referring to the current object in Vega or d3 as a \`\`\`datum\`\`\`: the d3 \`\`\`datum\`\`\` is quite different.) These all have a particular specification ID that you can find at http://spatialreference.org/. The standard longitude and latitude references we use in d3, as far as I can tell, are WGS84 format - EPSG:4326. HOwever, many geodata come in different coordinate systems: Census Canada, for example, uses Lambert Conformal Conic. The City of Vancouver uses EPSG:26910. The geodetic coordinate system used in the data you have should be listed in an accompanying file (in the .prj file if the original data format is a shapefile directory). So you have to need to find the coordinate system (there may or may not be an EPSG number), and then convert the coordinates from the original form to the standard EPSG:4326(lat,long). There are a number of ways to do this.
You can download the free qGIS tool (if you want to dive into the full spectrum of GIS and geodata) and export the layers to the apporpriate datum in the LAyers menu. You can also use the arcGIS tools if you have access to them (it's an expensive license, but most universities have full access). Or if you only have a couple of coordinates you can use an online pair conversion tool like https://mygeodata.cloud/cs2cs/.

Projection: d3 (and by extension the Vega and Vega-Lite grammars) supports many projections (take a look at them in this workbook: https://observablehq.com/@vega/vega-lite-cartographic-projections). We can typically use the \`\`\`Mercator\`\`\` projection for most cases. But you will probably need to play with the right scaling factor and centering to position your map as you want it. Here is another VERY POORLY documented fact: a d3 geoProjection takes coordinates in an [x,y] or [long,lat] format. In other words, longitude and latitude are switched from the normal way they will be expressed in many datasets - even after you have done the datum conversion. Typically the most useful workaround is simply to rename your data fields. Urk.` 35:Tb06,md `## File Formats Once you have your coordinate system set to [lat,long] you need to get it into the right file format. d3 recognizes two geo file formats: \`\`\`geojson\`\`\` and \`\`\`topojson\`\`\`. Each of these file formats is organized slightly differently and you need to examine your data to find the key level of the hierarchy to tell d3 what features to extract and process: often geofiles will have several feature categories, and you need to choose which to render.

The \`dparking\` dataset in this workbook is in geojson format. A GeoJSON file is a text file with a list of features and, for each one, a bunch of latitudes and longitudes describing its geometry. The dataset we use here contains a single \`FeatureCollection\` object. The individual data objects are in the \`features\` array and the \`features.geometry\` property contains the [long and lat] array of individual parking spots - explore the object hierarchy to get a feel for the structure of the data and the information.

The \`VanAreaValues2\` dataset is in topojson format. A TopoJSON file is similar, but instead of storing the latitudes and longitudes directly, it describes the topology (geometrical properties and spatial relations of figures) of a group of features, meaning the borders between them.So its structure is a little more complicated: its key categories are in the \`objects\` array. In the usa dataset below in this file, there are three objects:
\` objects: Object {counties: Object, states: Object, land: Object}\`.
In our Vancouver case, there is one object in the array - \`VancouverAreaSize\`). TopoJSON files are usually much smaller, but require a little bit of extra processing in the browser when you want to display one on a map, and can potentially lose details from the original latitudes/longitude.

d3 knows how to process these file types. The first visualization below shows vega maps geoshapes from a geojson file. OF course, you can always bring your data in in csv form as well, but rendering it will take a little more work. The second map shows data rendered from a csv file, where the longitude have to be explcitly identified as fields.

All GIS utilities support exporting geoJSON and topojson. (In either qGIS or arcGIS you typically do this using the \`Layers\` menu.) There are also online tools: I used [mapshaper](http://mapshaper.org). Mapshaper lets you load and inspect your geographic data and export it to geoJSON or topoJSON formats. It does NOT however let you convert your coordinate system. Once you have the data, you can set it up as a gist or host it somewhere and load the file using the regular \`d3.json()\` function. Warning: geodata files can be very big and gist will typically fail on many. Consider hosting on a more robust webserver.`36:T7bf,embed({ width: 900, height: 600, padding: { top: 0, left: 0, right: 0, bottom: 0 }, data: [ { name: "parkingVan", values: dparking1, //"format": {"type": "csv", "parse": "auto"}, "transform": [ { "type": "geopoint", "projection": "projection", "fields": ["longitude", "latitude"] }, { "type": "filter", "expr": "datum.x != null && datum.y != null" } ] }], scales: [], "signals": [ { "name": "hover", "value": null, "on": [ {"events": "symbol:mouseover", "update": "datum"}, {"events": "symbol:mouseout", "update": "null"} ] }, { "name": "dparkspots", "value": "Vancouver Disabled PArking Spaces", "update": "hover ? hover.NOTES + ' (' + hover.DESCRIPTN + ')' + ': ' + hover.LOCATION: 'Vancouver Disabled Parking'" } ], projections: [ { name: "projection", type: "mercator", scale: 200000, center: d3.geoCentroid(dparking) //"translate": [{"signal": "width / 2"}, {"signal": "height / 2"}] } ], "marks": [ { "type": "symbol", "from": {"data": "parkingVan"}, "encode": { "enter": { "size": {"value": 60}, "fill": {"value": "green"}, "fillOpacity": {"value": 0.8}, "stroke": {"value": "black"}, "strokeWidth": {"value": .5} }, "update": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "exit": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "hover": { "fill": {"value": "red"} } } }, { "type": "text", "interactive": false, "encode": { "enter": { "x": {"signal": "width", "offset": -5}, "y": {"value": 0}, "fill": {"value": "black"}, "fontSize": {"value": 10}, "align": {"value": "right"} }, "update": { "text": {"signal": "dparkspots"} } } } ] })37:Ta00,embed({ width: 900, height: 560, padding: { top: 0, left: 0, right: 0, bottom: 0 }, data: [ { name: "streetsVan", url: "https://gist.githubusercontent.com/drlynb/234c622e2f98da7c61393cbb33573bf8/raw/8e8fa5ae3dc76d94dcca3a50e3dce41b89eb7185/vanstreets.json", format: { type: "json", feature: "vanstreets" }, transform: [ { type: "geopath", projection: "projection" } ] }, { name: "parkingVan", values: dparking1, //"format": {"type": "csv", "parse": "auto"}, "transform": [ { "type": "geopoint", "projection": "projection", "fields": ["longitude", "latitude"] }, { "type": "filter", "expr": "datum.x != null && datum.y != null" } ] }], scales: [], "signals": [ { "name": "hover", "value": null, "on": [ {"events": "symbol:mouseover", "update": "datum"}, {"events": "symbol:mouseout", "update": "null"} ] }, { "name": "dparkspots", "value": "Vancouver Disabled PArking Spaces", "update": "hover ? hover.NOTES + ' (' + hover.DESCRIPTN + ')' + ': ' + hover.LOCATION: 'Vancouver Disabled Parking'" } ], projections: [ { name: "projection", type: "mercator", scale: 200000, center: d3.geoCentroid(Streets) //"translate": [{"signal": "width / 2"}, {"signal": "height / 2"}] } ], marks: [ { type: "path", from: { data: "streetsVan" }, encode: { enter: { stroke: { value: "#bbbbbb" } }, update: { path: { field: "path" } } } }, { "type": "symbol", "from": {"data": "parkingVan"}, "encode": { "enter": { "size": {"value": 60}, "fill": {"value": "green"}, "fillOpacity": {"value": 0.8}, "stroke": {"value": "black"}, "strokeWidth": {"value": .5} }, "update": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "exit": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "hover": { "fill": {"value": "red"} }}}, { "type": "text", "interactive": false, "encode": { "enter": { "x": {"signal": "width", "offset": -5}, "y": {"value": 0}, "fill": {"value": "black"}, "fontSize": {"value": 12}, "align": {"value": "right"} }, "update": { "text": {"signal": "dparkspots"} } } } ] })38:T976,parking1 = embed({ $schema: 'https://vega.github.io/schema/vega/v5.json', data:[ { name: "streetsVan1", url: "https://gist.githubusercontent.com/drlynb/234c622e2f98da7c61393cbb33573bf8/raw/8e8fa5ae3dc76d94dcca3a50e3dce41b89eb7185/vanstreets.json", format: { type: "json", feature: "vanstreets" }, transform: [ { type: "geopath", projection: "projection" } ] }, { name: "disabilities1", url: "https://gist.githubusercontent.com/StanisNowak/7432a7e657f857e1e98d2f0f10015db7/raw/8a211fcffbea75f7caa8583736048e096679727b/disabilityParking.geojson", format: { type: "json", property: "features" } }], "signals": [ { "name": "hover", "value": null, "on": [ {"events": "symbol:mouseover", "update": "datum.properties"}, {"events": "symbol:mouseout", "update": "null"} ] }, { "name": "dparkspots", "value": "Vancouver Disabled Parking Spaces", "update": "hover ? hover.properties.NOTES + ' (' + hover + ')' + ': ' + hover: 'Vancouver Disabled Parking Spaces'" } ], projections: [{ name: "projection", fit: {signal: "data('disabilities1')"}, size: {signal: "[800, 600]"}, //scale: 190000, //center: [-123.149648,49.265815], type: "mercator" }], marks: [ { type: "shape", from: {data: "disabilities1"}, encode: { enter: { opacity: {value: 0.50}, fill: {value: "blue"}, "tooltip": {"signal": "datum.properties.NOTES" } }, update: { opacity: {value: 0.50}, fill: {value: "blue"} }, "fill": {"value": "red"} }, transform: [ { type: "geoshape", projection: "projection", pointRadius: 6 } ] }, { type: "path", from: { data: "streetsVan1" }, encode: { enter: { stroke: { value: "#aaaaaa" } }, update: { path: { field: "path" } } } }, { "type": "text", "interactive": false, "encode": { "enter": { "x": {"signal": "width", "offset": -5}, "y": {"value": 0}, "fill": {"value": "black"}, "fontSize": {"value": 14}, "align": {"value": "right"} }, "update": { "text": {"signal": "dparkspots"} } } } ] })39:[] 32:{"id":"2efe1b1d428b35ee","slug":"geodata-wrangling-for-vega-and-d3-data-geog-3540-assignment","title":"Working with data using Vega and D3","thumbnail":"0721f3c047b4468c39f21467d61658ae7df1fca963ee8410e617a22e46d76166","access_level":"public","create_time":"$D2019-11-20T21:02:06.732Z","update_time":"$D2020-02-12T14:12:03.314Z","view_time":null,"first_public_time":"$D2020-02-12T14:10:13.520Z","has_unpublished_edits":null,"fork":{"id":"0699f7223f6acdeb","slug":"lyn-tries-maps-again","title":"Geodata wrangling for Vega-lite and d3","owner":{"id":"4a0762a387ec748f","login":"drlynb","name":""}},"stars":1,"starred":false,"owner":{"type":"team","id":"1522f7962f5f1185","login":"caglarkoylu","name":"Caglar Koylu","avatar_url":"https://avatars-next.observableusercontent.com/94b0c40bf8a6b04057088ea171864451b3e3b27b7346d866d2a7be21d077be76","tier":"starter_2024"},"version":405,"publish_version":405,"role":null,"authored":false,"authors":[{"type":"individual","id":"60e9af384dfce94c","login":"caglarkoylu","name":"Caglar Koylu","avatar_url":"https://avatars-next.observableusercontent.com/94b0c40bf8a6b04057088ea171864451b3e3b27b7346d866d2a7be21d077be76","tier":null}],"members":[{"type":"individual","id":"60e9af384dfce94c","login":"caglarkoylu","name":"Caglar Koylu","avatar_url":"https://avatars-next.observableusercontent.com/94b0c40bf8a6b04057088ea171864451b3e3b27b7346d866d2a7be21d077be76","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":8,"collection":null,"type":"notebook","body":{"title":"Working with data using Vega and D3","cells":[{"id":0,"value":"md `# Working with data using Vega and D3`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":19,"value":"md `I noticed that while I can get and process the \\\"states\\\" json file that is packaged as one of the vega-lite datasets (and subsequently use the \"values\" property as the processed array), I CAN'T do the same thing with regular files that i bring in from other sources. How do we process data so that d3 and Vega can understand it? To do that, here is a little introduction to some geocoding getarounds.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":323,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":361,"value":"$35","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":68,"value":"md `And now we try this with the Vancouver disabled parking spot data.\nThese spots plot perfectly in qGIS and mapshaper.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":145,"value":"md `### Vancouver Parking Spots mapped the easy way as geoshapes.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":147,"value":"viewof parking = embed({\n $schema: 'https://vega.github.io/schema/vega/v5.json',\n data:[{\n name: \"disabilities\",\n url: \"https://gist.githubusercontent.com/StanisNowak/7432a7e657f857e1e98d2f0f10015db7/raw/8a211fcffbea75f7caa8583736048e096679727b/disabilityParking.geojson\",\n format: {\n type: \"json\",\n property: \"features\"\n }\n }],\n projections: [{\n name: \"projection\",\n fit: {signal: \"data('disabilities')\"},\n size: {signal: \"[800, 600]\"},\n //scale: 190000,\n //center: [-123.149648,49.265815],\n type: \"mercator\"\n }],\n marks: [\n {\n type: \"shape\",\n from: {data: \"disabilities\"},\n encode: {\n update: {\n opacity: {value: 0.25},\n fill: {value: \"blue\"}\n }\n },\n transform: [\n {\n type: \"geoshape\",\n projection: \"projection\",\n pointRadius: 6\n }\n ]\n }\n ]\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":113,"value":"md `### Vancouver Parking Spots Mapped the gnarlier way with csv and geopoint`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":227,"value":"md `Here because i have the properties associated with each point at the same level in the data structure I can easily access them using the \"hover\" event, so i can use the text mark to show them. Mouse over a point and see. There's an odd aspect that causes some of the points to disappear after they have been hovered, so I needed to add in the exit{} event to make sure that the points re-rendered.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":77,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":116,"value":"md `And now finally let's bring in the streets map and see if I can plot the spaces on top of the streets. I added the hover code to see the data associated with each point.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":118,"value":"$37","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":163,"value":"md `Finally I am going to do this using the geoshapes - they look much nicer! and they are not so gnarly. Some things to notice here with these data:\n

I have to scale the map to the view. There seem to be a couple of ways to do this. I can use d3.geoCentroid(Streets) but then I need to set d3 parameters for the projection:\n\n d3 \n .geoMercator() \n .fitSize([100, 100], Streets) \n .scale()
\nor I can use a signal that takes the data and fits to it:
\n\nprojections: [{ \n name: \"projection\", \n fit: {signal: \"data('disabilities1')\"}, \n size: {signal: \"[800, 600]\"}, \n\n
I have added a tooltip that shows the NOTES field of the point data on mouseover. But for the life of me I can't see how to access additional properties in the same depth in the json file such as LOCATION (I want to concatenate the string). Next step.

`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":175,"value":"$38","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":129,"value":"md `## Data Modules and Libraries`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":133,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":136,"value":"embed = require('vega-embed@3')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":190,"value":"dp2 = d3.json(\"https://gist.githubusercontent.com/StanisNowak/7432a7e657f857e1e98d2f0f10015db7/raw/8a211fcffbea75f7caa8583736048e096679727b/disabilityParking.geojson\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":205,"value":"us10 = d3.json(\n \"https://cors-anywhere.herokuapp.com/https://vega.github.io/vega/data/us-10m.json\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":208,"value":"airports= d3.csv(\n \"https://cors-anywhere.herokuapp.com/https://vega.github.io/vega/data/airports.csv\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":223,"value":"dparking = d3.json(\"https://gist.githubusercontent.com/drlynb/6dffc7ccdaf774ee031dc46c909721b2/raw/8a8b8fb6b6b60e66980cbb9f897b198bac23dbf7/testparking.geojson\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":219,"value":"dparking1 = d3.csv(\"https://gist.githubusercontent.com/drlynb/2b4ac0221b8138ade06cd91886b95f69/raw/a45e5883da217c0c1b3237df307136b5ee86a7e9/disparking2.csv\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":369,"value":"vanAreaValues2 = d3.json(\"https://www.sfu.ca/~lyn/data/Urban/VancouverAreaSizeTopo.json\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":391,"value":"usa = d3.json('https://vega.github.io/vega-datasets/data/us-10m.json')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":229,"value":"Streets = d3.json(\"https://gist.githubusercontent.com/drlynb/234c622e2f98da7c61393cbb33573bf8/raw/8e8fa5ae3dc76d94dcca3a50e3dce41b89eb7185/vanstreets.json\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":265,"value":"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W39","stdlib":"1","titling":"auto"}} 33:"$32:body"