31:[] 2f:{"title":"OpenAQ Direct File Access from Open Data on AWS","cells":[{"id":0,"value":"# OpenAQ Direct File Access from Open Data on AWS","pinned":false,"hidden":false,"mode":"md"},{"id":572,"value":"

⚠️

This notebook demonstrates the new (as of November 2023) Open Data on AWS bucket file structure. We will keep the legacy file structure in place until the end of November 2023 and encourage migrating your existing code as soon as possible.

","pinned":false,"hidden":false,"mode":"md"},{"id":97,"value":"OpenAQ has changed how we store our data on S3. An improved folder structure will allow for more refined search of data on the OpenAQ platform. Let's take a look how to use this new and improved structure to find data for a single day.","pinned":false,"hidden":false,"mode":"md"},{"id":564,"value":"https://openaq-data-archive.s3.amazonaws.com/","pinned":false,"hidden":false,"mode":"md"},{"id":517,"value":"The S3 folder structure is as follows:\n\n```\n* records\n * csv.gz\n * locationid={OpenAQ location ID}\n * year={4 digit year}\n * month={2 digit month}\n * location-{locationid}-{year}{month}{day}.csv.gz\n```","pinned":false,"hidden":false,"mode":"md"},{"id":513,"value":"Currently data are stored as gzipped csv files. Below the ```records``` folder we will look in the ```csv.gz``` if other file formats are supported in the future a folder will be available at this level.","pinned":false,"hidden":false,"mode":"md"},{"id":373,"value":"For this demo we will use OpenAQ location 2178, an EPA monitor in Albuquerque, New Mexico, USA originally from airnow.gov. The first folder in the file path will be __locationid=2178__","pinned":false,"hidden":false,"mode":"md"},{"id":379,"value":"Next we can filter the data to the year and month we are interested in.","pinned":false,"hidden":false,"mode":"md"},{"id":100,"value":"## Choose a date","pinned":false,"hidden":false,"mode":"md"},{"id":456,"value":"Data are generated at the end of the day of the location, i.e. midnight of the local time.","pinned":false,"hidden":false,"mode":"md"},{"id":62,"value":"viewof date1 = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":122,"value":"For this date: ${date1.getUTCFullYear()}-${monthPad}-${date1.getUTCDate()}, we'll look in the __year=${date1.getUTCFullYear()}__ directory and __month=${monthPad}__ directory","pinned":false,"hidden":false,"mode":"md"},{"id":540,"value":"Finally, the file is named in the format ```location-{location ID}-{year}{month}{day}.csv.gz``` or in this example ```location-2178-${formatDate(date1)}.csv.gz```","pinned":false,"hidden":false,"mode":"md"},{"id":240,"value":"ultimately the URL for this file will be: ${url}","pinned":false,"hidden":false,"mode":"md"},{"id":233,"value":"This file includes the measurements for all parameters supported by the location. In the table below we can view the contents of the csv:","pinned":false,"hidden":false,"mode":"md"},{"id":230,"value":"Inputs.table(data)","pinned":false,"hidden":false,"mode":"ojs"},{"id":329,"value":"We can chart the series to see how the values change throughout the day (in local time). Choose a parameter and Y axis scale type below:","pinned":false,"hidden":false,"mode":"md"},{"id":317,"value":"viewof parameter = Inputs.select(parameters, {\n value: \"log\",\n label: \"Measurand\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":311,"value":"viewof scaleValue = Inputs.select([\"log\", \"linear\"], {\n value: \"log\",\n label: \"Y Axis Scale\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":38,"value":"Plot.plot({\n width: width,\n color: {\n type: \"categorical\",\n scheme: \"tableau10\"\n },\n x: { ticks: 24 },\n y: { type: scaleValue },\n marks: [\n Plot.line(series, {\n x: \"date\", \n y: \"value\",\n stroke: \"location\"\n }),\n Plot.dot(series, {\n r: 2,\n x: \"date\",\n y: \"value\",\n stroke: \"location\",\n fill: \"location\",\n title: d => `${d.value} ${d.units}`\n })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":500,"value":"To learn more about how to query data from the OpenAQ Open Data on AWS bucket read more at [https://docs.openaq.org](https://docs.openaq.org). If you have any questions or want to connect with others join our [Slack community](https://join.slack.com/t/openaq/shared_invite/zt-yzqlgsva-v6McumTjy2BZnegIK9XCVw)","pinned":false,"hidden":false,"mode":"md"},{"id":223,"value":"series = data.filter(o => o.parameter == parameter).map(o => {o.date = new Date(o.datetime); o.value = parseFloat(o.value); return o})","pinned":false,"hidden":false,"mode":"ojs"},{"id":320,"value":"parameters = [...new Set(data.map(x => x.parameter))].filter(e => e != null);","pinned":false,"hidden":false,"mode":"ojs"},{"id":219,"value":"data.map(o => {o.date = new Date(o.datetime); o.value = parseFloat(o.value); return o}).filter(o => o.measurand == \"o3\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":6,"value":"data = {\n const bytes = new Uint8Array(await res.arrayBuffer());\n const text = new TextDecoder().decode(gunzip(bytes));\n return parseCsv(text);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"parseCsv = value => {\n const rows = value.split('\\n').map(o => o.split(','));\n rows.pop();\n const columns = rows.shift();\n let data = [];\n for (const row of rows) {\n let record = {}\n for (let i = 0; i < row.length; i++) {\n record[columns[i]] = row[i]\n } \n data.push(record);\n }\n return data;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":3,"value":"res = await fetch(url);","pinned":false,"hidden":false,"mode":"ojs"},{"id":464,"value":"url = `https://openaq-data-archive.s3.amazonaws.com/records/csv.gz/locationid%3D2178/year%3D${date1.getUTCFullYear() }/month%3D${monthPad}/location-2178-${formatDate(date1)}.csv.gz`","pinned":false,"hidden":false,"mode":"ojs"},{"id":544,"value":"monthPad = month.toString().padStart(2,\"0\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":477,"value":"month = date1.getUTCMonth() + 1","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"import {gunzip} from \"@mourner/wasm-gunzip\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":64,"value":"max = new Date()","pinned":false,"hidden":false,"mode":"ojs"},{"id":67,"value":"function formatDate(date) {\n return [\n (date.getFullYear()).toString().padStart(4, \"0\"),\n (date.getMonth() + 1).toString().padStart(2, \"0\"),\n (date.getDate() + 1).toString().padStart(2, \"0\")\n ].join(\"\")\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":559,"value":"html``","pinned":true,"hidden":false,"mode":"ojs"},{"id":485,"value":"import {Plot} from \"@mkfreeman/plot-tooltip\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}