2b:null 2c:["$","title",null,{"children":"Cozy Collecting, Case Study: Getting a Decade's Worth of Weather Data in R | Periscopic | Observable"}] 31:T4d7,To cap off this 3-part series, here's a summary of this approach, going from an API to outputting and parsing JSON in R: - **Start Small**: Consult the documentation of your API to see how to access your desired endpoint and any query parameters necessary. Use `httr::GET` to grab a response, making sure it returns a valid response. If necessary, use `httr::GET` to [pass other things for authentication](https://httr.r-lib.org/articles/api-packages.html#authentication-1). Get it working with one call before expanding. - **Parameter Dataframe, One Row Per Call**: Make a dataframe of each call you'd like to make. Use `purrr::pmap` to iterate through options, getting a collection of [httr responses](https://httr.r-lib.org/reference/response.html). - **Check Results and Create Nested List**: Check status codes and [keep or discard](https://purrr.tidyverse.org/reference/keep.html) responses as needed. Take results and convert responses to a nested list of [content](https://httr.r-lib.org/reference/content.html). - **Export or Parse with Rectangling**: With the nested list, can either export the results as a flat file or use [rectangling methods](https://tidyr.tidyverse.org/articles/rectangle.html) to parse as a nested tibble.32:[{"id":"5a27edee020953ecc89c67a631bd1e4dac84d8edfb8f2039990a4c90db5d3bfbed47dcec28c1039f87fad0b5cee17f5eee3b021d47f7b1b6cf14bd299e661761","href":"https://static.observableusercontent.com/files/5a27edee020953ecc89c67a631bd1e4dac84d8edfb8f2039990a4c90db5d3bfbed47dcec28c1039f87fad0b5cee17f5eee3b021d47f7b1b6cf14bd299e661761","name":"p3-param-station-calls.png","type":"image/png","lastModified":1644009431046.867,"size":275865},{"id":"f538af365384072df350cb19b9d863a7bce0d3cbcb510c2ee8413241cb0fca8b5680df6ab3a1ce16722e09e65a3f5579863199e93aa1b536ec2f5e86ed3c37c2","href":"https://static.observableusercontent.com/files/f538af365384072df350cb19b9d863a7bce0d3cbcb510c2ee8413241cb0fca8b5680df6ab3a1ce16722e09e65a3f5579863199e93aa1b536ec2f5e86ed3c37c2","name":"p3-stations.png","type":"image/png","lastModified":1644009430097.304,"size":137198},{"id":"56a7d6eea05bd1bb2f330dee02bce41ae5e80849fc6c533ab293b03612742652fe9775930ea554546bb53e2324fb5353197385335c5b68627f7ad4032f044abf","href":"https://static.observableusercontent.com/files/56a7d6eea05bd1bb2f330dee02bce41ae5e80849fc6c533ab293b03612742652fe9775930ea554546bb53e2324fb5353197385335c5b68627f7ad4032f044abf","name":"p3-station-counts.png","type":"image/png","lastModified":1644009430864.261,"size":258662},{"id":"a13f303ca3e81f340329672dd985ae6ae1098233d503e9f741dff681ffebc9fb599ba1b90360cd1ad1a36a66a21061fe02313c613531934522907b91050d01a7","href":"https://static.observableusercontent.com/files/a13f303ca3e81f340329672dd985ae6ae1098233d503e9f741dff681ffebc9fb599ba1b90360cd1ad1a36a66a21061fe02313c613531934522907b91050d01a7","name":"p3-tbl-weather-data.png","type":"image/png","lastModified":1644009431057.324,"size":305388},{"id":"e0376e3012bf3892cfd9284b0ba7f3a4c3cbf45e8b49b0c4b3be53a6abc643db8488b6c13f8e39f3b3c82ac370ad85cee52f9e436ffcdd7e6ebefacdf7c987c8","href":"https://static.observableusercontent.com/files/e0376e3012bf3892cfd9284b0ba7f3a4c3cbf45e8b49b0c4b3be53a6abc643db8488b6c13f8e39f3b3c82ac370ad85cee52f9e436ffcdd7e6ebefacdf7c987c8","name":"justin-docanto-ZpxpMEbFGY4-unsplash.jpeg","type":"image/jpeg","lastModified":1644009437844.422,"size":195837},{"id":"a96a4e8ddf4950e6c9f8c1818b3c930d52b869b86992e731e289f5a1f130c75fe322235e7eb501aba0f752ce622273608c81a149961414f3dce76e54a4f2342f","href":"https://static.observableusercontent.com/files/a96a4e8ddf4950e6c9f8c1818b3c930d52b869b86992e731e289f5a1f130c75fe322235e7eb501aba0f752ce622273608c81a149961414f3dce76e54a4f2342f","name":"p3-selected-zipcodes.png","type":"image/png","lastModified":1644009429503.637,"size":72916},{"id":"1e8b29d116dced17c918178aa92acbaea656da224ad6e9b55e9742c106409afeb68b280a926097647996d8ed7bd86c41dda1ff35af64361a4b929bbec1c6d384","href":"https://static.observableusercontent.com/files/1e8b29d116dced17c918178aa92acbaea656da224ad6e9b55e9742c106409afeb68b280a926097647996d8ed7bd86c41dda1ff35af64361a4b929bbec1c6d384","name":"p3-timeseries-plot.png","type":"image/png","lastModified":1644009430967.116,"size":320046},{"id":"8482748f591e8355c3d429caea63985c02e26015ca98590182d530e062d058f4fec178dac1b44551f2c679aad407b6cb686fd00521ea34b830ed5ff4fa24532c","href":"https://static.observableusercontent.com/files/8482748f591e8355c3d429caea63985c02e26015ca98590182d530e062d058f4fec178dac1b44551f2c679aad407b6cb686fd00521ea34b830ed5ff4fa24532c","name":"p3-temperatures.png","type":"image/png","lastModified":1644009430530.597,"size":293274},{"id":"a790db14f0acb3d0bf9107c2930dba751f06f2454acaa9e2bc3a0e0f5387f1ba053765c0a6f75a161123d88f4fa6e52af49f8b0a593a8e2210fee286ec7ec7ee","href":"https://static.observableusercontent.com/files/a790db14f0acb3d0bf9107c2930dba751f06f2454acaa9e2bc3a0e0f5387f1ba053765c0a6f75a161123d88f4fa6e52af49f8b0a593a8e2210fee286ec7ec7ee","name":"p3-param-station-counts.png","type":"image/png","lastModified":1644009429623.373,"size":247179}] 2f:{"id":"a4713ee96f5946b0","slug":"cozy-collecting-case-study","title":"Cozy Collecting, Case Study: Getting a Decade's Worth of Weather Data in R","thumbnail":"d1ae86c8b20c99ef33ec1cb6a64837b0b7c5a6139d0b7c00faba8ddf1d769fc7","access_level":"public","create_time":"$D2022-02-04T21:08:52.931Z","update_time":"$D2022-02-04T21:44:43.777Z","view_time":null,"first_public_time":"$D2022-02-15T17:05:57.062Z","fork":false,"stars":2,"starred":false,"owner":{"type":"team","id":"3a75712dbd8df3bb","login":"periscopic","name":"Periscopic","avatar_url":"https://avatars-next.observableusercontent.com/2f6e98f71dc2af94cfb5e4c3b77652cd448d7c9420fc850bfa00e052a63ec480","tier":"starter_2024"},"version":214,"publish_version":214,"role":null,"authored":false,"authors":[{"type":"individual","id":"284cdf205719c86e","login":"zachperiscopic","name":"Zach Bogart","avatar_url":"https://avatars-next.observableusercontent.com/6f18b0401ce7dbd258bca1afb27a7a9d850d10197a42ad23e35e242ca095a4e9","tier":null}],"members":[{"type":"individual","id":"284cdf205719c86e","login":"zachperiscopic","name":"Zach Bogart","avatar_url":"https://avatars-next.observableusercontent.com/6f18b0401ce7dbd258bca1afb27a7a9d850d10197a42ad23e35e242ca095a4e9","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"Cozy Collecting, Case Study: Getting a Decade's Worth of Weather Data in R","cells":[{"id":0,"value":"# Cozy Collecting, Case Study: Getting a Decade's Worth of Weather Data in R","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":137,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":5,"value":"This post is the third and final part of a series explaining our methods for extracting data from APIs and unpacking nested JSONs in R.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":10,"value":"This walkthrough is split into three parts:\n\n1. 👋 **[Pinging an API, Creating JSON](https://observablehq.com/@zachperiscopic/cozy-collecting-part-1)**: Get data from an API and extract as JSON\n2. 🔨 **[Working with JSON](https://observablehq.com/@zachperiscopic/cozy-collecting-part-2)**: Use extracted JSON data in R\n3. ➡️ 🚀 **Case Study: Getting a Decade's Worth of Weather Data**: Example of method with a larger data request","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":14,"value":"# 🚀 Case Study: Getting a Decade's Worth of Weather Data","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":142,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":17,"value":"In [Part 1](https://observablehq.com/@zachperiscopic/cozy-collecting-part-1) and [Part 2](https://observablehq.com/@zachperiscopic/cozy-collecting-part-2), we just got a little bit of weather JSON from a few handshakes. Here, we'll move beyond the smaller examples and collect years worth of historical weather data from [NOAA Climate Data Online](https://www.ncdc.noaa.gov/cdo-web/webservices/v2#gettingStarted). \n\nKeep some Purell handy, it's time to do some serious handshaking. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":21,"value":"*Note: To clarify where code is coming from, we will often include the longer format of package function names (`httr::GET()` versus `GET()`).*","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":25,"value":"```\n# imports\nlibrary(httr) # accessing APIs\nlibrary(glue) # pass variables into strings\nlibrary(tibble) # nicer dataframes\nlibrary(purrr) # apply functions to lists\nlibrary(dplyr) # data wrangling\nlibrary(tidyr) # creating tidy data\nlibrary(jsonlite) # JSON parsing\n\nlibrary(ggplot2) # plotting\nlibrary(lubridate) # date handling\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":30,"value":"## 🌤️ Getting Historical Weather Data","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":34,"value":"To let the API know who we are, we can [request an API token from NOAA](https://www.ncdc.noaa.gov/cdo-web/token), and can store it with `Sys.setenv()`:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":38,"value":"```\nSys.setenv(\"CDO_API_KEY\" = \"\")\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":43,"value":"### 📌 Define the Locations","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":146,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":46,"value":"First we define the locations we're interested in. Looking at the [docs](https://www.ncdc.noaa.gov/cdo-web/webservices/v2#locations), the API's handshake requires a specific location id format, which we can build up with dplyr:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":51,"value":"```\n# grabbing zip codes\nselected_zipcodes <- tribble(\n ~place, ~zip_code,\n \"San Francisco\", 94128,\n \"New York\", 11430,\n \"Phoenix\", 85034\n ) %>% \n mutate(locationid = paste(\"ZIP\", zip_code, sep = \":\"))\n\nselected_zipcodes\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":151,"value":"

Dataframe of parameters for selected locations

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":54,"value":"We then ping the API to get all the weather stations in those zip codes that provide daily summary (\"GHCND\") data. Like [the OpenWeatherAPI from before](https://observablehq.com/@zachperiscopic/cozy-collecting-part-1), this API wants to know who it's shaking hands with. In this case, we authenticate using a token header, so we [include a header field to our GET call](https://httr.r-lib.org/reference/add_headers.html).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":58,"value":"```\n# iterate through zip codes,\nres_stations <- purrr::pmap(selected_zipcodes, ~with(list(...), {\n return(\n GET(\n \"https://www.ncdc.noaa.gov/cdo-web/api/v2/stations\",\n query = list(\n datasetid = \"GHCND\",\n locationid = locationid\n ),\n add_headers(token = Sys.getenv('CDO_API_KEY'))\n )\n )\n \n}))\n\n# convert to nested list\nlist_stations <- res_stations %>% map(~content(.))\n\n# rectangling methods on nested tibble\nstations <- tibble(data = list_stations) %>% \n unnest_wider(data) %>% \n unnest_longer(results) %>% \n unnest_wider(results)\n\nstations\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":158,"value":"

Resulting dataframe of weather station metadata

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":62,"value":"We now have the weather stations in our desired locations. Let's go get a lot of daily weather summaries.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":66,"value":"### ⏱️ Getting Data for the Past Decade","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":163,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":70,"value":"When we make a call, we get back metadata about all of the available items given the query parameters. Since this API limits calls to one-year increments and batches of one thousand items, we will make a call to figure out the extent of each year, then construct the necessary calls to get the data for every year.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":74,"value":"```\n# define date inputs for previous years\nnewest_year <- 2021\nyears <- 10\n\npast_years <- tibble(\n year = seq(newest_year, newest_year - years + 1, -1)\n ) %>% \n mutate(\n startdate = paste0(year, \"-01-01\"),\n enddate = paste0(year, \"-12-31\")\n )\n\n# parameters for extracting metadata\nparam_station_counts <- stations %>% \n select(station_name = name, datacoverage, stationid = id) %>% \n crossing(past_years) %>% \n mutate(\n limit = 1,\n datasetid = \"GHCND\"\n )\n\nparam_station_counts\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":169,"value":"

Parameter dataframe for initial API calls

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":78,"value":"We then iterate through to get the metadata for each year/station combination:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":82,"value":"```\n# go get metadata for each year\nres_station_counts <- purrr::pmap(param_station_counts, ~with(list(...), {\n return(\n GET(\n \"https://www.ncdc.noaa.gov/cdo-web/api/v2/data\",\n query = list(\n datasetid = datasetid,\n stationid = stationid,\n limit = limit,\n startdate = startdate,\n enddate = enddate\n ),\n add_headers(token = Sys.getenv('CDO_API_KEY'))\n )\n )\n}))\n\n# convert to nested list\nlist_station_counts <- res_station_counts %>% map(~content(.))\n\n# parse metadata as a tibble\nstation_counts <- tibble(data = list_station_counts) %>% \n hoist(data, \"metadata\") %>% \n unnest_wider(metadata) %>% \n unnest_wider(resultset) %>% \n bind_cols(param_station_counts) %>% \n select(api_count = count, station_name:enddate, datasetid)\n\nstation_counts\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":176,"value":"

Parameter dataframe, now with item counts as api_count column

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":87,"value":"We now have an item count for each call (`api_count` column). This will allow us to split up each call into thousand-item chunks for requesting data from the API. We can do some dplyr grouping to get an expanded tibble that is an itemized list of all the calls we'd like to make.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":90,"value":"```\n# use station counts to get data within range\nitems_per_call <- 1000\nparam_station_calls <- station_counts %>% \n # define offset and count for each station\n mutate(calls = ceiling(api_count / items_per_call)) %>% \n uncount(calls) %>% \n group_by(stationid, year) %>% \n mutate(offset = (row_number() - 1) * items_per_call + 1,\n limit = items_per_call) %>% \n ungroup() %>% \n select(-c(station_name, datacoverage))\n\nparam_station_calls\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":181,"value":"

Parameter dataframe, all API weather calls

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":94,"value":"Finally we make all those handshakes! We iterate through all the calls, getting back a big nested tibble of weather data:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":98,"value":"```\n# get weather data\nres_cdo_weather_data <- purrr::pmap(param_station_calls, ~with(list(...), {\n Sys.sleep(1.1)\n \n return(\n GET(\n \"https://www.ncdc.noaa.gov/cdo-web/api/v2/data\",\n query = list(\n datasetid = datasetid,\n stationid = stationid,\n startdate = startdate,\n enddate = enddate,\n offset = offset,\n limit = limit,\n units = \"metric\"\n ),\n add_headers(token = Sys.getenv('CDO_API_KEY'))\n )\n )\n}))\n\n# convert to nested list\nlist_cdo_weather_data <- res_cdo_weather_data %>% keep(~ status_code(.) == 200) %>% map(~content(.)) \n\n# parse metadata as a tibble\ntbl_weather_data <- tibble(api_call = list_cdo_weather_data) %>% \n unnest_wider(api_call) %>% \n unnest_longer(results) %>% \n unnest_wider(results)\n\ntbl_weather_data\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":189,"value":"

Final nested tibble of weather data (100K+ rows)

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":194,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":102,"value":"We have successfully fetched a decade's worth of data for our three locations (100K+ rows), parsing the results entirely in R! Now the visualization fun can begin. For example, we could look at the historical extreme temperatures in our chosen areas. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":106,"value":"```\n# temperature extremes\ntemperatures <- tbl_weather_data %>% \n select(-metadata) %>% \n filter(datatype %in% c(\"TMIN\", \"TMAX\")) %>% \n rename(temp_c = value) %>%\n mutate(\n date = lubridate::date(date),\n location_name = case_when(\n station == \"GHCND:USW00023234\" ~ \"San Francisco, CA\",\n station == \"GHCND:USW00094789\" ~ \"New York, NY\",\n station == \"GHCND:USW00023183\" ~ \"Phoenix, AZ\"\n )\n )\n\ntemperatures\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":201,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":111,"value":"Looking at the timeseries, we can notice the very hot summers in Phoenix, the far more constant environment around San Francisco, and the pronounced seasonality in New York.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":114,"value":"```\ntheme_set(theme_light())\n\n# decade-long faceted timeseries\ntemperatures %>% \n mutate(\n year = lubridate::year(date),\n day_of_year = lubridate::yday(date)\n ) %>% \n ggplot(aes(day_of_year, temp_c, color = datatype)) +\n geom_line() +\n facet_grid(year~location_name) +\n guides(color = \"none\") +\n labs(\n title = \"A Decade of Ups and Downs\",\n subtitle = \"Historical Extreme Temperatures\",\n x = \"Day of the Year\",\n y = \"Temperature (°C)\"\n )\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":206,"value":"

Historical timeseries of extreme temperatures

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":118,"value":"## 🤖 Wrap Up","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":122,"value":"Parsing JSON from APIs in R can be challenging and we hope this approach helps to alleviate some of those challenges. Using dataframes to define batches of API calls helps organize your data requests, and using rectangling methods on a nested tibble helps organize the outputs, regardless of the response formats you get back. The goal is for APIs not to be a limitation or frustration, but a source of inspiration. The data is out there and we hope this helps get you working with it.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":126,"value":"---\n# 🔥 Full Method","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":130,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"}],"files":"$W32","stdlib":"1","titling":"auto"}} 30:"$2f:body"