2e:null 2f:["$","title",null,{"children":"Moving analyses to the web using Observable and DuckDB | José M. Zavala González | Observable"}] 34:T7ea,canvas = { // Adjust the canvas size according to the screen's width // const canvasWidth = 300; const canvasWidth = width; // const canvasHeight = Math.min(1000, canvasWidth); const canvasHeight = canvasWidth; // (Optional) Paint the background const context = DOM.context2d(canvasWidth, canvasHeight); // context.imageSmoothingEnabled= false; // context.fillStyle = "hsl(216deg 100% 13%)"; // context.fillRect(0, 0, canvasWidth, canvasHeight); // Paint the background of the active drawing area const minLength = Math.min(canvasWidth, canvasHeight); const dx = 0.5*(canvasWidth - minLength); const dy = 0.5*(canvasHeight - minLength); // context.fillStyle = "hsl(100deg 100% 30%)"; context.fillStyle = 'black'; context.fillRect(dx, dy, minLength, minLength); // Calculate scales to position the pixels const xScale = d3.scaleLinear([0, pixels], [dx, dx+minLength]); const yScale = d3.scaleLinear([0, pixels], [dy+minLength, dy]); const unitLength = xScale(1) - xScale(0); // context.fillStyle = "hsl(30deg 100% 50%)"; // context.fillRect(xScale(0), yScale(0), unitLength, unitLength); // Color scale ("extent" of the total transformed pickups, and the corresponding color scale) const fillScale_transform_fn = Math.log10; // Log scale to smooth differences in orders of magnitude const fillScale = d3.scaleSequential( d3.extent(pickupAny.map( d => fillScale_transform_fn( Number(d.pickup) ) )), d3.interpolateViridis); // Viridis for colors // For each pixel... pickupAny.map(d => { // Determine its color const transformed_value = fillScale_transform_fn( Number(d.pickup) ); const color = fillScale( transformed_value ); context.fillStyle = color; context.strokeStyle = "rgba(1, 1, 1, 0)"; // Zero opacity // context.strokeStyle = color; // Paint it in its position context.fillRect(xScale(Number(d.x)), yScale(Number(d.y)), unitLength, unitLength); // context.stroke(); }) return context.canvas; }35:T42c,This notebook is based on a tutorial developed by Danielle Navarro called ["How to visualise a billion rows of data in R with Apache Arrow"](https://blog.djnavarro.net/posts/2022-08-23_visualising-a-billion-rows/). Her blog, ["Notes from a data witch"](https://blog.djnavarro.net/), is spectacular and you should check it out if you haven't yet. Danielle explains the challenges behind graphing millions of observations and implements a solution using the R language and her local computer. I love this visualization whether it's with sparse or densely-packed pixels and I want to share it with whoever wishes to see it. That said, not everyone has programming experience or using R. Even when they do, having to download an additional IDE and dataset files delays someone from the actual starting point: playing with the visualization. In this example, we attempt to translate the tutorial to a web-based environment that takes you directly to an interactive graphic. If all goes well, this example should be shareable with more ease and a reduced barrier to entry.36:T61c,## Outline for moving the analysis Replicating an R analysis on the web involves moving a few pieces which we'll be detailing as they arise. To do so, we'll rely on 4 tools: 1. **DuckDB-WASM** is completely portable. If you get its SQL to work in R or Python, it will work just as well on the web. The WebAssembly (WASM) allows us to conduct the heaviest calculations live on the user's web browser without need for additional servers to support it. 2. **ObservableHQ** facilitates focusing on the code for processing and visualizing data. It includes by default a [DuckDB client](https://observablehq.com/@observablehq/duckdb) and removes the need for having a bad time configuring WASM, organizing reactivity, promises, etc. 3.

**{dbplyr}** is an R package that translates the tidyverse verbes to valid SQL to send to databases like DuckDB. Specifically, the `show_query` function saves us from rewriting a long query manually. 5. **GitHub Pages** hosts a copy of the [nyc-taxi-tiny](https://github.com/jzavala-gonzalez/nyc-taxi-tiny) dataset that Voltron Labs [makes available](https://arrow.apache.org/docs/r/articles/dataset.html). The reasoning behind rehosting is that Pages allows sending data with sufficiently flexible [CORS](https://medium.com/@baphemot/understanding-cors-18ad6b478e2b) permissions (It includes the `Access-Control-Allow-Origin: *` header) so that WASM reads the data without worrying about cyberattacks or something like that.37:T469,## Summary The idea behind this notebook is demonstrating that **there are analyses that can reduce their barrier to entry by hosting them altogether on the web**. In this example, the user or developer doesn't need to download R and RStudio nor the parquet files for the nyc-taxi-tiny dataset. By using Observable, we also go around configuring WebAssembly, reactivity and deployment which typically requires knowledge about frontend and backend technologies. DuckDB is renowned for its speed but here its _portability_ stands out most. If a query we are testing in the R or Python version of DuckDB works, then that query will work on the web-based version, DuckDB-WASM. The {dbplyr} R package is spectacular here because we can use the dplyr API to write SQL that DuckDB accepts. It makes me wish there was a dbplyr version that ran on the browser so it could be more handily available for web-based projects. The visualization draws squares which colors are chosen based on the quantity of observations contained by said square. It uses a canvas element to maintain efficiency as the number of "pixels" drawn increases. 38:T40f,## 🦆 DuckDB and Arrow for magic The computer on which you're reading this can analyze thousands and millions of datapoints when you hand it tools that can support it. In Danielle's tutorial, this tool is the {arrow} R package. Arrow implements many tricks to agilize its calculations and that is reflected on how fast the R code runs. To fill Arrow's shoes in a web-based environment, we look towards DuckDB. DuckDB is a database system capable of running our data. It's compatible with the 'parquet' formatted dataset files we use and it can also read the files remotely across the web. Did I mention it is 🏃‍♀️💨_ ridiculously fast_? I understand it is so quick that it takes us longer to provide the data to DuckDB than for DuckDB to process it. Importantly, we use the WebAssembly (WASM) version of DuckDB so we can keep our work _inside the same computer_ where you read this. This falls in harmony with Observable which also executes its notebooks locally and saves us from figuring out how to run a remote server. 39:[] 32:{"id":"3bfff6bbb49ea6ac","slug":"moving-analyses-to-the-web-using-observable-and-duckdb","title":"Moving analyses to the web using Observable and DuckDB","thumbnail":"5697aa09bdf53b82c898d5cce401c1d98862e8bea9116b6be708460ebb2631c1","access_level":"public","create_time":"$D2023-01-11T01:46:44.160Z","update_time":"$D2023-01-11T03:18:14.528Z","view_time":null,"first_public_time":"$D2023-01-11T01:46:44.160Z","fork":{"id":"f2a54755449473eb","slug":"moviendo-analisis-a-la-web-con-observable-y-duckdb","title":"Moviendo análisis a la web con Observable y DuckDB","owner":{"id":"247b19faa48b7d89","login":"jzavala-gonzalez","name":"José M. Zavala González"}},"stars":3,"starred":false,"owner":{"type":"team","id":"247b19faa48b7d89","login":"jzavala-gonzalez","name":"José M. Zavala González","avatar_url":"https://avatars-next.observableusercontent.com/c80357963553ac341ea90ed3d79296a72d004aa90d9b89afd76f17e04dcf9277","tier":"starter_2024"},"version":757,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"2c62f0243f8f45b0","login":"jzavala-gonzalez","name":"José M. Zavala González","avatar_url":"https://avatars-next.observableusercontent.com/c80357963553ac341ea90ed3d79296a72d004aa90d9b89afd76f17e04dcf9277","tier":null}],"members":[{"type":"individual","id":"2c62f0243f8f45b0","login":"jzavala-gonzalez","name":"José M. Zavala González","avatar_url":"https://avatars-next.observableusercontent.com/c80357963553ac341ea90ed3d79296a72d004aa90d9b89afd76f17e04dcf9277","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Moving analyses to the web using Observable and DuckDB","cells":[{"id":0,"value":"# Moving analyses to the web using Observable and DuckDB","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":702,"value":"### [**TRADUCCIÓN EN ESPAÑOL DISPONIBLE AQUI**](https://observablehq.com/d/f2a54755449473eb)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":245,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":319,"value":"viewof pixels = Inputs.range([2, 1000], {label: \"Pixeles\", step: 1, value: 150})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":645,"value":"viewof year_drop = Inputs.select([...Array(8).keys()].map(i => (2009+i).toString()), {label: \"Año\", value: \"2009\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":334,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":459,"value":"\n

bservableHQ as a publishing platform

\n\nAn Observable notebook (what you're reading right now) simplifies publishing widely any visualizations. It provides a platform that is designed to create visuals without understanding how a website works as a prerequisite. However, Observable is still scaffolding over the fundamentals of the web (HTML, CSS, and Javascript), which guarantees some longevity for the visualization's code. \n\nIt allows users to interact with visuals efficiently, thanks to its reactivity model. That is how the viz responds rapidly to the sliders included in this notebook.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":449,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":358,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":472,"value":"$38","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":336,"value":"## Initializing the database\n\nTo start, we open the DuckDB-WASM client and create the `VIEW` of the dataset files hosted on Github Pages:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"dbInitial = DuckDBClient.of({})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":33,"value":"view_query = `\nCREATE OR REPLACE VIEW nyc_taxi_tiny AS\nSELECT *\nFROM parquet_scan(${(urls)});\n`","pinned":true,"hidden":true,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":383,"value":"The `view_query` references another cell called `urls` that is a string of a list of links pointing to each parquet file in the dataset:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":386,"value":"urls","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":389,"value":"Is this good practice for replicating in other projects? Probably not, since this kind of copy-pasting tends to open one up to an SQL injection attack. Nevertheless, it works for this quick example. This should be remedied as soon as DuckDB implements a clearer method for reading in remote datasets with Hive partitioning (e.g.'`.../year=2009/month=1/...`' in the filename).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":395,"value":"The next step is informing Observable that we must wait for DuckDB to register the `VIEW` of the dataset before we start sending queries. Observable usually knows how to determing the appropriate order of events, but the `VIEW` is created as a \"secondary effect\" and we have to specify the dependency explicitly. Mike Bostock explains the process in more detail in [this notebook](https://observablehq.com/@observablehq/derived-views-with-duckdb).\n\nIn practical terms: **Every time we execute a query that creates a VIEW or a new table, we define a \"new\" connection that depends on this initial query**. We utilize this pattern frequently throughout this analysis to ensure the notebook maintains the flow of its reactivity.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":42,"value":"dbView = dbInitial.query(view_query)","pinned":true,"hidden":true,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":91,"value":"con = dbView, dbInitial","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":95,"value":"SELECT *\nFROM nyc_taxi_tiny\nLIMIT 5;","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"","database":"var:con"},{"id":100,"value":"SELECT date_part('year', pickup_datetime) as pickup_year, date_part('month', pickup_datetime) as pickup_month, COUNT(*)\nFROM nyc_taxi_tiny\nGROUP BY pickup_year, pickup_month\nORDER BY pickup_year, pickup_month;","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"","database":"var:con"},{"id":671,"value":"SELECT pickup_longitude, pickup_latitude\nFROM nyc_taxi_tiny\nWHERE pickup_longitude IS NOT NULL","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"","database":"var:con"},{"id":401,"value":"To start, we create the dataset `VIEW` in the `dbView` cell. Next, we establish a connection (`con`), which is a copy of the initial connection (`dbInitial`) but materialized after `dbView` completes. The queries that use the new `VIEW` then reference `con` instead of `dbInitial`.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":340,"value":"## Taxi pickups query\n\nWe want to aggregate the coordinates for the taxi pickups to specific regions that we can visualize. The R code that Danielle uses in her tutorial to run this query is (more or less) the following:\n\n```r\n# Parameters\nx0 <- -74.05\ny0 <- 40.6\nspan <- 0.3\npixels <- 4000\n\n# The query\nnyc_taxi_tiny |>\n filter(\n !is.na(pickup_longitude),\n !is.na(pickup_latitude),\n pickup_longitude > x0,\n pickup_longitude < x0 + span,\n pickup_latitude > y0,\n pickup_latitude < y0 + span\n ) |>\n mutate(\n unit_scaled_x = (pickup_longitude - x0) / span,\n unit_scaled_y = (pickup_latitude - y0) / span,\n x = as.integer(round(pixels * unit_scaled_x)), \n y = as.integer(round(pixels * unit_scaled_y))\n ) |>\n count(x, y, name = \"pickup\")\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":418,"value":"It's super readable! The dplyr API creates the query one step at a time. However, DuckDB needs _SQL_ to execute queries. If we add one more line, we obtain a draft SQL query:\n\n```r\ntbl(con, 'nyc_taxi_tiny') |>\n ...\n count(x, y, name = \"pickup\") |>\n show_query() # Magia!\n```\n\nY el SQL que resulta:\n\n```sql\nSELECT x, y, COUNT(*) AS pickup\nFROM (\n SELECT\n *,\n CAST(ROUND(4000.0 * unit_scaled_x, CAST(ROUND(0.0, 0) AS INTEGER)) AS INTEGER) AS x,\n CAST(ROUND(4000.0 * unit_scaled_y, CAST(ROUND(0.0, 0) AS INTEGER)) AS INTEGER) AS y\n FROM (\n SELECT\n *,\n (pickup_longitude - -74.05) / 0.3 AS unit_scaled_x,\n (pickup_latitude - 40.6) / 0.3 AS unit_scaled_y\n FROM nyc_taxi_tiny\n WHERE\n (NOT((pickup_longitude IS NULL))) AND\n (NOT((pickup_latitude IS NULL))) AND\n (pickup_longitude > -74.05) AND\n (pickup_longitude < (-74.05 + 0.3)) AND\n (pickup_latitude > 40.6) AND\n (pickup_latitude < (40.6 + 0.3))\n ) q01\n) q02\nGROUP BY x, y\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":426,"value":"Ok, the process to create this query is actually a bit different than what is presented. Before receiving the SQL with `show_query()`, we have to translate opening the dataset using the {arrow} package (like in the tutorial) to use the {dbplyr} package with DuckDB. The details are hosted in [this Quarto notebook](https://jzavala-gonzalez.github.io/nyc-taxi-tiny/nyc_taxi_tiny_testing) on GitHub Pages. \n\nThe adjusted query such that Observable can enter the parameters chosen by the user is the following:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":105,"value":"CREATE OR REPLACE TABLE pickupRaw AS\nSELECT x, y, COUNT(*) AS pickup\nFROM (\n SELECT\n *,\n CAST(ROUND(${pixels} * unit_scaled_x, CAST(ROUND(0.0, 0) AS INTEGER)) AS INTEGER) AS x,\n CAST(ROUND(${pixels} * unit_scaled_y, CAST(ROUND(0.0, 0) AS INTEGER)) AS INTEGER) AS y\n FROM (\n SELECT\n *,\n (pickup_longitude - -74.05) / 0.3 AS unit_scaled_x,\n (pickup_latitude - 40.6) / 0.3 AS unit_scaled_y\n FROM nyc_taxi_tiny\n WHERE\n (NOT((pickup_longitude IS NULL))) AND\n (NOT((pickup_latitude IS NULL))) AND\n (pickup_longitude > -74.05) AND\n (pickup_longitude < (-74.05 + 0.3)) AND\n (pickup_latitude > 40.6) AND\n (pickup_latitude < (40.6 + 0.3))\n ) q01\n) q02\nGROUP BY x, y","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"pickupRawQuery","database":"var:con"},{"id":502,"value":"The query changes slightly from the exact result from 'show_query()' so that it is compatible with Observable's casting and string interpolation.\n\nHere are the results from that query:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":228,"value":"con_pickupRaw = pickupRawQuery, con","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":179,"value":"SELECT *\nFROM pickupRaw","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"pickupRaw","database":"var:con_pickupRaw"},{"id":429,"value":"An additional detail from the tutorial that we can replicate with DuckDB is calculating all the combinations of XY coordinats. We implicitly use a [`CROSS JOIN`](https://www.sqlshack.com/sql-cross-join-with-examples/) and the `generate_series` function from DuckDB's SQL dialect: ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":111,"value":"CREATE OR REPLACE TABLE xycombs AS\nSELECT series.generate_series as x, series2.generate_series as y\nFROM generate_series(0, CAST(${pixels} AS INTEGER)-1 ) as series,\n generate_series(0, CAST(${pixels} AS INTEGER)-1 ) as series2","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"xycombsQuery","database":"var:con"},{"id":231,"value":"con_xycombs = xycombsQuery, con","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":189,"value":"SELECT *\nFROM xycombs","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"","database":"var:con_xycombs"},{"id":431,"value":"The pickup query with all the XY combinations is created from a `JOIN` and the empty values filled in with `COALESCE`:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":234,"value":"con_pickupQuery = pickupRaw, xycombsQuery, con","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":164,"value":"CREATE OR REPLACE TABLE pickup AS\nSELECT xycombs.x, xycombs.y, COALESCE(pickup, 0) AS pickup\nFROM xycombs\nLEFT JOIN pickupRaw\nON xycombs.x = pickupRaw.x\nAND xycombs.y = pickupRaw.y","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"pickupQuery","database":"var:con_pickupQuery"},{"id":240,"value":"con_pickup = pickupQuery, con","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":195,"value":"SELECT *\nFROM pickup","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"pickup","database":"var:con_pickup"},{"id":342,"value":"## Create the visualization\n**The final visualization is at the top cell of the notebook with corresponding comments alongside each step.** To alleviate the weight of drawing ${(pixels**2).toLocaleString()} pixels, we can first filter our results to only include those pixels with at least one pickup.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":281,"value":"SELECT *\nFROM pickup\nWHERE pickup > 0","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","output":"pickupAny","database":"var:con_pickup"},{"id":523,"value":"This way we only have to draw ${pickupAny.length.toLocaleString()} pixels.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":40,"value":"# Appendix","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":433,"value":"## URLs for the dataset's files","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":617,"value":"// viewof years = Inputs.checkbox([...Array(14).keys()].map(i => (2009+i).toString()), {label: \"Year(s)\", value: ['2009']})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":647,"value":"years = [year_drop]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":619,"value":"yearstrs = years.map(s => 'year='+s)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":634,"value":"fileurls_filtered = fileurls.filter(u => yearstrs.some(ys => u.includes(ys)))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":631,"value":"fileurls","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":13,"value":"fileurls = await fetch(\"https://raw.githubusercontent.com/jzavala-gonzalez/nyc-taxi-tiny/main/fileurls.txt\").then(response => {\n if (!response.ok) throw new Error(response.status);\n return response.text().then(t => t.split('\\n'));\n}).catch(error => {return error;})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":47,"value":"urls = ('[' + fileurls_filtered.map(s => \"'\"+s+\"'\").join(',') + ']')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":662,"value":"`\nSELECT date_part('year', pickup_datetime) as pickup_year, date_part('month', pickup_datetime) as pickup_month, COUNT(*)\nFROM parquet_scan(${urls})\nGROUP BY pickup_year, pickup_month\nORDER BY pickup_year, pickup_month;\n`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":199,"value":"// Esto va a crashear si hay demasiados pixeles\n// Plot.plot({\n// width: width,\n// y: {reverse: true},\n// padding: 0,\n// color: {\n// scheme: \"bupu\",\n// },\n// marks: [\n// Plot.cell(pickupAny, {\n// x: d => Number(d.x),\n// y: d => Number(d.y),\n// fill: d => Math.log10(Number(d.pickup)) ,\n// inset: 0.0,\n// })\n// ]\n// })","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":614,"value":"# References\n\nDanielle Navarro. 2022. “How to Visualise a Billion Rows of Data in R with Apache Arrow.” August 23, 2022. https://blog.djnavarro.net/posts/2022-08-23_visualising-a-billion-rows.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"}],"files":"$W39","stdlib":"1","titling":"auto"}} 33:"$32:body"