2a:null 2b:["$","title",null,{"children":"File Loading Techniques | Alan Xu | Observable"}] 30:[] 2e:{"id":"6e261fdb41e704a9","slug":"file-loading-techniques","title":"File Loading Techniques","thumbnail":"efa8282c9ddba6b6b7cf1dc98fd9c0d5015de67dc492cd06063c91182eb6db8d","access_level":"public","create_time":"$D2018-11-04T20:50:21.579Z","update_time":"$D2022-03-03T01:20:02.737Z","view_time":null,"first_public_time":"$D2018-11-05T09:14:30.870Z","has_unpublished_edits":null,"fork":false,"stars":13,"starred":false,"owner":{"type":"team","id":"ff96f1b382d1b1d5","login":"alandelip","name":"Alan Xu","avatar_url":"https://avatars-next.observableusercontent.com/e9644a63c4f8f2430e70cae7f0c27b476afbaf70d5de98f4ed05b83c4552ac28","tier":"starter_2024"},"version":864,"publish_version":864,"role":null,"authored":false,"authors":[{"type":"individual","id":"d51cb173b79e5d19","login":"alandelip","name":"Alan Xu","avatar_url":"https://avatars-next.observableusercontent.com/e9644a63c4f8f2430e70cae7f0c27b476afbaf70d5de98f4ed05b83c4552ac28","tier":null}],"members":[{"type":"individual","id":"d51cb173b79e5d19","login":"alandelip","name":"Alan Xu","avatar_url":"https://avatars-next.observableusercontent.com/e9644a63c4f8f2430e70cae7f0c27b476afbaf70d5de98f4ed05b83c4552ac28","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":4,"collection":null,"type":"notebook","body":{"titling":"auto","title":"File Loading Techniques","cells":[{"id":0,"value":"md`# File Loading Techniques`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":22,"value":"md`\n- Loading large JSON file using streaming JSON powered by [**Oboe.js**](http://oboejs.com/)\n- Loading local files using [**Blob**](https://developer.mozilla.org/en-US/docs/Web/API/Blob)\n- Other loading techniques, including **Customized Stream Fetching**, **Shapefile** and **Socket**\n![](https://mdn.mozillademos.org/files/15817/Concept.png)\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":45,"value":"md`---\n## Loading large JSON files`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":40,"value":"md`When loading large files, a long download process will drain the patience and attention of users. Modern web techniques can help load pages component by component, and the same process can be duplicated when loading large files due to [Stream API](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API). With the concept of streaming, we can download large files and render data simultaneously. One excellent implementation is [*Oboe.js*](http://oboejs.com/) (and also see alternatives [*JSONStream*](https://github.com/dominictarr/JSONStream)).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":62,"value":"md`As can be seen in the following examples. We are loading a 3.77MB file with 28865 restaurant data derived from [Yelp Open Dataset](https://www.yelp.com/dataset/). The whole loading process can be 1 - 3 seconds if not optimized and it can be worse when coming to a larger file. A streaming json fetching process will allow the data to be processed while remaining data is still downloading.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":809,"value":"md`A trick of using Oboe.js and similar libraries is that it's using event listeners to listen on certain JSON object patterns. And whenever there's a object that matches the pattern, it will fire a event. Notice that the events can be fired so frequently that the main thread suspends to handle the event function calls instead of processing the data (Javascript is single-threaded). If so, streaming JSON loses all its charm. So here, web workers which work on background threads of the browser are recommended.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":739,"value":"mutable item_loading = {}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":518,"value":"mutable item_loaded = 0","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":574,"value":"worker = {\n mutable item_loaded = 0;\n createWorker('worker.js', (worker) => {\n worker.postMessage({\n url: \"https://raw.githubusercontent.com/AlanDelip/D3Viz/master/data/airport-data/las_vegas_stream.json\"\n });\n\n worker.onmessage = function (e) {\n if (e.data.load) {\n mutable item_loaded++;\n mutable item_loading = e.data.load;\n }\n }\n });\n \n return md`You can use Shift-Enter restart this for preview`;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":369,"value":"md`What's more, Oboe.js also allows for an abortion during the downloading phase. When certain requirements have acheieved, an abortion can be made to stop downloading. In the following example, a *till_id* param is designed to abort the downloading process once a certain id is met.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":838,"value":"mutable a_item_loading = {}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":841,"value":"mutable a_item_loaded = 0","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":847,"value":"abort_worker = {\n mutable a_item_loaded = 0;\n createWorker('worker.js', (worker) => {\n worker.postMessage({\n url: \"https://raw.githubusercontent.com/AlanDelip/D3Viz/master/data/airport-data/las_vegas_stream.json\",\n till_id: 122104\n });\n\n worker.onmessage = function (e) {\n if (e.data.load) {\n mutable a_item_loaded++;\n mutable a_item_loading = e.data.load;\n }\n }\n });\n \n return md`You can use Shift-Enter restart this for preview`;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":679,"value":"function createWorker(s, cb) {\n const worker = new window.Worker(\n window.URL.createObjectURL(\n new Blob(\n [\n `\nimportScripts('https://unpkg.com/oboe@2.1.4/dist/oboe-browser.min.js');\nself.onmessage = function(e){\n oboe(e.data.url)\n .node('{id name}', function(res) {\n if(e.data.till_id && res.id === e.data.till_id){\n this.abort();\n }\n self.postMessage({\n load: res\n });\n });\n};`\n ],\n {\n type: \"text/javascript\"\n }\n )\n )\n );\n if (typeof cb == \"function\") {\n cb(worker);\n }\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":50,"value":"md`---\n## Loading local files`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":376,"value":"md`Sometimes, we may allow user to upload their own data for visualization. Instead of uploading to a remote server, another way is using the blob of the browser. A blob is a storage unit that stores the user data in their own client-side browsers. The speed of uploading is significantly faster comparing to the traditional way. Moreover, technically it can work without the internet if the host site enables a service worker. However, it's not the perfect silver bullet. Note that for some browsers like Internet Explorer(Base64 solution may be adopted in cases like that).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":387,"value":"md`Here is an example of uploading a local image with the help of inputs toolkit from Observable official.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":383,"value":"viewof local_file = file({\n description: \"Only image files are allowed.\",\n accept: \"image/*\",\n multiple: false\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":391,"value":"blob = {\n let fileReader = new FileReader();\n let blob_url = URL.createObjectURL(local_file);\n return blob_url;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":386,"value":"viewof image = html`

`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":394,"value":"md`Also, we can upload local audio files. Remember that a local loading process is significantly faster than loading from an online url and providing both ways for loading can be even more friendly to users. The audio example can be seen at: [Music Viz](https://beta.observablehq.com/@alandelip/music-viz).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":400,"value":"md`---\n## Other loading techniques`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":419,"value":"md`Other loading techniques including **Customized Stream Fetching**, **Shapefile** and **Socket**.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":430,"value":"md`A **Customized Stream Fetching** process can technically fetch stream data of any format and process it simultaneously. Here is [an example of streaming CSV files](https://beta.observablehq.com/@mbostock/streaming-csv) by one of the founders of Observable. Notice that a customized streaming process can be complicated and likely to cause efficiency and memory issues, we should pay extra attention when creating our own stream fetching process.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":452,"value":"md`There are extra examples using Stream API at MDN Official:\n- [Simple stream pump](http://mdn.github.io/dom-examples/streams/simple-pump/)\n- [Simple random stream](http://mdn.github.io/dom-examples/streams/simple-random-stream/)\n- [Unpack chunks of a PNG](http://mdn.github.io/dom-examples/streams/png-transform-stream/)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":443,"value":"md`**Shapefile** can help use a streaming way to load map. A shapefile is an Esri vector data storage format for storing the location, shape, and attributes of geographic features. It is stored as a set of related files and contains one feature class. Here is [an example of streaming shapefiles](https://beta.observablehq.com/@mbostock/streaming-shapefiles).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":482,"value":"md`**Socket** is another way of stream loading. While socket works as long connection to the server, it can transmit data continuously. Here is [an example of Streaming Data Using Blockchain's webwsocket API](https://beta.observablehq.com/@tmcw/streaming-data).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":458,"value":"md`---\n## Dependencies & References`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":125,"value":"oboe = require(\"oboe@2.1.4\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":92,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":381,"value":"import {file} from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":475,"value":"md`[Oboe.js API](http://oboejs.com/api)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":477,"value":"md`[MDN Stream API](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":818,"value":"md`[MDN Web Worker API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":831,"value":"md`[MDN Service Worker API](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":483,"value":"md`[Shapefile Definition, ArcGIS](https://doc.arcgis.com/en/arcgis-online/reference/shapefiles.htm)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W30","stdlib":"1"}} 2f:"$2e:body"