2c:null 2d:["$","title",null,{"children":"From raw notebook source to Observable runtime V1 modules | Bryan Gin-ge Chen | Observable"}] 32:Tb8b,top = md`# From raw notebook source to Observable runtime V1 modules *Forked from [Alex Garcia](/@asg017)'s [notebook on using the Observable parser library in an Observable notebook](https://observablehq.com/d/5720deb83ebe0fc6)* **Update: Alex Garcia's [Unofficial ObservableHQ Compiler](https://observablehq.com/@asg017/an-unofficial-observablehq-compiler) lets you do much of the following for runtime V3 modules!**

## Every notebook has two faces One of the great features of Observable notebooks is that the code for every cell is easily accessible – either displayed beneath every cell (if the cell is "pinned") or available after a click in the left sidebar. Thus a cell can be represented as a JS Object containing a string for this displayed code string (let's call this the "raw source") and a boolean for the pin state. An Observable notebook then can be stored in an array of such cell objects. Indeed, this is essentially what is fetched when your browser loads an Observable notebook!^[[1]](#fn1) This array is the **first** of the two representations of an Observable notebook that I'll relate. It contains the raw source of each cell, and is more or less what we interact with when we're writing and editing a notebook. How does the raw source get turned into a live-running cell? In general, you can't copy and paste it directly into a standalone JS file or into a script tag in an HTML file. The solution, as given in [the "Downloading and Embedding Notebooks"](https://observablehq.com/@observablehq/downloading-and-embedding-notebooks) tutorial is to download a separate \`.js\` file which can be used with [the Observable runtime library](https://github.com/observablehq/runtime). Previously, such files defined modules in "Runtime version: 1" format, and I'll call them "runtime V1 modules" or "V1 modules" for short.^[[2]](#fn2) These JS files are the **second** representation of Observable notebooks, and they are (closer to) what the browser sees when it "runs" the notebook. If you've ever peeked at V1 module code (we'll see some examples below), you'll notice that it bears a certain resemblance to the raw source of the cells, but it's not immediately obvious how to get from one to the other. This notebook documents my attempt to **bridge the gap** and imitate roughly what the Observable servers do in generating the V1 module code. In other words, this will be a demonstration of how raw notebook cell source (like what's stored in the JSON response) can be transformed into "module" code that the Observable runtime library can understand. Along the way there'll be examples of the Observable runtime, inspector, and parser libraries in use. **Edit** (2019/09/27): Don't know why I didn't realize this when I wrote this notebook, but what this notebook provides a **COMPILER** from notebook JSON files to runtime V1 modules! `33:T80f,nbScratch = ({"id":"10ffc3f281b8b72f","creator":{"name":"Bryan Gin-ge Chen"},"version":745,"title":"scratch","nodes":[{"id":0,"value":"md`# scratch`","pinned":false},{"id":2,"value":"// node_id: 213\nnbdat = [\n {\"id\":40,\"value\":\"version = 80\",\"pinned\":true},\n ]","pinned":false},{"id":651,"value":"/* */\n// \n /* */ function /* */ x(a,b) {\n return a+b\n}","pinned":true},{"id":628,"value":"aabbasdfasdf","pinned":false},{"id":6,"value":"{\n const nbdat2 = nbdat.slice();\n const index = nbdat2.findIndex(({id}) => id === 47);\n nbdat2.splice(index,1,'XXXXX');\n return nbdat2;\n // return \"// node_id = 213\\n nbdat = \" + JSON.stringify(nbdat2);\n}","pinned":false},{"id":28,"value":"a = md`test`","pinned":false},{"id":653,"value":"x(b,c)","pinned":false},{"id":656,"value":"b= 3","pinned":false},{"id":658,"value":"c = 45","pinned":false},{"id":661,"value":"Element","pinned":true},{"id":663,"value":"Array","pinned":true},{"id":665,"value":"viewof bbb = html``","pinned":true},{"id":668,"value":"{\n return viewof bbb.value\n}","pinned":true},{"id":671,"value":"{\n yield [2]\n}","pinned":true},{"id":685,"value":"{\n await 3;\n}","pinned":true},{"id":678,"value":"async function abc() {\n return;\n}","pinned":true},{"id":681,"value":"async function* def() {\n yield;\n}","pinned":true},{"id":688,"value":"yield* [1,2,3]","pinned":true},{"id":691,"value":"mutable mut=2","pinned":true},{"id":695,"value":"{\n if (mut === 2)\n mutable mut = 3;\n return mutable mut;\n}","pinned":true},{"id":706,"value":"mut","pinned":true},{"id":709,"value":"import {a as importeda, a as importedb, a as importedc} from '3c69b22e8a78157b'","pinned":true},{"id":728,"value":"import {hljs as hljs1, hljs as hljs2} from '048035bf9a9c263e'","pinned":false},{"id":713,"value":"importeda","pinned":false},{"id":613,"value":"md`Final cell`","pinned":false},{"id":720,"value":"// ahgo)) }}} & ^&awei\\/vh 2 \\4\\\\ a2 } } } } } ))) ( ( a2))","pinned":false},{"id":744,"value":"23","pinned":false}]})34:T796,nbCoord = ({ id: "027035f3316c079f", slug: "coordinated-imports", trashed: false, likes: 12, forks: 0, fork_of: null, thumbnail: "ff7c362ec9f8c339e1b20338f62714b865066fa285c4a58b170da695f85d49f1", collections: [], publish_time: "2019-01-17T17:28:59.194Z", publish_version: 25, update_time: "2019-01-17T17:29:58.551Z", owner: { id: "074c414ad1d825f5", github_id: "230541", avatar_url: "https://avatars2.githubusercontent.com/u/230541?v=4", login: "mbostock", name: "Mike Bostock", bio: "Building a better computational medium. Founder @observablehq. Creator @d3. Former @nytgraphics. Pronounced BOSS-tock.", home_url: "https://bost.ocks.org/mike/", type: "individual" }, creator: { id: "074c414ad1d825f5", github_id: "230541", avatar_url: "https://avatars2.githubusercontent.com/u/230541?v=4", login: "mbostock", name: "Mike Bostock", bio: "Building a better computational medium. Founder @observablehq. Creator @d3. Former @nytgraphics. Pronounced BOSS-tock.", home_url: "https://bost.ocks.org/mike/" }, version: 25, title: "Coordinated Imports", nodes: [ { id: 0, value: "md`# Coordinated Imports\n\nThe chart below shows the population by age for the United States. Click a state in the map below to show the population distribution for that state instead.\n\nData: American Community Survey`", pinned: false }, { id: 19, value: "chart", pinned: true }, { id: 6, value: "viewof state", pinned: true }, { id: 15, value: "geography = state === null ? \"us:*\" : `state:${state}`", pinned: true }, { id: 10, value: "import {chart} with {geography} from \"@mbostock/u-s-population-by-age/2\"", pinned: true }, { id: 4, value: "import {viewof state} from \"@mbostock/u-s-state-map@27\"", pinned: true } ] })35:T4e6,code = md_`## Code So, how exactly does \`toV1\` work? How do we get from the raw source of each cell in the JSON to code for the V1 module? First, recall that each cell of a notebook is represented by an object in the JSON, and the part that we care about is just the \`value\` field, which is a string containing the source of that cell. On the other hand, the V1 module code consists of one or more runtime modules, which each contain an array of variables. Each cell of the notebook, roughly speaking, corresponds to one variable in the first module (called \`m0\`). The exceptions are import cells and viewof and mutable cells. Let's discuss cells which aren't any of these exceptions first. We have to take the source of a cell, which might look something like this: \`\`\`js a = x(b,c) \`\`\` and turn it into a "variable" object that looks like this: \`\`\`js { name: "a", inputs: ["x","b","c"], value: (function(x,b,c){return( x(b,c) )}) }, \`\`\` To do this, we'll make use of the Observable parser library. ~~More precisely, in this notebook we use Alex Garcia's version of the parser copied from the version on his site~~ Woo-hoo! As of version 1.1.1, we can now directly \`require\` the Observable parser: `36:Tf01,md_`If you dig through the parser output, you can see what goes into the \`name\` and \`inputs\` fields (hint: look in \`id\` and \`references\`), and then it's just a matter of trimming the cell source (using \`start\` and \`end\`) and wrapping it in another function call (or not) (this depends on whether the cell's type is parsed as "\`BlockStatement\`" or not). Ultimately, all this data and logic become embedded expressions in a big JS [template literal](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals). Cells which use \`viewof\` or \`mutable\` syntax are a tiny bit more complicated. Each \`viewof X\` cell becomes two variables, one for \`viewof X\` and one for \`X\` itself. Similarly, each \`mutable Y\` cell becomes three variables: \`initial Y\`, \`mutable Y\` and \`Y\`. Fortunately, these extra cells are constructed from a very simple pattern. For instance, the object corresponding to \`X\` always looks like this: \`\`\`js { name: "X", inputs: ["Generators","viewof X"], value: (G, _) => G.input(_) }, \`\`\` In addition to this wrinkle, we have to replace all references to \`viewof X\` and \`mutable Y\` in other cells by tokens like \`$0, $1,\` etc. (since ordinarily JS variables can't have spaces in them). To do this we use [\`acorn-walk\`](https://github.com/acornjs/acorn/tree/master/acorn-walk) to search each of the parsed cell objects which reference "\`viewof\`"s and "\`mutable\`"s. Lastly, I'll discuss import cells. Their raw source looks like this: \`\`\`js import {remoteFunction as localFunction} from 'other_notebook_id' \`\`\` In the V1 module code, this gives rise to a variable in \`m0\` that looks like this: \`\`\`js { from: "other_notebook_id", name: "localFunction", remote: "remoteFunction" }, \`\`\` as well as a new module (named \`m1\`, \`m2\`, ...) which contains variables corresponding to \`remoteFunction\` and all its dependencies. Creating the variable in \`m0\` is straightforward since the parsed import cell has a field "\`specifiers\`" containing an array of imported variables (though note that imported "\`viewof X\`" and "\`mutable Y\`" also import "\`X\`" and "\`Y\`", respectively). Filling the new module \`m*\` is trickier. It has to contain not only the definiton of \`remoteFunction\` but also all of its dependencies. In the helper function [\`getCellsFromModule\`](#getCellsFromModule), I opt for dynamically importing the imported notebook's V1 module from the Observable API (as we do at the beginning of [the first section](#demo)) and then adding the dependencies as strings using the [\`Function.prototype.toString()\`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/toString). Cells that contain \`import... with\` statements (indicated by an \`injections\` field in the parser output) are similar, except that the V1 modules generated by Observable's servers replace \`"other_notebook_id"\` with "\`this_notebook_id/N\`" for some integer N (to avoid conflicts between different imports of the same notebook). ### Addenda - *March 5, 2019*: The \`tag\` option attaches the ID of the notebook cell object to each variable and attaches the \`nodes\` array to the module. This can be useful if you want to display the source with each cell (as in [this notebook](https://observablehq.com/@bryangingechen/exporting-notebooks-with-their-source)). - *March 7, 2019*: You can now import modules that import modules. The implementation below assumes that the V1 module files served by Observable's API contain all necessary dependencies. - *May 2, 2020*: The code below now throws an error if the notebook uses the FileAttachment API. Unfortunately, V1 modules do not contain enough information to fetch uploaded FileAttachments. `37:T614,// nb is an object like the one returned by the Observable API /document/id endpoint // it must have id, title, creator.name, version, and nodes fields // if tag is true: // - add node_id to the variables // - add fake cells for comment-only cells // - add nodes object to the module and export it for convenience // (keeping this notebook-level data lets us style pinned vs unpinned cells) async function toV1(nb, tag = false) { const url = nb.id.match(/\//) ? nb.id : 'd/'+nb.id; const prefix = `// URL: https://observablehq.com/${url} // Title: ${nb.title} // Author: ${nb.creator && nb.creator.name} // Version: ${nb.version} // Runtime version: 1 `; const nbVersionedID = `${nb.id}@${nb.version}`; const {importStatements, variablesStr} = mainModule(nb.nodes, nbVersionedID, tag); const vars = ` const m0 = { id: "${nbVersionedID}", variables: [ ${variablesStr} ] }; `; // fill in nbModules with module IDs and variable strings, // using the V1 modules served by the API const nbModules = await getCellsFromModule(importStatements, nbVersionedID); // write the strings here: const modules = nbModules.map((module,i) => { return `const m${i+1} = { id: "${module.moduleID}", variables: [ ${module.str.join('\n')} ] };`; }).join('\n'); const end = `${tag ? ` const nodes = ${JSON.stringify(nb.nodes)}; ` : ''} const notebook = { id: "${nbVersionedID}", modules: [m0${nbModules.map((d,i) => `,m${i+1}`).join('')}]${tag ? `, nodes` : ''} }; export default notebook; `; return prefix+vars+modules+end; }38:T739,function notImportCell(value, parsed, node, tag) { const {inputs, fInputs, newValue} = getInputs(value, parsed); // pass "parsed" here since it gets mutated by "getInputs" // value will be "newValue" function varStr(parsed, value, cellName) { return ` {${cellName ? ` name: "${cellName}",` : ''}${inputs.length ? ` inputs: ${JSON.stringify(inputs)},` : ''} value: (${parsed.async ? 'async ' : ''}function${parsed.generator ? '*' : ''}(${fInputs})${parsed.body.type !== 'BlockStatement' ? `{return(` : ''} ${value.slice(parsed.body.start, parsed.body.end + parsed.indexShift)} )${parsed.body.type !== 'BlockStatement' ? `;})` : ''}${tag ? `, node_id: ${node.id}` : ''} },`; } if (parsed.id === null || parsed.id.type === "Identifier") // normal cell return varStr(parsed, newValue, parsed.id && parsed.id.name); else if (parsed.id.type === "ViewExpression") { // similar to the above, but we have an extra cell const viewofName = parsed.id.id.name; return `${varStr(parsed, newValue, `viewof ${viewofName}`)} { name: "${viewofName}", inputs: ["Generators","viewof ${viewofName}"], value: (G, _) => G.input(_)${tag ? `, node_id: ${node.id}` : ''} },`; } else if (parsed.id.type === "MutableExpression") { // similar to the above, but we have two extra cells const mutableName = parsed.id.id.name; return `${varStr(parsed, newValue, `initial ${mutableName}`)} { name: "mutable ${mutableName}", inputs: ["Mutable","initial ${mutableName}"], value: (M, _) => new M(_)${tag ? `, node_id: ${node.id}` : ''} }, { name: "${mutableName}", inputs: ["mutable ${mutableName}"], value: _ => _.generator${tag ? `, node_id: ${node.id}` : ''} },`; } }39:T7b8,// mutates "parsed" (adds field "indexShift") function getInputs(value, parsed) { const fInputs = []; // array of function inputs parsed.indexShift = 0; // character offset in `value`, after "viewof X" and "mutable Y" replacements let j = 0; // used to name the tokens that replace "viewof X" and "mutable Y" references in the cell const inputs = parsed.references.map((d)=> { // check references to see if any point to ViewExpressions or MutableExpressions if (d.type === "Identifier") { // ordinary input, function input is named the same fInputs.push(d.name); return d.name; } else if (d.type === "ViewExpression") { // "viewof" input, need to perform replacement const inputName = 'viewof '+d.id.name; // go through `value` and replace all "viewof XX" with "$j" walk.simple(parsed.body,{ ViewExpression(node) { const newName = `\$${j}`; value = value.slice(0,node.start + parsed.indexShift) + newName + value.slice(node.end + parsed.indexShift); parsed.indexShift += newName.length - (node.end - node.start); } }, cellWalk); fInputs.push(`\$${j++}`); return inputName; } else if (d.type === "MutableExpression") { // "mutable" input, need to perform replacement const inputName = 'mutable '+d.id.name; // go through `value` and replace all "mutable XX" with "$j.value" walk.simple(parsed.body,{ MutableExpression(node) { const newName = `\$${j}.value`; value = value.slice(0,node.start + parsed.indexShift) + newName + value.slice(node.end + parsed.indexShift); parsed.indexShift += newName.length - (node.end - node.start); } }, cellWalk); fInputs.push(`\$${j++}`); return inputName; } else console.log('unknown input type:', d); }); return {inputs, fInputs, newValue:value}; }3a:T1a4f,// Let's trust that importing the module served by the API gets us everything. async function getCellsFromModule(importStatements, parentID) { const imported = new Map(); // fetch and import all necessary modules await Promise.all( importStatements.map(({ body: { source: { value: id } } }) => { const url = id.match(/\//) ? id : 'd/' + id; if (!imported.has(id)) return importModule(`https://api.observablehq.com/${url}.js`).then( module => { imported.set(id, module); } ); }) ); const nbModules = []; const funcs = new Map(); // first pass, set up initial "top modules" and initial funcs for (let j = 0; j < importStatements.length; j++) { const { specifiers: funcs, source: { value: id }, injections } = importStatements[j].body; const topModuleID = injections ? `${parentID}/${j + 1}` : id; // we may find an import statement that has the same ID as one we already created // (this will not happen to imports with injections) let topModule = nbModules.find(({ moduleID }) => moduleID === topModuleID); if (topModule !== undefined) { // if we do, add our functions to that notebook module const newFuncNames = [ ...new Set(funcs.map(({ imported: { name } }) => name)) ]; topModule.funcNames = [ ...new Set(topModule.funcNames.concat(newFuncNames)) ]; // remove duplicates again for (let k = 0; k < newFuncNames.length; k++) topModule.queued.add(newFuncNames[k]); } // otherwise, we create a new notebook module else { const funcNames = [ ...new Set(funcs.map(({ imported: { name } }) => name)) ]; const queued = new Set(funcNames.concat(LibraryKeys)); const injMap = new Map( injections ? injections.map(inj => [inj.imported.name, inj]) : [] ); topModule = { funcNames, injMap, queued, str: [], moduleID: topModuleID, module: imported.get(id), moduleIndex: 0 }; nbModules.push(topModule); } } // process modules / variables let currModule; while ( (currModule = nbModules.find(({ funcNames }) => funcNames.length > 0)) ) { const { funcNames, injMap, queued, str, module, moduleIndex } = currModule; // `funcNames` is a queue that begins with the functions that are imported; // their dependencies will be added recursively // `queued` marks the functions that have already been added while (funcNames.length > 0) { const name = funcNames.shift(); // could use .pop() here too; // .shift() just gets the order closer to what the Observable API returns? if (injMap.has(name)) { // if this definition is injected from the base notebook using import...with str.push(` { from: "${parentID}", name: "${name}", remote: "${injMap.get(name).local.name}" },`); } else if (name === "FileAttachment") { throw new Error('FileAttachment API not supported!'); } // some DOM / Web APIs show up in "inputs" but aren't real cells, we should ignore them // however, 'define' and 'hljs' are things added to the window by Observable, so we should not ignore those! // (Probably comment those out if running this in a non-notebook environment) else if (!(window[name] && name !== 'define' && name !== 'hljs')) { // WARNING: for cells imported from "broken" notebooks where the same name is given to two different cells, // the following line grabs the first cell with that name, which differs from current Observable behavior. const modObj = module.default.modules[moduleIndex].variables.find( ({ name: varName }) => varName === name ); if (modObj.from) { // there's an import in your import // write import variable str.push(` { from: "${modObj.from}", name: "${name}", remote: "${modObj.remote}" },`); // where are you 'from' if (modObj.from !== parentID) { // if it's the parent, do nothing // search in other nbModules let innerModule = nbModules.find( ({ moduleID }) => moduleID === modObj.from ); if (innerModule !== undefined) { // the module is already in nbModules, so check "queued" for the remote name... if (!innerModule.queued.has(modObj.remote)) { // not in "queued" so it's not a duplicate innerModule.funcNames = innerModule.funcNames.push( modObj.remote ); innerModule.queued.add(modObj.remote); } } else { // otherwise, we create a new notebook module // dive into the other modules inside this import let newModuleIndex = module.default.modules.findIndex( (mod, ind) => ind !== moduleIndex && mod.variables.some( ({ name: varName }) => varName === modObj.remote ) ); if (newModuleIndex > 0) { // if found, push the remote name and module to nbModules const newFuncNames = [modObj.remote]; const newQueued = new Set(newFuncNames.concat(LibraryKeys)); innerModule = { funcNames: newFuncNames, injMap: new Map(), queued: newQueued, str: [], moduleID: modObj.from, module, moduleIndex: newModuleIndex }; nbModules.push(innerModule); } else console.log(`not found: ${modObj}`); // if this happens then probably the import is missing something! // let the runtime throw with some "could not resolve" } } } else { // non-import / "normal" variable if (modObj.inputs) { // queue the dependencies into this module's "funcNames" modObj.inputs .filter(str => !queued.has(str)) .forEach(newName => { funcNames.push(newName); queued.add(newName); }); } // append the variable definition str.push(` { name: "${name}",${ modObj.inputs ? ` inputs: ${JSON.stringify(modObj.inputs)},` : '' } value: ${modObj.value.toString()} },`); } } } } return nbModules; }3b:[] 30:{"id":"c5efd660b44c143c","slug":"from-raw-notebook-source-to-observable-runtime-v1-modules","title":"From raw notebook source to Observable runtime V1 modules","thumbnail":"0a9f3a3f82a73a2198a03be79a73faad546f39aae8f4722db240b2c6a806be54","access_level":"public","create_time":"$D2019-02-28T03:59:13.303Z","update_time":"$D2020-05-02T10:54:37.378Z","view_time":null,"first_public_time":"$D2019-03-01T21:00:33.051Z","fork":true,"stars":19,"starred":false,"owner":{"type":"team","id":"d2fd23101beb2a61","login":"bryangingechen","name":"Bryan Gin-ge Chen","avatar_url":"https://avatars-next.observableusercontent.com/73f0dccc0b7fc5c845e6a52a58e01a2ae432cc529aaada0d10ab602137e603c0","tier":"starter_2024"},"version":1702,"publish_version":1702,"role":null,"authored":false,"authors":[{"type":"individual","id":"5c13a1a27cf52564","login":"bryangingechen","name":"Bryan Gin-ge Chen","avatar_url":"https://avatars-next.observableusercontent.com/73f0dccc0b7fc5c845e6a52a58e01a2ae432cc529aaada0d10ab602137e603c0","tier":null}],"members":[{"type":"individual","id":"5c13a1a27cf52564","login":"bryangingechen","name":"Bryan Gin-ge Chen","avatar_url":"https://avatars-next.observableusercontent.com/73f0dccc0b7fc5c845e6a52a58e01a2ae432cc529aaada0d10ab602137e603c0","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":4,"collection":{"id":"938fbc4744e2d8f6","slug":"observable","title":"Observable","owner":{"id":"d2fd23101beb2a61","login":"bryangingechen","name":"Bryan Gin-ge Chen"}},"type":"notebook","body":{"title":"From raw notebook source to Observable runtime V1 modules","cells":[{"id":0,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1638,"value":"md_`\nFor convenience, here's how to import [the main function](#toV1) of this notebook:\n\\`\\`\\`js\nimport {toV1} from '@bryangingechen/from-raw-notebook-source-to-observable-runtime-v1-modules'\n\\`\\`\\`\n\nHere's a quick summary of the contents:\n- [First](#demo), I illustrate how to import an Observable server-generated runtime V1 module and embed it into a notebook cell\n- [In the second section](#hard), I begin with a notebook array of raw source strings and convert it to V1 module code and then embed it into a notebook cell.\n- [Finally](#code), I describe how the main function works. This mainly boils down to describing the format of the V1 module code and comparing it to what the Observable parser returns.\n\n*See also:* [\\`fromV1\\`](/@bryangingechen/fromv1), the \"reverse\" function which takes a V1 module object and outputs a notebook array.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1248,"value":"fn1 = md`[^](#top)[1]: You can check this out by having the \"Network\" tab in your browser's dev tools open when a notebook loads and finding the XHR request to \\`https://api.observablehq.com/document/(notebook ID)\\`. (Don't worry if you'd rather not check now, there's an example later on). The response from this endpoint is a JSON string which contains, among other metadata, an array called \\`nodes\\`, which is precisely of this form: the elements of that array are objects, they correspond to cells, and they each have a field \\`value\\` which contains a string with the raw source for that cell and a field \\`pinned\\` with a boolean saying whether the cell source is pinned or not!`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1259,"value":"fn2 = md`[^](#top)[2]: There are two distinct notions of \"module\" floating around here:\n - [ES6 modules](https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/) are collections of JS code that are suitable for being \"imported\" into other JS. \n - [(Observable) runtime modules](https://github.com/observablehq/runtime#modules) are namespaces for variables in an Observable runtime.\n\n~~So the \\`.js\\` files that you get from the \"Download code\" links are ES6 modules~~ You can get \\`.js\\` files that define one or more Observable V1 runtime modules by navigating to \\`https://api.observablehq.com/@user/notebook-name.js\\`. I *could* be referring to them as \"module modules\", but instead I'll just call them modules.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1113,"value":"demo = md`## Loading a notebook into the runtime the easy way\n\nIn this first section, I'll show how to embed a full Observable notebook as a cell. Compare for example [the \"Ouroboros\" demo](https://observablehq.com/@jashkenas/ouroboros-a-notebook-embeds-itself) by Jeremy Ashkenas.\n\nFirst, let's grab the JS source of the V1 module code for a [scratch notebook](https://observablehq.com/d/10ffc3f281b8b72f), just to see what it looks like:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":803,"value":"scratchTruth = d3.text('https://api.observablehq.com/d/10ffc3f281b8b72f@745.js')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1123,"value":"md`We can dynamically import this module from that URL into our notebook as follows (\\`importModule\\` is [a polyfill for dynamic import](/@bryangingechen/dynamic-import-polyfill) so that this will all work in Firefox 65):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":933,"value":"moduleScratchTruth = importModule('https://api.observablehq.com/d/10ffc3f281b8b72f@745.js')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1125,"value":"md`Having done so, we now use the Observable runtime to embed that notebook in a cell (using the default [inspector](https://github.com/observablehq/inspector)):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":941,"value":"// v1 runtime\n{\n const hello = html`

`;\n observable.Runtime.load(moduleScratchTruth.default, lib, observable.Inspector.into(hello));\n return hello;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1595,"value":"requireWarning = md`⚠️ **Warning!** Embedding an imported copy of the runtime in a notebook like this may lead to issues with \\`require\\`. [See this forum post](https://talk.observablehq.com/t/require-notebook-runtime/1740/). One possible workaround is to use the \\`require\\` provided by an imported copy of the standard library (defined in the cell [\\`lib\\`](#lib)) (see [\\`require_\\`](#require_)), at least for all cells which could be evaluated **after** the cell that embeds the runtime. This turns out to break syntax highlighting as well, so we also have to define a [\\`md_\\`](#md_). A better solution is to [embed notebooks in iframes](https://observablehq.com/@mbostock/embedded-notebook).\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":360,"value":"hard = md`## Turning notebook JSON files into a runtime V1 module\n\nNow, let's examine the JSON data corresponding to that [scratch notebook](https://observablehq.com/d/10ffc3f281b8b72f), which I've previously extracted from the Observable API response from \\`https://api.observablehq.com/document/10ffc3f281b8b72f\\` (you can't visit that link directly, you need to communicate an HTTP request with valid session info):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":757,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1106,"value":"md`Let's focus on the \\`nodes\\` array. Its elements are objects with 3 fields: \\`id\\`, \\`pinned\\`, and \\`value\\`:\n- \\`id\\`: an integer associated to the cell, used internally by the Observable editor\n- \\`pinned\\`: a boolean, if true then the notebook source is pinned open in the editor,\n- \\`value\\`: a string giving the source of the cell.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1050,"value":"md`The main function below, [\\`toV1\\`](#toV1), can take the notebook JSON object above and turn it into a string which is suitable for loading as a module and passing to an instance of the Observable runtime:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":761,"value":"scratchV1 = toV1(nbScratch)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1054,"value":"md`Now we convert that module source code into a data URL and dynamically import it:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":931,"value":"moduleScratchNew = importModule(`data:text/javascript;base64,${utoa(scratchV1)}`)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1056,"value":"md`Finally, embed it in a cell as we did above:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":944,"value":"// v1 runtime\n{\n const hello = html`

`;\n observable.Runtime.load(moduleScratchNew.default, lib, observable.Inspector.into(hello));\n return hello;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1155,"value":"md`So far so good. The sharp-eyed among you may have noticed one difference. The penultimate cell is 1 here, but 2 above. The reason is has to do with the way that I'm (not) dealing with notebook imports from notebooks with multiply-defined cells (see the notes in the source of [\\`getCellsFromModule\\`](#getCellsFromModule)).\n\nHere's a more complicated example, Mike Bostock's [\"Coordinated Imports\"](https://observablehq.com/@mbostock/coordinated-imports) notebook (slightly modified to use [an older version of U.S. State Map](https://observablehq.com/d/3eda9b2ecfc88051@27) that avoids the FileAttachment API, which is awkward to use with the v1 module format):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1290,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1490,"value":"coord = toV1(nbCoord);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1292,"value":"{\n if (navigator.userAgent.match('HeadlessChrome')) return;\n const hello = html`

`;\n const coord = await toV1(nbCoord);\n const moduleCoord = await importModule(\n `data:text/javascript;base64,${utoa(coord)}`\n );\n observable.Runtime.load(\n moduleCoord.default,\n lib,\n observable.Inspector.into(hello)\n );\n return hello;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":801,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2,"value":"// parser = require('https://iamprettydamn.cool/parser.js') // dist/parser.js hosted on Alex Garcia's site\n// parser = require('https://gistcdn.githack.com/bryangingechen/70cbbc03de1c598cb9ce0fef88fc383d/raw/8c27c7f48d1b0a6ae1ce432db0e2f0cc5618467a/parser.js') // re-hosted on raw.githack.com\nparser = require(\"@observablehq/parser@1\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1138,"value":"md`\nThe Observable parser library's \\`parseCell\\` function takes as input the raw source and returns an abstract syntax tree of the cell, plus important meta info, including e.g. whether the cell is asynchronous, or a generator, as well as an array of dependencies of the cell:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":810,"value":"scratchParsed = nbScratch.nodes.map(({value}) => parser.parseCell(value))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1179,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":146,"value":"$37","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1422,"value":"function mainModule(nodes, nbVersionedID, tag) {\n const importStatements = [];\n const variablesStr = nodes.map((node,i) => { // loop over cells\n const {value} = node;\n let parsed;\n try {\n parsed = parser.parseCell(value); // parse\n } \n catch (e) {\n throw new Error(`${e}, in cell ${i}`);\n }\n if (parsed.hasOwnProperty('references')) { // if NOT an import statement\n return notImportCell(value, parsed, node, tag);\n } \n else if (parsed.body && parsed.body.type === \"ImportDeclaration\") { // no \"references\" field => notebook import (?)\n importStatements.push(parsed);\n return importCell(nbVersionedID, importStatements, parsed, node, tag);\n } \n else if (parsed.body === null) {\n // comment-only cell?\n if (tag) return ` {\n node_id: ${node.id}\n },`;\n } \n else {\n throw new Error(`unknown type at cell ${i}: ${JSON.stringify(node)}`);\n }\n }).join('\\n');\n return {importStatements, variablesStr};\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":505,"value":"LibraryKeys = {\n const L = new observable.Library;\n const p = [];\n for (const l in L)\n p.push(l);\n return p;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1431,"value":"$38","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1438,"value":"$39","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1447,"value":"function importCell(nbVersionedID, importStatements, parsed, node, tag) {\n const {specifiers, source:{value:importedID}, injections} = parsed.body;\n const fromID = injections ? // the ID changes if the injections field is present (even if it's empty []!)\n `${nbVersionedID}/${importStatements.length}` :\n importedID;\n \n function importStr(localName, importedName) {\n return ` {\n from: \"${fromID}\",\n name: \"${localName}\",\n remote: \"${importedName}\"${tag ? `,\n node_id: ${node.id}`\n : ''}\n },`;\n }\n\n return specifiers.map((spec) => {\n const {imported:{name:importedName}, local:{name:localName}, view} = spec;\n return `${importStr(localName, importedName)}${view ? // extra cell if we import viewof\n '\\n' + importStr(`viewof ${localName}`, `viewof ${importedName}`) :\n ''}${spec.mutable ? // extra cell if we import mutable\n '\\n' + importStr(`mutable ${localName}`, `mutable ${importedName}`) : ''}`\n }).join('\\n');\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":459,"value":"$3a","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":107,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":377,"value":"md`This function is used to base64-encode the JS strings in the dataURLs. [\\`btoa()\\`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/btoa) would be the natural choice, but it doesn't work well if the notebook contains unicode:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":301,"value":"// https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/btoa#Unicode_strings\n// possibly deprecated...\nfunction utoa(str) {\n return window.btoa(unescape(encodeURIComponent(str)));\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":117,"value":"d3 = require_('d3-fetch@1')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1586,"value":"lib = {\n toV1; // delay creating this until after toV1 is done\n // once the \"require\" in \"lib\" is called, the built-in require should be considered broken\n return new observable.Library;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1626,"value":"require_ = lib.require()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1619,"value":"md_ = lib.md()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":127,"value":"observable = require('@observablehq/runtime@3')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":518,"value":"import {importModule} from '@bryangingechen/dynamic-import-polyfill'","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":858,"value":"walk = require('acorn-walk@7')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1192,"value":"md`\\`parser.walk\\` contains a \"walker\" object used above via [acorn-walk's \\`simple\\` method](https://github.com/acornjs/acorn/tree/master/acorn-walk#interface):`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":880,"value":"cellWalk = parser.walk //walk.make(parser.walk)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1673,"value":"md`## LICENSE\n\nAny original code here is made available under the terms of the ISC License.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1671,"value":"LICENSE = `Copyright 2019\n\nPermission to use, copy, modify, and/or distribute this software for any purpose\nwith or without fee is hereby granted, provided that the above copyright notice\nand this permission notice appear in all copies.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH\nREGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND\nFITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,\nINDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS\nOF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER\nTORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF\nTHIS SOFTWARE.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W3b","stdlib":"1","titling":"auto"}} 31:"$30:body"