Observable Framework 1.12.0-alpha.3 GitHub️ 2.4k

Converting notebooks

Framework’s built-in convert command helps you convert an Observable notebook to standard Markdown for use with Observable Framework. To convert a notebook, you need its URL; pass it to the convert command like so:

npm run observable convert <notebook-url>

The above command assumes you’re running convert within an existing project. Outside of a project, you can use npx:

npx "@observablehq/framework@latest" convert <notebook-url>

You can convert multiple notebooks by passing multiple URLs:

npm run observable convert <url1> <url2> <url3>

The convert command currently only supports public notebooks. To convert a private notebook, you can (temporarily) make the notebook public unlisted by clicking Share… on the notebook and choosing Can view (unlisted) under Public access. Please upvote #1578 if you are interested in support for converting private notebooks.

For example, to convert D3’s Zoomable sunburst:

npm run observable convert "https://observablehq.com/@d3/zoomable-sunburst

This will output something like:

┌   observable convert 
│
◇  Downloaded zoomable-sunburst.md in 443ms
│
◇  Downloaded flare-2.json in 288ms
│
└  1 notebook converted; 2 files written

The convert command generates files in the source root of the current project by default (typically src); you can change the output directory using the --output command-line flag. The command above generates two files: zoomable-sunburst.md, a Markdown file representing the converted notebook; and flare-2.json, an attached JSON file.

Due to differences between Observable Framework and Observable notebooks, the convert command typically won’t produce a working Markdown page out of the box; you’ll often need to make further edits to the generated Markdown. We describe these differences below, along with examples of manual conversion.

The convert command has minimal “magic” so that its behavior is easier to understand and because converting notebook code into standard Markdown and JavaScript requires human interpretation. Still, we’re considering making convert smarter; let us know if you’re interested.

JavaScript syntax

Framework uses vanilla JavaScript syntax while notebooks use a nonstandard dialect called Observable JavaScript. A JavaScript cell in a notebook is technically not a JavaScript program (i.e., a sequence of statements) but rather a cell declaration; it can be either an expression cell consisting of a single JavaScript expression (such as 1 + 2) or a block cell consisting of any number of JavaScript statements (such as console.log("hello");) surrounded by curly braces. These two forms of cell require slightly different treatment. The convert command converts both into JavaScript fenced code blocks.

Expression cells

Named expression cells in notebooks can be converted into standard variable declarations, typically using const. So this:

foo = 42

Becomes this:

const foo = 42;

Variable declarations in Framework don’t implicitly display. To inspect the value of a variable (such as foo above), call display explicitly.

Framework allows multiple variable declarations in the same code block, so you can coalesce multiple JavaScript cells from a notebook into a single JavaScript code block in Framework. Though note that there’s no implicit await when referring to a variable declared in the same code block, so beware of promises.

Anonymous expression cells become expression code blocks in Framework, which work the same, so you shouldn’t have to make any changes.

1 + 2

While a notebook is limited to a linear sequence of cells, Framework allows you to interpolate dynamic values anywhere on the page: consider using an inline expression instead of a fenced code block.

Block cells

Block cells are used in notebooks for more elaborate definitions. They are characterized by curly braces ({…}) and a return statement to indicate the cell’s value. Here is an abridged typical example adapted from D3’s Bar chart:

chart = {
  const width = 960;
  const height = 500;

  const svg = d3.create("svg")
      .attr("width", width)
      .attr("height", height);

  return svg.node();
}

To convert a named block cell to vanilla JavaScript: delete the cell name (chart), assignment operator (=), and surrounding curly braces ({ and }); then replace the return statement with a variable declaration and a call to display as desired.

const width = 960;
const height = 500;

const svg = d3.create("svg")
    .attr("width", width)
    .attr("height", height);

const chart = display(svg.node());

For an anonymous block cell, omit the variable declaration. To hide the display, omit the call to display; you can use an inline expression (e.g., ${chart}) to display the chart elsewhere.

If you prefer, you can instead convert a block cell into a function such as:

function chart() {
  const width = 960;
  const height = 500;

  const svg = d3.create("svg")
      .attr("width", width)
      .attr("height", height);

  return svg.node();
}

Then call the function from an inline expression (e.g., ${chart()}) to display its output anywhere on the page. This technique is also useful for importing a chart definition into multiple pages.

Imports

Notebooks often import other notebooks from Observable or open-source libraries from npm. Imports require additional manual conversion.

If the converted notebook imports other notebooks, you should convert the imported notebooks, too. Extract the desired JavaScript code from the imported notebooks into standard JavaScript modules which you can then import in Framework.

In Framework, reactivity only applies to top-level variables declared in fenced code blocks. If the imported code depends on reactivity or uses import-with, you will likely need to do some additional refactoring, say converting JavaScript cells into functions that take options.

Some notebooks use require to load libraries from npm. Framework discourages the use of require and does not include built-in support for it because the asynchronous module definition (AMD) convention has been superseded by standard JavaScript modules. Also, Framework preloads transitive dependencies using static analysis to improve performance, and self-hosts imports to eliminate a runtime dependency on external servers to improve security and give you control over library versioning. So this:

regl = require("regl")

Should be converted to a static npm import:

import regl from "npm:regl";

The code above imports the default export from regl. For other libraries, such as D3, you should use a namespace import instead:

import * as d3 from "npm:d3";

You can import d3-require if you really want to a require implementation; we just don’t recommend it.

Likewise, instead of resolve or require.resolve, use import.meta.resolve. So this:

require.resolve("regl")

Should be converted to:

import.meta.resolve("npm:regl")

Some notebooks use dynamic import to load libraries from npm-backed CDNs such as jsDelivr and esm.sh. While you can use dynamic imports in Framework, for security and performance, we recommend converting these into static imports. So this:

isoformat = import("https://esm.sh/isoformat")

Should be converted to:

import * as isoformat from "npm:isoformat";

If you do not want to self-host an import, say because you want the latest version of the library to update without having to rebuild your app, you can load it from an external server by providing an absolute URL:

import * as isoformat from "https://esm.sh/isoformat";

Generators

In notebooks, the yield operator turns any cell into a generator. In vanilla JavaScript, the yield operator is only allowed within generator functions. Therefore in Framework you’ll need to wrap a generator cell declaration with an immediately-invoked generator function expression (IIGFE). So this:

foo = {
  for (let i = 0; i < 10; ++i) {
    yield i;
  }
}

Can be converted to:

const foo = (function* () {
  for (let i = 0; i < 10; ++i) {
    yield i;
  }
})();

Since variables are evaluated lazily, the generator foo above will only run if it is referenced by another code block. If you want to perform asynchronous side effects, consider using an animation loop and the invalidation promise instead of a generator.

If you need to use await with the generator, too, then use async function* to declare an async generator function instead.

Views

In notebooks, the nonstandard viewof operator is used to declare a reactive value that is controlled by a user interface element such as a range input. In Framework, the view function performs the equivalent task with vanilla syntax. So this:

viewof gain = Inputs.range([0, 11], {value: 5, step: 0.1, label: "Gain"})

Can be converted to:

const gain = view(Inputs.range([0, 11], {value: 5, step: 0.1, label: "Gain"}));

In other words: replace viewof with const, and then wrap the input declaration with a call to view. The view function both displays the given input and returns the corresponding value generator so you can define a top-level reactive value.

Mutables

In notebooks, the nonstandard mutable operator is used to declare a reactive value that can be assigned from another cell. In Framework, the Mutable function performs the equivalent task with vanilla syntax. So this:

mutable foo = 42

Can be converted to:

const foo = Mutable(42);
const setFoo = (x) => (foo.value = x);

Then replace any assignments to mutable foo with calls to setFoo. Note that setFoo must be declared in the same code block as foo, and that outside of that block, foo represents the value; any code that depends on foo will update reactively after setFoo is invoked.

Standard library

As part of our modernization efforts with Framework, we’ve pruned deprecated methods from the standard library used in notebooks. The following notebook built-ins are not available in Framework:

For convenience, we’ve linked to the implementations above so that you can see how they work, and if desired, copy the code into your own Framework app as vanilla JavaScript. For example, for a 2D canvas, you can replace DOM.context2d with:

function context2d(width, height, dpi = devicePixelRatio) {
  const canvas = document.createElement("canvas");
  canvas.width = width * dpi;
  canvas.height = height * dpi;
  canvas.style = `width: ${width}px;`;
  const context = canvas.getContext("2d");
  context.scale(dpi, dpi);
  return context;
}

For md, we recommend writing literal Markdown. To parse dynamic Markdown, you can also import your preferred parser such as markdown-it from npm.

In addition to the above removals, a few of the built-in methods have changed:

The Framework standard library also includes several new methods that are not available in notebooks. These are covered elsewhere: Generators.dark and dark; Generators.now; Generators.width and resize; display; and sql.

File attachments

Framework’s FileAttachment includes a few new features:

And two removals:

For the latter, file.arrow now imports npm:apache-arrow internally, and thus uses the same version of Arrow as if you imported Arrow directly.

In Framework, implicit imports of recommended libraries are normal npm imports, and thus are self-hosted, giving you control over versioning. If a requested library is not in your npm cache, then by default the latest version will be downloaded. You can request a more specific version either by seeding the npm cache or by including a semver range in the import specifier (e.g., import * as d3 from "npm:d3@6").

Because Framework defaults to the latest version of recommended libraries, you will typically get a more recent version than what is available in notebooks. As of August 2024, here is a comparison of recommended library versions between notebooks and Framework:

In Framework, the html and svg built-in template literals are implemented with Hypertext Literal which automatically escapes interpolated values. The dot template literal implements responsive dark mode & better styling. And Framework has several additional recommended libraries that are not available in notebooks: ReactDOM, React, duckdb, echarts, mapboxgl, and vg.

Sample datasets

Like recommended libraries, Framework’s built-in sample datasets (e.g., aapl and penguins) are backed by npm imports that are self-hosted.

Cell modes

The convert command only supports code cell modes: Markdown, JavaScript, HTML, TeX, and SQL. It does not support non-code cell modes: data table and chart. You can use the “Convert to SQL” or “Convert to JavaScript” feature to convert data table cells and chart cells to their code equivalents prior to conversion. Alternatively, you can manually replace data table cells with Inputs.table (see #23 for future enhancements), and chart cells with Observable Plot’s auto mark.

Databases

Database connectors can be replaced by data loaders.

Secrets

We recommend using a .env file with dotenv to store your secrets (such as database passwords and API keys) in a central place outside of your checked-in code; see our Google Analytics dashboard example.