Bringing the TypeScript Language Server to Observable

The editor hints found in popular code editors such as VSCode (e.g., autocomplete and embedded documentation) have become indispensable for developers. So, we set out to implement similar functionality within Observable.

We learned that the architecture that powers VSCode is modular and that the “brains” of VSCode can be used outside of VSCode, even in a browser. After a few twists and turns, we were able to implement editor hints within Observable by integrating with the TypeScript Language Server.

This is how we did it.

Code editors often use the TypeScript Language Server (tsserver) to provide editor hints for TypeScript as well as JavaScript. For JavaScript, completions are inferred either from JavaScript source found in the project or from any d.ts files that have been loaded into the project (e.g., all of the node or browser APIs).

tsserver also consumes any JSDoc annotations it finds in order to display signature help descriptions, as well as any JSDoc type annotations to provide autocompletions, etc.

Here is an example from the TypeScript Playground:

The question we had when starting this project was whether we could get these editor hints into Observable, which runs in the browser and uses CodeMirror to as an editor. VSCode is powered by Microsoft’s open-source editor called Monaco, so it wasn’t clear how we could integrate with CodeMirror, if at all. Luckily, the Monaco editor is implemented in a modular way, with a clear separation between the editor and the language services.

The Monaco editor doesn’t do any of the heavy lifting for the editor hints, but is more of the presentation layer that relies on the various language services (tsserver being one of them) to get the editor hint information. In theory this means we could either host a TS Language Server that the browser can query, or we could try to run tsserver in the browser to remove the round trip over the internet.

We quickly discovered that tsserver has a dependency on the file system in order to read from and write to the various files in a TypeScript (or JavaScript) project, which meant there wouldn’t be a way to run tsserver in the browser. It wasn’t until we happened upon this incredibly helpful CodeMirror forum thread that the path forward was clear. The answer was hidden in plain sight: do what the TypeScript Playground does and use @typescript/vfs rather than tsserver.

Until we found the above CodeMirror forum thread and especially the responses by Sid, it hadn’t occurred to us that the TypeScript Playground is actually already doing the thing we’re trying to do: run tsserver in the browser. We assumed (without ever actually checking) that it must be communicating over the internet with a hosted tsserver somewhere.Sid invested a lot of time in that thread outlining how he got the TypeScript language service running in the browser and integrated with CodeMirror. @typescript/vfs was developed by Microsoft in order to enable implementation of the TypeScript Playground by removing tsserver’s dependency on the file system. (VFS here stands for Virtual File System.)

Here we express our gratitude for Sid and Microsoft for paving the way. All that was left for us was to connect everything together. Let's do it now!

First, we import typescript and @typescript/vfs.

vfs = import("https://cdn.jsdelivr.net/npm/@typescript/vfs@1.4.0/+esm")
ts = (await import("https://cdn.jsdelivr.net/npm/typescript@4.8.4/+esm")).default

Then we initialize a "Virtual TypeScript Environment".

This uses the createDefaultMapFromCDN function which will load all of the browser lib.*.d.ts files, which is how we have types and documentation for browser APIs like document.querySelector().

tsEnv = {
  const {createSystem, createVirtualTypeScriptEnvironment, createDefaultMapFromCDN} = vfs;
  const fsMap = await createDefaultMapFromCDN({target: ts.ScriptTarget.ES2021}, ts.version, true, ts);
  const system = createSystem(fsMap);
  const tsEnv = createVirtualTypeScriptEnvironment(system, [], ts, {allowJs: true});
  tsEnv.createFile("/index.js", " "); // This is where our code will go. Note: can’t be empty 😅
  return tsEnv;
}

As long as we keep the virtual TypeScript environment in sync with our editor, we will be able to leverage all of the smarts that the language server provides.

Our editor in this notebook will be an html textarea (with a little extra to emit changes to the cursor offset as well). We will also query the language service for any auto completions we can show.

We represent each cell in a notebook as a separate file in the VFS, and keep the TypeScript virtual environment in sync with changes made in the CodeMirror cell editors.Then, we can query the language service for signature help and autocompletions at any position in the CodeMirror editor. What remains is simply the UI work around integrating these autocompletions and signature help items with CodeMirror.Fortunately, CodeMirror has very robust frameworks for both autocompletions as well as tooltips.

Next, we moved the virtual TypeScript environment to a WebWorker. That way, we could then take advantage of Webpack 5’s automatic code-splitting for WebWorkers to ensure that the download of the virtual TypeScript environment and the download of the Observable application would be done separately.

This also alleviated concerns about performance because WebWorkers are run in a separate thread, so any heavy lifting that the TypeScript language service is doing will not block the main thread and the UI will still be responsive.

Now, when we use native browser APIs in a cell, we get the autocompletions and signature help:

There are still some challenges we’re working through.

The first issue we had to face is the fact that Observable isn’t JavaScript. There are some tweaks and additions we’ve made to JavaScript in order to facilitate the reactivity and express certain functionality in Observable.In order for the TypeScript language service to be able to make sense of a given cell, we need to transpile that cell into something that is valid JS first, and maintain a Source Map that will allow us to map from a position in the CodeMirror editor to a position in the transpiled JavaScript file that is in the TypeScript virtual environment.

Another issue is that the Observable Runtime does really nice things like ensure that a Promise is resolved before evaluating downstream cells. Unfortunately, we don’t have a way yet to make TypeScript aware of all of these transformations, so there are some instances where cross-cell inference is incorrect.

So, if you reference a cell that is defined as a Promise in another cell, you will get completions for a promise, not what the Promise resolves to. We will be improving cross-cell inference over time.

Another issue is that within an IDE, if you see a named TypeScript interface in, say, a Signature help tooltip, there is a way to drill down on that interface and see what its definition is. This is not yet implemented in Observable, so sometimes you will see a reference to a named interface or type, but not have a way to “drill down” and view that interface’s definition. We intend to improve this experience so you can view the d.ts files from the virtual TypeScript environment.

Finally, some things we would like to do in the future are:

• Observable users should be able to use JSDoc annotations on cell functions to provide Signature help and autocompletions for their libraries.

• We want to use TypeScript’s Automatic Type Acquisition library to fetch the types (if they exist) for any user-required (or imported) modules.

• We also want to provide editor hints for all of Observable’s standard Library.