Custom database clients for use with Observable SQL cells.

Visnu Pitiyanuvath

Observable

Connecting to databases

Mike Bostock

DatabaseClient Specification

Hello, DuckDB

Alex Garcia

Observable Ambassadors

Hello, Datasette

Hello, QuestDB

Hello, HEAVY.AI!

Artjoms Iskovs

Seafowl

Seafowl database client

Hello, Dolt

Robert Harris

Cobus Theunissen

Hello, Trino/Presto

Whether your data is hosted on the web, stored in a database, or saved in a local file, you can access it from Observable with these techniques.

Getting Data In and Out

Resources for working with SQL in Observable

Database Clients

Would you entertain S3 import as a feature request? I think it could supplant much of what my team is using postgres for at the moment https://duckdb.org/docs/guides/import/s3_import

Ben Ayre

I believe this will require a custom build of duckdb-wasm, since it will require the HTTPFS extension. But we can look into it.

https://github.com/duckdb/duckdb-wasm/issues/839
https://github.com/duckdb/duckdb-wasm/pull/880

Thanks for sharing those links. Led me to https://github.com/duckdb/duckdb-wasm/issues/1040 and https://observablehq.com/@kimmolinna/duckdb-parquet-and-aws-s3 which demonstrates a working implementation 💫

Great find! Here’s a fork of that notebook that uses Observable’s DuckDBClient:

https://observablehq.com/d/f2de401dd3ab54d8

I’ve sent it as a suggestion to @kimmolinna.

Thanks @mbostock! I will change all my notebooks to use DuckDBClient of stdlib during weekend.

Kimmo Linna

Here's a quick wrapper I've put together to simplify the setup. Thanks for sharing @kimmolinna! https://observablehq.com/@bayre/duckdb-s3

@mbostock,  I want to define table from the buffer that it's possible to use different ways (AWS-SDK etc.) to use AWS, Scaleway, Cloudflare etc. Now I do it like this but this isn't so elegant way so could it possible to add this somehow to the standard client:

 class BlobFile extends Blob {
  constructor (arrayBuffer,extension) {
    super([arrayBuffer]);
    Object.defineProperty(this,"name",{value: "d."+extension});
    Object.defineProperty(this,"_url",{value: "blob:"});
  }
  async url(){
    return (await this._url) +"";
  }
 }

DuckDBClient.of({'test': {file:new BlobFile(arrayBuffer,"parquet")}})