Observable

Custom database clients for use with Observable SQL cells.

Database Clients

Would you entertain S3 import as a feature request? I think it could supplant much of what my team is using postgres for at the moment https://duckdb.org/docs/guides/import/s3_import

Ben Ayre

I believe this will require a custom build of duckdb-wasm, since it will require the HTTPFS extension. But we can look into it.

https://github.com/duckdb/duckdb-wasm/issues/839
https://github.com/duckdb/duckdb-wasm/pull/880

Mike Bostock

Thanks for sharing those links. Led me to https://github.com/duckdb/duckdb-wasm/issues/1040 and https://observablehq.com/@kimmolinna/duckdb-parquet-and-aws-s3 which demonstrates a working implementation 💫

Great find! Here’s a fork of that notebook that uses Observable’s DuckDBClient:

https://observablehq.com/d/f2de401dd3ab54d8

I’ve sent it as a suggestion to @kimmolinna.

Thanks @mbostock! I will change all my notebooks to use DuckDBClient of stdlib during weekend.

Kimmo Linna

Here's a quick wrapper I've put together to simplify the setup. Thanks for sharing @kimmolinna! https://observablehq.com/@bayre/duckdb-s3

@mbostock,  I want to define table from the buffer that it's possible to use different ways (AWS-SDK etc.) to use AWS, Scaleway, Cloudflare etc. Now I do it like this but this isn't so elegant way so could it possible to add this somehow to the standard client:

 class BlobFile extends Blob {
  constructor (arrayBuffer,extension) {
    super([arrayBuffer]);
    Object.defineProperty(this,"name",{value: "d."+extension});
    Object.defineProperty(this,"_url",{value: "blob:"});
  }
  async url(){
    return (await this._url) +"";
  }
 }

DuckDBClient.of({'test': {file:new BlobFile(arrayBuffer,"parquet")}})