Taras Novak

DuckDB Data Import

SQLite to DuckDB

Data Package 📦 DuckDB

DuckDB Data Tables

DuckDB

Tabular data tools and diagraming notebooks.

Tables

My duckdb often runs out of memory if I insert more than 1000 rows via INSERT, seems low and like I should change something. Any ideas?

Tom Larkworthy

Tom,

This notebook only provides an overview of existing Observable data tools you can use to view DuckDB instance information.

It's hard to elaborate on your particular scenario without knowing the nature of your dataset, as there are different ways of importing data into DuckDB via:

- batched Inserts: https://duckdb.org/docs/data/insert
- compressed CSV files: https://duckdb.org/docs/data/csv
- Appender: https://duckdb.org/docs/data/appender
- and Parquet Files: https://duckdb.org/docs/data/parquet

Also, it depends on the dataset you are trying to import, and how many columns each record has. If you are working with a wide column dataset that has hundreds or thousands of columns in it, I would suggest you explore parquet data format: https://parquet.apache.org/docs

You can also set your memory limit via DuckDB Configuration settings (https://duckdb.org/docs/sql/configuration) briefly covered at the end of this notebook.

I would suggest you post your question in duckdb github repository (https://github.com/duckdb/duckdb/issues) with more information and a sample of the dataset you are trying to import.