2a:null 2b:["$","title",null,{"children":"Explore 🤗 Hugging Face datasets with Parquet | Hugging Face | Observable"}] 30:T425,parquetFile === null ? md`❤️‍🩹 No parquet file found. Try to fix the search.` : md`## Contents of the parquet file The parquet file [${parquetFile.filename}](${ parquetFile.url }) (${parquetFile.size.toLocaleString("en-US", { style: "unit", unit: "byte", unitDisplay: "long" })}) has been loaded. It corresponds to: |dataset|config|split| |---|---|---| |[${dataset.id}](https://huggingface.co/datasets/${ dataset.id }) |${parquetFile.config}|${parquetFile.split}| Note also that this split can be explored on the Hub with the dataset viewer: https://huggingface.co/datasets/${ dataset.id }/viewer/${parquetFile.config}/${ parquetFile.split }. Read the datasets docs to get more details about the concepts of [splits](https://huggingface.co/docs/datasets/load_hub#splits) and [configs](https://huggingface.co/docs/datasets/load_hub#configurations). Explore the contents of the parquet file with DuckDB, using SQL queries: `31:T77f,## Epilogue The parquet conversion is done by the [dataset viewer](https://huggingface.co/docs/dataset-viewer), and the list of parquet files is published through its [API](https://huggingface.co/docs/datasets-server/parquet). Join the growing community on the [forum](https://discuss.huggingface.co/) or [Discord](https://discord.com/invite/JfAtkvEtRb) today, and give the [dataset viewer repository](https://huggingface.co/docs/datasets-server/index) a ⭐️ if you’re interested in the latest updates! ## FAQ

What is the dataset viewer API?

The dataset viewer API is a lightweight web API for visualizing and exploring all types of datasets - computer vision, speech, text, and tabular - stored on the Hugging Face Hub. As datasets increase in size and data type richness, the cost of preprocessing (storage and compute) these datasets can be challenging and time-consuming. To help users access these modern datasets, the dataset viewer runs a server behind the scenes to generate the API responses ahead of time and stores it in a database so they are instantly returned when you make a query through the API. Let the dataset viewer API take care of the heavy lifting so you can:
Get instantaneous responses to information, such as the dataset splits, column and data types, about a dataset through a simple REST API.
Download and preview the first 100 rows of any dataset.
Access the dataset as parquet files.

How do I get a dataset's parquet files?

To get the list of parquet files for the dataset duorc, fetch the following endpoint: https://datasets-server.huggingface.co/parquet?dataset=duorc

32:[] 2e:{"id":"175c6dd9410bb252","slug":"explore-hugging-face-datasets-with-parquet","title":"Explore 🤗 Hugging Face datasets with Parquet","thumbnail":"b744d311f9c1153c2997fd1bf4a9c2b15f1d7cd9aa2d2e051926d28ac2c3098c","access_level":"public","create_time":"$D2022-12-13T10:56:33.890Z","update_time":"$D2024-04-05T13:38:03.217Z","view_time":null,"first_public_time":"$D2022-12-13T14:27:13.425Z","has_unpublished_edits":null,"fork":{"id":"160d69ba8c2b1c98","slug":"parquet-quick-view","title":"Parquet quick view","owner":{"id":"08f80506ae76fdf3","login":"huggingface","name":"Hugging Face"}},"stars":12,"starred":false,"owner":{"type":"team","id":"08f80506ae76fdf3","login":"huggingface","name":"Hugging Face","avatar_url":"https://avatars-next.observableusercontent.com/798cf3379d3840e4759a43470c74a7225fe5d81c88a08e763c446c93eff8797c","tier":"pro"},"version":479,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"bf1872cca880d345","login":"severo","name":"Sylvain Lesage","avatar_url":"https://avatars-next.observableusercontent.com/7713195c83e5bfd8090ffe70c871c23e743c711e9e712675c99a8960ca8c7944","tier":null}],"members":[{"type":"team","id":"08f80506ae76fdf3","login":"huggingface","name":"Hugging Face","avatar_url":"https://avatars-next.observableusercontent.com/798cf3379d3840e4759a43470c74a7225fe5d81c88a08e763c446c93eff8797c","tier":"pro","role":"editor"},{"type":"individual","id":"bf1872cca880d345","login":"severo","name":"Sylvain Lesage","avatar_url":"https://avatars-next.observableusercontent.com/7713195c83e5bfd8090ffe70c871c23e743c711e9e712675c99a8960ca8c7944","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":6,"collection":{"id":"efa944eb2bf1c5e6","slug":"datasets","title":"Datasets","owner":{"id":"08f80506ae76fdf3","login":"huggingface","name":"Hugging Face"}},"type":"notebook","body":{"titling":"auto","title":"Explore 🤗 Hugging Face datasets with Parquet","cells":[{"id":0,"value":"md`# Explore 🤗 Hugging Face datasets with Parquet\n\nThe datasets hosted on the [Hugging Face Hub](https://huggingface.co) are now automatically converted to [parquet](https://arrow.apache.org/docs/python/parquet.html). It makes them very easy to explore, thanks to the [DuckDB client](https://observablehq.com/@observablehq/duckdb) integrated in Observable. Let's dive in.\n\n_Note: the parquet conversion is still in beta, and accessible only to datasets up to 500 MB. Open an issue in https://github.com/huggingface/dataset-viewer/ if you want to get access for a bigger dataset._`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":121,"value":"## Select a dataset from the Hugging Face Hub\n\nFirst, select one of the ${datasets.length.toLocaleString('en-US')} datasets published on the Hugging Face Hub.\n\nBy default, the first dataset in the list will be selected. Click on the radio button on the left to select a different one.\n\nUse the search box to query the dataset name, tags or description, eg\n\n blimp","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":80,"value":"viewof search = Inputs.search(datasets, {\n label: \"Search datasets\",\n query: \"\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":83,"value":"viewof dataset = Inputs.table(sortedSearch, {\n format: {\n id: (x) =>\n htl.html`${x}`\n },\n columns: [\n \"id\",\n \"downloads\",\n \"description\",\n \"tags\",\n \"citation\",\n \"cardData\",\n \"paperswithcode_id\",\n \"siblings\",\n \"gated\"\n ],\n width: {\n id: 200,\n downloads: 100,\n description: 300,\n tags: 200\n },\n layout: \"fixed\",\n sort: \"downloads\",\n reverse: true,\n multiple: false,\n value: sortedSearch[0]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":124,"value":"dataset === null\n ? md`❤️‍🩹 No dataset found. Try to fix the search.`\n : parquet_response.error\n ? md`❤️‍🩹 No parquet file found for the dataset [${dataset.id}](https://huggingface.co/datasets/${dataset.id}). Try another dataset.\n\nThe [dataset viewer API](https://datasets-server.huggingface.co/parquet?dataset=${dataset.id}) returned an error:\n> ${parquet_response.error}\n`\n : md`\n ## Select the parquet file\n\nThe dataset [${dataset.id}](https://huggingface.co/datasets/${dataset.id}) has ${parquet_response.parquet_files.length} parquet files.\n\n_Note: we used the dataset viewer API](https://huggingface.co/docs/datasets-server/parquet) to get the list of parquet files programmatically._\n\nSelect a parquet file to explore its contents.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":99,"value":"viewof parquetFile = Inputs.table(parquet_files, {\n format: {\n dataset: (x) =>\n htl.html`${x}`\n },\n multiple: false,\n value: parquet_files[0]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":131,"value":"$30","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":276,"value":"SELECT * FROM parquet","pinned":true,"hidden":false,"mode":"sql","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"var:parquetdb"},{"id":351,"value":"parquetFile === null\n ? md`❤️‍🩹 No parquet file found. Try to fix the search.`\n : md`\n## Visualize the data\n\nWe have the data. We can now do any kind of visualization. For example, here is the frequency of every word in the string columns:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":409,"value":"chart","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":427,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":48,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":356,"value":"stringColumns = (await parquetdb.describeColumns({ table: \"parquet\" }))\n .filter((d) => d.type === \"string\")\n .map((d) => d.name)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":369,"value":"text = (await parquetdb.query(`SELECT ${stringColumns.join(\",\")} FROM parquet`))\n .flatMap((d) => stringColumns.flatMap((column) => d[column].split(\" \")))\n .join(\" \")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":404,"value":"import {chart} with {text} from \"@wolfiex/word-count-bubble-chart\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":447,"value":"sortedSearch = search.sort((a, b) => d3.descending(a.downloads, b.downloads))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":92,"value":"parquet_response = {\n if (dataset === null) {\n return [];\n }\n const r = await fetch(\n `https://datasets-server.huggingface.co/parquet?dataset=${dataset.id}`,\n { headers: { authorization: `Bearer ${token}` } }\n );\n return await r.json();\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":194,"value":"parquet_files = parquet_response.parquet_files || []","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":232,"value":"parquetdb = {\n if (parquetFile === null) {\n return null;\n }\n return DuckDBClient.of({\n parquet: {\n file: { url: () => parquetFile.url, name: parquetFile.filename }\n }\n });\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":77,"value":"import { datasets } from \"@huggingface/search-the-hugging-face-datasets\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":469,"value":"viewof token = Inputs.text({\n label: html`HF Token (settings)`\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W32","stdlib":"1"}} 2f:"$2e:body"