2c:null 2d:["$","title",null,{"children":"EdelweissData docs - create and publish a dataset | Daniel Bachler | Observable"}] 32:T5df,md`This notebook is part of the [set of interactive documentation notebooks](https://observablehq.com/collection/@danyx/edelweissdata-interactive-documentation) for the [EdelweissData](https://edelweissdata.com) data mangement platform. You can change any code snippet at any time and the rest of the notebook will update immedately with the new changes (and run HTTP requests again where needed). This notebook shows you how to create a new dataset from a csv file, including setting description and metadata and publishing the dataset. In later notebooks you will learn more about the details of the authentication scheme, the query language etc.. ## Creating a dataset In order to create a dataset, you first need to create an In-Progress Dataset. In this stage you can make as many changes to the dataset as you want. Once you are okay with the data in the dataset and want to make the dataset available to others, you can publish it, thus creating a new version. A published dataset is versioned and as such cannot be modified. In order to modify a published dataset you will need to publish a new version by creating another In-Progress Dataset, apply the new changes you'd like to make and then publish a new version. Keep in mind that the old version will still be available. When accessing datasets, you can specify in the URL if you want to retrieve the dataset at a specific version (identified by the integer version number starting at 1) or whatever is the latest published version.` 33:T835,md`### 3. Upload the Schema At this point we have our data stored as CSV in EdelweissData. However, it is currently stored as a bunch of string values. In order to make the data interesting and allow Edelweiss Data make sense of it, we need to supply a schema. The schema defines the datatype of the columns in the data. The data types could be simple data types like \`string\`, \`integer\` or they could be more advanced datatypes like \`DateTime\` or [Smiles](https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system) Here are the list of Datatypes currently supported | Data Type | Representation | | -- | -- | | String | xsd:string | | Url | xsd:anyURI | | Boolean | xsd:boolean | | Integer | xsd:integer | | Float | xsd:double | | DateTime | xsd:dateTime | | Date | xsd:date | | DatasetId | edelweiss:datasetid | | SMILES | cheminf:CHEMINF_000018 | | Image | https://schema.org/image | | Json | http://edamontology.org/format_3464 | There are currently two ways to define the Schema 1. Inference - We can tell EdelweissData to infer the schema 2. Upload Schema - We supply the correct schema as json #### 3a. Schema Inference Edelweiss Data can infer the schema based on some heuristics. Schema inference can only infer basic information like the data type. If you use schema inference, consider augmenting the returned schema (e.g. with richer descriptions for each column if you have them) and uploading it again (see the Schema Upload section below for details) To do this, we simply call the [/datasets/{datasetId}/in-progress/schema/infer](https://api.edelweissdata.com/docs/index.html#operations-In_Progress-inferSchemaOfInProgressDataset) endpoint as follows`34:[{"id":"c3ef8d853ce27c6178f8e1817c61cc73abb1fbe3ee2e419ad8ff8c41318c83cdda8e9c5e490cf5d0e985edbc5496139353b8f7851fbaac9e79a52e3cd2a1b9be","href":"https://static.observableusercontent.com/files/c3ef8d853ce27c6178f8e1817c61cc73abb1fbe3ee2e419ad8ff8c41318c83cdda8e9c5e490cf5d0e985edbc5496139353b8f7851fbaac9e79a52e3cd2a1b9be","name":"test.csv","type":"text/csv","lastModified":1598278112851.153,"size":49},{"id":"4a5ddf8d7cf0a605709ca6297ee1c25c081dd1c61536325a742eb8c8e738d6ce0a02b523ff2cf9749014f55e6766862f04ca5afe214c0d4049ac873f18c286f8","href":"https://static.observableusercontent.com/files/4a5ddf8d7cf0a605709ca6297ee1c25c081dd1c61536325a742eb8c8e738d6ce0a02b523ff2cf9749014f55e6766862f04ca5afe214c0d4049ac873f18c286f8","name":"dataset-lifecycle.png","type":"image/png","lastModified":1598265706301.449,"size":22818}] 30:{"id":"02872bcf6af7ecf4","slug":"edelweissdata-docs-create-and-publish-a-dataset","title":"EdelweissData docs - create and publish a dataset","thumbnail":"5c45e108be1ee629c597889052bb6a7565219e983959784414c7199f766be93d","access_level":"public","create_time":"$D2020-08-24T10:38:20.048Z","update_time":"$D2020-09-03T09:16:02.322Z","view_time":null,"first_public_time":"$D2020-08-24T17:00:56.131Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"06e54191865d69b8","login":"danyx","name":"Daniel Bachler","avatar_url":"https://avatars-next.observableusercontent.com/872b63ff84951b97aaf3e8a0ee3bacbeaed9371a037583c8df3968bc81211304","tier":"starter_2024"},"version":296,"publish_version":296,"role":null,"authored":false,"authors":[{"type":"individual","id":"439eeaf56f71bbf5","login":"danyx","name":"Daniel Bachler","avatar_url":"https://avatars-next.observableusercontent.com/872b63ff84951b97aaf3e8a0ee3bacbeaed9371a037583c8df3968bc81211304","tier":null}],"members":[{"type":"individual","id":"439eeaf56f71bbf5","login":"danyx","name":"Daniel Bachler","avatar_url":"https://avatars-next.observableusercontent.com/872b63ff84951b97aaf3e8a0ee3bacbeaed9371a037583c8df3968bc81211304","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":{"id":"f7c98af31f6a8ffa","slug":"edelweissdata-interactive-documentation","title":"EdelweissData interactive documentation","owner":{"id":"06e54191865d69b8","login":"danyx","name":"Daniel Bachler"}},"type":"notebook","body":{"titling":"auto","title":"EdelweissData docs - create and publish a dataset","cells":[{"id":0,"value":"md`# EdelweissData docs - create and publish a dataset`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":2,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8,"value":"md`**Dataset Lifecycle Flow**\n\n${await FileAttachment(\"dataset-lifecycle.png\").image()}\n\nFor detailed information about specific endpoints of the EdelweissData API you can visit the [Swagger Docs](https://api.edelweissdata.com/docs/index.html)\n\n## Getting Started\nThe steps to publish a new Dataset are as follows\n\n1. Create a dataset\n2. Upload the data\n3. Upload the schema\n4. Upload metadata and description\n5. Set the visibility of the dataset\n6. Publish the dataset\n\nAs a first step we store the base URL of the EdelweissData API server we want to interact with:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"baseUrl = \"https://api.edelweissdata.com\" // Points to the specific Edelweiss API url\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":213,"value":"uiUrl = \"https://edelweissdata.com\" // The URL of the user interface - only used for direct linking","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":42,"value":"md`### Authentication\n\nBefore you can create a dataset you first need to retrieve an Access Token.\n\nThe Access Token is what identifies you when interacting with EdelweissData. The easiest way to retrieve this token for your account is to visit https://edelweissdata.com/datasets/manage (logging in if you are not logged in already) and clicking the \"Get API Token\" in the upper right of the screen. The token you get there is only valid for a short time - if you need longer lived tokens please refer to the [Authentication documentation notebook](/@danyx/edelweissdata-docs-authentication) for more details. \n\nThe token acts like a password so treat it with similar care.\n\nThe field below allows you to paste your access token and stores it locally in your browsers cache so that it will be available only for you and not be stored with this notebook if you should make a copy of it.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":27,"value":"// In normal Javascript code you would just\nviewof token = secret('accessToken', {\n description: \"Enter your EdelweissData access token here\",\n submit: \"Save token\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":91,"value":"md`### 1. Create the dataset`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":75,"value":"md`Now we are ready to perform the call to the [/datasets/create](https://api.edelweissdata.com/docs/index.html#operations-Create-createInProgressDataset) endpoint. We have to supply an object with the name of the new dataset and make sure to set the authorization header is filled correctly` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":122,"value":"createDatasetFn = function() {\n const data = { name: 'my-dataset' };\n\n let fetchOptions = {\n method: 'POST',\n headers: {\n 'Content-Type': 'application/json',\n Authorization: `bearer ${token}`\n },\n body: JSON.stringify(data)\n };\n\n return fetch(`${baseUrl}/datasets/create`, fetchOptions)\n .then(response => response.json())\n .catch(error => {\n console.error('Error:', error);\n });\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":70,"value":"// This is ObservableHQ specific plumbing\nviewof createDatasetButton = html`

${Object.assign(html`