2d:null 2e:["$","title",null,{"children":"Data Curation Workflow | Michael Deagen | Observable"}] 33:T445,# Data Curation Workflow ##### (see Appendix below for additional links and resources) Curation involves *selecting*, *organizing*, and *looking after* items in a collection. You can think of our data curation workflow as one big ETL process comprising several smaller ETL processes. The source of data is typically a human-readable data representation such as a table or chart embedded in a PDF document, without the raw data itself. The goal of this workflow is to make data machine-interpretable (via digital extraction and metadata annotation) so that the data become searchable. To curate data from published research articles, the process is decomposed into specific stages and tasks that enable us to manage data quality over time. Depending on the complexity of the datasets or the capabilities of the database schema or ontology at a particular point in time, this process can be highly iterative! ### Workflow at a Glance The graph below shows the workflow, where nodes represent stages of curation and edges describe tasks. **Hover over labels for more detailed descriptions.**34:Tfd4,dot`digraph x { node [style=filled]; "Wishlist" [fillcolor="#857c8d", fontcolor=white, tooltip= "A research article of known digital object identifier (DOI) sits in a virtual Wishlist until selected for curation."]; "Wishlist" -> "Initialized" [label="TravelerSetup", labeltooltip= "A Curation Traveler is created, which is a README.md document that lives alongside the curation job through its life cycle."]; "Initialized" [fillcolor="#e65f5c", fontcolor=white, tooltip= "Once a Curation Traveler and job ID have been created, data collection and preparation process can begin."]; "Initialized" -> "Outlined" [label="NotebookSetup", labeltooltip= "To keep track of the tables and charts that will be curated, we set up an Observable notebook with placeholder Tables and Vega-Lite chart specs that will be automatically populated once the data are uploaded."]; "Outlined" [fillcolor="#acf7c1", fontcolor=black, tooltip= "Once the notebook has been created, the curation job sits at the Outlined stage and is ready for data collection."]; "Outlined" -> "Digitized" [label="DataGathering", labeltooltip= "To extract data from charts and tables, the curator can use tools like WebPlotDigitizer and Tabula. In most cases, the WebPlotDigitizer project files will be created during the NotebookSetup task."]; "Digitized" [fillcolor="#52ad9c", fontcolor=white, tooltip= "If the notebook passes a visual inspection once the data have been uploaded to it, then the curation job sits at the Digitized stage."]; "Digitized" -> "Wrangled" [label="Wrangling", labeltooltip= "Although the data are mostly tidy when extracted from charts or tables, some wrangling is necessary in order to prepare the data for further processing. Since this process may be revisted, wrangling should be done programmatically."]; "Wrangled" [fillcolor="#ffff82", fontcolor=black, tooltip= "Datasets sitting at the Wrangled stage are ready for semantic annotation. However, some re-wrangling may be necessary in order to accommodate the semantic metadata."]; "Wrangled" -> "Annotated" [label="Semantics", labeltooltip= "Applying semantic metadata enables the datasets to become searchable by applying standardized terminology to the dataset attributes from a controlled vocabulary."]; "Annotated" [fillcolor="#5c0029", fontcolor=white, tooltip= "Once the data have been sufficiently described with metadata, the curation job sits at the Annotated stage until it has been uploaded to the knowledge graph."]; "Annotated" -> "Searchable" [label="Integration", labeltooltip= "Annotated datasets are converted to RDF triples, which get uploaded as 'nanopublications' to the knowledge graph. Nanopublications track the provenance of data and enable any later corrections to the data to be reflected in the knowledge graph."]; "Searchable" [tooltip= "Data at the Searchable stage have been nominally curated and are accessible in the knowledge graph. If error correction(s) or addition(s) need to be made to the dataset, the Curation Traveler can be used to track these updates."]; "Annotated" -> "Wrangled"; "Wrangled" -> "Templated" [label="Mapping", style=dashed, labeltooltip= "An alternative curation path for conventional polymer nanocomposite data is through manual or programmatic data entry into Excel templates. Manual work can be fast when the task is straightforward but becomes difficult to replicate for complex datasets."]; "Templated" -> "Wrangled" [style=dashed]; "Templated" [fillcolor="#207245", fontcolor=white, tooltip= "At this stage, data have been mapped to Excel templates and are ready for upload via the XML-based upload form. Each 'sample' requires its own Excel workbook."]; "Templated" -> "Searchable" [label="Upload", style=dashed, labeltooltip= "Each sample Excel workbook and accompanying file attachments must be uploaded one at a time to the site, awaiting a confirmation email for each sample to ensure proper ingest."]; "Searchable" -> "Outlined"; "Searchable" -> "Digitized"; "Searchable" -> "Wrangled"; "Searchable" -> "Annotated"; }`35:[] 31:{"id":"1e10701acf96ffd1","slug":"data-curation-workflow","title":"Data Curation Workflow","thumbnail":"21332c4d30546a97e06dac6d271459a57c43daa6c5301525b33f5eb2f8e84069","access_level":"public","create_time":"$D2021-05-17T13:48:26.160Z","update_time":"$D2021-05-18T20:47:21.375Z","view_time":null,"first_public_time":"$D2021-05-18T20:38:51.301Z","has_unpublished_edits":null,"fork":false,"stars":2,"starred":false,"owner":{"type":"team","id":"5ca8a3812ea4a326","login":"mdeagen","name":"Michael Deagen","avatar_url":"https://avatars-next.observableusercontent.com/0b8e876e2eac1058faa6c1a2624346f3e72350b8a7e6c9acaec6a9ad920b0476","tier":"starter_2024"},"version":349,"publish_version":349,"role":null,"authored":false,"authors":[{"type":"individual","id":"d3611c0325cb99e9","login":"mdeagen","name":"Michael Deagen","avatar_url":"https://avatars-next.observableusercontent.com/0b8e876e2eac1058faa6c1a2624346f3e72350b8a7e6c9acaec6a9ad920b0476","tier":null}],"members":[{"type":"individual","id":"d3611c0325cb99e9","login":"mdeagen","name":"Michael Deagen","avatar_url":"https://avatars-next.observableusercontent.com/0b8e876e2eac1058faa6c1a2624346f3e72350b8a7e6c9acaec6a9ad920b0476","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":5,"collection":{"id":"07bad1e1751e43d5","slug":"documentation","title":"MaterialsMine Docs","owner":{"id":"5ca8a3812ea4a326","login":"mdeagen","name":"Michael Deagen"}},"type":"notebook","body":{"title":"Data Curation Workflow","cells":[{"id":18,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":0,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":108,"value":"### What is the *Curation Traveler?*\n\nThe \"traveler\" is a digital document (with the filename \"README.md\") in [Markdown format](https://www.markdownguide.org/) that provides documentation throughout a curation job's life cycle. Because of the diverse range of tasks involved in data curation, the process benefits from collaboration. The traveler allows curators to log the process and keep track of the the status and any issues during curation.\n\nThe example below shows what the curation traveler looks like when opened in a text editor such as Notepad. We use the filename README.md because Github will recognize and automatically render the markdown formatting when you view the directory in the remote repository.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":173,"value":"

\n # Curation Traveler
\n* **curation_job_id:** lee2012tailoring
\n* **DOI:** 10.1021/ma3016085
\n* **status:** NotebookSetup
\n* **notebook:** https://observablehq.com/d/b3114cc98be438ca
\n
\n---
\n
\n### LOG
\n
\n2021-04-15: Initialized traveler [MED]
\n
\n2021-04-30: Initialized notebook, moved to NotebookSetup [MED]\n

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":217,"value":"The log section of the Traveler is there for curators to document the curation process. It is a good idea to update the Traveler when updates are made to the repo, such as a commit.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":336,"value":"### Using Github to Organize Curation\n\n*Git* is a powerful tool for version control ubiquitously used on collaborative software projects. Here, we use a [Github repository](https://github.com/mdeagen/mm-curation-notebooks) (repo) to remotely collaborate and stay organized. Each curation job has its own directory containing the curated data and metadata, supporting code, and a curation traveler. You can use [Github Desktop](https://desktop.github.com/), a GUI that helps you more easily organize commits containing multiple files.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":150,"value":"### Setting up a Curation Job\n\nOnce you have a DOI for a research article, you can view the article page by visiting doi.org/*{insert DOI here}* in your browser (you will be automatically redirected to the article page). Searching the DOI on a service such as [Google Scholar](https://scholar.google.com/) should return the article as a search result, where you can view bibliographic information, a portion of the abstract, and linked articles. Here, you can retrieve the *BibTeX* information (click on the closing quotation icon, then BibTeX) and use that ID as the curation job ID.\n\nThe example below shows the BibTeX information for *lee2012tailoring*:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":156,"value":"

\n @article{lee2012tailoring,\n title={Tailoring the photomechanical response of glassy, azobenzene-functionalized polyimides by physical aging},\n author={Lee, By Kyung Min and Koerner, Hilmar and Wang, David H and Tan, Loon-Seng and White, Timothy J and Vaia, Richard A},\n journal={Macromolecules},\n volume={45},\n number={18},\n pages={7527--7534},\n year={2012},\n publisher={ACS Publications}\n}\n

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":171,"value":"Create a folder in the Curation Jobs directory with the curation job ID as the folder name. In this folder, you can save the BibTeX information (e.g. as a text file called scholar.txt) and the Curation Traveler (README.md).\n\nIf you use Google Scholar, add library links via the settings in order to easily access articles through your institution log-in. You can save the research article and article images in this directory within your local repository, but note that the Github repository has been configured to ignore any files with .pdf, .jpg, or .png file extensions, so these types of files will not appear in the remote repository.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":194,"value":"### Setting up a Notebook\n\n[This template](https://observablehq.com/d/e7da314cc30f8e4d) can be used for setting up a new Observable notebook. Simply create a Fork of the template and fill out the relevant information. The notebook can then be shared with collaborators using Observable's new share feature. Setting up the notebook involves defining which .csv files will be expected and how those files will be rendered. For visual charts, we use [Vega-Lite](https://vega.github.io/vega-lite/docs/) specifications to create an interactive version of the chart. (Creating Vega-Lite specs requires familiarity with the language, but there are templates in the repository that provide boilerplate examples to build upon.)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":203,"value":"### Digital Extraction of Data\n\nFor data tables, you can use [Tabula](https://tabula.technology/) to auto-extract the information (although, it is important to check the extracted output for errors). For charts such as line and scatter plots, we use [WebPlotDigitizer (WPD)](https://apps.automeris.io/wpd/) to decode the data from a chart image. WPD allows projects to be saved (as .tar files), which allows us to collaborate and save progress on data extraction tasks. In order to tidy the output from WebPlotDigitizer into a format that can be interpreted by Vega-Lite, we use the home-built [TidyCatWPD](https://mdeagen.shinyapps.io/TidyCatWPD/) tool (link to tutorial on app page).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":288,"value":"### Semantic Annotation of Metadata\n\nWe are trying out a new method for providing context to a dataset through Semantic Data Dictionaries (SDDs). This involves assembling a multi-sheet workbook in Excel or Google Sheets that describes the semantic relationships between fields of a tabular dataset. The SDD exists as a piece of metadata alongside a dataset which defines how the dataset should be interpreted, and is compiled to a SETLr script.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":122,"value":"---\n\n## Appendix\n\n- [Glossary of terms & acronyms](https://observablehq.com/@mdeagen/glossary-of-terms-acronyms)\n- [Curation Job tracking sheet](https://docs.google.com/spreadsheets/d/1k7Sx2OQHwkNQ59uqKm8tzx3700xWdx1SnfkoW3b_eRw/edit#gid=0)\n- [Github repository for curation](https://github.com/mdeagen/mm-curation-notebooks)\n\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W35","stdlib":"1","titling":"auto"}} 32:"$31:body"