2b:null 2c:["$","title",null,{"children":"2. Data Ingestion and Tagging | Kalle Westerling | Observable"}] 31:T5cf,## Data Ingestion of Historical Newspaper Materials Let us look at an example to understand what the data ingestion process looked like. We will start with an individual advertisement from _The News-Palladium_, July 28, 1936, 10, which advertised Miss Fay Norman’s Gay Boy Revue: ${await FileAttachment("image.png").image()} Advertisement from _The News-Palladium_.

In this advertisement, we can find some nice datapoints, such as: - the **names of performers/persons involved with the troupe** (Fay Norman, Connie Walsh); - the **name of the venue** (Dwan’s Recreational Gardens in the Dwan Hotel); - the **name of the revue** (The Feather Parade); - the **number of other performers** (at least 13–14 unnamed more, granted that the named Connie Walsh and/or Fay Norman were 1–2 of the 15 numbered). A fundamental assumption was made regarding people such as Fay Norman, who were sometimes mentioned as part of the troupe’s name. I always assumed that they were part of the revue as I am privileging relationships, which means that _whether someone like Norman was present or not does not fundamentally matter_. People such as Norman were overseeing the operations, and thus, everyone in the group would know who they were, and they would know all of the other performers. Thus, such names are attached to every single appearance of company who has names of people attached to them. 32:T551,sources = () => { const svg = d3.create('svg') .attr("title", "Illustration") .attr('width', width) .attr('height', 190) const cols = [0, 310]; const rows = [10, 70, 130]; const widths = 300; const heights = 50; const sources = [ { source: 'LOC’s “Chronicling America”', x: cols[0], y: rows[0], },{ source: 'NYPL research collections', x: cols[1], y: rows[0], },{ source: 'ProQuest’s Historical Newspapers', x: cols[0], y: rows[1], },{ source: 'newspapers.com', x: cols[1], y: rows[1], },{ source: 'PQ’s Entertainment Industry Magazines', x: cols[0], y: rows[2], },{ source: 'Other research collections', x: cols[1], y: rows[2], }, ] svg.selectAll('rect') .data(sources) .join('rect') .attr('x', d=>d.x) .attr('y', d=>d.y) .attr('height', heights) .attr('width', widths) .attr('fill', 'lightblue') const text = svg.selectAll('text') .data(sources) .join('text') .text(d => d.source) .attr('x', d=>d.x + widths/2) .attr('y', d=>d.y + heights / 2 + 4) .attr('fill', 'black') .attr('text-anchor', 'middle') .attr('font-size', 15) return svg.node() }33:[{"id":"4ac0a852e660f4bb6b9a53bce60c7e37f595574dbf4efe9162fa4a79f4399b89527f316949adaa9279e3d4ddcead10132adf7251f2c004d8f5e084c3f1662447","href":"https://static.observableusercontent.com/files/4ac0a852e660f4bb6b9a53bce60c7e37f595574dbf4efe9162fa4a79f4399b89527f316949adaa9279e3d4ddcead10132adf7251f2c004d8f5e084c3f1662447","name":"image.png","type":"image/png","lastModified":1639674624657.014,"size":59106},{"id":"f94af92ab541fff47bd558f5b44a1da9e9bfbf6309b3a03eefdfa0e26c71a9e5a2e47c1ce5b93d80d5910005f47c3a7e6a0501e8adc58d91d06cd4701c0d247e","href":"https://static.observableusercontent.com/files/f94af92ab541fff47bd558f5b44a1da9e9bfbf6309b3a03eefdfa0e26c71a9e5a2e47c1ce5b93d80d5910005f47c3a7e6a0501e8adc58d91d06cd4701c0d247e","name":"image@1.png","type":"image/png","lastModified":1639675814132.346,"size":236865},{"id":"e0d015aeacfa140947e890d037d1945bafb1b4216fc4ecd0dfe223e4596c6d9f7fd20d936e8428fbaed15e5f25115d206f3442767dc8f8abe6cef641721cdd0f","href":"https://static.observableusercontent.com/files/e0d015aeacfa140947e890d037d1945bafb1b4216fc4ecd0dfe223e4596c6d9f7fd20d936e8428fbaed15e5f25115d206f3442767dc8f8abe6cef641721cdd0f","name":"image@2.png","type":"image/png","lastModified":1639676258920.081,"size":291940}] 2f:{"id":"32b96d00ec5c64eb","slug":"data-ingestion-and-tagging","title":"2. Data Ingestion and Tagging","thumbnail":"470a3fc1852f01a3695e796bcb60d4509e29dc91492c8d470a61d5b6429856c5","access_level":"public","create_time":"$D2021-12-16T16:33:40.164Z","update_time":"$D2022-01-16T10:54:24.714Z","view_time":null,"first_public_time":"$D2022-01-05T11:42:35.937Z","fork":false,"stars":4,"starred":false,"owner":{"type":"team","id":"32362545e9f3330d","login":"kallewesterling","name":"Kalle Westerling","avatar_url":"https://avatars-next.observableusercontent.com/f759a45dfbde6f2da561a90bcc8852bd0efd30d6a12ae3031f4d0bedc2464b65","tier":"starter_2024"},"version":823,"publish_version":823,"role":null,"authored":false,"authors":[{"type":"individual","id":"a4098a3e48d12fd5","login":"kallewesterling","name":"Kalle Westerling","avatar_url":"https://avatars-next.observableusercontent.com/f759a45dfbde6f2da561a90bcc8852bd0efd30d6a12ae3031f4d0bedc2464b65","tier":null}],"members":[{"type":"individual","id":"a4098a3e48d12fd5","login":"kallewesterling","name":"Kalle Westerling","avatar_url":"https://avatars-next.observableusercontent.com/f759a45dfbde6f2da561a90bcc8852bd0efd30d6a12ae3031f4d0bedc2464b65","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":1,"collection":{"id":"f56ba7f1433fe336","slug":"dissertation-data","title":"Dissertation data","owner":{"id":"32362545e9f3330d","login":"kallewesterling","name":"Kalle Westerling"}},"type":"notebook","body":{"title":"2. Data Ingestion and Tagging","cells":[{"id":671,"value":"# 2. Data Ingestion and Tagging","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":678,"value":"

Over several years, I collected, closely read, tagged, and structured approximately ${numberOfSources} unique sources into a large-scale dataset. The sources consisted mostly of advertisements, short notices, and photos from newspapers. Such primary sources were found in a number of different places, but one of the most important details to point out here is that a project like mine (and most certainly, at this scale) would not have been possible prior to the digitization of newspaper collections and the introduction of constantly improving OCR technologies.

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":0,"value":"paragraph1 = md`I closely read the material, tagged it, and structured it in a database format. The database was built as a Google Sheet, drawing on recent scholarly observation about the tool’s accessibility and collaborative aspects for future expansion and availability of the dataset. Simply put, the dataset was, in this way, made open for exploration by anyone with an Internet browser.^{[1](#footnote1)} Another beneficial feature of Google Sheets is that it bridges skills and technical knowledge gaps within the research community. In place of heavy databases and high engagement thresholds through technological skills, Google Sheets offers a lightweight, accessible infrastructure that makes the data readily available in several formats. That means that the research community can retrieve and examine and use it for any number of future tasks.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":684,"value":"The most prominent base assumption in any of my work with the sources was that none of the sources were “lying” or “fake.” Rather, they were taken at face value as my interest here was in the relationships between the artists, the cooccurrence of their performances. If the source was “lying,” it likely did so with some base in a “truthy” idea that the artists were related in one way or another, even if it was not the case that they performed together at that particular given time.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":510,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":130,"value":"md`\n## Sources\n\nThe databases and collections that were consulted included the following:\n\n${sources()}\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":507,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":164,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":495,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":343,"value":"### Tagging and Inferring\n\nAll of the datapoints for each individual source were collected into a data sheet that was structured, in its most basic conception, in the following format:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":247,"value":"Inputs.table(exampleData, tableSettings)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":277,"value":"You may ask yourself where the **Benton Harbor, MI** comes from, in the `location` column. The location of any given venue in the dataset was inferred from research on each individual venue. In this case, since we knew that the the Dwan’s Recreational Gardens was located in the Dwan Hotel, it was fairly straightforward to find other sources that pointed to the location of the venue:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":284,"value":"FileAttachment(\"image@1.png\").image()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":503,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":289,"value":"### Repeat the Process — Creating a Dataset\n\nThis process was repeated, manually, with all the ${numberOfSources} sources in the dataset.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":330,"value":"$${await FileAttachment(\"image@2.png\").image()}\n\nExamples of other advertisements, and smaller and longer articles, as well as photos of performers participating in the Gay Boy Revue, from the example above.\n\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":364,"value":"If we now look at the ${formatComma(counters.selectedSources)} sources in the current dataset that mentions the Gay Boy Revue, you can see that it contains many more names (${formatComma(counters.performers)}), venues (${formatComma(counters.venues)}), and locations (${formatComma(counters.locations)}).\n\nHere is the complete table of the current datapoints that mention Fay Norman’s Gay Boy Revue, loaded from the source. Note that the table is sortable, which means that you can click the headers to order the data differently. It has also been outfitted with links for each source that has been tagged with the correct ID for the image on newspapers.com for easy access to the source itself (granted that you have access to the website).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":392,"value":"Inputs.table(selectDataset, tableSettings)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":775,"value":"Table that illustrates all the connections tagged in each of the sources that mention Fay Norman’s Gay Boy Revue.","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":625,"value":"As you can see in the table above, the data has been cleaned somewhat—especially in the revue name, where the their names have been standardized across all of the datapoints. The original metadata is preserved in the original dataset, but the filtered and cleaned data appears here. The [next section](https://observablehq.com/d/b4992b572ab19d86?collection=@kallewesterling/dissertation-data) explains the data cleaning more in detail.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":692,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":652,"value":"footnotes = md`---\n\n## Footnotes`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":654,"value":"footnote1 = footnote(\"paragraph1\", 1, `\nMatthew Lincoln, “Best Practices for Using Google Sheets in Your Data Project,” Matthew Lincoln’s Personal Blog, March 26, 2018, https://matthewlincoln.net/2018/03/26/best-practices-for-using-google-sheets-in-your-data-project.html. See also discussions about the inclusive and accessible aspect of Google Sheets for building collaborative datasets in Melinda A. Cro and Sara K. Kearns, “Developing a Process-Oriented, Inclusive Pedagogy: At the Intersection of Digital Humanities, Second Language Acquisition, and New Literacies,” DHQ: Digital Humanities Quarterly 14, no. 1 (2020): n.p., http://www.digitalhumanities.org/dhq/vol/14/1/000443/000443.html.\n \n `)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":710,"value":"smallNav({\n prev: {url: nav.intro.url},\n next: {url: nav.cleaning.url}\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":145,"value":"scriptDisclaimer","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":742,"value":"fullWidth","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":755,"value":"tachyons","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":540,"value":"exampleData = {\n const sliced = selectDataset.filter(d=>(d.date === \"1936-07-28\" || d.date === \"1936-08-01\") && d.venue === \"Dwan's Recreation Gardens\" && !d.performer.includes('Unnamed')).slice(0,5);\n sliced[sliced.length-1] = Object.assign(sliced[sliced.length-1], {\n performer: '... etc ...'\n })\n return sliced\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":7,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":578,"value":"numberOfSources = formatComma(Math.ceil([...new Set(data.map(d=>d['Source clean']))].length/100)*100)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":572,"value":"selectDataset = data\n .filter(d => d.Year > 1929 && d.Year < 1940)\n .filter(d => d['Normalized Revue Name'] == \"Fay Norman's Gay Boy Revue\" && \n d['Normalized performer'] !== '—')\n .map(d => {\n let link = '';\n if (d['Newspaper_ID'])\n link += `https://www.newspapers.com/image/${d['Newspaper_ID']}`\n if (!link && d['EIMA_ID'])\n link += `https://search.proquest.com/docview/${d['EIMA_ID']}`\n return {\n date: d['Date'],\n performer: d['Normalized performer'],\n revue: 'Fay Norman’s Gay Boy Revue',\n venue: d['Normalized Venue'].replace('—', ''),\n location: d['Normalized City'].replace('—', ''),\n source: d['Source clean'],\n link\n }\n })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":406,"value":"counters = ({\n selectedSources: selectDataset.length,\n performers: new Set(selectDataset.map(d=>d.performer)).size,\n venues: new Set(selectDataset.map(d=>d.venue)).size,\n dates: new Set(selectDataset.map(d=>d.date)).size,\n locations: new Set(selectDataset.map(d=>d.location)).size\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":516,"value":"tableSettings = ({\n columns: [\n 'date', 'performer', 'revue', 'venue', 'location', 'source'\n ],\n sort: 'date', \n layout: 'auto', \n disable: true, \n rows: 30, \n multiple: false,\n format: {\n source: d => {\n const o = selectDataset.find(e=>e.source === d && e.link);\n if (o) {\n return html`${d}`;\n } else {\n return d;\n }\n }\n }\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":568,"value":"import { rawData as data, formatComma, footnote, nav, navigate, fullWidth, scriptDisclaimer, smallNav, tachyons, ga } from '@kallewesterling/drag-dataset'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":820,"value":"ga()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"}],"files":"$W33","stdlib":"1","titling":"auto"}} 30:"$2f:body"