44:["$","title",null,{"children":"3. Data Cleaning | Kalle Westerling | Observable"}] 4c:T4ca,paragraph1 = html`

Digital Humanities scholars have asked for more transparent methods and power structures that categorize data cleaning.^[1]

I wanted to spend quite an extensive time on manual data cleaning because it would allow me to ensure that no valuable data or nuance was lost in the cleaning. While I built an application that assisted me to determine whether there were similar names in the dataset, I maintained, in the end, the authority (as the primary investigator of this project) to make the final decision about which name would be the “canonical” one. A great deal of manual labor went into identifying names of performers and venues that needed to be “normalized.” An example can be seen below, where what can be called “canonical” names were kept in a separate column from the original spellings and names. Google Sheets was chosen as a technology to use in this step, drawing on Matthew Lincoln’s work with the Getty Research Institute’s Provenance Index Remodel project.`4d:T60f,While most of the data cleaning, as should be clear from the table above, aimed at consolidating spellings (see, for example Billy “Senorita” Herrera’s name), there were some other notable features of my cleaning that may need some addressing: - **Inferences of identity**. For example Mike Ames (to Adrian Ames) and Arica (to Arica Wild). This is an example of where domain knowledge is important, and an iterative process is also particularly important. (Which is, in addition, one of the main reasons that it is hard to determine when a dataset is “done.”) - **Inferences of location**. For example Unnamed performer at Bird's Cage 1 (to Bobby Brennan). Here is another example oft the iterative process where I found that Bobby Brennan had been named as present around the same date, which made me infer that at least one of the unnamed performers in another source was also Bobby Brennan. - **Removal of some indeterminacy**. For example Billy ? (into Billy) and Mitzi ? (into Mitzi). - **Splitting of teams**. For example Darryl and Gilbert (into Darryl? Gilbert?) and Mildred and Maurice (into Maurice —). In the first example, I remain unsure whether Darryl or Gilbert from the team was the performer in drag, and the remaining question marks in the dataset indicates that more research is necessary. In the latter example, I have found evidence that it was Maurice from the Mildred and Maurice team who appeared in drag, and thus only Maurice remains (with missing last name indicated by the dash that follows their first name).4e:T6e0,Here, as well, there are some noticeable corrections that need to be addressed: - **Addition of “Theatre.”** For example Binghamton (into Binghamton Theatre). They were added to more easily determine whether a venue was a nightclub, or a vaudeville theatre. Those were the two predominant categories I used in a simplifying manner. Nightclubs, for example, also incorporated “inns,” “cellars,” “halls,” “gardens,” “cafes,” and “taverns.” - **Inferences of names**. For example Villa Venice (from Albert Bouche’s Club). Whenever I had domain knowledge through additional research, such as knowing that Albert Bouche was the proprietor of Villa Venice, I added this information into the dataset, despite the information lacking in the source itself. - **Removal of “The.”** For example Stables (from The Stables). For the dataset to be more lenient but also have a better ability to sort data by the dominant name, I decided to remove any leading “The” from names. - **Moving “Club” to the end of the name**. For example Club Ballyhoo (to Ballyhoo Club). In any given name that contained the word “Club,” I decided to move it to the end, thus facilitating sorting and data processing. - **Need of city specificity**. Try, in the table above to deselect the “Show venue locations” button, and it will reveal one of the major data cleaning jobs that I did on the venue names. Since multiple cities could have clubs with the same name (before or after cleaning), I also needed to connect each venue with each city in which it was located. For example, when you toggle on and off the button, see how the value for the corrected “Ballyhoo Club” changes from _one_ corrected entry, into _three_ corrected entries.4f:[] 48:{"id":"b4992b572ab19d86","slug":"data-cleaning","title":"3. Data Cleaning","thumbnail":"1e72ab5f163b2139022472b9830af67b91f7ffd6b44f9f789d98db9451103f60","access_level":"public","create_time":"$D2021-12-16T22:01:59.391Z","update_time":"$D2022-01-31T08:52:17.544Z","view_time":null,"first_public_time":"$D2022-01-05T11:49:10.573Z","has_unpublished_edits":null,"fork":false,"stars":2,"starred":false,"owner":{"type":"team","id":"32362545e9f3330d","login":"kallewesterling","name":"Kalle Westerling","avatar_url":"https://avatars-next.observableusercontent.com/f759a45dfbde6f2da561a90bcc8852bd0efd30d6a12ae3031f4d0bedc2464b65","tier":"starter_2024","bio":"Research Application Manager with the Alan Turing Institute / PhD from @GC_CUNY / formerly Research Software Engineer @LivingwMachines and @dhinstitutes project mgr","followed_by_you":false,"follows_you":false},"version":354,"publish_version":354,"role":null,"authored":false,"authors":[{"type":"individual","id":"a4098a3e48d12fd5","login":"kallewesterling","name":"Kalle Westerling","avatar_url":"https://avatars-next.observableusercontent.com/f759a45dfbde6f2da561a90bcc8852bd0efd30d6a12ae3031f4d0bedc2464b65","tier":null,"bio":"Research Application Manager with the Alan Turing Institute / PhD from @GC_CUNY / formerly Research Software Engineer @LivingwMachines and @dhinstitutes project mgr","followed_by_you":false,"follows_you":false}],"members":[{"type":"individual","id":"a4098a3e48d12fd5","login":"kallewesterling","name":"Kalle Westerling","avatar_url":"https://avatars-next.observableusercontent.com/f759a45dfbde6f2da561a90bcc8852bd0efd30d6a12ae3031f4d0bedc2464b65","tier":null,"role":"editor"}],"comments":[{"id":"7892d62ee8c68847","reply_to_id":null,"content":"I don't think this rendering like you want.. the first link is a dead end (even though it reads like it points to same location as the second).","create_time":"$D2022-01-25T17:56:35.865Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"fc88ff3942f7785e","login":"josephsmann","name":"Joe","avatar_url":"https://avatars-next.observableusercontent.com/8089eca3d0d1bfc18ba5f7492ea1574d05db45eebbc8e0f3af1c72105b236d4c","tier":null}}],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":{"id":"f56ba7f1433fe336","slug":"dissertation-data","title":"Dissertation data","owner":{"id":"32362545e9f3330d","login":"kallewesterling","name":"Kalle Westerling"}},"type":"notebook","body":{"titling":"auto","title":"3. Data Cleaning","cells":[{"id":0,"value":"# 3. Data Cleaning","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":3,"value":"$4c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":246,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":195,"value":"## Example of Data Cleaning, Part 2: Venue Names","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":149,"value":"Following the same process, and encountering many of the similar concerns above, I cleaned the names of venues in the dataset.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":219,"value":"viewof showCities = Inputs.checkbox([\"Show venue locations\"], {value: [\"Show venue locations\"]})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":162,"value":"viewof selectedVenues = Inputs.table(getAllCorrectedVenues(), {\n layout: 'fixed',\n rows: 21,\n sort: 'Corrected',\n width: {\n 'Corrected': 300\n }\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":266,"value":"Table that illustrates all the cleaned venue names in the dataset, in comparison with what they were before cleaning.","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":202,"value":"$4e","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":289,"value":"footnotes = md`---\n\n## Footnotes`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":292,"value":"footnote1 = footnote(\"paragraph1\", 1, `\nKatie Rawson and Trevor Muñoz, “Against Cleaning,” in Debates in the Digital Humanities 2019, ed. Matthew K. Gold and Lauren F. Klein (Minneapolis, MN and London: University of Minnesota Press, 2019), n.p., https://dhdebates.gc.cuny.edu/read/untitled-f2acf72c-a469-49d8-be35-67f9ac1e3a60/section/07154de9-4903-428e-9c61-7a92a6f22e51.`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":271,"value":"smallNav({\n prev: {url: nav.ingestion.url},\n next: {url: nav.dates.url}\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":275,"value":"scriptDisclaimer","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":153,"value":"getAllCorrectedVenues = () => {\n\n const hasVenues = (d) => {\n return d.Venue !== '—' && d['Normalized Venue'] !== '—';\n }\n\n const isDiffVenues = (d) => {\n return !(d.Venue === d['Normalized Venue']);\n }\n \n const corr = {};\n \n data.filter(d=>d.Year > 1929 && d.Year < 1940).filter(d=>hasVenues(d) && isDiffVenues(d)).forEach(d=>{\n const orig = d.Venue,\n norm = showCities.length ? `${d['Normalized Venue']} (${d['Normalized City']})` : d['Normalized Venue'];\n if (!Object.keys(corr).includes(norm))\n corr[norm] = []\n corr[norm].push(orig);\n corr[norm] = [...new Set(corr[norm])].sort();\n })\n\n return Object.entries(corr).map((d)=>{\n let [norm, origs] = d;\n return {\n 'Names before correction': origs.join(', '),\n 'Corrected': norm,\n };\n })\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":47,"value":"getAllCorrectedNames = () => {\n\n const hasPerformers = (d) => {\n return d.Performer !== '—' && d['Normalized performer'] !== '—';\n }\n\n const isDiffPerformers = (d) => {\n return !(d.Performer === d['Normalized performer']);\n }\n \n const corr = {};\n \n data.filter(d=>d.Year > 1929 && d.Year < 1940).filter(d=>hasPerformers(d) && isDiffPerformers(d)).forEach(d=>{\n const orig = d.Performer,\n norm = d['Normalized performer'];\n if (!Object.keys(corr).includes(norm))\n corr[norm] = []\n corr[norm].push(orig);\n corr[norm] = [...new Set(corr[norm])].sort();\n })\n\n return Object.entries(corr).map((d)=>{\n let [norm, origs] = d;\n return {\n 'Names before correction': origs.join(', '),\n 'Corrected': norm,\n 'Occurrences': getPerformerCount(norm),\n };\n })\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":337,"value":"getPerformerCount = (name) => data.filter(d=>d['Normalized performer'] === name).length","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":24,"value":"import { rawData as data, footnote, nav, navigate, scriptDisclaimer, fullWidth, tachyons, smallNav, ga } from \"@kallewesterling/drag-dataset\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":317,"value":"tachyons","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":312,"value":"fullWidth","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":321,"value":"ga()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W4f","stdlib":"1"}} 24:null