2d:null 2e:["$","title",null,{"children":"Text Extraction and Cleaning | Maeve Kane | Observable"}] 33:T630,## Text Cleaning The names included in the transcription are then "cleaned" or regularized using OpenRefine. Seventeenth and eighteenth century names were highly variable in spelling. Some individuals appeared with a differently spelled name each time they appeared in a baptism, even when the register was kept by the same officiant. Officiants who spoke different languages from their congregants--for example, a Dutch officiant recording the names of German, English, French, or Haudenosaunee parents--added further complexity. Robert Livingston, for example, was recorded with Robbert, Robart, and Robbart, Livingstone, Levinston, Levingstone, Livinstein, and many other variations. As much as possible, surnames rather than patronymics are used, and spellings were standardized across congregations with a processing .json file created with OpenRefine to regularize and make reproducible the cleaning steps. In cases where it was not clear if a common name was the same person, further archival research and cross-referencing relationships within the network were used to disambiguate relationships. (For example, if John Smith appeared as a parent with both Mary Smith and Alida Smith in 1740, and Mary Smith again in 1742, there are probably two John Smiths.) Part of the goal of data cleaning is to ensure that individuals only appear once in the network and that each individual is connected to all of the others they were actually connected with because the source/target pairs must be an exact match. Further details on the cleaning process will be added later.34:[] 31:{"id":"ecbd6e2d690acebb","slug":"text-extraction-and-cleaning","title":"Text Extraction and Cleaning","thumbnail":"d082833e6ee35b79484fb28bf5e07f4c64f383cd82f9596c93b29c26f1d8119a","access_level":"public","create_time":"$D2022-06-20T17:15:24.430Z","update_time":"$D2022-09-03T17:11:59.258Z","view_time":null,"first_public_time":"$D2022-06-20T17:27:58.796Z","has_unpublished_edits":null,"fork":false,"stars":4,"starred":false,"owner":{"type":"team","id":"aa9e7a8a29012c93","login":"mkane2","name":"Maeve Kane","avatar_url":"https://avatars-next.observableusercontent.com/10f27e44a96d5e36afe9f72cf1af8b5efa3b0f668929d6850e6e7d05867b630c","tier":"starter_2024"},"version":67,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"c33c0c6945323f32","login":"mkane2","name":"Maeve Kane","avatar_url":"https://avatars-next.observableusercontent.com/10f27e44a96d5e36afe9f72cf1af8b5efa3b0f668929d6850e6e7d05867b630c","tier":null}],"members":[{"type":"individual","id":"c33c0c6945323f32","login":"mkane2","name":"Maeve Kane","avatar_url":"https://avatars-next.observableusercontent.com/10f27e44a96d5e36afe9f72cf1af8b5efa3b0f668929d6850e6e7d05867b630c","tier":null,"role":"editor"}],"comments":[{"id":"460248b213fae8fa","reply_to_id":null,"content":"Note that Observable Inputs are now available in notebooks by default. Instead of importing Table, you can use Inputs.table().","create_time":"$D2022-06-21T14:29:04.965Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"07362516b5994994","login":"mootari","name":"Fabian Iwand","avatar_url":"https://avatars-next.observableusercontent.com/0d7defa821f38094c03bad23b9b360a5364e6e97e21fc238c39ddc48db7994ad","tier":null}}],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":{"id":"bef43644fd2a5fba","slug":"baptismal-networks","title":"Baptismal Networks","owner":{"id":"aa9e7a8a29012c93","login":"mkane2","name":"Maeve Kane"}},"type":"notebook","body":{"title":"Text Extraction and Cleaning","cells":[{"id":0,"value":"# Text Extraction and Cleaning","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"The baptismal networks surveyed here have been drawn from both manuscript and print sources, with a goal of adding further networks in the future. For both manuscript and printed baptismal records, the records are first transcribed (if manuscript) and reformatted where necessary for both manuscript and print sources while preserving the original spelling. This raw data file, which records one event per row and one name per column, copied for later cleaning and manipulation, and the original copy preserved in case later reference is needed.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":32,"value":"## Text Extraction","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":54,"value":"Many scans of published baptismal registers like the NYC Lutheran and Totowa Dutch Reformed ones below have text appropriate for high quality OCR that only needs to be formatted and regularized. This text often has some OCR errors like \"EUsasbeth\" for \"Elizabeth\" or catches formatting marks like | as letter characters I or 1. These can be programmatically corrected or used to re-format the data in tabular form ready for cleaning and analysis.\n\nHowever, off the shelf OCR products often do not capture the structure of the text in such a way that relationships between parents and godparents can be detected. Text often needs to be pre-processed in a structured way so the OCR output can be extracted and cleaned for analysis.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":37,"value":"Sample of New York Dutch Lutheran records, image and OCR text without processing, Year Book of the Holland Society for 1903, 38.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":39,"value":"

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":41,"value":"1704, April 17th, in our church at New York, Antje, y. d. of Pieter and Trintje Van Boschkerk, b. Dec. 26, 1703, at Constapels Hoeck. Witnesses: Major De Bruyn and Gertje, wife of Bernt Christiaensen Van Hoorn. \n\n1704, May ist, in our church at New York, Sophia, y. d. of Johannes Buttinscher and wife EUsabeth, b. at the River Raritons, Dec. 9, 1702. Witnesses: The child's mother, Pieter Woglem and his wife Margareta.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":43,"value":"---\nSample of the Totowa register, image with OCR text before processing or cleaning. 61.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":48,"value":"

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":50,"value":"Ouders 1 Hendrick Francisco I Marytje Snyer I getuygen I Jacob Francisco Antje Snyer\nNaem Elizabeth gebore den 9 July gedoopt den 8 ang: 1756","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":52,"value":"---\n\nSample of the Port Richmond Dutch Reformed records, image with OCR text before and after processing, NYGBR 36: 177.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":56,"value":"

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":58,"value":"| Port Richmond OCR text | Text |\n| --- | --- |\n| Before processing | May 5- 5- 5- 5- 5. 5- 5- 5. Sept. Ephraim Thealer Derek Kroessen Thomas Morgen Lambert Janzen Lambert Gerritzer1 Jores Hooghlandt Johan Staats Hendrick Van Pelt Annetie Mattheus De Decker Hendrick Van Dyck Thomas Possel Pieter Praal Lambert Janzen Thomas Morgen Annetie , Jan Arent C. Aafye Martha WITNESSES Jan Van Pelt Marritie Parat Jan Pieterszen Wog- gelom Hendricka Stroekelf Margrietie SegerGerrissen Eliesabet Ariesmet Nickasa Abraham Winnifret |\n| After processing | 5. Ephraim Thealer Margrietie Seger Gerrissen Eliesabet Ariesmet \\n5. Derck Kroessen Nickasa Cornelis Vielen Caatje Bogardus \\n5. Thomas Morgen Abraham Arent Praal Tryntie Barents \\n5. Lambert Janzen Winnifret Jan Wauterzen Tryntie Hendricksen |","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":35,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":8,"value":"

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":11,"value":"After cleaning, the baptisms were formatted into source/target pairs. Each baptism is composed of multiple source/target pairs. For example, in the baptism of the child Cornelius Sixbury with parents William Sixbury and Mary Sixbury, and godparents John Hough and Mary Phillips, the source/target pairs created are:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":13,"value":"Inputs.table(sixbury)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":19,"value":"This creates a mini network with each baptism. If Mary Sixbury becomes a godmother in another baptism, her participation in both baptisms creates a connection between two mini networks and so grows the network over time. Baptised children are not included in any of the networks because they necessarily do not have any other connections besides the parents and godparents.\n\nThis source/target file is analyzed with Python Network X, a suite of algorithms for network analysis. [Analysis code is available here](https://colab.research.google.com/drive/1-xUNQxVywRwCUHtU-cOZMOQydvzPlKmr?usp=sharing).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":16,"value":"sixbury = [\n { source: \"William Sixbury\", target: \"Mary Sixbury\" },\n { source: \"William Sixbury\", target: \"John Hough\" },\n { source: \"William Sixbury\", target: \"Mary Phillips\" },\n { source: \"Mary Sixbury\", target: \"John Hough\" },\n { source: \"Mary Sixbury\", target: \"Mary Phillips\" },\n { source: \"John Hough\", target: \"Mary Phillips\" }\n]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":29,"value":"","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"}],"files":"$W34","stdlib":"1","titling":"auto"}} 32:"$31:body"