2e:null 2f:["$","title",null,{"children":"Searching for Stereographs in the Getty Collection | David Newbury | Observable"}] 34:T5ee,md` _Before we get started, a note about this post. It's actually an interactive notebook—each of the sections of this document are live code that updates on-the-fly. If you click in the left margin, you'll be able to see the code for that section—a "cell" in Observable terminology. You can tweak it and watch the document update. Thus lets us describe what we're doing and show you how you might do it yourself at the same time!_ ## What's a Stereograph? ![stereograph image of The Temple at Maharraka](https://media.getty.edu/iiif/image/9efbcbf7-7d77-4acc-8d2a-f74378db6a21/full/!700,/0/default.jpg) Stereograph images are the original 3D technology—they're cards that contain a stereo pair of images, taken with two cameras side by side. The two cameras capture images that are slightly offset, just like your eyes, and if you can find a way to view one image with one eye and one with the other, your brain will stitch them together into a 3D view. Getty has a huge photography collection, and as part of that there's a ton of stereograph cards. In this post, we're going to find all of those images. There's not a list of these published, though, so we're going to have to find them within the data that we have about the collection. Getty has recently released several new APIs for their Museum's collection. In this post, I'm going to use three of them—the ActivityStream API, the Linked.Art API, and the IIIF Presentation API. We'll first look at how we can use the ActivityStream API. `35:T60b,paragraph3 = md`The JSON response tells us the API lists ${as_collection.totalItems.toLocaleString()} events, oldest to newest. That's too many items to show at once, so it breaks them into pages, providing links to the to the [first](${ as_collection.first.id }) and [ last](${ as_collection.last.id }) pages. The API provides the both because then we can read oldest-to-newest or newest-to-oldest. We'll try both of those later in this post, and explain when you'd want to do each. The API is [JSON-LD](https://json-ld.org), which means that these documents are [RDF](https://www.w3.org/RDF/), the standard behind Linked Data. There are interesting implications to that, but for what it means that each JSON document you see is going to have a \`context\`, a \`type\`, and an \`id\` key. \`context\` links to a mapping from the JSON to RDF, \`type\` lets us know what kind of document we're looking at, and \`id\` lets us know what the permanent identifier this document is. You'll notice \`id\`'s value is the URL of the document: because tis is Linked Data, you can look up an identifier and should get the resource it identifies. You'll see that the \`first\` and \`last\` properties have an \`id\` and a \`type\`. They're references to ActivityStream [OrderedCollectionPage](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-orderedcollectionpage) documents. ### OrderedCollectionPages As you'd expect from the name, these are pages of event data. We'll start with the [\`first\`](${ as_collection.first.id }) page:`36:T4bb,md` ### AS Objects Each of these ActivityStream Objects describe a single event that took place. In our case, we have an event for every time a record was changed within our API. ~~~json ${JSON.stringify(as_first_page.orderedItems[0], null, 2)} ~~~ One difference here is that while these records all have a Type, the type isn't \`Object\`, as you might expect: instead, this one is of type \`${ as_first_page.orderedItems[0].type }\`. Getty's API uses three types ([Create](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-create), [Update](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-update), & [Delete](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-delete)), which are subclasses of the AS [Object](https://www.w3.org/TR/activitystreams-core/#asobject) type and are defined (along with others) in the [Activity Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/). They're ActivityStream Objects, they're just specific types of Objects. Each also includes a \`created\` key, which records the creation date (\`${ as_first_page.orderedItems[0].created }\`) and a \`object\` key, which is a reference to the record that changed, with \`id\` and \`type\` keys. `37:T4f2,viewof all_hmo_ids = { let currentPage = as_first_page; let currentPageCounter = 0; let seenIds = 0; let ids = new Set(); let deleted_ids = []; let i = this || ids; if (paused) { return yield i; } while (true) { // Add all the items currentPage.orderedItems .filter(obj => obj.object.type == desired_type) .forEach(obj => ids.add(obj.object.id)); // Delete any ones that are of type "Delete". currentPage.orderedItems .filter(obj => obj.type == "Delete") .forEach(obj => ids.delete(obj.object.id)); // Track our progress through the crawl currentPageCounter++; seenIds += currentPage.orderedItems.length; // Build the presentation const element = html`

Crawling page ${currentPageCounter}. We have crawled ${seenIds.toLocaleString()} out of ${as_collection.totalItems.toLocaleString()} total event records.
Thus far, there have been ${ids.size.toLocaleString()} records of type ${desired_type}.

`; element.value = ids; yield element; // Retrieve the next page if (currentPage.next) { currentPage = await crawl(currentPage.next.id); } else { break; } } }38:T6ca,viewof reverse_crawled_HMOs = { let currentPage = await crawl(as_collection.last.id); let currentPageCounter = 0; let seenIds = crawled_data.size; let ids = new Set([...crawled_data]); let foundCrawledDate = false; let lastSeenDate = Date(); // let i = this || ids; // if (paused) return yield i; while (!foundCrawledDate) { // For this way, we want to process the items backwards, so we reverse the items. const reversed_items = [...currentPage.orderedItems].reverse(); reversed_items.forEach(obj => { // If we're processing at an event that was created on or before our last crawl, stop! lastSeenDate = new Date(obj.created); if (lastSeenDate < crawled_date && !foundCrawledDate) { foundCrawledDate = true; } if (obj.object.type == desired_type && !foundCrawledDate) { // Delete any ones that are of type "Delete". // There's a race condition here where, if an object is deleted, then recreated within the // same page, this code won't work. But we don't recreate objects, so it's OK. if (obj.type == "Delete") { ids.delete(obj.object.id); } // Add this item to the list of items we've seen. else { ids.add(obj.object.id); } } }); // Track our progress through the crawl currentPageCounter++; seenIds += currentPage.orderedItems.length; // Build the presentation yield reverse_crawler_output( lastSeenDate, ids.size, currentPageCounter, ids ); // Retrieve the previous page if (currentPage.prev && !foundCrawledDate) { currentPage = await crawl(currentPage.prev.id); } else { break; } } }39:T627,viewof stereoGraphs = { const startTime = Date.now(); let stereographUrls = []; let errors = []; let counter = 0; for (let batch of batchedRequests) { // track how far along we've got counter += chunkSize; // get a batch of data from the API let resultsBatch = await Promise.allSettled( batch.map(async obj => crawl(obj)) ); // process the data resultsBatch.forEach(result => { if (result.status == "fulfilled") { if (isStereograph(result.value)) { stereographUrls.push(result.value.id); } } else { errors.push(result.reason.config.url); } }); // do some time calculations to provide an estimate const currentTime = Date.now(); const elapsedTime = currentTime - startTime; const estimate = (reverse_crawled_HMOs.size / counter) * elapsedTime - elapsedTime; // render the view let view = html`

Processing ${counter} of ${ reverse_crawled_HMOs.size } URLs. We've found ${ stereographUrls.length } stereographs and had ${errors.length} errors.
It's taken ${Math.round( elapsedTime / 1000, 2 )} seconds and will probably take ${Math.round( estimate / (60 * 1000), 2 )} minutes to complete.

${JSON.stringify(errors, null, 2)}

`; view.value = stereographUrls; yield view; } }3a:[{"id":"1259eda395d75c6a874fd56197eb80c3b45e4905e0dc3ec5e8b7415ca2a66a96d5b9fe386003a26929fcb6dd603dda6e57c2740c39fd88fc4ca3884931a9f9c0","href":"https://static.observableusercontent.com/files/1259eda395d75c6a874fd56197eb80c3b45e4905e0dc3ec5e8b7415ca2a66a96d5b9fe386003a26929fcb6dd603dda6e57c2740c39fd88fc4ca3884931a9f9c0","name":"saved_record.json","type":"application/json","lastModified":1598115736099.503,"size":14046119}] 32:{"id":"4e62276620b9ef41","slug":"searching-for-stereographs-in-the-getty-collection","title":"Searching for Stereographs in the Getty Collection","thumbnail":"f7cb9aceece5060a9cc6e65a262079f29a98a7436bad43f36ca28bee540d7ca0","access_level":"public","create_time":"$D2020-08-21T19:08:27.069Z","update_time":"$D2020-09-16T22:22:56.539Z","view_time":null,"first_public_time":"$D2020-08-24T16:47:59.123Z","fork":false,"stars":2,"starred":false,"owner":{"type":"team","id":"33dca7dc15beb7dd","login":"workergnome","name":"David Newbury","avatar_url":"https://avatars-next.observableusercontent.com/f8f77a4b19bf001c6f32ddd77c5f782adec49678da493bd22a2034213e2c7921","tier":"starter_2024"},"version":963,"publish_version":963,"role":null,"authored":false,"authors":[{"type":"individual","id":"bbe74beccd55ec27","login":"workergnome","name":"David Newbury","avatar_url":"https://avatars-next.observableusercontent.com/f8f77a4b19bf001c6f32ddd77c5f782adec49678da493bd22a2034213e2c7921","tier":null}],"members":[{"type":"individual","id":"bbe74beccd55ec27","login":"workergnome","name":"David Newbury","avatar_url":"https://avatars-next.observableusercontent.com/f8f77a4b19bf001c6f32ddd77c5f782adec49678da493bd22a2034213e2c7921","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"Searching for Stereographs in the Getty Collection","cells":[{"id":0,"value":"md`# Searching for Stereographs in the Getty Collection`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":197,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":861,"value":"paragraph_intro_to_activitystreams = md`\n## An Introduction to ActivityStreams\n\n An ActivityStream is a [W3C Standard](https://www.w3.org/TR/activitystreams-core/) that describes a set of activities. This standard emerged from social media—you can think of it as a Twitter feed.^{[[1](#footnote1)]} Each activity has information about what happened, who did it, when they did it, and what they did it to. Getty uses this to create a feed for each API we publish to track when we change the data. You can think of it as if we were tweeting a link to each record that changed! For ${as_collection.summary}, this feed is available at: \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":54,"value":"viewof ACTIVITY_STREAM_URL = {\n // This is the URL of the activity stream\n const url = \"https://data.getty.edu/museum/collection/activity-stream\";\n\n // however, we'd like to present it as a clickable link, so we're also using\n // some Observable magic to create a view of the data.\n const element = html`${url}`;\n \n // setting a value on the element allows us to expose the URL as a variable\n // to the rest of the document, at ACTIVITY_STREAM_URL.\n element.value = url;\n return element;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":68,"value":"paragraph2 = md`\n\n### OrderedCollections\nThat link returns a JSON document describing the ActivityStream [OrderedCollection](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-orderedcollection). an OrderedCollection is the highest-level document for an ActivityStream — it describes the collection of all the events that make up the stream.^{[[2](#footnote2)]} \n\nYou can see that document below. Remember, the code in the next cell is a live view into the data, and is showing it as it was the moment you started reading.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10,"value":"as_collection = await crawl(ACTIVITY_STREAM_URL)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":43,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":95,"value":"as_first_page = await crawl(as_collection.first.id)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":120,"value":"paragraph4 = md`\nThis document looks similar to the previous one. First, it contains two keys that are references to other documents. First, [\\`partOf\\`](${as_first_page.partOf.id}), which is a link back up to the parent OrderedCollection. There's also [\\`next\\`](${as_first_page.partOf.id}), which is the next page in the list. If this wasn't the first page, there'd also be a link to \\`prev\\`, for the previous page.^{[[3](#footnote3)]} \n\nIt also contains an array of ${as_first_page.orderedItems.length} \\`orderedItems\\`, each of which is a single ActivityStream [Object](https://www.w3.org/TR/activitystreams-core/#asobject).\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":829,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":894,"value":"paragraph_hmo = md`\n### API Records\n\n~~~json\n${JSON.stringify(as_first_page.orderedItems[0].object, null, 2)}\n~~~\n\n This event above describes a change to a record about a \\`${\n as_first_page.orderedItems[0].object.type\n }\\`, which is a type from the [Linked.Art](https://linked.art) ontology. Including this type lets us process activities per-resource type, without having to make an additional network call for each one to figure out what resource type it is.^{[[4](#footnote4)]}\n\nThe ActivityStream specification has other parts to it, but these four sections describe all of the ActivityStream API we use, so this is enough for us to move on to actually using it to retrieve information. \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":919,"value":"md`\n\n## Crawling the Stream\n\nTo find all the stereographs, first we need to collect a list of the collection objects that are present in the API. We'll do this by crawling each of the ActivityStream pages and gathering the ids for all the collection objects. Since the ActivityStream records all activities for the API, we will need to filter the API records using their type property. \n\nSince we know that the API uses the [Linked.art](https://linked.art) ontology, we know that collection objects will have the type **HumanMadeObject**. If we wanted instead to crawl to find other types, we could select those types from the dropdown below.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":308,"value":"viewof desired_type = html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":343,"value":"md`\n### Crawling Oldest-to-Newest\n\nCrawling is easy: start at the first page, look at each api record, gather the ids of each one that's a ${desired_type}, then follow the \\`next\\` link to access the next page. We don't know how many pages there will be, but we do know how many records there are, so we can track our progress that way. \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":705,"value":"viewof paused = {\n let element = html``;\n return element;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":256,"value":"$37","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":466,"value":"all_hmo_ids","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":354,"value":"paragraph6 = md`\nCrawling this way works, but is slow: we have to go through many, many pages of records to scan for our records. Thankfully, this is not how we intend for people to use this API. If you remember, the top level collection contained two links: one for [\\`first\\`](${as_collection.first.id}), but also one for [\\`last\\`](${as_collection.last.id}). \n\n### Crawling Newest-to-Oldest\n\nCrawling from [\\`first\\`](${as_collection.first.id}) lets you see everything, but what we've found is that most uses of an API like this are interested in what's _new_. If you're interested in new things, we can assume that you have a old copy of the data around. We can also assume that you know how old that data is.\n\nWe can initialize a javascript [Set](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set) with data captured into a file from a previous crawl:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":381,"value":"crawled_data = new Set(await FileAttachment(\"saved_record.json\").json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":413,"value":"md`We also know that this crawl was completed on ${crawled_date.toLocaleString()}.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":389,"value":"crawled_date = new Date(\"2020-05-12T17:09:10\");\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":418,"value":"md`These two bits of information will let us start a crawl starting from the most recent records, and crawl backwards in time until we hit the point where we ended.` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":420,"value":"$38","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":464,"value":"reverse_crawled_HMOs","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":782,"value":"md`\n## Using the Crawl\n\n...we can parallelize this, so we're going to do this in batches. Each batch is a set of ${chunkSize.toLocaleString()} records, which results in ${batchedRequests.length.toLocaleString()} batches.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":689,"value":"chunkSize = 15","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":683,"value":"batchedRequests = {\n let arr = [...reverse_crawled_HMOs];\n let chunks = [];\n for (let q = 0; q <= Math.floor(arr.length / chunkSize); q++) {\n chunks.push(arr.slice(q * chunkSize, q * chunkSize + chunkSize));\n }\n // return [\n // [\n // \"https://data.getty.edu/museum/collection/object/5aa4059d-b4e4-4a20-9869-8c30df8cd09f\"\n // ]\n // ];\n return chunks; //.slice(4051, 4150);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":582,"value":"md`This crawl takes significantly less time, but still ensures we have an updated set of ids.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":617,"value":"$39","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":664,"value":"stereoGraphs","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":628,"value":"isStereograph = function(data) {\n const aatObjectType = \"http://vocab.getty.edu/aat/300435443\";\n return data.referred_to_by\n .filter(obj => obj.classified_as.some(type => type.id == aatObjectType))\n .some(obj => obj.content == \"Stereograph\");\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":749,"value":"getObjectTypes = function(data) {\n const aatObjectType = \"http://vocab.getty.edu/aat/300435443\";\n return data.referred_to_by\n .filter(obj => obj.classified_as.some(type => type.id == aatObjectType))\n .map(obj => obj.content);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":209,"value":"md`## Footnotes`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":864,"value":"footnote1 = footnote(\n \"paragraph_intro_to_activitystreams\",\n 1,\n \"ActivityStreams, and ActivityPub, which is built on top of it, is actually the standard underneath Mastadon, a federated competitor to Twitter.\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":211,"value":"footnote2 = footnote(\"paragraph2\", 2, \"We're using the Ordered variants because it's important for us to know which events happened before others—our Create events need to happen before our Delete events, for instance.\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":888,"value":"footnote3 = footnote(\n \"paragraph4\",\n 3,\n `This is an example of a hypermedia API. Hypermedia APIs assume that you should be able to 'follow your nose' and navigate an API—you shouldn't have to know implicitly how to get to the next page, for example, that link should be included in the API.`\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":906,"value":"footnote4 = footnote(\n \"paragraph_hmo\",\n 4,\n `There's an interesting tradeoff here: the more information we include here, the easier it is to decide what to do without having to make those extra network calls. However, the ActivityStream is meant to be a permanent record and we don't want to have to go back and update each of these if data changes in the referred-to record. Our compromise was to only include the type of record: we're pretty confident that people won't turn into events, and it's very common to need to process different types of resource differently—for this post, we're only interested in objects, for example.\n
\nWe can't guarantee that about most of the data, though, so to get access to the actual information about that resource, you'd have to dereference the id and parse the data that's returned.\n`\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":168,"value":"X______APPENDICES______X = md`## Appendices`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":24,"value":"axios = require('axios');","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":181,"value":"function toc(selector = \"h1,h2,h3\", heading = \"Table of Contents\") {\n return Generators.observe(notify => {\n let headings = [];\n\n \n function observed() {\n const h = Array.from(document.querySelectorAll(selector));\n if (h.length !== headings.length || h.some((h, i) => headings[i] !== h)) {\n notify(html`${heading}

${DOM.text(h.textContent)}`,\n {onclick: e => (e.preventDefault(), h.scrollIntoView())}\n );\n })}`);\n }\n }\n\n const observer = new MutationObserver(observed);\n observer.observe(document.body, {childList: true, subtree: true});\n observed();\n return () => observer.disconnect();\n });\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":204,"value":"import {footnote} from \"@jashkenas/markdown-style-footnotes\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":227,"value":"css = html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":430,"value":"reverse_crawler_output = function(date, size, pagenum, ids) {\n const element = html`\n
\n We've crawled ${pagenum} pages, and are processing records from ${date.toLocaleString()}.\n
\n We're working backwards toward ${crawled_date.toLocaleString()}.\n
\n Thus far, there have been ${size.toLocaleString()} records of type ${desired_type}.\n
`;\n element.value = ids;\n return element;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":490,"value":"crawl = async function(url) {\n return axios.get(url).then(function(response) {\n return response.data;\n });\n // .catch(err => {\n // return { error: true, url: url, message: err };\n // });\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":177,"value":"\ntable_of_contents = toc(\"h1,h2,h3,h4\");","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":837,"value":"md`\n### About Collection Metadata, Searching, and Complexity\n\nOne of the interesting things we've learned about our collection data is that it's almost impossible to determine what questions people might ask. Particularly as our data gets richer and more complex, the range of possible questions continues to increase. We also continue to learn more things about our collection—every day, we add new data to our collection management system, and thus to our APIs. At this point, a full copy of our the collection data that we serve is ~1.5gb of JSON, and that changes every single day.\n\nThis presents a problem for us as a data providing organization—how do we let people know what's changed? We don't change _everything_ every day, just selected records. The answer that we've come up with is to use a API standard called ActivityStreams.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W3a","stdlib":"1","titling":"auto"}} 33:"$32:body"