32:T4f5,## Todo - Improve initial folder listing if the `findingaid` response no series but files, such as in the [BATO](https://webservices.memorix.nl/archives/findingaid/BATO?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba), [AAND](https://webservices.memorix.nl/archives/findingaid/AAND?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba) or [HEME](https://webservices.memorix.nl/archives/findingaid/HEME?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba) archives. In this case there's no `meta.scans` property to filter the list. The following query might then be useful: `https://webservices.memorix.nl/archives/findingaiditems/AAND/?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba&withscans=1` (I did not use this because it lacks information about the folder). Currently the list of files is filtered for items with scancount > 0, this means that a subfolder containing scans is omitted, such as in the case of the BARB archive: `https://webservices.memorix.nl/archives/tree/BARB/2?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba`. - Improve scan query if more than 50 scans per item (e.g. in the BATO archive). - Remove archives from the list that do not seem to contain scans such as [BAKS](https://webservices.memorix.nl/archives/findingaid/BAKS?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba)33:Tacb,## Problematiek De objecten binnen een `findingaid` bestaan uit verschillende typen: `series`, `subseries`, `file` en `item`. Archieven hebben een verschillende opbouw: - `series > file > item > scans` - `series > subseries > file > item > item > scans` - `series > subseries > file > scans` - `item > items > scans` Nadat de hoofdmappen (`series`) zijn opgevraagd met `findingaid`, kunnen de submappen worden opgevraagd met verschillende API calls: `tree` (voor `subseries` en `files` met children) en `file` (voor `items` en `files` zonder children). De scans van een `file` of `item` kunnen worden opgevraagd met `scans`. Zie ook deze beperkte API documentatie: [Picturae Webkitchen - Memorix Archives](http://demo.webservices.picturae.pro/docs/archives/recipes-and-ingredients/). Het probleem is dat de `tree` API response geen duidelijke indicatie geeft of de children nog méér children hebben. Zitten er `items` in een `file`? Zitten er aanvullende `items` in een `item`? Twee voorbeelden: - [RAVE/7](https://webservices.memorix.nl/archives/file/RAVE/7?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba): File 7.6 en 7.7 bevatten ieder 2 scans. Bij 7.6 vallen die direct onder de `file`; bij 7.7 zitten ze verstopt in twee `items`. Alleen 7.7 is op te vragen met een `tree` query. De `scancount` property van 7.6 is 2 en `scans` bevat een array; bij 7.7 is de `scancount` property 0 en is de array leeg. De `boolean_has_nested_scans` property is bij beide `true` (dus onbruikbaar). - [BERL/4.2.55](https://webservices.memorix.nl/archives/tree/BERL/4.2.55?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba): Deze `file` bevat twee `items`, beide met scancount 0 en zonder `boolean_has_nested_scans` property. 4.2.55.2 bevat echter 259 items mét scans. De enige indicatie hiervoor is de `meta.scans` property in de 4.2.55 response. In de [zoekportal](https://zoeken.hetnieuweinstituut.nl/nl/archieven/file/110318163) zijn deze scans ook verborgen en komen pas bovendrijven na een [zoekopdracht](https://zoeken.hetnieuweinstituut.nl/nl/archieven/file/110319414). De meest betrouwbare manier om te achterhalen of achter een bepaald object scans schuilgaan is de `meta.scans` property die een array bevat van de `children.cpath` properties. Als vervolgens de scanarray leeg is voor het desbetreffende child, moet de tree worden opgevraagd. Het zou echter kunnen dat een `file` scans bevat en óók nog een subitem met scans... (maar hier heb ik geen voorbeeld van gevonden). De array met scans van een file of item bevat maximaal 10 scans. De property `scancount` geeft aan hoeveel scans er in totaal zijn. Als het er meer dan 10 zijn, moeten de betreffende scans apart worden opgevraagd om de volledige array te krijgen. Dit gebeurt met de `getMoreScans` functie.34:T421,// From: https://observablehq.com/@tophtucker/tree-input function treeInput( tree, { name = getName, children = getChildren, open = 0, value = tree } = {} ) { const id = DOM.uid().id; const form = htl.html``; set(value)(); return form; function set(value) { return function () { form.value = value; form.dispatchEvent(new Event("input", { bubbles: true })); }; } function inner(node, { checked = false, open } = {}) { const item = htl.html` ${name(node)}`; return node.children?.length ? htl.html`

${item}

${children(node).map((d) => inner(d, { open: 0 }))}

` : htl.html`

${item}

`; } }35:[{"id":"e241bddeae9b9977cc4949349446d9c0937ee58b5e277b12d4e4c636e723df7ab6472e5886ee00f405559e5760557eee1668df70cd06b76489303e9ee4018860","href":"https://static.observableusercontent.com/files/e241bddeae9b9977cc4949349446d9c0937ee58b5e277b12d4e4c636e723df7ab6472e5886ee00f405559e5760557eee1668df70cd06b76489303e9ee4018860","name":"findingaids.json","type":"application/json","lastModified":1676035900226.705,"size":555739}] 30:{"title":"Georeferencing the collection of The New Institute with Allmaps","cells":[{"id":8,"value":"# Georeferencing the collection of The New Institute with Allmaps\n\nSelect an archive containing scans:","pinned":false,"hidden":false,"mode":"md"},{"id":270,"value":"viewof selectedFindingAidFromList = Inputs.select(findingAidsWithScans.findingAids, {format: x => x.ead.eadheader.eadid + ' ' + x.title.replaceAll(\"&\",\"&\")})","pinned":false,"hidden":false,"mode":"ojs"},{"id":830,"value":"Select an archival folder (only folders containing scans are shown):","pinned":false,"hidden":false,"mode":"md"},{"id":692,"value":"viewof selectedFolder = treeInput(folderListing, {\n children: (d) => d.children,\n name: (d) =>\n htl.html`${d.title}`,\n open: 1,\n value: folderListing.children[0]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1084,"value":"Click the button to fetch scans for the selected (sub)folder. Please try subfolders first to avoid loading large numbers of scans! It is not possible to fetch scans when the entire archive is selected.","pinned":false,"hidden":false,"mode":"md"},{"id":1074,"value":"viewof selectedFolderClicked = Inputs.button(\"Fetch scan tree\", {value: null, reduce: () => selectedFolder})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1262,"value":"Use the tree below to filter the list of scans if needed:","pinned":false,"hidden":false,"mode":"md"},{"id":1167,"value":"viewof selectedFolderScans = treeInput(scansPerChild, {\n children: (d) => d.children,\n name: (d) =>\n htl.html`${d.name} ${d.id_sort ? '(' + d.id_sort + ')' : \"\"}`,\n open: 1,\n value: scansPerChild\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":791,"value":"Found ${listOfScans.length} ${scanOrScans}. Click links to open ${scanOrScans} in the [Allmaps Editor](https://editor.allmaps.org/#/). Or click the button below to copy the list of `info.jsons` to the clipboard and use [this notebook](https://observablehq.com/@allmaps/manifest-from-iiif-images) to generate a IIIF Manifest.","pinned":false,"hidden":false,"mode":"md"},{"id":1226,"value":"Inputs.button(\"Copy info.jsons to clipboard\", {\n value: null,\n reduce: () => navigator.clipboard.writeText(exportScans.join(\"\\n\"))\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1329,"value":"viewof page = Inputs.range([1, listedScansChunks.length], {\n label: \"Page\",\n step: 1,\n value: 1\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1331,"value":"viewof pageSize = Inputs.range([50, 200], {\n label: \"Page size\",\n step: 50,\n value: 100\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1333,"value":"viewof thumbSize = Inputs.range([100, 300], {\n label: \"Thumbnail size\",\n step: 100,\n value: 200\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1343,"value":"$${listedScansChunks.length > 0 ? listedScansChunks[page -1].map(item => ` $\"${item.name}\"$ `).join(\"\\n\") : \"_No images found; please fetch_\"}","pinned":false,"hidden":false,"mode":"md"},{"id":620,"value":"## Folder Tree","pinned":false,"hidden":false,"mode":"md"},{"id":551,"value":"function folderApiUrl (path) {\n const url = findingAidItemQuery.url + selectedFindingAidId + '/' + path + findingAidItemQuery.extension\n return url\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":874,"value":"function filterTreeForItemsContainingScans (tree) {\n // Er staat soms een punt achter een item in de lijst...\n const filesWithScans = Object.keys(tree.meta.scans).map(key => key.replace(/\\.$/,\"\"))\n const children = tree.children\n const results = children.filter(child => filesWithScans.includes(child.cpath))\n return results\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":390,"value":"async function fetchFolderTree(tree) {\n let newArray = []\n for (const folder of tree) {\n if (folder.c_level === \"series\" || folder.c_level === \"subseries\" ) {\n const url = folderApiUrl(folder.cpath)\n const results = await fetchJson(url)\n const filteredResults = filterTreeForItemsContainingScans(results)\n const recursiveResults = await fetchFolderTree(filteredResults)\n if (recursiveResults.length !== 0) {\n newArray.push({\n title:folder.title,\n c_level:folder.c_level,\n cpath:folder.cpath,\n id_sort:folder.id_sort,\n children:recursiveResults\n })\n }\n }\n if (folder.c_level === \"file\") { // && folder.scancount !== 0\n newArray.push({\n title:folder.title,\n c_level:folder.c_level,\n cpath:folder.cpath,\n scancount:folder.scancount,\n id_sort:folder.id_sort,\n scans:folder.scans\n })\n }\n }\n return newArray\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":729,"value":"folderListing = ({title: selectedFindingAidId, c_level: \"findingaid\", children: await fetchFolderTree(selectedFindingAidSeries)})","pinned":false,"hidden":false,"mode":"ojs"},{"id":618,"value":"## Selected Archive","pinned":false,"hidden":false,"mode":"md"},{"id":354,"value":"selectedFindingAidId = selectedFindingAidFromList.ead.eadheader.eadid","pinned":false,"hidden":false,"mode":"ojs"},{"id":308,"value":"selectedFindingAidUrl = findingAidQuery.url + selectedFindingAidId + findingAidQuery.extension","pinned":false,"hidden":false,"mode":"ojs"},{"id":300,"value":"selectedFindingAid = fetchJson(selectedFindingAidUrl)","pinned":false,"hidden":false,"mode":"ojs"},{"id":614,"value":"selectedFindingAidSeries = selectedFindingAid.findingAid.tree\n\n// {\n// const folders = selectedFindingAid.findingAid.tree\n// let folderArray = []\n// for (const folder of folders) {\n// if (folder.c_level === \"file\" && folder.scancount !== 0) {\n// folderArray.push(folder)\n// }\n// else if (folder.c_level === \"series\" || folder.c_level === \"subseries\") {\n// folderArray.push(folder)\n// }\n// }\n// return folderArray\n// }","pinned":false,"hidden":false,"mode":"ojs"},{"id":10,"value":"findingAidQuery = ({\n url: \"https://webservices.memorix.nl/archives/findingaid/\",\n extension:\n \"?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba&withscans=1&limit=100\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":340,"value":"findingAidItemQuery = ({\n url: \"https://webservices.memorix.nl/archives/tree/\",\n extension:\n \"?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba&withscans=1&limit=100\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":622,"value":"## Fetching Scans","pinned":false,"hidden":false,"mode":"md"},{"id":213,"value":"function getScansUrl(cpath) {\n return `https://webservices.memorix.nl/archives/scans/${selectedFindingAidId}/${cpath}?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba&limit=50`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":978,"value":"async function fetchFileTree(file) {\n let newArray = []\n const filteredFile = filterTreeForItemsContainingScans(file)\n for (const item of filteredFile) {\n if (item.scans.length === 0) {\n const url = folderApiUrl(item.cpath)\n const results = await fetchJson(url)\n const recursiveResults = await fetchFileTree(results)\n if (recursiveResults.length !== 0) {\n newArray.push({\n name: item.title,\n c_level: item.c_level,\n cpath: item.cpath,\n scancount: item.scancount,\n id_sort: item.id_sort,\n children: recursiveResults\n })\n // newArray.push(recursiveResults)\n }\n }\n else {\n const scans = await getMoreScans(item)\n newArray.push(scans)\n }\n }\n return newArray\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1039,"value":"async function getMoreScans (item) {\n if (item.scans.length !== 0 && item.scancount <= 10) {\n return {\n name: item.title,\n type: item.c_level,\n path: item.cpath,\n id_sort: item.id_sort,\n scancount: item.scancount,\n scans: item.scans\n }\n }\n else {\n const cpath = item.cpath\n const scansUrl = getScansUrl(cpath)\n const results = await fetchJson(scansUrl)\n return {\n name: item.title,\n type: item.c_level,\n path: item.cpath,\n id_sort: item.id_sort,\n scancount: item.scancount,\n scans: results.scans.scans\n }\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":975,"value":"async function fetchScans (selectedFolder) {\n let result\n if (selectedFolder !== null) {\n if (selectedFolder.c_level === \"file\" && selectedFolder.scans.length > 0) {\n const scans = await getMoreScans(selectedFolder)\n return [scans]\n }\n else if (selectedFolder.c_level !== \"findingaid\") {\n const selectedFolderUrl = folderApiUrl(selectedFolder.cpath)\n const selectedFolderListing = await fetchJson(selectedFolderUrl)\n const scans = fetchFileTree(selectedFolderListing)\n return scans\n }\n else {\n return []\n }\n }\n else {\n return []\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1023,"value":"scansPerChild = {\n if (selectedFolderClicked === null) {\n return ({name: \"Click button above to fetch scans\", children: []})\n }\n else {\n return ({name: selectedFolder.title, id_sort: selectedFolder.id_sort, children: await fetchScans(selectedFolderClicked)})\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1179,"value":"function scansFromSelectedFolders (selectedFolderScans) {\n let scanArray = []\n if (selectedFolderScans.children) {\n const children = selectedFolderScans.children\n for (const child of children) {\n if (child.scans) {\n const scans = child.scans\n scanArray.push(scans)\n }\n else {\n const results = scansFromSelectedFolders(child)\n scanArray.push(results)\n }\n }\n }\n else if (selectedFolderScans.scans) {\n const scans = selectedFolderScans.scans\n scanArray.push(scans)\n }\n return scanArray\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1187,"value":"listOfScans = scansFromSelectedFolders(selectedFolderScans).flat(10)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1338,"value":"listedScansChunks = {\n const arr = listOfScans;\n const chunkSize = pageSize;\n let chunks = [];\n\n for (let i = 0; i < arr.length; i += chunkSize) {\n const chunk = arr.slice(i, i + chunkSize);\n chunks.push(chunk);\n }\n return chunks;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1209,"value":"scanOrScans = listOfScans.length === 1 ? 'scan' : 'scans'","pinned":false,"hidden":false,"mode":"ojs"},{"id":96,"value":"## Base URLs for image services and Allmaps","pinned":false,"hidden":false,"mode":"md"},{"id":163,"value":"iiifBase = ({ url:'https://images.memorix.nl/nai/iiif/' })","pinned":false,"hidden":false,"mode":"ojs"},{"id":165,"value":"thumbBase = ({ url:'https://images.memorix.nl/nai/thumb/200x200/', extension:'.jpg' })\n// Maximum size = 1000x1000","pinned":false,"hidden":false,"mode":"ojs"},{"id":94,"value":"topViewBase = ({ url:'https://images.memorix.nl/nai/topviewjson/beeldbank/' })","pinned":false,"hidden":false,"mode":"ojs"},{"id":167,"value":"dziBase = ({ url:'https://images.memorix.nl/nai/deepzoom/', extension:'.dzi' })","pinned":false,"hidden":false,"mode":"ojs"},{"id":1212,"value":"allmapsEditorBase = ({\n url: \"https://editor.allmaps.org/#/collection?url=\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1291,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":835,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":811,"value":"## Functions","pinned":false,"hidden":false,"mode":"md"},{"id":27,"value":"fetchJson = (url) => fetch(url).then((response) => response.json())","pinned":false,"hidden":false,"mode":"ojs"},{"id":807,"value":"## Exports","pinned":false,"hidden":false,"mode":"md"},{"id":1371,"value":"exportScans = listOfScans.map(\n (scan) => `https://images.memorix.nl/nai/iiif/${scan.id}/info.json`\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":687,"value":"## Imports","pinned":false,"hidden":false,"mode":"md"},{"id":689,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":816,"value":"getName = (d) => d.name","pinned":false,"hidden":false,"mode":"ojs"},{"id":818,"value":"getChildren = (d) => d.children","pinned":false,"hidden":false,"mode":"ojs"},{"id":275,"value":"findingAidsWithScans = FileAttachment(\"findingaids.json\").json()\n// Source: https://webservices.memorix.nl/archives/findingaids?apiKey=cf47033c-74c2-11e4-b116-123b93f75cba&withscans=1&limit=381&order=id_sort+asc","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W35","stdlib":"1","titling":"auto"}