3e:null 3f:["$","title",null,{"children":"23andme Manifest | lorenasandoval88 | Observable"}] 45:T7f0,

Workflow

PGP File Collection → Chip Manifests → Overlap & Fingerprints → Classification Results

46:T5ea9,# Running the 23andMe Chip Manifest Pipeline in Google Cloud ## Overview This pipeline builds genomic-coordinate manifests for known **23andMe v2, v3, v4, and v5** files stored in Google Cloud Storage. The manifests are the first step in the chip-classification workflow: ```text Known 23andMe files │ ▼ Build chip manifests (v2 / v3 / v4 / v5) │ ▼ Core coordinate sets │ ▼ Compare overlap across chips │ ▼ Create chip-specific fingerprints │ ▼ Apply fingerprints to unlabeled files │ ▼ Infer chip version ``` The manifest stage answers: > **Which genomic coordinates are consistently present in known files from each 23andMe chip version?** The resulting core-coordinate sets are later compared to identify coordinates that distinguish one chip version from the others. --- # 1. Reference Files The current Build 37 reference cohorts are: | Chip | Known reference files | Files required for 95% core | |---|---:|---:| | v2 | 3 | 3 | | v3 | 17 | 17 | | v4 | 74 | 71 | | v5 | 63 | 60 | The default threshold is: ```text MIN_PRESENCE = 0.95 ``` A coordinate is therefore included in the **core manifest** when it is present in at least 95% of successfully processed files for that chip. ### Important for v2 Only **3 standalone v2 Build 37 reference files** are currently available. Therefore: ```text 3 / 3 = 100% 2 / 3 = 66.7% ``` With a 95% threshold: ```text v2 core = coordinates present in all 3 files ``` So for the current v2 reference set: ```text 95% core = 100% intersection ``` This is expected because of the small number of v2 reference files. --- # 2. What `index.js` Does The same `index.js` program is run separately for: ```text v2 v3 v4 v5 ``` For each chip version, the program: 1. Reads the consolidated participant metadata from Google Cloud Storage. 2. Finds valid 23andMe files. 3. Selects files for the requested chip version. 4. Requires the requested genome build. 5. Excludes ambiguous or combined-chip files. 6. Keeps one file per participant by default. 7. Downloads each selected genotype file from GCS. 8. Reads `.txt`, `.txt.gz`, `.gz`, or `.zip` files. 9. Parses chromosome and genomic position. 10. Normalizes chromosome names. 11. Deduplicates repeated physical coordinates within each file. 12. Counts how many reference files contain each coordinate. 13. Calculates coordinate prevalence. 14. Creates: - union manifests, - 95% core manifests, - 100% intersection manifests. 15. Saves all output files to Google Cloud Storage. --- # 3. Project Folder The Cloud Shell project folder is: ```text ~/v3v4v5-manifest ``` The folder name is retained from the original v3/v4/v5 pipeline, but the updated code also supports **v2**. Recommended structure: ```text v3v4v5-manifest/ ├── Dockerfile ├── index.js ├── package.json └── package-lock.json ``` `index.js` contains the manifest-building code. The output files are not stored in this local folder. They are written directly to: ```text gs://all23_0710/chip_manifests/ ``` --- # 4. Input Metadata The recommended metadata source is: ```text gs://all23_0710/updatedParticipantsList/ participants_1020_consolidated_with_build_07102026_2053.json ``` Set it with: ```bash export INPUT_METADATA_JSON="updatedParticipantsList/participants_1020_consolidated_with_build_07102026_2053.json" ``` The metadata is used to identify: - participant ID, - data type, - valid 23andMe status, - chip version from the filename, - genome build, - GCS path, - publication date, - file size. The script can inspect: ```text participant.files[] ``` and: ```text participant.files[].genomeBuildFiles[] ``` --- # 5. Chip-Version Selection The pipeline now accepts: ```text v2 v3 v4 v5 ``` Examples: ```bash export CHIP_VERSION="v2" ``` ```bash export CHIP_VERSION="v3" ``` ```bash export CHIP_VERSION="v4" ``` ```bash export CHIP_VERSION="v5" ``` The script is run separately for each chip version. --- # 6. Filename Rules Reference files must contain an unambiguous full-chip filename pattern. Accepted examples: ```text genome_person_v2_Full_20180101120000.txt genome_person_v3_Full_20140101120000.txt genome_person_v4_Full_20180705194728.txt genome_person_v5_Full_20260429131650.txt ``` The recognized patterns are: ```text _v2_Full _v3_Full _v4_Full _v5_Full ``` Combined or ambiguous files are excluded. Examples: ```text genome_person_v2_v3_Full.txt genome_person_v3_v5_Full.txt genome_person_v3_v4_Full.txt genome_person_v3_v4_v5_Full.txt genome_person_combined_v5_Full.txt ``` These files are useful for later testing, but they should **not** be used to construct a clean single-chip reference manifest. --- # 7. File-Selection Requirements By default, a file must satisfy all of the following: ```text valid23File === true ``` and: ```text dataType contains "23andMe" ``` and: ```text genomeBuild matches requested build ``` and: ```text filename contains one unambiguous chip version ``` and: ```text chip version matches CHIP_VERSION ``` and: ```text GCS genotype object exists in a supported format ``` Supported formats include: ```text .txt .txt.gz .gz .zip ``` --- # 8. One File Per Participant Default: ```text ONE_FILE_PER_PARTICIPANT = true ``` The program keeps one file for each: ```text participantId | chipVersion | genomeBuild ``` This prevents one participant with multiple copies of the same chip file from contributing more weight to the manifest than another participant. Additional files are recorded in: ```text excluded_input_files.json ``` with reasons such as: ```text additional_file_for_same_participant ``` --- # 9. Genome Build The current manifests use: ```text GENOME_BUILD = 37 ``` Recognized Build 37 labels include: ```text 37 GRCh37 hg19 build 37 assembly 37 ``` The genome build is taken from the consolidated metadata rather than inferred from the genotype coordinates. --- # 10. Variant Identity The manifest is based on **physical genomic position**, not marker ID. The primary identity is: ```text chromosome:position ``` Example: ```text 18:29172937 ``` Suppose one file contains: ```text i3002758 18 29172937 ``` and another contains: ```text rs28933979 18 29172937 ``` These are treated as the same physical genomic coordinate: ```text 18:29172937 ``` The coordinate is counted once, while both marker IDs can still be retained as aliases. Conceptually: ```json { "chromosome": "18", "position": 29172937, "coordinate": "18:29172937", "rsidAliases": [ "rs28933979" ], "internalIdAliases": [ "i3002758" ] } ``` This is important because the goal is to identify the physical genomic positions represented by each chip, rather than depend on how a marker happens to be labeled in a particular 23andMe export. --- # 11. Chromosome Normalization Accepted chromosomes are: ```text 1-22 X Y XY MT ``` A leading: ```text chr ``` is removed. Examples: ```text chr1 → 1 chr22 → 22 chrX → X ``` The following mitochondrial labels are normalized to: ```text MT ``` from: ```text M MTR MITO ``` Invalid chromosome values are skipped. --- # 12. Per-File Coordinate Deduplication Within each genotype file, coordinates are keyed by: ```text chromosome:position ``` For example, if a file contains: ```text i3002758 18 29172937 rs28933979 18 29172937 ``` the coordinate: ```text 18:29172937 ``` is counted only once for that file. The associated marker IDs remain available as aliases. This prevents duplicate rows from artificially increasing coordinate prevalence. --- # 13. Coordinate Prevalence For each genomic coordinate, the script records: ```text filesPresent ``` which means: ```text number of successfully processed files containing that coordinate ``` and: ```text fileCount ``` which means: ```text total number of successfully processed files used for that chip manifest ``` Presence is calculated as: ```text presenceRate = filesPresent / fileCount ``` and: ```text presencePercent = presenceRate × 100 ``` Example: ```json { "chromosome": "1", "position": 123456, "coordinate": "1:123456", "filesPresent": 19, "fileCount": 20, "presenceRate": 0.95, "presencePercent": 95, "manifestClass": "core" } ``` This coordinate occurs in: ```text 19 of 20 files ``` and therefore meets the default: ```text 95% core threshold ``` --- # 14. Manifest Types Three main coordinate sets are created. ## Union The union contains every genomic coordinate observed in at least one successfully processed reference file. ```text presence > 0% ``` ## Core The core contains coordinates satisfying: ```text presenceRate >= MIN_PRESENCE ``` With the current setting: ```text presenceRate >= 0.95 ``` Therefore: ```text Core = coordinates present in at least 95% of files ``` ## Intersection The intersection contains coordinates satisfying: ```text filesPresent === successfulFileCount ``` Therefore: ```text Intersection = coordinates present in 100% of files ``` Conceptually: ```text Union └── every observed coordinate Core └── coordinates present in ≥95% of files Intersection └── coordinates present in 100% of files ``` --- # 15. Main Environment Variables ## `CHIP_VERSION` Allowed: ```text v2 v3 v4 v5 ``` --- ## `GENOME_BUILD` Current: ```text 37 ``` --- ## `MIN_PRESENCE` Current: ```text 0.95 ``` Meaning: ```text 95% presence required for the core manifest ``` --- ## `MANIFEST_MODE` Current: ```text core ``` Allowed: ```text union core intersection ``` With: ```text MANIFEST_MODE=core MIN_PRESENCE=0.95 ``` the main: ```text manifest.json ``` contains the 95% core manifest. --- ## `REQUIRE_VALID_23_FILE` Current: ```text true ``` Requires: ```text valid23File === true ``` --- ## `REQUIRE_23ANDME_TYPE` Current: ```text true ``` Requires `dataType` to contain: ```text 23andMe ``` --- ## `ONE_FILE_PER_PARTICIPANT` Current: ```text true ``` --- ## `FILE_LIMIT` Current: ```text 0 ``` Meaning: ```text process all matching files ``` A positive number can be used for testing. Example: ```bash export FILE_LIMIT="2" ``` --- ## `INCLUDE_NON_RS_IDS` Current: ```text true ``` This retains non-rs marker labels such as: ```text i4000013 ``` in addition to true rsIDs such as: ```text rs12345 ``` --- # 16. Build the Google Cloud Container Enter the project directory: ```bash cd ~/v3v4v5-manifest ``` Set the project and region: ```bash export PROJECT_ID="$(gcloud config get-value project)" export REGION="us-east4" ``` Check: ```bash echo "PROJECT_ID=$PROJECT_ID" echo "REGION=$REGION" ``` Build the container: ```bash gcloud builds submit \ --project="$PROJECT_ID" \ --tag="gcr.io/$PROJECT_ID/v3v4v5-manifest:latest" \ . ``` You only need to rebuild the container after changing: ```text index.js Dockerfile package.json dependencies ``` Once built, the same container image can be used for v2, v3, v4, and v5. --- # 17. Run the v2 Manifest ## Deploy ```bash gcloud run jobs deploy manifest-v2-95pct \ --image="gcr.io/$PROJECT_ID/v3v4v5-manifest:latest" \ --region="$REGION" \ --project="$PROJECT_ID" \ --tasks=1 \ --parallelism=1 \ --memory=8Gi \ --cpu=2 \ --task-timeout=3600s \ --max-retries=1 \ --set-env-vars="BUCKET_NAME=all23_0710,INPUT_METADATA_JSON=updatedParticipantsList/participants_1020_consolidated_with_build_07102026_2053.json,CHIP_VERSION=v2,GENOME_BUILD=37,MIN_PRESENCE=0.95,MANIFEST_MODE=core,REQUIRE_VALID_23_FILE=true,REQUIRE_23ANDME_TYPE=true,ONE_FILE_PER_PARTICIPANT=true,FILE_LIMIT=0,NODE_OPTIONS=--max-old-space-size=6144" ``` The v2 job uses: ```text 8 GiB Cloud Run memory 6 GiB Node.js heap ``` because generation of the large manifest JSON structures can require more than Node's default heap limit. ## Execute ```bash gcloud run jobs execute manifest-v2-95pct \ --region="$REGION" \ --project="$PROJECT_ID" \ --wait ``` Expected reference files: ```text 3 ``` Output: ```text gs://all23_0710/chip_manifests/v2_build37_95pct/ ``` --- # 18. Run the v3 Manifest ## Deploy ```bash gcloud run jobs deploy manifest-v3-95pct \ --image="gcr.io/$PROJECT_ID/v3v4v5-manifest:latest" \ --region="$REGION" \ --project="$PROJECT_ID" \ --tasks=1 \ --parallelism=1 \ --memory=8Gi \ --cpu=2 \ --task-timeout=3600s \ --max-retries=1 \ --set-env-vars="BUCKET_NAME=all23_0710,INPUT_METADATA_JSON=updatedParticipantsList/participants_1020_consolidated_with_build_07102026_2053.json,CHIP_VERSION=v3,GENOME_BUILD=37,MIN_PRESENCE=0.95,MANIFEST_MODE=core,REQUIRE_VALID_23_FILE=true,REQUIRE_23ANDME_TYPE=true,ONE_FILE_PER_PARTICIPANT=true,FILE_LIMIT=0,NODE_OPTIONS=--max-old-space-size=6144" ``` ## Execute ```bash gcloud run jobs execute manifest-v3-95pct \ --region="$REGION" \ --project="$PROJECT_ID" \ --wait ``` Expected reference files: ```text 17 ``` Output: ```text gs://all23_0710/chip_manifests/v3_build37_95pct/ ``` --- # 19. Run the v4 Manifest ## Deploy ```bash gcloud run jobs deploy manifest-v4-95pct \ --image="gcr.io/$PROJECT_ID/v3v4v5-manifest:latest" \ --region="$REGION" \ --project="$PROJECT_ID" \ --tasks=1 \ --parallelism=1 \ --memory=4Gi \ --cpu=2 \ --task-timeout=3600s \ --max-retries=1 \ --set-env-vars="BUCKET_NAME=all23_0710,INPUT_METADATA_JSON=updatedParticipantsList/participants_1020_consolidated_with_build_07102026_2053.json,CHIP_VERSION=v4,GENOME_BUILD=37,MIN_PRESENCE=0.95,MANIFEST_MODE=core,REQUIRE_VALID_23_FILE=true,REQUIRE_23ANDME_TYPE=true,ONE_FILE_PER_PARTICIPANT=true,FILE_LIMIT=0" ``` ## Execute ```bash gcloud run jobs execute manifest-v4-95pct \ --region="$REGION" \ --project="$PROJECT_ID" \ --wait ``` Expected reference files: ```text 74 ``` Output: ```text gs://all23_0710/chip_manifests/v4_build37_95pct/ ``` --- # 20. Run the v5 Manifest ## Deploy ```bash gcloud run jobs deploy manifest-v5-95pct \ --image="gcr.io/$PROJECT_ID/v3v4v5-manifest:latest" \ --region="$REGION" \ --project="$PROJECT_ID" \ --tasks=1 \ --parallelism=1 \ --memory=4Gi \ --cpu=2 \ --task-timeout=3600s \ --max-retries=1 \ --set-env-vars="BUCKET_NAME=all23_0710,INPUT_METADATA_JSON=updatedParticipantsList/participants_1020_consolidated_with_build_07102026_2053.json,CHIP_VERSION=v5,GENOME_BUILD=37,MIN_PRESENCE=0.95,MANIFEST_MODE=core,REQUIRE_VALID_23_FILE=true,REQUIRE_23ANDME_TYPE=true,ONE_FILE_PER_PARTICIPANT=true,FILE_LIMIT=0" ``` ## Execute ```bash gcloud run jobs execute manifest-v5-95pct \ --region="$REGION" \ --project="$PROJECT_ID" \ --wait ``` Expected reference files: ```text 63 ``` Output: ```text gs://all23_0710/chip_manifests/v5_build37_95pct/ ``` --- # 21. Final Google Cloud Storage Structure After all four jobs complete: ```text gs://all23_0710/chip_manifests/ │ ├── v2_build37_95pct/ ├── v3_build37_95pct/ ├── v4_build37_95pct/ └── v5_build37_95pct/ ``` Each chip directory contains the same general output structure: ```text manifest.json variant_manifest_union.json variant_manifest_core.json variant_manifest_intersection.json coordinate_manifest.json coordinates_core.json coordinates_intersection.json rsid_manifest.json rsid_observation_manifest.json rsids_core.json rsids_intersection.json marker_id_manifest.json marker_id_observation_manifest.json marker_ids_union.json marker_ids_core.json marker_ids_intersection.json selected_input_files.json excluded_input_files.json processed_files.json failed_files.json summary.json ``` --- # 22. What the Main Output Files Mean ## `manifest.json` This is the main manifest selected by: ```text MANIFEST_MODE ``` With: ```text MANIFEST_MODE=core MIN_PRESENCE=0.95 ``` it contains the detailed 95% core manifest. --- ## `variant_manifest_union.json` Detailed records for: ```text all observed genomic coordinates ``` --- ## `variant_manifest_core.json` Detailed records for: ```text coordinates present in at least 95% of successful files ``` --- ## `variant_manifest_intersection.json` Detailed records for: ```text coordinates present in 100% of successful files ``` --- ## `coordinate_manifest.json` Coordinate-level records including prevalence information. --- ## `coordinates_core.json` A compact list of core coordinates such as: ```text 1:752566 1:768448 1:779322 ... ``` This is the most important compact file for the next **fingerprint-building** step. --- ## `coordinates_intersection.json` Compact coordinates observed in every successfully processed reference file. --- # 23. Audit Files The pipeline also records which files were included or excluded. ## `selected_input_files.json` Files selected after metadata filtering and deduplication. ## `excluded_input_files.json` Files rejected during selection. Possible reasons include: ```text valid23File_is_not_true dataType_missing dataType_is_not_23andMe genome_build_mismatch_or_unknown chip_version_mismatch filename_contains_combined filename_contains_multiple_chip_versions additional_file_for_same_participant ``` ## `processed_files.json` Files successfully parsed and actually used to build the manifest. ## `failed_files.json` Selected files that could not be processed. --- # 24. Summary File Each chip directory contains: ```text summary.json ``` The summary records information such as: - chip version, - genome build, - output path, - selection settings, - metadata counts, - selected file count, - successfully processed file count, - unique participant count, - failed file count, - minimum presence threshold, - manifest mode, - union counts, - core counts, - intersection counts, - coordinate counts, - marker-ID counts, - rsID counts, - internal-ID counts, - output file paths. --- # 25. Check a Completed Manifest Example for v2: ```bash gcloud storage ls \ gs://all23_0710/chip_manifests/v2_build37_95pct/ ``` View its summary: ```bash gcloud storage cat \ gs://all23_0710/chip_manifests/v2_build37_95pct/summary.json ``` For another chip, change: ```text v2_build37_95pct ``` to: ```text v3_build37_95pct v4_build37_95pct v5_build37_95pct ``` --- # 26. Primary Files Used for Fingerprint Construction The most important outputs for the next stage are: ```text gs://all23_0710/chip_manifests/v2_build37_95pct/coordinates_core.json gs://all23_0710/chip_manifests/v3_build37_95pct/coordinates_core.json gs://all23_0710/chip_manifests/v4_build37_95pct/coordinates_core.json gs://all23_0710/chip_manifests/v5_build37_95pct/coordinates_core.json ``` These contain the physical genomic coordinates consistently observed on each chip. The detailed versions are: ```text v2_build37_95pct/variant_manifest_core.json v3_build37_95pct/variant_manifest_core.json v4_build37_95pct/variant_manifest_core.json v5_build37_95pct/variant_manifest_core.json ``` --- # 27. Manifest → Fingerprint The manifest and fingerprint are different stages. First: ```text Known v2 files → v2 core manifest Known v3 files → v3 core manifest Known v4 files → v4 core manifest Known v5 files → v5 core manifest ``` Then the four core sets are compared: ```text v2 core ─┐ v3 core ─┤ v4 core ─┼── compare coordinate overlap v5 core ─┘ │ ├── shared coordinates │ └── chip-specific coordinates │ ▼ chip fingerprints ``` The fingerprint stage determines which core coordinates are unique to a particular chip relative to the other chip versions. For example: ```text v5 fingerprint = v5 core coordinates minus coordinates also present in v2, v3, or v4 core ``` After adding v2, the fingerprints for **all four chips should be recalculated**. Do not simply add a v2 fingerprint to the old v3/v4/v5 fingerprints. The comparison should become: ```text v2 vs v3 vs v4 vs v5 ``` because adding v2 can change which coordinates are truly specific to v3, v4, or v5. --- # 28. Fingerprint → Classification After the chip-specific fingerprints are generated, they are applied to previously unlabeled PGP 23andMe files. For each file: ```text 1. Parse genomic coordinates 2. Compare against v2 fingerprint 3. Compare against v3 fingerprint 4. Compare against v4 fingerprint 5. Compare against v5 fingerprint 6. Calculate match percentages 7. Assign a provisional chip classification ``` Conceptually: ```text Unknown 23andMe file │ ├── v2 fingerprint match % ├── v3 fingerprint match % ├── v4 fingerprint match % └── v5 fingerprint match % │ ▼ classification ``` Possible classifications can include: ```text v2 v3 v4 v5 combined-chip classifications unknown unknown_build unsupported_build ``` --- # 29. Full Workflow ```text PGP 23andMe genotype files │ ▼ Consolidated participant metadata │ ▼ Select known single-chip Build 37 files │ ┌──────┼──────┬──────┐ ▼ ▼ ▼ ▼ v2 v3 v4 v5 │ │ │ │ ▼ ▼ ▼ ▼ Core Core Core Core manifest manifest manifest manifest └──────┼──────┴──────┘ ▼ Cross-chip comparison │ ▼ Shared vs chip-specific coordinates │ ▼ Chip fingerprints │ ▼ PGP files without a known chip version │ ▼ Fingerprint match percentages │ ▼ Chip classification ``` --- # 30. Key Definitions ## Manifest A **manifest** describes the genomic coordinates observed across known files from a particular chip version. ```text Manifest = characteristic genomic-coordinate content of a known chip ``` ## Core Manifest A **core manifest** contains coordinates observed in at least the selected minimum percentage of reference files. Current threshold: ```text ≥95% ``` ## Fingerprint A **fingerprint** is the subset of core coordinates that distinguishes one chip from the other chip versions. ```text Fingerprint = chip-specific coordinates ``` ## Classification **Classification** applies the fingerprints to a previously unlabeled 23andMe file to infer which chip version or combination of versions best matches its coordinate content. --- # 31. Important Production Note The production pipeline runs as a **Google Cloud Run Job**. You normally do **not** run: ```bash node index.js ``` for the production manifest builds. Instead: ```text gcloud builds submit │ ▼ container image │ ▼ gcloud run jobs deploy │ ▼ gcloud run jobs execute │ ▼ index.js runs inside Cloud Run ``` `node index.js` is useful only for local testing in Cloud Shell. --- # 32. Final Manifest Output Locations ```text v2: gs://all23_0710/chip_manifests/v2_build37_95pct/ v3: gs://all23_0710/chip_manifests/v3_build37_95pct/ v4: gs://all23_0710/chip_manifests/v4_build37_95pct/ v5: gs://all23_0710/chip_manifests/v5_build37_95pct/ ``` The primary compact coordinate files for the fingerprint stage are: ```text v2_build37_95pct/coordinates_core.json v3_build37_95pct/coordinates_core.json v4_build37_95pct/coordinates_core.json v5_build37_95pct/coordinates_core.json ``` --- # Summary The pipeline is designed as four consecutive stages: ```text 1. Build manifests ↓ 2. Compare chip coordinate overlap ↓ 3. Derive chip-specific fingerprints ↓ 4. Classify previously unlabeled 23andMe files ``` In short: ```text Manifest = what coordinates characterize each known chip Fingerprint = what coordinates distinguish that chip from the others Classification = use those fingerprints to infer the chip version of an unlabeled file ```47:Te76,toc = { const slugify = text => text .toLowerCase() .normalize("NFKD") .replace(/[\u0300-\u036f]/g, "") .replace(/[^a-z0-9]+/g, "-") .replace(/^-|-$/g, ""); return function toc({ selector = "h2",//, h2, h3", title = "Table of Contents" } = {}) { const nav = document.createElement("nav"); nav.className = "notebook-toc"; let scheduled = false; function getAllHeaders() { const cells = [ ...document.querySelectorAll(".observablehq") ]; const headers = cells.flatMap(cell => [...cell.querySelectorAll(selector)] ); // Remove duplicates and headings inside the TOC. return [...new Set(headers)] .filter(header => !nav.contains(header)) .filter(header => header.textContent.trim()) .filter(header => header.isConnected); } function render() { scheduled = false; const headers = getAllHeaders(); const usedIds = new Set( [...document.querySelectorAll("[id]")] .filter(element => !headers.includes(element)) .map(element => element.id) ); for (const header of headers) { if (!header.id) { const base = slugify(header.textContent.trim()) || "section"; let id = base; let number = 2; while (usedIds.has(id)) { id = `${base}-${number++}`; } header.id = id; } usedIds.add(header.id); } const levels = headers.map(header => Number(header.tagName.substring(1)) ); const minimumLevel = levels.length ? Math.min(...levels) : 1; nav.replaceChildren(); const heading = document.createElement("strong"); heading.textContent = title; nav.appendChild(heading); const links = document.createElement("div"); links.className = "notebook-toc-links"; if (headers.length === 0) { const emptyMessage = document.createElement("div"); emptyMessage.textContent = "No headings found."; emptyMessage.style.color = "#777"; emptyMessage.style.marginTop = "0.5rem"; links.appendChild(emptyMessage); } for (const header of headers) { const level = Number( header.tagName.substring(1) ); const link = document.createElement("a"); link.href = `#${header.id}`; link.textContent = header.textContent.trim(); link.dataset.level = level; link.style.display = "block"; link.style.marginLeft = `${(level - minimumLevel) * 1.25}rem`; link.style.padding = "0.2rem 0"; link.style.textDecoration = "none"; link.onclick = event => { event.preventDefault(); header.scrollIntoView({ behavior: "smooth", block: "start" }); history.replaceState( null, "", `#${header.id}` ); }; links.appendChild(link); } nav.appendChild(links); } function scheduleRender() { if (scheduled) return; scheduled = true; requestAnimationFrame(render); } render(); const observer = new MutationObserver(mutations => { const onlyTocChanges = mutations.every(mutation => nav.contains(mutation.target) ); if (!onlyTocChanges) { scheduleRender(); } }); observer.observe(document.body, { childList: true, subtree: true, characterData: true }); invalidation.then(() => { observer.disconnect(); }); return nav; }; }48:[{"id":"fd72b1b73402db44a6540182449a970bf70cbed003de78e07aba10aebd0c7e68299be4eb8705076ad023ccee2c92a8fe2aed6d631326f3e265b7135fbb2e6a6e","href":"https://static.observableusercontent.com/files/fd72b1b73402db44a6540182449a970bf70cbed003de78e07aba10aebd0c7e68299be4eb8705076ad023ccee2c92a8fe2aed6d631326f3e265b7135fbb2e6a6e","name":"Dockerfile.txt","type":"text/plain","lastModified":1790612368681,"size":120},{"id":"32d78f09aab810176deb14070a0782b0bc8a7b01cdc9e4c8b8d8ca37cc9432d53deefb90468111b41710b220f3a2aff93767d37cb1db9db22350641406e30a9e","href":"https://static.observableusercontent.com/files/32d78f09aab810176deb14070a0782b0bc8a7b01cdc9e4c8b8d8ca37cc9432d53deefb90468111b41710b220f3a2aff93767d37cb1db9db22350641406e30a9e","name":"package.json","type":"application/json","lastModified":1790612337179,"size":109},{"id":"0dbf8709f0d421f2b4f3bae0149f9ac7f933510b5630241b7ecc8e61f34198fee4032896bd86ca3ed77f562b8efdf74884795bcfbe85610ac71cdf1c94350d0c","href":"https://static.observableusercontent.com/files/0dbf8709f0d421f2b4f3bae0149f9ac7f933510b5630241b7ecc8e61f34198fee4032896bd86ca3ed77f562b8efdf74884795bcfbe85610ac71cdf1c94350d0c","name":"v5_manifest.zip","type":"application/x-zip-compressed","lastModified":1790733515658,"size":10830045},{"id":"719dfbb325c1e4eaf4717aca28c5c1ac8727e9da07eef2abc90d41becb938a27ab6126458f1f6a368e2afbd7b47bc85ef95fb32e9f129bdf15b3f056ce0647ed","href":"https://static.observableusercontent.com/files/719dfbb325c1e4eaf4717aca28c5c1ac8727e9da07eef2abc90d41becb938a27ab6126458f1f6a368e2afbd7b47bc85ef95fb32e9f129bdf15b3f056ce0647ed","name":"v4_manifest.zip","type":"application/x-zip-compressed","lastModified":1790733435532,"size":10005093},{"id":"b8f85cac5410b0364f8de07b8b206b96ec71f9bf5f171b03b86355d528ad33c3ca353456495888cd85643835ba6ac44ffa195659578f81ac6011576e6965c2f8","href":"https://static.observableusercontent.com/files/b8f85cac5410b0364f8de07b8b206b96ec71f9bf5f171b03b86355d528ad33c3ca353456495888cd85643835ba6ac44ffa195659578f81ac6011576e6965c2f8","name":"v3_manifest.zip","type":"application/x-zip-compressed","lastModified":1790732127924,"size":15739317},{"id":"daf05ec6d477077505437ccb16aa9aa4f3309354d2808e051e01841e584b9cd2f925a3a94ab183630975b82cfb5f3aee1cb9f1d8a9cc99cb82e59328b4905b7a","href":"https://static.observableusercontent.com/files/daf05ec6d477077505437ccb16aa9aa4f3309354d2808e051e01841e584b9cd2f925a3a94ab183630975b82cfb5f3aee1cb9f1d8a9cc99cb82e59328b4905b7a","name":"index.js","type":"text/javascript","lastModified":1790963517873,"size":45396},{"id":"1d10b621147acabbc5f221ccb26b2643a067382e936b844c76ace5656971b2334c5f425e6a306a93f0caed038e5ff54957d6ac1b08574687b21039b350d9c8ab","href":"https://static.observableusercontent.com/files/1d10b621147acabbc5f221ccb26b2643a067382e936b844c76ace5656971b2334c5f425e6a306a93f0caed038e5ff54957d6ac1b08574687b21039b350d9c8ab","name":"image23.png","type":"image/png","lastModified":1790964712619,"size":1586688},{"id":"52221dff23872e3ca8dd9876f3737f8db4b28cf9eb40bc6f97a37f0ffb602389eb07afbd14243d9ccf3fb3821326233a079e416cced7ee30c961c95a543a2e85","href":"https://static.observableusercontent.com/files/52221dff23872e3ca8dd9876f3737f8db4b28cf9eb40bc6f97a37f0ffb602389eb07afbd14243d9ccf3fb3821326233a079e416cced7ee30c961c95a543a2e85","name":"v2_manifest.zip","type":"application/x-zip-compressed","lastModified":1790965480542,"size":9019376}] 43:{"id":"6177fd62081c915e","slug":"23andme-manifest","title":"23andme Manifest","thumbnail":"49f6952075d907caf5b443941ac7541f09af5a8ac03670597f8a94203ac9550b","access_level":"public","create_time":"$D2026-07-21T15:50:38.994Z","update_time":"$D2026-10-02T19:14:12.595Z","view_time":null,"first_public_time":"$D2026-07-21T17:06:13.162Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"5a5147324ed15c62","login":"lorenasandoval88","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":"starter_2024","bio":""},"version":515,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"5c479a2c5ae5f52f","login":"lorenasandoval","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":null,"bio":""}],"members":[{"type":"individual","id":"5c479a2c5ae5f52f","login":"lorenasandoval","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":null,"role":"editor"},{"type":"team","id":"5a5147324ed15c62","login":"lorenasandoval88","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":{"id":"1476de78fbd46e52","slug":"23andme-data","title":"23andme-open-data","owner":{"id":"5a5147324ed15c62","login":"lorenasandoval88","name":""}},"type":"notebook","body":{"titling":"auto","title":"23andme Manifest","cells":[{"id":208,"value":"# 23andme Manifest","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":282,"value":"$45","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":288,"value":"

\n\n

\n Building an Empirical 23andMe Marker List in Google Cloud\n

\n\n

\n Confirmed Build 37 files were used to create empirical chip-reference manifests:\n 3 v2, 17 v3, 74 v4, and\n 63 v5. Genomic coordinates shared consistently within each chip\n were used to define core manifests, which were then compared to identify\n shared and chip-specific coordinates for downstream fingerprinting and classification.\n

\n\n

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":168,"value":"

\n \n

\n","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":204,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":170,"value":"v5 = (await (await FileAttachment(\"v5_manifest.zip\").zip())\n .file(\"v5_manifest.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":172,"value":"v4 = (await (await FileAttachment(\"v4_manifest.zip\").zip())\n .file(\"v4_manifest.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":261,"value":"v3 = (await (await FileAttachment(\"v3_manifest.zip\").zip())\n .file(\"v3_manifest.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":270,"value":"v2 = (await (await FileAttachment(\"v2_manifest.zip\").zip())\n .file(\"v2_manifest.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":294,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":146,"value":"$46","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":246,"value":"","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9,"value":"---\n## Helper Functions","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":7,"value":"$47","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W48","stdlib":"1"}} 44:"$43:body"