2c:null 2d:["$","title",null,{"children":"Preparation of 23andMe files for Imputation | lorenasandoval88 | Observable"}] 32:T3a0a,--- # Google Cloud Code This Google Cloud pipeline prepares **63 23andMe v5 / GRCh37 genotype files** for submission to the **Michigan Imputation Server**. `clean23.js` removes unusable and duplicate genotype records, `convert_bcf.js` converts each cleaned participant into a reference-aware, indexed single-sample BCF, and `merge.js` combines all 63 samples into a multi-sample cohort, applies **≥90% variant call-rate QC**, retains **biallelic SNPs**, and generates **chromosome-specific VCF.gz files for chr1–chr22**. --- ## `clean23.js` # 23andMe v5 Build37 Cleaning Summary - **63 files processed successfully** - **40,221,431** input variant rows - **38,681,638** autosomal rows retained for cleaning - **1,539,793** non-autosomal rows removed - **889,810** no-call rows removed - **288,533** unsupported genotypes removed - **37,474,740** final retained coordinates - **1,264** metadata/header rows preserved ### Duplicate-coordinate handling Duplicates were identified using: ```text chromosome:position ``` Across the 63 files: - **28,796** duplicate rows - **28,555** same-genotype duplicates - **241** both-missing duplicates - **0** called/missing duplicates - **0** conflicting genotype duplicates When different marker IDs represented the same coordinate and genotype, they were collapsed to one record and the **rsID was preferred**. Example: ```text i123 1 12345 AG rs123 1 12345 AG ``` became: ```text rs123 1 12345 AG ``` No conflicting called genotypes were found across the 63 files. The purpose of `clean23.js` is to create a **clean, non-duplicated, autosomal genotype input for the next VCF/BCF conversion step**. --- ## `convert_bcf.js` # 23andMe TXT → Single-Sample BCF Conversion `convert_bcf.js` runs as a **Cloud Run Job** and converts each cleaned 23andMe TXT file into a standard single-sample BCF file using `bcftools`. The job reads the cleaning manifest: ```text gs://all23_0710/v5_imputation_63/clean23/cleaning_manifest.json ``` and processes only files marked as successfully cleaned. The participant ID extracted during cleaning is retained as the BCF sample name. Example: ```text Cleaned file: huD2AD97_3395_genome_Jodi_Riggins_v5_Full_20180217093249.cleaned.txt ↓ BCF sample name: huD2AD97 ``` ### Cloud Run Environment The Cloud Run job uses: - **4 CPUs** - **8 GiB memory** - **1 task** - **sequential processing of the 63 participants** - the `all23_0710` bucket mounted at: ```text /gcs ``` The Cloud Storage mount allows the job to read and write directly to: ```text gs://all23_0710/ ``` without requiring the reference genome or genotype files to remain in Cloud Shell storage. ### GRCh37 Reference Genome BCF conversion requires a reference genome so that each 23andMe genotype can be represented using standard VCF/BCF `REF`, `ALT`, and `GT` fields. The job uses the **1000 Genomes GRCh37 reference**: ```text human_g1k_v37.fasta ``` On the first successful run, the script: ```text download GRCh37 gzip ↓ validate download ↓ decompress gzip ↓ convert to BGZF ↓ samtools faidx ↓ validate chromosomes 1–22 ↓ cache reference in GCS ``` The persistent reference cache is stored at: ```text gs://all23_0710/reference/grch37/ ``` with: ```text human_g1k_v37.fasta.bgz human_g1k_v37.fasta.bgz.fai human_g1k_v37.fasta.bgz.gzi ``` Later Cloud Run executions can reuse this cached reference rather than downloading it again. The script also verifies that the reference uses chromosome names: ```text 1 2 3 ... 22 ``` rather than: ```text chr1 chr2 ... ``` so the reference naming matches the cleaned 23andMe files. ### BCF Conversion Each cleaned TXT file has four genotype columns: ```text ID CHROM POS AA ``` Example: ```text rs123 1 12345 AG ``` The conversion is performed with the equivalent of: ```text bcftools convert -c ID,CHROM,POS,AA -s PARTICIPANT_ID -f GRCh37_REFERENCE --tsv2vcf CLEANED_FILE -Ob ``` The `-Ob` option produces a compressed binary BCF file. ### REF/ALT Genotype Representation The original 23andMe genotype: ```text rs123 1 12345 AG ``` is converted into standard VCF/BCF representation. For example, if GRCh37 contains: ```text REF = A ALT = G ``` then: ```text 23andMe genotype BCF GT AA 0/0 AG 0/1 GG 1/1 ``` where: ```text 0 = reference allele 1 = alternate allele ``` Conceptually: ```text CHROM POS ID REF ALT FORMAT huD2AD97 1 12345 rs123 A G GT 0/1 ``` `GT` identifies the sample field as a genotype, while `0/1` is the genotype value for that participant. ### Per-Participant Conversion QC For each participant, the script records: - input coordinates - BCF record count - total rows processed - rows skipped - missing genotypes - homozygous reference genotypes - heterozygous reference/alternate genotypes - homozygous alternate genotypes - heterozygous two-alternate genotypes Example: ```text Sample name: huD2AD97 Input coordinates: 586,313 BCF records: 586,313 Rows total: 586,313 Rows skipped: 0 Missing GTs: 0 Hom RR: 436,172 Het RA: 95,342 Hom AA: 54,799 Het AA: 0 ``` The genotype categories mean: ```text Hom RR = homozygous reference = 0/0 Het RA = heterozygous REF/ALT = 0/1 Hom AA = homozygous alternate = 1/1 Het AA = two different ALT alleles = 1/2 ``` For this example: ```text 436,172 95,342 54,799 0 ------- 586,313 ``` which exactly matches the number of BCF records. During the conversion run, checked participants showed: ```text Input coordinates = BCF records Rows skipped = 0 Missing GTs = 0 Het AA = 0 ``` indicating that the cleaned SNP rows were being converted without loss. ### Output Files For each participant, the Cloud Run job creates: ```text huD2AD97.bcf huD2AD97.bcf.csi ``` The `.bcf` file contains the participant's standardized genotype data. The `.bcf.csi` file is the BCF index and allows `bcftools` to rapidly access genomic regions without scanning the entire file. The files are stored at: ```text gs://all23_0710/v5_imputation_63/convert_bcf/single_sample_bcf/ ``` Example: ```text single_sample_bcf/ ├── huA08F4D.bcf ├── huA08F4D.bcf.csi ├── huC8B936.bcf ├── huC8B936.bcf.csi ├── hu7ECB9C.bcf ├── hu7ECB9C.bcf.csi └── ... ``` ### Conversion Logs A conversion log is also generated for each participant: ```text gs://all23_0710/v5_imputation_63/convert_bcf/logs/ ``` Example: ```text huD2AD97.bcftools_convert.log ``` These logs preserve the `bcftools` conversion statistics and can be reviewed for conversion problems. ### Conversion Manifest After processing the cohort, `convert_bcf.js` creates: ```text gs://all23_0710/v5_imputation_63/convert_bcf/conversion_manifest.json ``` The manifest records: - participant ID - source cleaned file - BCF output path - CSI index path - conversion log path - number of input coordinates - number of BCF records - `bcftools` QC statistics - successful/failed conversion status - reference genome information - Cloud Run execution metadata The conversion manifest provides a reproducible record of the **TXT → BCF conversion step** before cohort merging. --- ## `merge.js` # 63-Sample BCF Merge and Cohort QC `merge.js` runs as a **Cloud Run Job** and combines the **63 single-sample BCF files** into one multi-sample cohort for Michigan Imputation Server preparation. The input BCF files are read from: ```text gs://all23_0710/v5_imputation_63/convert_bcf/single_sample_bcf/ ``` Each participant contributes one indexed BCF: ```text huXXXXXX.bcf huXXXXXX.bcf.csi ``` Before merging, the script verifies that: - exactly **63 BCF files** are present - every BCF has a corresponding **CSI index** - every BCF contains exactly **one sample** - all sample IDs are unique ### Missing Genotypes Missing sites are preserved as: ```text ./. ``` They are **not converted to homozygous-reference genotypes (`0/0`)**. This distinction is important because: ```text 0/0 = observed homozygous reference genotype ./. = genotype not observed for that participant ``` For example: ```text SNP huA huB huC rs123 0/0 0/1 ./. ``` means that `huC` does not have an observed genotype at that site. The merge therefore intentionally does **not** use: ```text --missing-to-ref ``` or: ```text -0 ``` --- ### Cohort Merge The 63 single-sample BCF files are combined using: ```text bcftools merge ``` Pipeline: ```text 63 single-sample BCFs ↓ bcftools merge ↓ one 63-sample cohort BCF ``` The merged cohort contained: - **63 participants** - **614,637 unique variant sites** The merged cohort is stored at: ```text gs://all23_0710/v5_imputation_63/merge/cohort/merged_63samples.bcf ``` with index: ```text merged_63samples.bcf.csi ``` --- ### Cohort Call-Rate QC After merging, cohort-level genotype statistics are calculated with: ```text bcftools +fill-tags ``` The following tags are added: ```text NS F_MISSING AC AN ``` where: ```text NS = number of samples with called genotypes F_MISSING = fraction of samples with missing genotypes AC = alternate allele count AN = total number of called alleles ``` A **90% minimum variant call rate** is then applied. With 63 participants: ```text 63 × 0.90 = 56.7 ``` Therefore, each retained SNP must have a genotype call in at least: ```text 57 / 63 participants ``` This allows a maximum of: ```text 6 / 63 participants missing ``` at a retained site. The implemented filter is: ```text INFO/F_MISSING <= 0.10 ``` Examples: ```text 63 called + 0 missing → PASS 60 called + 3 missing → PASS 57 called + 6 missing → PASS 56 called + 7 missing → FAIL ``` The **57 value refers to the number of participants that must have a genotype call at each SNP**. It does not mean that only 57 participant files were retained. All **63 participants remain in the cohort**. --- ### Biallelic SNP Filter The cohort is also restricted to: ```text biallelic SNPs ``` The final retained-variant rule is therefore: ```text biallelic SNP AND INFO/F_MISSING <= 0.10 ``` This removes sites that either: - fail the **90% cohort call-rate requirement** - are not SNPs - are multiallelic --- ### Cohort QC Results The completed merge and cohort-QC step produced: | Result | Count | | --- | ---: | | Participants | **63** | | Merged variant sites | **614,637** | | Minimum called participants per site | **57 / 63** | | Maximum missing participants per site | **6 / 63** | | Retained biallelic SNPs | **487,366** | | Removed records | **127,271** | | Percent retained | **79.29%** | | Percent removed | **20.71%** | The **127,271 removed records** represent the combined effect of the: ```text ≥90% call-rate filter + biallelic SNP restriction ``` Therefore, not every removed record was necessarily removed because of missing genotypes alone. The final QC cohort is stored at: ```text gs://all23_0710/v5_imputation_63/merge/qc/cohort_callrate90_biallelic_snps.bcf ``` with index: ```text cohort_callrate90_biallelic_snps.bcf.csi ``` --- ### Chromosome-Specific VCF Output The quality-controlled cohort is split into chromosomes **1–22**. Each chromosome file: - contains all **63 participant sample columns** - contains only the retained cohort variants - is written as a **bgzip-compressed VCF** - has a corresponding **tabix `.tbi` index** Output: ```text chr1.vcf.gz chr1.vcf.gz.tbi chr2.vcf.gz chr2.vcf.gz.tbi ... chr22.vcf.gz chr22.vcf.gz.tbi ``` The chromosome files are stored at: ```text gs://all23_0710/v5_imputation_63/merge/chromosomes/ ``` --- ### Variants Retained Per Chromosome | Chromosome | Retained variants | | --- | ---: | | chr1 | 38,474 | | chr2 | 40,158 | | chr3 | 33,983 | | chr4 | 31,915 | | chr5 | 29,355 | | chr6 | 36,270 | | chr7 | 27,384 | | chr8 | 25,553 | | chr9 | 21,119 | | chr10 | 24,806 | | chr11 | 24,215 | | chr12 | 23,479 | | chr13 | 17,643 | | chr14 | 15,981 | | chr15 | 15,006 | | chr16 | 16,099 | | chr17 | 14,228 | | chr18 | 14,610 | | chr19 | 11,157 | | chr20 | 12,045 | | chr21 | 6,942 | | chr22 | 6,944 | Total: ```text 487,366 retained variants ``` --- ### Merge Pipeline Summary ```text 63 single-sample BCF files ↓ validate BCF + CSI + sample IDs ↓ bcftools merge ↓ 63-sample cohort 614,637 merged variant sites ↓ bcftools +fill-tags ↓ NS / F_MISSING / AC / AN ↓ ≥90% cohort call rate ≥57 of 63 participants called ↓ biallelic SNPs only ↓ 487,366 retained variants ↓ split chromosomes 1–22 ↓ 22 VCF.gz files + 22 tabix indexes ↓ Michigan Imputation Server ``` --- ### Cloud Run Execution `merge.js` was run as the Cloud Run Job: ```text Job: v5-merge-qc Execution: v5-merge-qc-xmv7b Task index: 0 Task count: 1 bcftools: 1.16 Threads: 4 ``` The completed run took: ```text Started: 2026-08-21T14:16:39.589Z Finished: 2026-08-21T14:18:10.716Z Elapsed: 91 seconds ``` or approximately: ```text 1 minute 31 seconds ``` --- ### Merge Manifest A complete machine-readable record of the merge and QC step is stored at: ```text gs://all23_0710/v5_imputation_63/merge/merge_manifest.json ``` The manifest contains: - all **63 participant IDs** - number of input BCF files - number of merged variant sites - missing-genotype behavior - call-rate threshold - minimum called samples - maximum missing samples - `NS`, `F_MISSING`, `AC`, and `AN` tags - retained and removed variant counts - per-chromosome variant counts - BCF and VCF output paths - index paths - Cloud Run execution information - runtime --- ### Michigan Imputation Server Input The final output from `merge.js` consists of **22 chromosome-specific GRCh37 / hg19 VCF.gz files** prepared for submission to the Michigan Imputation Server. ```text 63 participants ↓ 614,637 merged sites ↓ ≥90% cohort call rate ↓ ≥57 of 63 participants called per SNP ↓ biallelic SNPs only ↓ 487,366 retained SNPs ↓ chr1–chr22 VCF.gz ↓ Michigan Imputation Server ↓ phasing ↓ genotype imputation ``` Final submission directory: ```text gs://all23_0710/v5_imputation_63/merge/chromosomes/ ``` The purpose of `merge.js` is to convert the **63 reference-aware single-sample BCF files into one quality-controlled cohort and then generate chromosome-specific VCF files ready for full Michigan Imputation Server processing**. 33:Ta23,# Michigan Imputation Server The **63 single-sample BCF files should not be submitted directly** to the Michigan Imputation Server. Michigan Imputation Server expects **bgzip-compressed VCF files, separated by chromosome and sorted by genomic position**. ## Google chell Install bcftools ```bash sudo apt-get update sudo apt-get install -y bcftools ``` ### Initial Chromosome 22 Test Before submitting the full cohort, chromosome 22 can be used as a small validation test. ```text 63-sample cohort ↓ chr22 ↓ Michigan Imputation Server ↓ Quality Control Only ``` If chromosome 22 passes QC successfully, the full set of chromosomes 1–22 can then be submitted for imputation. Michigan performs additional QC including: - reference-panel overlap - allele consistency checks - variant call rate - per-sample call rate - allele-frequency comparisons - strand and allele alignment checks ### Recommended Michigan Settings | Setting | Recommended starting choice | | --- | --- | | **Mode** | **Quality Control Only** for the initial chr22 test | | **Genome build** | **hg19 / GRCh37** | | **Phasing** | **Eagle** because the 23andMe genotypes are unphased | | **Reference panel** | **1000 Genomes Phase 3 v5** for mixed or unknown ancestry | | **Allele Frequency Check** | **Off** when the cohort contains mixed or uncertain ancestry | | **Imputation quality filter** | Not important for QC-only; imputation-quality filtering can be applied after the final imputation run | ### Reference Panel For this cohort, **1000 Genomes Phase 3 v5** is a reasonable starting reference because it includes samples from multiple global populations. ```text 23andMe cohort ↓ Eagle phasing ↓ 1000 Genomes Phase 3 reference ↓ Minimac4 imputation ↓ dense imputed genotypes ``` If the cohort were known to be predominantly European, a larger predominantly European reference panel such as HRC could also be considered. ### Final Submission Files The final Michigan-ready files should look like: ```text chr1.vcf.gz chr1.vcf.gz.tbi chr2.vcf.gz chr2.vcf.gz.tbi ... chr22.vcf.gz chr22.vcf.gz.tbi ``` Each VCF should contain all 63 participants as sample columns. Conceptually: ```text CHROM POS ID REF ALT FORMAT huA08F4D huC8B936 ... hu88D4F9 1 ... ... ... ... GT 0/1 0/0 ... ./. ``` The next processing stage is therefore: ```text single-sample BCF ↓ merge ↓ call-rate QC ↓ chromosome split ↓ bgzip + tabix ↓ Michigan Imputation Server ```34:T2556,html`

23andMe v5 Cohort Merge

Preparation of 63 23andMe v5 / GRCh37 samples for Michigan Imputation Server

Merge completed successfully.
All 63 participants were retained and chromosome-specific VCF files were generated for chromosomes 1–22.

${mergeManifest.input.bcfFiles}

Input BCF files

${mergeManifest.merge.records.toLocaleString()}

Merged variant sites

${mergeManifest.qc.retainedRecords.toLocaleString()}

Variants retained

${mergeManifest.qc.removedRecords.toLocaleString()}

Variants removed

${mergeManifest.chromosomes.length}

Chromosome VCFs

${mergeManifest.timing.elapsedSeconds}s

Cloud Run runtime

Pipeline

63 single-sample BCF files ↓ merge participants ↓ 614,637 cohort variant sites ↓ calculate NS / F_MISSING / AC / AN ↓ require ≥57 of 63 participants called ↓ retain biallelic SNPs ↓ 487,366 retained variants ↓ split chromosomes 1–22 ↓ 22 bgzip-compressed VCF files ↓ Michigan Imputation Server

Cohort Call-Rate Filter

A ${mergeManifest.qc.minimumCallRate * 100}% variant call-rate threshold was applied across the 63 participants.

A SNP therefore had to contain a genotype call in at least ${mergeManifest.qc.minimumCalledSamples} of 63 participants. A maximum of ${mergeManifest.qc.maximumMissingSamples} participants could have a missing genotype at a retained site.

63 × 0.90 = 56.7
→ minimum called samples = 57
→ maximum missing samples = 6

Missing genotypes remained encoded as ./.. They were not converted to homozygous-reference genotypes (0/0).

The cohort filter was:

${mergeManifest.qc.filterExpression}

Additional cohort tags calculated with bcftools +fill-tags: ${mergeManifest.qc.fillTags.join(", ")}.

The final dataset was also restricted to ${mergeManifest.qc.variantType}.

Variant Retention

Merged variant sites	${mergeManifest.qc.mergedRecords.toLocaleString()}
Retained	${mergeManifest.qc.retainedRecords.toLocaleString()} (${( mergeManifest.qc.retainedRecords / mergeManifest.qc.mergedRecords * 100 ).toFixed(2)}%)
Removed	${mergeManifest.qc.removedRecords.toLocaleString()} (${( mergeManifest.qc.removedRecords / mergeManifest.qc.mergedRecords * 100 ).toFixed(2)}%)

Chromosome-Specific Output

The ${mergeManifest.qc.retainedRecords.toLocaleString()} retained variants were split into 22 chromosome-specific VCF files. Every chromosome file contains all ${mergeManifest.input.expectedSamples} participants.

${mergeManifest.chromosomes.map(d => html` `)}

Chromosome	Samples	Variants
chr${d.chromosome}	${d.samples}	${d.records.toLocaleString()}

Participants

${mergeManifest.input.sampleIds.length} participant IDs are present in the merged cohort:

Show participant IDs

${mergeManifest.input.sampleIds.map(id => html`

${id}

`)}

Michigan Imputation Files

Genome build: ${mergeManifest.michigan.genomeBuild}

Final chromosome files:


    gs://all23_0710/v5_imputation_63/merge/chromosomes/

Files generated:

chr1.vcf.gz
chr1.vcf.gz.tbi
...
chr22.vcf.gz
chr22.vcf.gz.tbi

These files are prepared for submission to the Michigan Imputation Server for phasing and full genotype imputation.

Output Files

Merged cohort BCF	`${mergeManifest.merge.bcf}`
QC cohort BCF	`${mergeManifest.qc.bcf}`
Chromosome VCF directory	`gs://all23_0710/v5_imputation_63/merge/chromosomes/`

Cloud Run

Job	${mergeManifest.cloudRun.job}
Execution	${mergeManifest.cloudRun.execution}
Task	${mergeManifest.cloudRun.taskIndex}
bcftools	${mergeManifest.tools.bcftools}
Threads	${mergeManifest.tools.threads}
Runtime	${mergeManifest.timing.elapsedSeconds} seconds (${Math.floor(mergeManifest.timing.elapsedSeconds / 60)} min ${mergeManifest.timing.elapsedSeconds % 60} sec)

Generated: ${new Date(mergeManifest.generatedAt).toLocaleString()}

`35:T8a9,# PGS Catalog Model Selection **Diseasome** focuses on **PGS Catalog models with ≤1,000 variants** so that PRS can be calculated efficiently in the browser using variants directly observed in 23andMe files. To identify models that are well suited for direct scoring, Diseasome compares each PGS model against empirical marker sets derived from **23andMe v4 and v5 files**. - **Marker sets:** generated from **20 v4 and 20 v5 files**, retaining chromosome-position coordinates present in **100% of files** for each chip version. - **Model filter:** PGS models are restricted to **≤1,000 variants** and can then be filtered by the percentage of model variants directly represented on the selected 23andMe chip. - **Higher v5 coverage:** the v5 marker set contains more common coordinates and rsIDs than v4 and therefore provides substantially greater overlap with PGS Catalog models. - **High-overlap models:** this difference is especially apparent at stringent overlap thresholds. At **≥80% overlap**, 121 models qualify for v5 compared with only 39 for v4. - **Direct PRS use case:** models with high chip overlap can be scored directly from 23andMe data without first imputing missing variants, supporting fast, browser-based PRS calculation. - **Imputation comparison:** selected models can also be used to compare **direct 23andMe PRS, imputed PRS, and Veritas whole-genome sequencing (WGS) PRS**, with phasing and imputation performed using **Eagle2 + Minimac4**. - **Validation limitation:** only **2 participants** currently have both a v5 23andMe file and Veritas WGS data, so the v5 imputation/WGS comparison will primarily serve as a **proof-of-concept**. | Result | v4 | v5 | |---|---:|---:| | Common coordinates | 596,198 | 622,461 | | rsIDs | 552,402 | 608,532 | | PGS ≥0% overlap | 1,759 | 1,759 | | PGS ≥80% overlap | 39 | 121 | | PGS ≥90% overlap | 22 | 58 | | PGS ≥95% overlap | 15 | 32 | | PGS 100% overlap | 13 | 22 | | Veritas participants | 9 | 2 | Overall, **v5 provides substantially better direct coverage of small PGS models than v4**, making it the stronger chip version for evaluating high-overlap browser-based PRS and for comparison with imputed and WGS-derived scores.36:T42f,function toc(selector = "h1", heading = "Table of Contents") { return Generators.observe(notify => { let headings = []; function observed() { const h = Array.from(document.querySelectorAll(selector)); if (h.length !== headings.length || h.some((h, i) => headings[i] !== h)) { notify(html`${heading}

${DOM.text(h.textContent)}

`); } } const observer = new MutationObserver(observed); observer.observe(document.body, { childList: true, subtree: true }); observed(); return () => observer.disconnect(); }); }37:[{"id":"cc6d965d376506b6f4709a64a35258f7bec649a6f988994cf459ce583a5c7dca9c4ce469362d4633816f52bbe8bdd927baca688054f78343406554d05ff6a80b","href":"https://static.observableusercontent.com/files/cc6d965d376506b6f4709a64a35258f7bec649a6f988994cf459ce583a5c7dca9c4ce469362d4633816f52bbe8bdd927baca688054f78343406554d05ff6a80b","name":"participants_1020_consolidated_with_build_and_profile_07102026_2053.json","type":"application/json","lastModified":1786978997461.873,"size":10617236},{"id":"803e5bdbc991da23ee44ed23bbba9325fa8f66d20f5ac355322ab47eeacbde172703a301af524bb626291b729806f02e1cb4513af62da23c8169ce165be950fb","href":"https://static.observableusercontent.com/files/803e5bdbc991da23ee44ed23bbba9325fa8f66d20f5ac355322ab47eeacbde172703a301af524bb626291b729806f02e1cb4513af62da23c8169ce165be950fb","name":"merge_manifest.json","type":"application/json","lastModified":1787341887860.05,"size":8507},{"id":"79457794fc4c81c5fa98c542409892cad2e66cb1e5a4fce9fc0bf6ebdbf612c1c80dfa4c93de517fea79950954eb0c2fd12fbeda169de637cbb0f89541fda6b2","href":"https://static.observableusercontent.com/files/79457794fc4c81c5fa98c542409892cad2e66cb1e5a4fce9fc0bf6ebdbf612c1c80dfa4c93de517fea79950954eb0c2fd12fbeda169de637cbb0f89541fda6b2","name":"image.png","type":"image/png","lastModified":1787667992048.158,"size":1895558},{"id":"9da44d5ab9370a47ea1fe3f71c1efe2d40ccee9acdc2d7a960eb8e458b0efa6d7c06b1979b75fe4121eb37982e8fb491b63365cb130cf064fb64e69e30581100","href":"https://static.observableusercontent.com/files/9da44d5ab9370a47ea1fe3f71c1efe2d40ccee9acdc2d7a960eb8e458b0efa6d7c06b1979b75fe4121eb37982e8fb491b63365cb130cf064fb64e69e30581100","name":"merge.js","type":"application/javascript","lastModified":1787668507801.393,"size":22352},{"id":"0392a66846eb44b9504d53259b481f9df6c579f874a9a9c4a6790a8fc771dfd575c72d66bed7c445755fe2ecf96062571805629345cafe0d76c73e2c99dcac28","href":"https://static.observableusercontent.com/files/0392a66846eb44b9504d53259b481f9df6c579f874a9a9c4a6790a8fc771dfd575c72d66bed7c445755fe2ecf96062571805629345cafe0d76c73e2c99dcac28","name":"clean.js","type":"application/javascript","lastModified":1787668573075.377,"size":30904},{"id":"c300e5144c25d15770622df7f34593fb64b9d53f820e65804ebe81529d6765514a70fea7d4197a238359de5b864f2f494461b959af5dcb3f5ae1fe615a520b75","href":"https://static.observableusercontent.com/files/c300e5144c25d15770622df7f34593fb64b9d53f820e65804ebe81529d6765514a70fea7d4197a238359de5b864f2f494461b959af5dcb3f5ae1fe615a520b75","name":"convert_bcf.js","type":"application/javascript","lastModified":1787668607473.074,"size":52391},{"id":"7602464f514448c8e369704791bd22ee7b4f847aec317739fef1eeb728a7e058d918ac99c1275afab055101ec63c788a373e5c206a4de93d3da265880b577487","href":"https://static.observableusercontent.com/files/7602464f514448c8e369704791bd22ee7b4f847aec317739fef1eeb728a7e058d918ac99c1275afab055101ec63c788a373e5c206a4de93d3da265880b577487","name":"overlap_0_100_v4.json","type":"application/json","lastModified":1786467271250,"size":979514},{"id":"c260b0849fca2b0fd6f5451ef7dc270f875a04c683b60ecd0bff87359c025ca0e46d84ab136a4c26e92c92bef1f90fe90b27d48619cb1212f9fce4d9516f02a3","href":"https://static.observableusercontent.com/files/c260b0849fca2b0fd6f5451ef7dc270f875a04c683b60ecd0bff87359c025ca0e46d84ab136a4c26e92c92bef1f90fe90b27d48619cb1212f9fce4d9516f02a3","name":"overlap_0_100_v5.json","type":"application/json","lastModified":1786467325368,"size":979437}] 30:{"id":"71957bff92c491d5","slug":"imputation-of-23andme-files","title":"Preparation of 23andMe files for Imputation","thumbnail":"99ed4088426759b74c2903e0570153775106d417b38e8bd1ba241470dd690d00","access_level":"public","create_time":"$D2026-08-10T17:44:52.902Z","update_time":"$D2026-09-16T14:21:29.640Z","view_time":null,"first_public_time":"$D2026-08-19T18:19:29.295Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"5a5147324ed15c62","login":"lorenasandoval88","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":"starter_2024","bio":""},"version":300,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"5c479a2c5ae5f52f","login":"lorenasandoval","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":null,"bio":""}],"members":[{"type":"individual","id":"5c479a2c5ae5f52f","login":"lorenasandoval","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":null,"role":"editor"},{"type":"team","id":"5a5147324ed15c62","login":"lorenasandoval88","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":{"id":"8bdfe82d7003e3b0","slug":"imputation","title":"imputation","owner":{"id":"5a5147324ed15c62","login":"lorenasandoval88","name":""}},"type":"notebook","body":{"titling":"auto","title":"Preparation of 23andMe files for Imputation","cells":[{"id":143,"value":"# Preparation of 23andMe files for Imputation","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":112,"value":"toc()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":174,"value":"# Overview\n\n

\n This Google Cloud pipeline prepares the\n 63 23andMe v5 / Build 37 genotype files\n for conversion to VCF/BCF and eventual\n \n Michigan Imputation Server\n \n submission.\n

\n\n

\n\n\n\n

\n  \n63 raw 23andMe v5 Build37 TXT files\n                ↓\n  \n            clean23.js\n                ↓\n  \n      63 cleaned autosomal TXT files\n                ↓\n  \n           convert_bcf.js\n                ↓\n  \n       63 single-sample BCF files\n          + 63 CSI indexes\n                ↓\n  \n          merge.js merge 63 samples\n                ↓\n  \n        one cohort BCF\n                ↓\n  \n        cohort call-rate QC\n                ↓\n  \n         split chr1–chr22\n                ↓\n  \n        bgzip VCF + tabix\n                ↓\n  \n      Michigan Imputation Server\n

\n\n

\n \n

\n\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":105,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":94,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":120,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":131,"value":"mergeManifest = FileAttachment(\"merge_manifest.json\").json()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":128,"value":"$34","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":126,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":92,"value":"# 23andMe File Selection\n\nWe select **23andMe v5 files mapped to genome build 37** from the consolidated PGP participant dataset. [Explore the participant and file selection here](https://observablehq.com/@lorenasandoval88/23andme-files-from-the-pgp-using-google-cloud).\n\nFile selection is performed at the **individual file level**, rather than the participant level. This allows participants who have uploaded multiple chip versions (for example, separate v4 and v5 files) to still contribute a qualifying v5 file.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":24,"value":"participants = FileAttachment(\n \"participants_1020_consolidated_with_build_and_profile_07102026_2053.json\"\n).json()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":33,"value":"v5Build37Files = participants.flatMap(p =>\n (p.files ?? [])\n .filter(f =>\n isV5File(f) &&\n String(f.genomeBuild) === \"37\" &&\n f.dataType === \"23andMe\" &&\n f.valid23File === true\n )\n .map(f => ({\n participantId: p.id,\n\n filename: f.filename,\n\n chipVersion: \"v5\",\n\n genomeBuild: f.genomeBuild,\n\n dataType: f.dataType,\n\n valid23File: f.valid23File,\n\n downloadUrl: f.downloadUrl,\n\n publishedDate: f.publishedDate,\n\n profileUrl: p.profileUrl\n }))\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":57,"value":"v5Build37Summary = ({\n matchingFiles: v5Build37Files.length,\n\n uniqueParticipants: new Set(\n v5Build37Files.map(d => d.participantId)\n ).size\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":36,"value":"v5Build37Participants =\n new Set(\n v5Build37Files.map(d => d.participantId)\n ).size","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":45,"value":"### 2 Duplicate files\nThere are 63 participants with v5 files, however, two have duplicate files, of which we pull the first file only, leaving 63 files total. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":41,"value":"v5Build37FirstFile = {\n const seen = new Set();\n\n return v5Build37Files.filter(d => {\n if (seen.has(d.participantId)) return false;\n\n seen.add(d.participantId);\n return true;\n });\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":39,"value":"Inputs.table(v5Build37FirstFile, {\n columns: [\n \"participantId\",\n \"filename\",\n \"chipVersion\",\n \"genomeBuild\",\n \"valid23File\",\n \"publishedDate\",\n \"downloadUrl\"\n ],\n width: { \n participantId: 90,\n filename: 200\n },\n rows: 10\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"### Users with 23andMe and WGS Veritas files. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":17,"value":"\nThere are 35 with Veritas files total, and the array breakdown among them is v3=17, v4=9, v2=4, v5=2. \n\nOnly 2 users are v5: hu7ECB9C and hu8209C4. \n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":89,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":21,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":176,"value":"v4Overlap = await FileAttachment(\"overlap_0_100_v4.json\").json()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":185,"value":"v4Scores = v4Overlap.scores","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":178,"value":"v5Overlap = await FileAttachment(\"overlap_0_100_v5.json\").json()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":189,"value":"v5Scores = v5Overlap.scores","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":193,"value":"overlapCounts = ({\n v4: v4Scores.filter(d => d.overlap >= overlapThreshold / 100).length,\n v5: v5Scores.filter(d => d.overlap >= overlapThreshold / 100).length\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":192,"value":"viewof overlapThreshold = Inputs.range([0, 100], {\n step: 1,\n value: 80,\n label: \"Minimum PGS overlap (%)\"\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":195,"value":"md`\n### PGS models with ≥${overlapThreshold}% overlap\n\n| Chip | PGS models |\n|---|---:|\n| v4 | ${overlapCounts.v4.toLocaleString()} |\n| v5 | ${overlapCounts.v5.toLocaleString()} |\n`","pinned":false,"hidden":false,"mode":"js","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":26,"value":"---\n# Helper Functions","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":28,"value":"getFileChipVersions = f => {\n const filename = f.filename ?? \"\";\n\n return [\n ...new Set(\n filename\n .match(/v[2-5]/gi)\n ?.map(d => d.toLowerCase()) ?? []\n )\n ];\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":31,"value":"isV5File = f => {\n const versions = getFileChipVersions(f);\n\n return (\n versions.length === 1 &&\n versions[0] === \"v5\"\n );\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":61,"value":"v5DuplicateParticipants = Array.from(\n d3.group(v5Build37Files, d => d.participantId),\n ([participantId, files]) => ({\n participantId,\n numberOfFiles: files.length,\n filenames: files.map(f => f.filename).join(\" | \")\n })\n)\n.filter(d => d.numberOfFiles > 1)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":65,"value":"({\n files: v5Build37Files.length,\n participants: new Set(\n v5Build37Files.map(d => d.participantId)\n ).size\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":109,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W37","stdlib":"1"}} 31:"$30:body"