2c:null 2d:["$","title",null,{"children":"23andMe v2/v3/v4/v5 Overlap | lorenasandoval88 | Observable"}] 32:T1409,

Workflow

Overlap & Fingerprints

Known 23andMe v2, v3, v4, and v5 Build 37 files were used as reference input files to identify genomic positions characteristic of each chip version.

Input files

v2 · 3 files v3 · 17 files v4 · 74 files v5 · 64 files

For each chip version, genomic coordinates from the known files were compared. Positions consistently associated with that version were retained as the chip fingerprint.

Resulting chip fingerprints

v2 111

v3 285,648

v4 59,332

v5 429,929

Values indicate fingerprint genomic positions.

33:T2780, ## Fingerprint Summary | Chip | Core markers | Shared with ≥1 other chip | Chip-specific | Shared across all 3 | |---|---:|---:|---:|---:| | **v3** | 959,702 | 607,163 | 352,539 | 105,026 | | **v4** | 596,247 | 536,264 | 59,983 | 105,026 | | **v5** | 622,753 | 192,201 | 430,552 | 105,026 | ## Composition of Each Chip Core | Chip | Shared | Chip-specific | |---|---:|---:| | **v3** | 63.3% | 36.7% | | **v4** | 89.9% | 10.1% | | **v5** | 30.9% | 69.1% | ### Shared vs. Chip-Specific Markers

v3 — 959,702 core markers

63.3% shared

36.7% specific

v4 — 596,247 core markers

89.9% shared

10.1% specific

v5 — 622,753 core markers

30.9% shared

69.1% specific

--- ## Chip-Specific Fingerprints These are the core genomic coordinates that distinguish each chip from **both** of the other chip versions. | Fingerprint | Unique coordinates | |---|---:| | **v3-specific** | 352,539 | | **v4-specific** | 59,983 | | **v5-specific** | 430,552 |

Number of chip-specific coordinates

v3	352,539
v4	59,983
v5	430,552

--- ## Pairwise Chip Overlap | Comparison | Shared coordinates | |---|---:| | **v3 ∩ v4** | 528,126 | | **v3 ∩ v5** | 184,063 | | **v4 ∩ v5** | 113,164 | | **v3 ∩ v4 ∩ v5** | 105,026 |

Pairwise overlap

v3 ↔ v4	528,126
v3 ↔ v5	184,063
v4 ↔ v5	113,164
All 3	105,026

### Understanding the Pairwise Counts The pairwise overlap values are **not mutually exclusive**. The **105,026 positions found on all three chips** are included within each pairwise overlap: - v3 ∩ v4 - v3 ∩ v5 - v4 ∩ v5 Therefore, the pairwise counts should not simply be added together. Subtracting the all-three intersection from each pairwise overlap gives the positions shared by **only two** chip versions: | Region | Coordinates | |---|---:| | **v3 only** | 352,539 | | **v4 only** | 59,983 | | **v5 only** | 430,552 | | **v3 + v4 only** | 423,100 | | **v3 + v5 only** | 79,037 | | **v4 + v5 only** | 8,138 | | **v3 + v4 + v5** | 105,026 | --- # Key Findings ## v3 The v3 core contains **959,702 genomic positions**, the largest of the three core manifests. Of these: - **607,163** are shared with v4 and/or v5. - **352,539** are specific to v3. - **36.7%** of the v3 core provides a v3-specific coordinate fingerprint. - v3 shares **528,126 positions with v4**, indicating substantial overlap between these two chip generations. ## v4 The v4 core contains **596,247 genomic positions**. Of these: - **536,264** are also present in v3 and/or v5. - **59,983** are specific to v4. - approximately **89.9%** of the v4 core overlaps at least one other chip. - approximately **88.6%** of the v4 core is also present in v3. v4 therefore has the **smallest chip-specific coordinate fingerprint** of the three versions. ## v5 The v5 core contains **622,753 genomic positions**. Of these: - **192,201** are shared with v3 and/or v4. - **430,552** are specific to v5. - approximately **69.1%** of the v5 core is absent from both v3 and v4. v5 therefore has the **largest chip-specific coordinate fingerprint** and is the most distinct of the three based on these core genomic positions. --- ## Overall Comparison

                         CORE MARKERS

v3    959,702
      ├── 607,163 shared
      └── 352,539 v3-specific

v4    596,247
      ├── 536,264 shared
      └──  59,983 v4-specific

v5    622,753
      ├── 192,201 shared
      └── 430,552 v5-specific

The overall pattern shows: - **v3 and v4 have substantial coordinate overlap.** - **v4 has relatively few positions unique to v4.** - **v5 is considerably more distinct from v3 and v4.** - **105,026 positions are represented in all three chip cores.** --- # Application to Unknown 23andMe Files [link](https://observablehq.com/@lorenasandoval88/23andme-chip-classification) The chip-specific coordinate sets can now be used as **reference fingerprints** for 23andMe files whose chip version is unknown. For an unknown genotype file, its genomic positions can be compared against:

v3-specific fingerprint: 352,539 possible positions
v4-specific fingerprint:  59,983 possible positions
v5-specific fingerprint: 430,552 possible positions

Because the fingerprint sets differ substantially in size, classification should use the **proportion of each fingerprint detected**, rather than comparing raw match counts alone. For example:

v3 match rate =
v3-specific coordinates found in unknown file
÷ 352,539

v4 match rate =
v4-specific coordinates found in unknown file
÷ 59,983

v5 match rate =
v5-specific coordinates found in unknown file
÷ 430,552

This allows chip version to be investigated using the **actual pattern of genomic positions contained in the file**, rather than relying only on filenames such as genome.txt. Unknown files should be compared using the same genome build because genomic coordinates can differ between genome assemblies. --- # Analysis Process ## Overview This analysis compared the **core genomic coordinates** from the 23andMe v3, v4, and v5 Build 37 chip manifests. Each marker is represented by its physical genomic position: `chromosome:position` For example: `1:752566` The comparison separated the core positions for each chip into two main groups: - **Shared markers** — positions also present in at least one of the other chip versions. - **Chip-specific markers** — positions present in that chip but absent from both other chip versions. The analysis also calculated: - pairwise overlap between v3 and v4 - pairwise overlap between v3 and v5 - pairwise overlap between v4 and v5 - positions shared across all three chips --- ## Input Core Manifests ```text gs://all23_0710/chip_manifests/v3_build37_95pct/coordinates_core.json gs://all23_0710/chip_manifests/v4_build37_95pct/coordinates_core.json gs://all23_0710/chip_manifests/v5_build37_95pct/coordinates_core.json ``` --- ## Workflow

v3 core coordinates ─────┐
                         │
v4 core coordinates ─────┼────► Compare chromosome:position
                         │
v5 core coordinates ─────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │ Coordinate overlap  │
                    └─────────────────────┘
                         │           │
                         ▼           ▼
                    Shared       Chip-specific
                    markers         markers
                         │           │
                         └─────┬─────┘
                               ▼
                    Chip fingerprint files

--- ## Run the Analysis The comparison script is saved as:

index.js

Run it with:

cd ~/chip-fingerprints
node index.js

The script: 1. reads the v3, v4, and v5 core coordinate manifests, 2. converts each coordinate list into a set, 3. compares the sets using `chromosome:position`, 4. identifies coordinates shared with at least one other chip version, 5. identifies coordinates specific to each chip, 6. calculates pairwise and three-way overlap, and 7. writes the resulting fingerprint files to Google Cloud Storage. --- ## Output Files

gs://all23_0710/chip_manifests/chip_fingerprints/
├── v3_chip_fingerprint.json
├── v4_chip_fingerprint.json
└── v5_chip_fingerprint.json

Each fingerprint contains:

chip core markers
│
├── sharedMarkers
│   └── also present in at least one other chip core
│
└── chipSpecificMarkers
    └── absent from both other chip cores

--- ## Generated Fingerprint Files

v3_chip_fingerprint.json
  ├── 607,163 shared markers
  └── 352,539 v3-specific markers

v4_chip_fingerprint.json
  ├── 536,264 shared markers
  └── 59,983 v4-specific markers

v5_chip_fingerprint.json
  ├── 192,201 shared markers
  └── 430,552 v5-specific markers

34:[{"id":"ed0649bb4e328c0cef0740a389b353a345f35b4c53205b80d02905bcd4383ec3760ac29b75723654a21a9f805e41bb95be7c5a8635b07f55ea49fe1aa0320793","href":"https://static.observableusercontent.com/files/ed0649bb4e328c0cef0740a389b353a345f35b4c53205b80d02905bcd4383ec3760ac29b75723654a21a9f805e41bb95be7c5a8635b07f55ea49fe1aa0320793","name":"index.js","type":"text/javascript","lastModified":1790699333279,"size":9896},{"id":"c1f7342f31378b57b0300813e3613b89b83c3a4c270bc291c7a6218b8a6af35c3c0d232bed2f20185c5d2cef7c46660d17e980bfc7ac08dae5b7f8c281cf574a","href":"https://static.observableusercontent.com/files/c1f7342f31378b57b0300813e3613b89b83c3a4c270bc291c7a6218b8a6af35c3c0d232bed2f20185c5d2cef7c46660d17e980bfc7ac08dae5b7f8c281cf574a","name":"v5_build37_input_files.json","type":"application/json","lastModified":1790953117767,"size":80806},{"id":"f585f7a15d8563b7ffab4050ca046b95e7b254c0005808de5a3eef553dd4414faf928b082c4de1c71ad9952399d3a4b2a709da82786b7f7f7c29dd69be5a3b5c","href":"https://static.observableusercontent.com/files/f585f7a15d8563b7ffab4050ca046b95e7b254c0005808de5a3eef553dd4414faf928b082c4de1c71ad9952399d3a4b2a709da82786b7f7f7c29dd69be5a3b5c","name":"v4_build37_input_files.json","type":"application/json","lastModified":1790953085651,"size":93344},{"id":"476a91b1e0a892cd28c8af766f6deec22d43cb9c8bb841cb93e55ca5dca8e078ddad60290bb13729e75b44773d3803d49943682f4afa49d4b4bbb9de8dd2c213","href":"https://static.observableusercontent.com/files/476a91b1e0a892cd28c8af766f6deec22d43cb9c8bb841cb93e55ca5dca8e078ddad60290bb13729e75b44773d3803d49943682f4afa49d4b4bbb9de8dd2c213","name":"v3_build37_input_files.json","type":"application/json","lastModified":1790953052902,"size":21193},{"id":"73eea87219a6950a34cd08c12d6879e310306f5271aac840e8c400b5e459c7e5a35fd7474bbd7a4022ea75efe7cb47181a89aaa6732755a8a12cb44b83de4d3c","href":"https://static.observableusercontent.com/files/73eea87219a6950a34cd08c12d6879e310306f5271aac840e8c400b5e459c7e5a35fd7474bbd7a4022ea75efe7cb47181a89aaa6732755a8a12cb44b83de4d3c","name":"v3_variant_manifest_core.zip","type":"application/x-zip-compressed","lastModified":1790955014791,"size":15636587},{"id":"709b8a75913debffe7e5ac9733b4212ab81709745238754902fa3b4df07f5476049fd80252b6e17c3a72df2ca8cdf4514bf5a62ac56406e7c73892e3355c57a2","href":"https://static.observableusercontent.com/files/709b8a75913debffe7e5ac9733b4212ab81709745238754902fa3b4df07f5476049fd80252b6e17c3a72df2ca8cdf4514bf5a62ac56406e7c73892e3355c57a2","name":"v5_variant_manifest_core.zip","type":"application/x-zip-compressed","lastModified":1790956067587,"size":10763904},{"id":"21cbaa46ea32ce325d69474b10931ebf65dad2b60612883c1f532a34c1c428aad99e1f9df1f36caf84c8c49ac30b46bde0c680cb2a35d515f89172aad15a287c","href":"https://static.observableusercontent.com/files/21cbaa46ea32ce325d69474b10931ebf65dad2b60612883c1f532a34c1c428aad99e1f9df1f36caf84c8c49ac30b46bde0c680cb2a35d515f89172aad15a287c","name":"v4_variant_manifest_core.zip","type":"application/x-zip-compressed","lastModified":1790956678820,"size":9930916},{"id":"20478c8f53c868f6c008842d0ed161e0b86d4900bc8ab65abee6c8b1630f97cf69df51dae9458da13e48a57c868090c68c639f64270c129f9e54fa9d4d425f19","href":"https://static.observableusercontent.com/files/20478c8f53c868f6c008842d0ed161e0b86d4900bc8ab65abee6c8b1630f97cf69df51dae9458da13e48a57c868090c68c639f64270c129f9e54fa9d4d425f19","name":"v3_chip_fingerprint.json","type":"application/json","lastModified":1790950636812,"size":17881400},{"id":"9d893fb133cf3c79d34e64a33b8facd654a109e022dd4f4eca678bdee0c87dae0f1ebc43164622c264fff8d7698e5590ac70fc16de0652be9ad51cd3edab5338","href":"https://static.observableusercontent.com/files/9d893fb133cf3c79d34e64a33b8facd654a109e022dd4f4eca678bdee0c87dae0f1ebc43164622c264fff8d7698e5590ac70fc16de0652be9ad51cd3edab5338","name":"v5_chip_fingerprint.json","type":"application/json","lastModified":1790950472687,"size":11598119},{"id":"1aa5277bb1378b53d0432ef236c7fbcb300f6e79db3a0e6efa2316375c39e5c193677eb9f7daac81f1aea4f6a7a2388f071c39bdfbc46083671c67df1cc8e3cf","href":"https://static.observableusercontent.com/files/1aa5277bb1378b53d0432ef236c7fbcb300f6e79db3a0e6efa2316375c39e5c193677eb9f7daac81f1aea4f6a7a2388f071c39bdfbc46083671c67df1cc8e3cf","name":"v4_chip_fingerprint.json","type":"application/json","lastModified":1790950718386,"size":11099852},{"id":"fe29c8fec76bd4956389a5947cf4debd1669cc34057a5d3b9a66fad0149fe69ab584248c872ed225a7f5abe7952dfcaa09d09db4ab46cc096f2bb6c9cf13d6c0","href":"https://static.observableusercontent.com/files/fe29c8fec76bd4956389a5947cf4debd1669cc34057a5d3b9a66fad0149fe69ab584248c872ed225a7f5abe7952dfcaa09d09db4ab46cc096f2bb6c9cf13d6c0","name":"image.png","type":"image/png","lastModified":1790967802287,"size":1341169},{"id":"999fbda367b11384764a2f859c85a34c04b46ed00b648607f10a6f28bb54c83e4eea57bcf31deb64705771649234fdf0b76a73541f396d9d7d8a6bffe63695be","href":"https://static.observableusercontent.com/files/999fbda367b11384764a2f859c85a34c04b46ed00b648607f10a6f28bb54c83e4eea57bcf31deb64705771649234fdf0b76a73541f396d9d7d8a6bffe63695be","name":"v2_chip_fingerprint.json","type":"application/json","lastModified":1791397034284,"size":10149391},{"id":"dc9177f7c3b8104c2703450dcf55f2e44f168b6d4ee1da831c3c539c3f116f664df204021c6f717e9b593a892503613c6d7de75d363349cf04461e6cf6720583","href":"https://static.observableusercontent.com/files/dc9177f7c3b8104c2703450dcf55f2e44f168b6d4ee1da831c3c539c3f116f664df204021c6f717e9b593a892503613c6d7de75d363349cf04461e6cf6720583","name":"v2_variant_manifest_core.zip","type":"application/x-zip-compressed","lastModified":1791399265592,"size":8969102}] 30:{"id":"e06c47b44903d567","slug":"23andme-v3v4v5-overlap","title":"23andMe v2/v3/v4/v5 Overlap","thumbnail":"5e933c9138839b51f4c6864ef9da575c11f990662c843bcb6eb95ad8af70c82c","access_level":"public","create_time":"$D2026-09-28T14:14:07.177Z","update_time":"$D2026-10-07T18:56:19.862Z","view_time":null,"first_public_time":"$D2026-09-29T16:56:23.794Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"5a5147324ed15c62","login":"lorenasandoval88","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":"starter_2024","bio":""},"version":409,"publish_version":409,"role":null,"authored":false,"authors":[{"type":"individual","id":"5c479a2c5ae5f52f","login":"lorenasandoval","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":null,"bio":""}],"members":[{"type":"team","id":"5a5147324ed15c62","login":"lorenasandoval88","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c8926e2434569cc1f72aeac62929e824fdec3f0ec037d4a41a38a7728d14e950","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":{"id":"1476de78fbd46e52","slug":"23andme-data","title":"23andme-open-data","owner":{"id":"5a5147324ed15c62","login":"lorenasandoval88","name":""}},"type":"notebook","body":{"titling":"auto","title":"23andMe v2/v3/v4/v5 Overlap","cells":[{"id":64,"value":"# 23andMe v2/v3/v4/v5 Overlap","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":59,"value":"$32","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":78,"value":"

\n\n${html``}\n\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":91,"value":"v2_core = (await (await FileAttachment(\"v2_variant_manifest_core.zip\").zip())\n .file(\"v2_variant_manifest_core.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":90,"value":"v2_intersection = (await (await FileAttachment(\"v3_chip_fingerprint.json\").json()))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":12,"value":"v3_core = (await (await FileAttachment(\"v3_variant_manifest_core.zip\").zip())\n .file(\"v3_variant_manifest_core.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":83,"value":"v3_intersection = (await (await FileAttachment(\"v3_chip_fingerprint.json\").json()))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":31,"value":"v4_core = (await (await FileAttachment(\"v4_variant_manifest_core.zip\").zip())\n .file(\"v4_variant_manifest_core.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":85,"value":"v4_intersection = (await (await FileAttachment(\"v4_chip_fingerprint.json\").json()))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":34,"value":"v5_core = (await (await FileAttachment(\"v5_variant_manifest_core.zip\").zip())\n .file(\"v5_variant_manifest_core.json\")\n .json())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":50,"value":"v5_intersection = (await (await FileAttachment(\"v5_chip_fingerprint.json\").json()))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":39,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10,"value":"\n\n## Results\n\nThe v3 and v4 arrays show substantial continuity, with **88.6% of v4 core coordinates also present in v3**. In contrast, v5 represents a major shift in array content: **69.1% of v5 core coordinates are absent from both v3 and v4**.\n\n---\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":3,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7,"value":"","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W34","stdlib":"2"}} 31:"$30:body"