44:["$","title",null,{"children":"Nanopore Phage Pipeline | WabushOnce | Observable"}] 4c:Ta6a40,{ // ========================================================== // Nanopore phage pipeline — WSL script generator (Observable) // dedup → (picked-plaque samples: host-read removal) → Filtlong → Flye → Medaka → Flye-vs-Medaka report // v3: several phages at once; a time on every step + timing table; Autocycler falls back to Flye when it fails // or fragments; host-read removal never leaves < HOST_MIN_KEEP_PCT of bases and lists its samples up front // → longest contig → Pharokka → Phold // v4: library kit setting (transposase kits: no read-end detection); Autocycler vs Flye chosen by alignment, host // match and read depth (depth alone never removes sequence); host-matching contig ends trimmed; coverage-dip // report; end-of-run flags for low depth, small contigs and sequences shared between samples // v5: Autocycler improvements: host-read removal for all samples (host genome), genome size from a full-read Flye // draft, that Flye assembly added as a weighted vote, default assemblers flye raven canu (miniasm if no Canu) // v6: Flye never sees more than a capped read set: a first Flye on the best reads (cap in Mb) gives the genome size, // then Flye at several depths of the best reads (default 200x 100x 50x) are added to Autocycler as votes // v7: the PyTorch / GPU check uses the Python that Phold itself runs with (from the phold script's first line), // not whichever python3 is first on PATH, and prints that interpreter if the check fails; Phold runs without // LD_LIBRARY_PATH / PYTHONPATH / PYTHONHOME left over from other environments (e.g. Dorado's bundled libraries) // v8: closest PhagesDB phage for every genome (local PhagesDB FASTA, minimap2); new genome-start option rotates a // circular genome to start where that relative starts (same strand); terL when there is no close relative // v9: PhagesDB headers are read (host, phage name, length, cos overhang / end type, cluster): every genome gets // a unique ID in the index (all headers start "Mycobacterium phage …"), the report shows the relative's // cluster, host and ends, and the genome start is set on the relative's 3' overhang when it is found there // v10: the PhagesDB data table (tab-delimited) adds subcluster, temperate, GC%, end type, overhang, terminal repeat, // ORFs, tRNAs, morphotype, host strain and accession for the closest phage; GC% and gene counts are compared // v11: coverage.html shows the closest PhagesDB phage (PhagesDB and GenBank links, cluster, ends, GC, genes) with a // genome comparison: aligned blocks by identity, inversions, regions found in only one of the two genomes // v15: PhageTracker import in Tracker Sync (no Claude step): the generator loads the sample set's phage IDs, the // pipeline writes /phage.json and puts phage_id / RUD-ID in phagetracker_import_.json; genome checks // (closest PhagesDB/GenBank phage, host DNA %, CheckV, similarity) run at the end of each run and go into the // import; the script carries its version; form values are remembered and lab-wide paths come from Lab settings. // v14: (also) overhang check recognises PhagesDB entries that carry the 3' overhang at the right end // v14: time limit per Autocycler subset assembly and per Flye vote (a stuck assembler is stopped and skipped // instead of halting the whole run); time limit for Phold, and a Phold failure no longer fails the phage // (it keeps the Pharokka annotation and the run continues) // v13: new defaults: combine FASTQs per folder, no Filtlong --target_bases, Accuracy (Autocycler) mode, // Medaka on the best reads capped at a target depth // v12: "Copy run command" button: runs the newest phage_pipeline*.sh in your Windows Downloads folder from WSL; // the PhagesDB search also reports more distant relatives (secondary hits down to 20% of the best score) // ========================================================== const wrap = document.createElement("div"); wrap.style.maxWidth = "980px"; wrap.style.fontFamily = "system-ui, -apple-system, Segoe UI, sans-serif"; wrap.innerHTML = `

Windows and WSL paths are accepted; Windows drive paths are converted to WSL paths. Each sample runs: deduplicate → Filtlong → Flye → Medaka → Flye-vs-Medaka report → longest contig → Pharokka → Phold. One failed sample no longer stops the batch, and finished stages are skipped when you rerun.

1. Input and output folders

Parent folder — Windows or WSL path

Include subfolders This folder only One FASTQ file

No path entered

Output parent folder — Windows or WSL path

No path entered

Folder behavior

Combine the FASTQs in each folder Each FASTQ file separately

Overwrite completed samples and rerun everything Reports only: keep every existing assembly as it is (whatever the assembly settings) and rebuild the coverage reports and genome checks; phages without an assembly are skipped

Example output for reads in …\\reads\\PhageA\\barcode07.fastq.gz:

1b. Sample names

Dorado sample sheet — recommended

No sample sheet: samples are named from folders

Optional. The sample sheet is the CSV file the Nanopore Workflow app makes, listing which phage is on which barcode. With it, every phage is named exactly as in the sheet, and reads that are not in the sheet are left out. Without it, phages are named after their folders (see Sample naming).

Sample naming

reads\\PhageA\\fastq_pass\\x.fastq.gz → PhageA (first option) or fastq_pass (second). A FASTQ directly in the parent folder is named after its file. If two samples would get the same name, they fall back to path-based names so nothing is overwritten.

Prefix contig names in the polished FASTA with the sample name (e.g. PhageA_contig_1) Skip Dorado's unclassified reads (reads with no barcode are not assembled)

2. Conda environments

Filtlong + Flye environment

Medaka environment

minimap2 (installed with Medaka) is used for the Flye-vs-Medaka report.

Pharokka environment

Phold environment

Conda base folder — optional

3. Read QC, preparation + Filtlong

Before Filtlong, the first occurrence of each read ID is kept and later duplicates removed (report saved per sample).

NanoPlot report for each sample NanoComp report comparing all samples

NanoPlot / NanoComp threads

Both run on the deduplicated reads before Filtlong, named by phage, from the Filtlong + Flye environment. Each writes a NanoStats.txt that the Nanopore Workflow app's Dorado tab can import.

Minimum read length (bp)

Minimum mean read Q (Phred)

Target bases (Mb) Use --target_bases

Keep best reads (%) Use --keep_percent

Host-read removal for DNA prepared from a picked plaque

DNA from a picked plaque can carry a lot of host DNA. Lysate and purified DNA are not affected: only the samples listed here are cleaned, and the cleaned reads then go to Filtlong, the assembler, Medaka, genome-end detection and the coverage report. Two methods: No host genome: a quick Flye --meta draft of the sample finds the high-depth (phage) contigs and only reads that align to them are kept; this also removes other contaminants. Host genome: reads are mapped to the host's genome FASTA with minimap2 and the matches are discarded. If the host carries a prophage (or you are not sure), reads that also match the phage draft are protected, so a phage related to the prophage is not thrown away. The share of reads that were phage is saved to qc/host_removal.txt, the summary table and the PhageTracker export (a host_depleted read set). If any step fails, the sample is assembled from all reads, as before.

Remove host reads

Method

Picked-plaque samples (phage names as in the FASTQ folder names; barcode numbers also work)

The button reads the sample set in section 9 and lists every barcode whose DNA source is Plaque (Lysate and DNA samples are left out) and fills in their names. It needs an editor sign-in in the Tracker Sync cell. Prefer names: a barcode number matches that barcode in whatever run you process, so a list typed for one sample set can catch the wrong samples in another. The script prints which samples it will clean before it starts.

Phage read: minimum aligned length (%) Phage contig: minimum depth (fraction of the deepest) Safety: keep all reads if less than this % of bases would be left

4. Assembly

Genome size group (fills the three genome-size settings below)

Sets "Estimated genome size" (Flye), "Smallest genome-size estimate" and "First Flye run: at most this many Mb". Too small starves a large genome of reads; too large overloads Flye on a small phage. Run the two groups as separate batches.

Assembly mode

Accuracy mode assembles several read subsets with several assemblers and keeps only the sequence they agree on, then finds the physical genome ends and rotates circular genomes.

Assembler threads (Flye / Autocycler)

Ryzen 7 9700F: 8 cores / 16 threads. With several phages at once, these threads are shared between them.

Phages at once

Most steps use only 1–2 threads, so running 3–4 phages side by side is much faster. Each phage writes its own log, added to the main log when it finishes. Medaka and Phold take turns on the GPU. With 16 GB of RAM, keep this at 4 or less.

Work on the WSL drive (faster)

Each phage is assembled in this Linux folder and copied to the output folder when it finishes. WSL reads and writes much faster there than on C:\ (/mnt/c), above all for Canu in accuracy mode. Needs free space for the phages running at once (a few GB each). Only used when the output folder is on /mnt/….

Autocycler environment

Assemblers (space separated)

Any of: flye raven miniasm canu redbean metamdbg myloasm. Missing ones are skipped (Canu missing → miniasm is used instead); at least 2 needed. Canu is more accurate than miniasm on small genomes at subset depth.

Read subsets per assembler

Time limit per subset assembly and per Flye vote (minutes)

On difficult samples an assembler (most often Canu) can run for hours. After this time it is stopped and that assembly is skipped (logged as "timed out"); the phage carries on with the others. Phage-sized assemblies normally take 1–5 minutes. 0 = no limit.

Genome size for Autocycler

flye = from a Flye assembly of all the reads (the high-depth contigs; host and other low-depth contigs don't count). auto = Autocycler's estimate from the reads, which contaminating reads inflate. Or a fixed size like 50k.

Add Flye assemblies at several read depths to Autocycler as votes Read depths for the Flye votes (×, best reads by Filtlong) Weight of each Flye vote First Flye run: at most this many Mb of the best reads

Flye can fail or fragment with too much data (e.g. over ~20 Mb for a small phage), so it never gets all the reads blindly. The first run (best reads up to this many Mb) gives the genome size; then Flye runs on the best reads at each depth listed (a depth above what's available becomes "all reads"; below 30× is skipped; near-duplicates are run once). The longer, higher-quality reads span repeats and low-depth regions (DNA modifications, PCR bias). The read sets overlap, so they are votes, not independent subsets: each counts this many times in Autocycler's clustering and consensus, and together they can't outvote the 12 subset assemblies. About a minute per Flye run.

Cross-check with Flye when Autocycler's genome isn't circular

Flye also assembles the same reads (about a minute). The longer assembly is kept unless its extra sequence matches the host genome; low read depth alone never removes sequence (PCR and DNA modifications can lower depth), it is only flagged. Host-matching contig ends are trimmed. If Autocycler fails, Flye is used ("flye_fallback").

Reads for Autocycler (Mb)

The best reads up to this many megabases (all reads when there are fewer). Does not depend on the genome-size estimate, which can be far too small for a new phage. 0 = use the depth cap below instead.

Maximum read depth for Autocycler (×)

Only used when "Reads for Autocycler" is 0: best reads up to this depth of the estimated genome size; 0 = no cap.

Smallest genome-size estimate (bp)

An estimated genome size below this is raised to it (a high-depth repeat or a less-modified, over-amplified region can make the estimate far too small). 0 = off. A number typed in "Autocycler genome size" is used as is.

Read type for Autocycler

Genome start (circular genomes)

"Match the closest PhagesDB phage": the genome is rotated (and flipped if needed) to start where its closest PhagesDB relative starts, so it follows the cluster's convention. If no relative covers at least half of the genome, or its start can't be placed confidently, the genome starts at terL instead. Native/ligation kits also show read pile-ups at cos/pac ends; Rapid kits do not.

PhagesDB genomes FASTA (for the closest phage and the reference start)

PhagesDB data table (tab-delimited; optional)

The PhagesDB phage list ("Phage Name, Host, Cluster, Subcluster, …"). Adds the closest phage's subcluster, temperate or not, GC%, end type and overhang, terminal repeat, ORF and tRNA counts, morphotype and GenBank accession. A name without an extension also finds .txt, .tsv or .tab.

The all-genomes FASTA from PhagesDB (Data downloads). Indexed once (a few minutes) into a phagesdb_index folder beside it, and re-indexed automatically when you download a newer file. Every genome gets its closest PhagesDB phages listed in qc/closest_phagesdb.tsv and the summary table. Leave blank to skip.

BLAST buttons in the coverage report: database lookup Worker (optional)

Optional Cloudflare Worker, only used to look up the current PhagesDB database (/phagesdb-version) when the pipeline cannot read the list from PhagesDB itself. Both BLAST buttons submit a normal form (POST) straight to NCBI and PhagesDB.

PhagesDB BLAST form: submission URL · sequence field name · database (blank = automatic)

Use the same values as phagesdbBlastUrl and sequenceFieldName in your BLAST notebook if they differ.

dnaapler environment — optional

Keep every individual assembly (large; for troubleshooting)

Library kit

Look for physical genome ends (read start/stop pile-ups) — ligation kits only

Flye (Standard mode, and the Flye runs in Accuracy mode: first run for the genome size, the votes and the cross-check)

Nanopore read type

Estimated genome size

Flye polishing iterations

Minimum overlap (bp)

0 = Flye chooses automatically.

Assembly coverage

0 = disabled. Requires a genome size.

Use --meta Use --scaffold

5. Medaka

Reads used for polishing

Phage runs often have thousands-fold coverage; capping speeds Medaka up without losing accuracy.

Target depth for Medaka (×)

Medaka threads

Medaka model

Example: r1041_e82_400bps_sup_v5.2.0.

Use --bacteria (Medaka ≥ 2.0 model trained on native, host-methylated DNA; ignores the model field)

6. Flye vs Medaka report

Align the Medaka consensus to the Flye assembly and list every change

Writes qc/<sample>.flye_vs_medaka.summary.tsv (per contig: length change, substitutions, insertions, deletions, homopolymer indels, identity) and …changes.tsv (every edit with positions in both assemblies and sequence context).

7. Coverage and consensus report

Map reads back to the polished assembly and draw an interactive coverage / consensus plot

Flag bases with read agreement below (%)

Flag coverage below (×)

Reads for the coverage plots

Writes <sample>.coverage.html (opens in any browser, no software needed): coverage along each contig, the percentage of reads agreeing with the consensus at every base, Medaka's edits, and the Pharokka/Phold gene map, with drag-to-zoom and hover read-outs. Uses the same reads Medaka used. Also keeps a per-base table (qc/<sample>.coverage.tsv.gz) and a BAM file you can open in IGV. Needs samtools, which is normally installed with Medaka.

8. Annotation of the longest contig

The longest polished contig is written to <sample>.longest_contig.fasta with the header ><sample>, so Pharokka/Phold LOCUS names match the sample. Other contigs are listed in the log.

Run Pharokka

Pharokka database (WSL path)

Gene predictor

Pharokka threads

Fast mode (skips PHROG HMM search)

Run Phold on the Pharokka GenBank

Phold database (WSL path)

ProstT5 hardware

Phold threads

ProstT5 batch size

Phold time limit per phage (minutes)

Phold normally takes under a minute per phage on the GPU. If it runs longer it is stopped (freeing the GPU for the next phage) and tried once on the CPU with the same limit; if that also fails, the phage keeps its Pharokka annotation. 0 = no limit.

Install missing Pharokka/Phold databases (large downloads; otherwise the script stops before assembling if a database is missing)

9. PhageTracker export

Write phagetracker_import_<time>.json for importing into the Rudner lab PhageTracker database

Sample set identifier

Sequencing run ID — optional

SharePoint link to the output folder — optional

The file collects, for every sample: read counts and quality (raw and Filtlong), the barcode, both assemblies with their contigs, coverage and Medaka statistics, every Pharokka/Phold gene and the genome checks (section 9b). Import it in Tracker Sync → 4. Import results: a preview shows which phage each folder matches, then one click writes the records. Phage records themselves are not changed until you approve a genome.

Recommended: links every output folder to its phage record by ID (written to <phage>/phage.json), so renamed phages or folders can't be imported into the wrong record. Needs the sample set identifier above and an editor sign-in in the Tracker Sync cell.

10. Finish

Make a SharePoint_Upload folder with only the files worth keeping

Include the read alignments (.bam, tens–hundreds of MB per phage; for IGV) Include the reads (unique.fastq.gz; usually already uploaded from Dorado's Phage_FASTQ)

Same layout as the output folder (<phage>/<file>), so upload its contents into the SharePoint folder whose link is in section 9 and the PhageTracker links work. Files are hard-linked, so the folder takes almost no extra disk space.

Open the output folder in Windows Explorer

Download .sh

Recommended: Download .sh, then paste the run command into WSL. It runs the newest phage_pipeline*.sh in your Downloads folder (also a renamed copy such as phage_pipeline (1).sh):

Show the generated script

`; const $ = (id) => wrap.querySelector(`#${id}`); // v19: highlight the fields usually set for each run; collapsible sections (remembered); long notes become numbered // footnotes at the bottom of their section (#n after the field's label: hover to read, click to jump) // v20: section 4 keeps the everyday settings; the rest goes into a collapsed "Advanced assembly settings" block (function advancedAssembly() { const h = [...wrap.querySelectorAll(".phage-box > h3")].find((x) => x.textContent.trim().startsWith("4.")); if (!h) return; const box = h.parentElement; const fieldOf = (id) => { const el = wrap.querySelector("#" + id); return el && (el.closest(".phage-field") || el.closest("label.phage-check")); }; const adv = document.createElement("details"); adv.className = "phage-advanced"; adv.innerHTML = '

Advanced assembly settings ' + "(assemblers, Flye votes, time limits, Flye options, PhagesDB and BLAST paths — usually left as they are)

"; const advAcc = document.createElement("div"); advAcc.id = "advAcc"; advAcc.className = "phage-grid"; advAcc.style.cssText = "border-left:3px solid #7c3aed;padding-left:12px;margin:10px 0"; for (const id of ["autocyclerEnv", "acAssemblers", "acCount", "acTimeout", "acGenomeSize", "acFlyeVote", "acMaxDepth", "acReadType", "phagesdbFasta", "phagesdbTable", "ncbiBlastProxy", "phagesdbBlastUrl", "dnaaplerEnv", "acKeep"]) { const f = fieldOf(id); if (f && !advAcc.contains(f)) advAcc.appendChild(f); } // the Flye genome size stays visible (with the other genome-size settings) const gs = fieldOf("genomeSize"), basic = fieldOf("assemblyMode"); if (gs && basic) basic.parentElement.appendChild(gs); const flye = document.createElement("div"); flye.style.margin = "8px 0"; box.querySelectorAll(".std-label, .std-only").forEach((e) => flye.appendChild(e)); adv.appendChild(advAcc); adv.appendChild(flye); box.appendChild(adv); })(); // v22: host-read removal laid out as a stack: removal choice → sample list (only these samples) → method (when on) // → host genomes as a table (name | genome FASTA | prophage) → fine-tuning (collapsed) (function hostLayout() { const rmSel = wrap.querySelector("#hostRemoval"); if (!rmSel) return; const rm = rmSel.closest(".phage-field"), grid = rm.parentElement; const method = wrap.querySelector("#hostMethodField"), plaque = wrap.querySelector("#plaqueField"); const params = wrap.querySelector("#hostParamFields"), refs = wrap.querySelector("#hostRefFields"); const intro = grid.previousElementSibling; // the long explanation above the controls const stack = document.createElement("div"); stack.className = "host-stack"; grid.parentElement.insertBefore(stack, grid); // fold the explanation behind one line if (intro && intro.classList.contains("phage-note") && intro.textContent.length > 300) { const d = document.createElement("details"); d.innerHTML = '

About host-read removal (methods, prophages, what is saved)

'; intro.parentElement.insertBefore(d, intro); intro.style.marginBottom = "4px"; d.appendChild(intro); } stack.appendChild(rm); plaque.style.gridColumn = ""; stack.appendChild(plaque); stack.appendChild(method); // host genomes: a 3-column table const title = document.createElement("div"); title.className = "host-refs-title"; title.id = "hostRefsTitle"; refs.insertBefore(title, refs.firstChild); const note = refs.querySelector(":scope > .phage-note"); const table = document.createElement("div"); table.className = "host-table"; table.innerHTML = '

Host name

Genome FASTA (Windows or WSL path; may be .gz)

Does it carry a prophage?

'; refs.querySelectorAll(".host-row").forEach((row, i) => { const lab = (row.querySelector("label") || {}).textContent || `Host ${i + 1}`; const c1 = document.createElement("div"), c2 = document.createElement("div"), c3 = document.createElement("div"); c1.innerHTML = `

${lab}

`; c1.appendChild(row.querySelector(`#hostName${i + 1}`)); const g = row.querySelector(`#hostGenome${i + 1}`); g.style.marginTop = "0"; c2.innerHTML = '

'; c2.appendChild(g); c2.appendChild(row.querySelector(`#hostGenomePreview${i + 1}`)); const pr = row.querySelector(`#hostProphage${i + 1}`); pr.style.marginTop = "0"; c3.innerHTML = '

'; c3.appendChild(pr); table.append(c1, c2, c3); row.remove(); }); refs.style.marginTop = "0"; refs.appendChild(table); if (note) { const det = document.createElement("details"); det.innerHTML = '

How the host genomes are used

'; det.appendChild(note); refs.appendChild(det); } stack.appendChild(refs); // fine-tuning: three numbers side by side, collapsed const det = document.createElement("details"); det.id = "hostParamBox"; det.innerHTML = '

Host-removal fine-tuning (usually left as is)

'; const pg = document.createElement("div"); pg.className = "host-params"; params.querySelectorAll("input").forEach((inp) => { const f = document.createElement("div"); f.className = "phage-field"; const l = inp.previousElementSibling && inp.previousElementSibling.tagName === "LABEL" ? inp.previousElementSibling : null; if (l) { l.style.marginTop = "0"; f.appendChild(l); } f.appendChild(inp); pg.appendChild(f); }); det.appendChild(pg); params.replaceChildren(det); params.classList.remove("phage-field"); stack.appendChild(params); if (!grid.children.length) grid.remove(); })(); (function tidyForm() { const KEY_FIELDS = ["parentFolder", "singleFastq", "sampleSheet", "outputFolder", "overwrite", "hostRemoval", "plaqueSamples", "gsizeGroup", "assemblyMode", "acMaxMb", "libKit", "ptSampleSet", "ptSeqRun", "makeUpload"]; const COLLAPSED = ["2.", "5.", "6.", "8.", "9b."]; const LONG = 150; for (const id of KEY_FIELDS) { const el = wrap.querySelector("#" + id); if (!el) continue; const box = el.closest(".phage-field") || el.closest("label.phage-check") || el.parentElement; if (box) box.classList.add("phage-key"); } let open = {}; try { open = JSON.parse(localStorage.getItem("phagePipelineSections") || "{}") || {}; } catch (e) { open = {}; } const saveOpen = () => { try { localStorage.setItem("phagePipelineSections", JSON.stringify(open)); } catch (e) { /* ignore */ } }; const setters = []; wrap.querySelectorAll(".phage-box").forEach((box, bi) => { const h3 = box.querySelector(":scope > h3"); if (!h3) return; const title = h3.textContent.trim(); const num = title.split(" ")[0]; // long static notes -> footnotes const notes = [...box.querySelectorAll(".phage-note")].filter((n) => !n.id && n.textContent.trim().length > LONG && !n.querySelector("input, select, button, textarea, [id]")); const list = document.createElement("ol"); list.className = "phage-fn"; let k = 0; for (const n of notes) { const prev = n.previousElementSibling; const field = n.closest(".phage-field"); const label = prev && prev.tagName === "LABEL" ? prev : field ? field.querySelector("label") : null; if (!label) continue; k += 1; // v26: the note opens under its field from an ⓘ button next to the label const btn = document.createElement("button"); btn.type = "button"; btn.className = "phage-info"; btn.textContent = "i"; btn.title = "Explanation (click to show or hide)"; btn.setAttribute("aria-expanded", "false"); n.classList.add("phage-pop"); btn.addEventListener("click", (ev) => { ev.preventDefault(); ev.stopPropagation(); const o = !n.classList.contains("open"); n.classList.toggle("open", o); btn.setAttribute("aria-expanded", o ? "true" : "false"); }); label.appendChild(btn); } // collapsible body const body = document.createElement("div"); body.className = "phage-body"; while (h3.nextSibling) body.appendChild(h3.nextSibling); if (list.children.length) body.appendChild(list); box.appendChild(body); h3.classList.add("phage-toggle"); h3.title = "Click to show or hide this section"; const hint = document.createElement("span"); hint.className = "phage-closed-hint pp-sum"; hint.textContent = "(closed — click to open)"; h3.appendChild(hint); (wrap.__hints = wrap.__hints || {})[num] = hint; const set = (o) => { body.style.display = o ? "" : "none"; h3.dataset.open = o ? "1" : "0"; hint.style.display = o ? "none" : ""; }; set(title in open ? !!open[title] : !COLLAPSED.includes(num)); h3.addEventListener("click", () => { const o = h3.dataset.open !== "1"; set(o); open[title] = o; saveOpen(); }); setters.push((o) => { set(o); open[title] = o; }); }); // the example output tree is long: fold it (with its caption) behind a summary line wrap.querySelectorAll(".phage-tree").forEach((tree) => { const det = document.createElement("details"); const sum = document.createElement("summary"); sum.textContent = "Show the example output folder"; sum.style.cssText = "cursor:pointer;font-size:13px;margin:6px 0"; const cap = tree.previousElementSibling; tree.parentNode.insertBefore(det, tree); det.appendChild(sum); if (cap && !cap.matches("input, select, label, .phage-grid, .phage-field") && cap.textContent.length < 160) det.appendChild(cap); det.appendChild(tree); }); // legend + open/close all const first = wrap.querySelector(".phage-box"); if (first) { const lg = document.createElement("div"); lg.className = "phage-legend"; lg.innerHTML = 'Highlighted= fields you usually set for each run' + '· click a section title to show or hide it · ⓘ = explanation of a setting' + ''; lg.addEventListener("click", (ev) => { const b = ev.target.closest("button[data-all]"); if (!b) return; for (const f of setters) f(b.dataset.all === "1"); saveOpen(); }); first.parentNode.insertBefore(lg, first); } })(); // v20: run profiles — built-in ones fill a few related settings; your own save the whole form (this browser only) (function profiles() { const BUILTIN = { actino: { label: "Actinobacteriophage batch (~40–70 kb)", v: { assemblyMode: "accuracy", genomeSize: "50k", acMinGsize: 50000, flyeFirstMb: 10, acMaxMb: 30, overwrite: false, reportsOnly: false } }, large: { label: "Large phages (~90–150 kb, e.g. Staphylococcus)", v: { assemblyMode: "accuracy", genomeSize: "135k", acMinGsize: 130000, flyeFirstMb: 20, acMaxMb: 30, overwrite: false, reportsOnly: false } }, reports: { label: "Rebuild reports only (keep all assemblies)", v: { reportsOnly: true, overwrite: false, cpRun: true, makeUpload: true } }, test: { label: "Assembly test (no annotation, checks, export or upload)", v: { runPharokka: false, runPhold: false, cpRun: false, exportPt: false, makeUpload: false, runNanoplot: false, runNanocomp: false } }, full: { label: "Full run (annotation, checks, export and upload on)", v: { runPharokka: true, runPhold: true, cpRun: true, exportPt: true, makeUpload: true, runNanoplot: true, runNanocomp: true } }, }; const KEY = "phagePipelineProfiles"; const load = () => { try { return JSON.parse(localStorage.getItem(KEY) || "{}") || {}; } catch (e) { return {}; } }; const store = (o) => { try { localStorage.setItem(KEY, JSON.stringify(o)); return true; } catch (e) { return false; } }; const bar = document.createElement("div"); bar.className = "phage-profiles"; bar.innerHTML = 'Run profile: ' + '' + '' + ''; const legend = wrap.querySelector(".phage-legend"); (legend || wrap.querySelector(".phage-box")).insertAdjacentElement(legend ? "beforebegin" : "beforebegin", bar); const sel = bar.querySelector("#profileSelect"), st = bar.querySelector("#profileStatus"); const fill = (keep) => { const mine = load(); sel.innerHTML = '' + Object.entries(BUILTIN).map(([k, p]) => ``).join("") + (Object.keys(mine).length ? '' + Object.keys(mine).sort().map((n) => `" : ""); sel.value = keep || ""; }; const apply = (values) => { let n = 0; for (const [id, v] of Object.entries(values)) { const el = wrap.querySelector("#" + id); if (!el) continue; if (el.type === "checkbox") el.checked = !!v; else el.value = v; el.dispatchEvent(new Event("input", { bubbles: true })); el.dispatchEvent(new Event("change", { bubbles: true })); n += 1; } return n; }; sel.addEventListener("change", (ev) => { ev.stopPropagation(); const v = sel.value; if (!v) return; const vals = v.startsWith("b:") ? BUILTIN[v.slice(2)].v : load()[v.slice(2)]; const n = vals ? apply(vals) : 0; st.textContent = `${n} setting(s) applied.`; }); bar.querySelector("#profileSave").addEventListener("click", () => { const name = bar.querySelector("#profileName").value.trim(); if (!name) { st.textContent = "Enter a name first."; return; } const vals = {}; wrap.querySelectorAll("input[id], select[id], textarea[id]").forEach((el) => { if (["profileSelect", "profileName", "gsizeGroup"].includes(el.id) || bar.contains(el)) return; vals[el.id] = el.type === "checkbox" ? el.checked : el.value; }); const mine = load(); mine[name] = vals; st.textContent = store(mine) ? `Saved "${name}" (${Object.keys(vals).length} settings).` : "Could not save (browser storage unavailable)."; fill("u:" + name); }); bar.querySelector("#profileDelete").addEventListener("click", () => { const v = sel.value; if (!v.startsWith("u:")) { st.textContent = "Only profiles you saved can be deleted."; return; } const mine = load(); delete mine[v.slice(2)]; store(mine); fill(""); st.textContent = "Deleted."; }); // the profile controls are not settings: keep them out of the form memory and the script for (const el of bar.querySelectorAll("input, select")) { el.addEventListener("input", (ev) => ev.stopPropagation()); el.addEventListener("change", (ev) => { if (el !== sel) ev.stopPropagation(); }); } fill(""); })(); let downloadUrl = null; let phageMapV15 = null; // {set: identifier, rows: [...] } from get_sample_set_pipeline_map function phageMapTextV15() { if (!phageMapV15 || !phageMapV15.rows.length) return ""; const want = $("ptSampleSet").value.trim().toLowerCase().replace(/[^a-z0-9]+/g, ""); if (phageMapV15.key !== want) return ""; const clean = (v) => String(v == null ? "" : v).replace(/[\t\r\n]+/g, " ").trim(); return ["barcode\tphage\tlinked_phage\tphage_id\tlab_phage_id"].concat( phageMapV15.rows.filter((r) => r.phage_id).map((r) => [r.barcode, r.phage, r.linked_phage, r.phage_id, r.lab_phage_id].map(clean).join("\t"))).join("\n"); } // closest_phages.py (same tool as the Closest phages generator), written by the script for the genome checks. const CLOSEST_PY = `#!/usr/bin/env python3 """ Closest PhagesDB phages (and closest GenBank phages) for a folder of phage genomes. For every subfolder of --input (one per phage) the genome FASTA is found, and all genomes are compared at once with minimap2 (asm20) to the PhagesDB genome collection (Actinobacteriophages-All.fasta). Writes closest_phages.html (sortable, linked table), closest_phages.tsv (one row per phage) and closest_phages_top_hits.tsv (the best hits for every phage). The PhagesDB index is shared with the Nanopore phage pipeline (folder "phagesdb_index" next to the FASTA), so it is built only once per downloaded PhagesDB file. Optionally, complete phage genomes are downloaded once from NCBI (NCBI Datasets: by host genus, or all Caudoviricetes) and searched the same way, giving BLAST-style columns for the closest GenBank phage (description, accession, per. ident, query cover, aligned length, subject length). Genome checks (optional): the share of each genome that matches the host genome(s) (host DNA), CheckV completeness, contamination and quality, and a VIRIDIC-style intergenomic similarity matrix of all the genomes with species (>= 95%) and genus (>= 70%) groups (ICTV thresholds), shown as a heatmap. v2 (2026-11-02): every contig >= --min-contig (5 kb) of a genome is searched (joined with N spacers), not only the longest; the report shows how much of the closest phage your contigs cover (complete / fragmented / partial / no relative), near-equal matches ("ties"), the ICTV-style similarity to the closest phage (same species / same genus / different genus), phages in the run that are the same phage (>= 98% similar), isolation hosts (--isolation-hosts), links to the pipeline's coverage.html and its warnings. --check lists the genomes and FASTA files without searching. """ import argparse import gzip import html import json import os import re import subprocess import sys import time import urllib.parse FASTA_EXT = (".fasta", ".fa", ".fna", ".fas", ".fasta.gz", ".fa.gz", ".fna.gz") TOOL_VERSION = "closest_phages v2.4 (2026-11-02)" SPACER = "N" * 100 MAX_CONTIG = 1_000_000 # longer contigs are bacterial (host) DNA, not phage: not searched # between the contigs of one genome in the query FASTA def open_text(path): return gzip.open(path, "rt") if path.endswith(".gz") else open(path, errors="replace") def read_fasta(path): recs, name, desc, chunks = [], None, "", [] with open_text(path) as fh: for line in fh: line = line.rstrip("\\r\\n") if line.startswith(">"): if name is not None: recs.append((name, desc, "".join(chunks))) parts = line[1:].strip().split(None, 1) name = parts[0] if parts else "contig" desc = parts[1] if len(parts) > 1 else "" chunks = [] elif line.strip(): chunks.append(line.strip()) if name is not None: recs.append((name, desc, "".join(chunks))) return recs def write_fasta(fh, name, seq, width=80): fh.write(">" + name + "\\n") for i in range(0, len(seq), width): fh.write(seq[i:i + width] + "\\n") def merge(iv): out = [] for s, e in sorted(iv): if out and s <= out[-1][1]: out[-1][1] = max(out[-1][1], e) else: out.append([s, e]) return out # ---------------------------------------------------------------- PhagesDB header + data table def pdb_header(header): h = header.strip() out = {"host": "", "phage": "", "length": "", "overhang": "", "ends": "", "cluster": ""} m = re.match(r"(.+?)\\s+phage\\s+([^\\s,;]+)", h, re.I) if m: out["host"], out["phage"] = m.group(1).strip(), m.group(2).strip() else: out["phage"] = h.split(None, 1)[0] if h else "" m = re.search(r"([\\d,]+)\\s*bp", h) if m: out["length"] = m.group(1).replace(",", "") m = re.search(r"(\\d+)[- ]base\\s+([35])['’′]?\\s*overhang\\s*\$([A-Za-z]+)\$", h, re.I) if m: out["overhang"] = m.group(3).upper() out["ends"] = f"{m.group(1)}-base {m.group(2)}' overhang ({m.group(3).upper()})" elif re.search(r"circularly permuted", h, re.I): out["ends"] = "circularly permuted" elif re.search(r"terminal repeat", h, re.I): out["ends"] = "direct terminal repeats" m = re.search(r"Cluster\\s+([A-Za-z0-9]+)", h) if m: out["cluster"] = m.group(1) elif re.search(r"\\bSingleton\\b", h, re.I): out["cluster"] = "Singleton" return out TABLE_COLS = ["in_table", "subcluster", "temperate", "gc_pct", "end_type", "overhang_len", "term_rpt_len", "orfs", "trnas", "morphotype", "host_strain", "accession", "genome_length_table"] def none_(v): v = str(v or "").strip() return "" if v.lower() in ("none", "null", "nan", "n/a") else v def read_table(path): rows = {} if not path or not os.path.isfile(path): return rows raw = open(path, "rb").read() text = "" for enc in ("utf-8-sig", "cp1252", "latin-1"): try: text = raw.decode(enc) break except UnicodeDecodeError: continue lines = text.splitlines() if not lines: return rows cols = [c.strip() for c in lines[0].split("\\t")] for line in lines[1:]: f = line.split("\\t") if not f or not f[0].strip(): continue d = {c: (f[i].strip() if i < len(f) else "") for i, c in enumerate(cols)} rows[d.get("Phage Name", f[0]).strip().lower()] = d return rows def prepare_phagesdb(fasta, table, threads): """IDs FASTA, metadata table and minimap2 index, in phagesdb_index next to the FASTA (same files as the Nanopore phage pipeline uses, so either one builds them once).""" pdb_dir = os.path.join(os.path.dirname(os.path.abspath(fasta)), "phagesdb_index") os.makedirs(pdb_dir, exist_ok=True) st = os.stat(fasta) stem = os.path.join(pdb_dir, f"{os.path.basename(fasta)}.{st.st_size}_{int(st.st_mtime)}.v9") tsig = "none" if table: ts = os.stat(table) tsig = f"{ts.st_size}_{int(ts.st_mtime)}" ids, meta, index = stem + ".ids.fasta", f"{stem}.table_{tsig}.meta.tsv", stem + ".asm20.mmi" if not (os.path.isfile(ids) and os.path.isfile(meta) and os.path.getsize(meta) > 0): print("Reading the PhagesDB genomes" + (" and data table" if table else "") + " (once per downloaded file)…", flush=True) tab = read_table(table) used, n, matched = set(), 0, 0 with open_text(fasta) as fin, open(ids + ".part", "w") as fo, open(meta + ".part", "w") as fm: fm.write("id\\tphage\\thost\\tlength_bp\\tends\\toverhang\\tcluster\\theader\\t" + "\\t".join(TABLE_COLS) + "\\n") for line in fin: if line.startswith(">"): header = line[1:].strip() f = pdb_header(header) base = re.sub(r"[^A-Za-z0-9._-]+", "_", f["phage"] or "phage").strip("_") or "phage" pid, k = base, 1 while pid in used: k += 1 pid = f"{base}_{k}" used.add(pid) n += 1 fo.write(f">{pid}\\n") extra = {c: "" for c in TABLE_COLS} row = tab.get(f["phage"].lower()) if row: matched += 1 sub, clu = none_(row.get("Subcluster")), none_(row.get("Cluster")) f["cluster"] = sub or clu or f["cluster"] end_type = none_(row.get("End Type")) oh_len, oh_seq = none_(row.get("Overhang Length (bp)")), none_(row.get("Overhang Sequence")).upper() trl = none_(row.get("Term Rpt Length")) if oh_seq and re.fullmatch(r"[ACGT]+", oh_seq): f["overhang"] = oh_seq if end_type: bits = [end_type] + ([f"{oh_len}-base"] if oh_len else []) + \\ ([f"({f['overhang']})"] if f["overhang"] else []) + ([f"terminal repeat {trl} bp"] if trl else []) f["ends"] = " ".join(bits) if none_(row.get("Host")): f["host"] = none_(row.get("Host")) extra.update({"in_table": "yes", "subcluster": sub or clu, "temperate": none_(row.get("Temperate?")), "gc_pct": none_(row.get("GC%")), "end_type": end_type, "overhang_len": oh_len, "term_rpt_len": trl, "orfs": none_(row.get("# ORFs")), "trnas": none_(row.get("# tRNAs")), "morphotype": none_(row.get("Morphotype")), "host_strain": none_(row.get("Host")), "accession": none_(row.get("Accession #")), "genome_length_table": none_(row.get("Genome Length(bp)"))}) clean = header.replace("\\t", " ") fm.write(f"{pid}\\t{f['phage']}\\t{f['host']}\\t{f['length']}\\t{f['ends']}\\t{f['overhang']}\\t{f['cluster']}" f"\\t{clean}\\t" + "\\t".join(extra[c].replace("\\t", " ") for c in TABLE_COLS) + "\\n") elif line.strip(): fo.write(line.strip() + "\\n") if n == 0: sys.exit("No sequences in the PhagesDB FASTA.") os.replace(ids + ".part", ids) os.replace(meta + ".part", meta) print(f" {n} PhagesDB genomes" + (f"; {matched} matched to the data table ({len(tab)} rows)" if tab else "")) if not os.path.isfile(index): print("Indexing the PhagesDB genomes for minimap2 (once per downloaded file; a few minutes)…", flush=True) r = subprocess.run(["minimap2", "-x", "asm20", "-t", str(threads), "-d", index + ".part", ids], capture_output=True, text=True) if r.returncode != 0: sys.exit("minimap2 could not index the PhagesDB genomes:\\n" + r.stderr[-800:]) os.replace(index + ".part", index) m = {} with open(meta) as fh: cols = next(fh).rstrip("\\n").split("\\t") for line in fh: f = line.rstrip("\\n").split("\\t") m[f[0]] = dict(zip(cols, f)) return index, m # ---------------------------------------------------------------- NCBI phage genomes DATASETS_URL = "https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/v2/linux-amd64/datasets" def datasets_tool(ncbi_dir): """NCBI Datasets command-line tool: on PATH, or downloaded once next to the NCBI genomes.""" import shutil import urllib.request found = shutil.which("datasets") if found: return found local = os.path.join(ncbi_dir, "datasets") if os.path.isfile(local) and os.access(local, os.X_OK): return local print(f"Downloading the NCBI Datasets tool → {local}", flush=True) urllib.request.urlretrieve(DATASETS_URL, local + ".part") os.chmod(local + ".part", 0o755) os.replace(local + ".part", local) return local def ncbi_download(ncbi_dir, mode, hosts, stem): """Complete phage genomes from NCBI into STEM.fasta + STEM.meta.tsv (accession, description, host, length). Host-genus mode: NCBI's --host filter only matches the exact host taxon (a phage of "Staphylococcus epidermidis" is not returned for "Staphylococcus") and skips records without a host, so all complete Caudoviricetes genomes are downloaded once and kept when the host lineage includes one of the genera, the host name starts with one, or the title starts with it ("Staphylococcus phage CF9", "... virus ..."). The per-host downloads are added too, in case a matching phage is not classified as Caudoviricetes.""" import zipfile tool = datasets_tool(ncbi_dir) genera = {h.lower() for h in hosts} if mode == "all": jobs = [["Caudoviricetes", None, False]] else: jobs = [["Caudoviricetes", None, True]] + [["10239", h, False] for h in hosts] # 10239 = Viruses def wanted(desc, host_name, lineage): if any(x in genera for x in lineage): return True if host_name and host_name.split()[0].lower() in genera: return True m = re.match(r"\\s*(?:uncultured\\s+)?([A-Za-z]+)\\s+(?:phage|virus|bacteriophage)\\b", desc or "", re.I) return bool(m and m.group(1).lower() in genera) seen, n = set(), 0 with open(stem + ".fasta.part", "w") as fo, open(stem + ".meta.tsv.part", "w") as fm: fm.write("accession\\tdescription\\thost\\tlength_bp\\n") for taxon, host, filt in jobs: label = f"viruses with host {host}" if host else (f"{taxon} (kept: phages of {', '.join(hosts)})" if filt else taxon) print(f" NCBI Datasets: complete genomes of {label} …" + (" (large, once: roughly 0.5–1 GB; the matching phages are kept)" if taxon == "Caudoviricetes" else ""), flush=True) # Big downloads go to a Linux folder (writing them to /mnt/c can fail); a Caudoviricetes package you # downloaded yourself can be reused by saving it as NCBI_phages/Caudoviricetes_complete.zip. tmp_dir = os.path.join(os.path.expanduser("~"), ".cache", "closest_phages") os.makedirs(tmp_dir, exist_ok=True) reuse = os.path.join(ncbi_dir, "Caudoviricetes_complete.zip") if taxon == "Caudoviricetes" else "" keep_zip = bool(reuse) and os.path.isfile(reuse) and (time.time() - os.path.getmtime(reuse)) / 86400 <= NCBI_MAX_AGE_DAYS zpath = reuse if keep_zip else os.path.join(tmp_dir, f"ncbi_{os.getpid()}_{re.sub(r'[^A-Za-z0-9]+', '_', label)[:40]}.zip") if keep_zip: print(f" using {reuse} (delete it to download a newer package)", flush=True) else: cmd = [tool, "download", "virus", "genome", "taxon", taxon, "--complete-only", "--include", "genome", "--filename", zpath] if host: cmd += ["--host", host] r = subprocess.run(cmd, capture_output=True, text=True) if r.returncode != 0 or not os.path.isfile(zpath): msg = (r.stderr or r.stdout or "").strip()[-800:] if os.path.exists(zpath): os.remove(zpath) if taxon == "Caudoviricetes": # without it the GenBank set would be badly incomplete, so stop the GenBank search here raise RuntimeError("NCBI Datasets could not download the Caudoviricetes genomes:\\n" + msg + "\\nTip: download it yourself with\\n datasets download virus genome taxon Caudoviricetes " f"--complete-only --include genome --filename '{reuse or 'Caudoviricetes_complete.zip'}'\\n" "and run the script again.") print(f"⚠️ NCBI Datasets failed for {label} (skipped):\\n{msg}", flush=True) continue hostmap, lineages = {}, {} with zipfile.ZipFile(zpath) as z: names = z.namelist() rep_name = next((x for x in names if x.endswith("data_report.jsonl")), None) if rep_name: with z.open(rep_name) as rh: for raw in rh: try: d = json.loads(raw) except ValueError: continue acc = d.get("accession", "") hd = d.get("host") or {} h = hd.get("organismName") or hd.get("organism_name") or hd.get("name") or "" if acc: hostmap[acc] = h lineages[acc] = {str(x.get("name", "")).lower() for x in (hd.get("lineage") or []) if isinstance(x, dict)} fna = next((x for x in names if x.endswith(".fna")), None) added = 0 if fna: acc, desc, chunks, keep = None, "", [], False def flush(): nonlocal n, added if acc and keep: seq = "".join(chunks) write_fasta(fo, acc, seq) fm.write(f"{acc}\\t{desc.replace(chr(9), ' ')}\\t{hostmap.get(acc, host or '')}\\t{len(seq)}\\n") n += 1 added += 1 with z.open(fna) as fh: for raw in fh: line = raw.decode("utf-8", "replace").rstrip("\\r\\n") if line.startswith(">"): flush() parts = line[1:].split(None, 1) acc = parts[0] if parts else "" desc = parts[1] if len(parts) > 1 else "" keep = acc not in seen and (not filt or wanted(desc, hostmap.get(acc, ""), lineages.get(acc, set()))) if keep: seen.add(acc) chunks = [] elif keep: chunks.append(line.strip()) flush() if not keep_zip: os.remove(zpath) print(f" {added:,} genomes", flush=True) if n == 0: raise RuntimeError("NCBI returned no genomes; check the host names") os.replace(stem + ".fasta.part", stem + ".fasta") os.replace(stem + ".meta.tsv.part", stem + ".meta.tsv") with open(stem + ".info.txt", "w") as fh: fh.write(f"downloaded\\t{time.strftime('%Y-%m-%d %H:%M')}\\nmode\\t{mode}\\nhosts\\t{' '.join(hosts)}\\ngenomes\\t{n}\\n") NCBI_MAX_AGE_DAYS = 90 def ncbi_age_days(stem): try: with open(stem + ".info.txt") as fh: for line in fh: k, _, v = line.rstrip("\\n").partition("\\t") if k == "downloaded": return (time.time() - time.mktime(time.strptime(v.strip(), "%Y-%m-%d %H:%M"))) / 86400 except (OSError, ValueError): pass return None def prepare_ncbi(ncbi_dir, mode, hosts, local_fasta, refresh, threads): """Index of NCBI phage genomes (downloaded once, or a FASTA you downloaded yourself) and its metadata.""" os.makedirs(ncbi_dir, exist_ok=True) if local_fasta: st = os.stat(local_fasta) stem = os.path.join(ncbi_dir, f"{os.path.basename(local_fasta)}.{st.st_size}_{int(st.st_mtime)}") if not os.path.isfile(stem + ".meta.tsv"): with open_text(local_fasta) as fin, open(stem + ".fasta", "w") as fo, open(stem + ".meta.tsv", "w") as fm: fm.write("accession\\tdescription\\thost\\tlength_bp\\n") acc, desc, chunks = None, "", [] for line in list(fin) + [">"]: line = line.rstrip("\\r\\n") if line.startswith(">"): if acc: seq = "".join(chunks) write_fasta(fo, acc, seq) fm.write(f"{acc}\\t{desc}\\t\\t{len(seq)}\\n") parts = line[1:].split(None, 1) acc = parts[0] if parts else None desc = (parts[1] if len(parts) > 1 else "").replace("\\t", " ") chunks = [] else: chunks.append(line.strip()) else: # v4: host mode filters a Caudoviricetes download (older "hosts_"/"hosts3_" sets missed many phages) tag = "all_Caudoviricetes" if mode == "all" else "hosts4_" + "_".join(sorted(h.lower() for h in hosts))[:120] stem = os.path.join(ncbi_dir, f"ncbi_phages_{re.sub(r'[^A-Za-z0-9_.-]+', '_', tag)}") age = ncbi_age_days(stem) if os.path.isfile(stem + ".meta.tsv") else None if age is not None and age > NCBI_MAX_AGE_DAYS and not refresh: print(f"The NCBI genomes were downloaded {age:.0f} days ago; downloading them again to include new phages…", flush=True) refresh = True if refresh or not os.path.isfile(stem + ".meta.tsv"): print("Downloading complete phage genomes from NCBI (once; refresh with the generator's checkbox)…", flush=True) ncbi_download(ncbi_dir, mode, hosts, stem) for ext in (".asm20.mmi",): if os.path.exists(stem + ext): os.remove(stem + ext) index = stem + ".asm20.mmi" if not os.path.isfile(index): print("Indexing the NCBI phage genomes for minimap2 (once per download)…", flush=True) r = subprocess.run(["minimap2", "-x", "asm20", "-t", str(threads), "-d", index + ".part", stem + ".fasta"], capture_output=True, text=True) if r.returncode != 0: raise RuntimeError("minimap2 could not index the NCBI genomes:\\n" + r.stderr[-800:]) os.replace(index + ".part", index) meta = {} with open(stem + ".meta.tsv") as fh: next(fh) for line in fh: f = line.rstrip("\\n").split("\\t") if f and f[0]: desc = f[1] if len(f) > 1 else "" host = f[2] if len(f) > 2 else "" if not host: # e.g. "Staphylococcus phage X, complete genome" -> Staphylococcus mh = re.match(r"\\s*([A-Z][a-z]+)\\s+(?:sp\\.\\s+)?phage\\b", desc) host = mh.group(1) if mh else "" meta[f[0]] = {"description": desc, "host": host, "length": f[3] if len(f) > 3 else ""} info = {} if os.path.isfile(stem + ".info.txt"): for line in open(stem + ".info.txt"): k, _, v = line.rstrip("\\n").partition("\\t") info[k] = v return index, meta, info def paf_lines(cmd, what): """Run minimap2 and return only the 12 standard PAF columns, read line by line. v2.1: the CIGAR strings of -c alignments of hundreds of genomes can take several GB when the whole output is held in memory, which can stop WSL.""" import tempfile out = [] with tempfile.TemporaryFile(mode="w+") as err: p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=err, text=True, bufsize=1 << 20) for line in p.stdout: f = line.split("\\t", 12)[:12] if len(f) == 12: out.append(f) if p.wait() != 0: err.seek(0) raise RuntimeError(f"minimap2 ({what}) failed:\\n" + err.read()[-800:]) return out def mem_note(): """free memory, for the progress lines (helps when WSL runs out of memory)""" try: for line in open("/proc/meminfo"): if line.startswith("MemAvailable:"): return f" [{int(line.split()[1]) / 1048576:.1f} GB free]" except OSError: pass return "" def gpos(qoff, name): """contig query name -> (genome id, offset of the contig in the genome's coordinates)""" return qoff.get(name, (name, 0)) if qoff else (name, 0) def search(index, qpath, threads, qoff=None): """minimap2 (asm20) of all query contigs against an index -> {genome: {target: totals}} (contigs of one genome are placed one after the other, see genome_query).""" hits = {} for f in paf_lines(["minimap2", "-c", "-x", "asm20", "-N", "100", "-p", "0.2", "-t", str(threads), index, qpath], "search"): g, off = gpos(qoff, f[0]) d = hits.setdefault(g, {}).setdefault(f[5], {"tlen": int(f[6]), "iv": [], "tiv": [], "st": [], "match": 0, "block": 0}) d["st"].append(f[4]) d["iv"].append((int(f[2]) + off, int(f[3]) + off)) d["tiv"].append((int(f[7]), int(f[8]))) d["match"] += int(f[9]) d["block"] += int(f[10]) return hits def rank_hits(hits_q, L): ranked = [] for tid, d in (hits_q or {}).items(): cov = sum(e - s for s, e in merge(d["iv"])) tcov = sum(e - s for s, e in merge(d["tiv"])) ident = 100.0 * d["match"] / d["block"] if d["block"] else 0 ranked.append({"score": cov * ident, "id": tid, "tlen": d["tlen"], "ident": round(ident, 2), "qcov": round(100.0 * cov / L, 1) if L else 0, "aln": cov, "taln": tcov, "scov": round(100.0 * tcov / d["tlen"], 1) if d["tlen"] else 0}) ranked.sort(key=lambda x: -x["score"]) return ranked def aln_blocks(d, offs, min_len=150, cap=400): """Where each searched contig aligns on the relative: [[start, end, contig index, strand], ...] (merged).""" by = {} for (qs, qe), (ts, te), st in zip(d["iv"], d["tiv"], d.get("st") or ["+"] * len(d["iv"])): k = next((i for i, o in enumerate(offs) if o[0] <= qs < o[1]), 0) by.setdefault((k, st), []).append((ts, te)) out = [[a, b, k, st] for (k, st), iv in by.items() for a, b in merge(iv) if b - a >= min_len] out.sort(key=lambda x: (x[0], x[1])) return out[:cap] def ictv_sim(ident, aln_q, aln_t, la, lb): """Approximate VIRIDIC-style similarity: 2 x identical bases / (both lengths) x 100.""" if not la + lb: return 0.0 return round(min(100.0, 200.0 * (float(ident) / 100.0) * min(aln_q, aln_t) / (la + lb)), 1) def ictv_rank(v): return "same species" if v >= 95 else ("same genus" if v >= 70 else "different genus") def url_ncbi(acc): return "https://www.ncbi.nlm.nih.gov/nuccore/" + urllib.parse.quote(str(acc)) # ---------------------------------------------------------------- genome checks def host_check(qpath, host_paths, threads, qoff=None, glen=None): """Share of each query genome aligned (>= 85% identity) to the host genome(s) -> {qid: (pct, host label)}.""" import tempfile labels, out = [], {} with tempfile.TemporaryDirectory() as tmp: hf = os.path.join(tmp, "hosts.fa") with open(hf, "w") as fo: for i, hp in enumerate(host_paths, 1): label = re.sub(r"\\.(fasta|fa|fna|fas)(\\.gz)?$", "", os.path.basename(hp), flags=re.I) labels.append(label) for name, _, seq in read_fasta(hp): write_fasta(fo, f"H{i}__{name}", seq) paf = paf_lines(["minimap2", "-c", "-x", "asm20", "-t", str(threads), hf, qpath], "host check") spans, per_host, qlen = {}, {}, {} for f in paf: if int(f[10]) == 0 or int(f[9]) / int(f[10]) < 0.85: continue h = int(f[5].split("__", 1)[0][1:]) - 1 q, off = gpos(qoff, f[0]) qlen[q] = (glen or {}).get(q) or int(f[1]) spans.setdefault(q, []).append((int(f[2]) + off, int(f[3]) + off)) per_host.setdefault(q, {}).setdefault(h, []).append((int(f[2]) + off, int(f[3]) + off)) for q, iv in spans.items(): cov = sum(e - s for s, e in merge(iv)) best = max(per_host[q].items(), key=lambda kv: sum(e - s for s, e in merge(kv[1])))[0] out[q] = (round(100.0 * cov / qlen[q], 1), labels[best]) return out def find_checkv_db(db): """The CheckV database folder (the one holding genome_db and hmm_db), at DB or one level below.""" if not db or not os.path.isdir(db): return "" if os.path.isdir(os.path.join(db, "genome_db")) and os.path.isdir(os.path.join(db, "hmm_db")): return db for sub in sorted(os.listdir(db), reverse=True): if sub.endswith(".incomplete"): continue d = os.path.join(db, sub) if os.path.isdir(os.path.join(d, "genome_db")) and os.path.isdir(os.path.join(d, "hmm_db")): return d return "" def run_checkv(qpath, out_dir, bin_dir, db, download, threads): """CheckV end_to_end on the query genomes -> {qid: quality_summary row}.""" env = dict(os.environ) if bin_dir: env["PATH"] = bin_dir + os.pathsep + env.get("PATH", "") import shutil if not shutil.which("checkv", path=env["PATH"]): raise RuntimeError("CheckV was not found" + (f" in {bin_dir}" if bin_dir else "")) dbdir = find_checkv_db(db) if not dbdir: if not download: raise RuntimeError(f"no CheckV database in {db}") os.makedirs(db, exist_ok=True) print(f"Downloading the CheckV database into {db} (once; a few GB)…", flush=True) r = subprocess.run(["checkv", "download_database", db], env=env, capture_output=True, text=True) dbdir = find_checkv_db(db) if r.returncode != 0 or not dbdir: raise RuntimeError("CheckV database download failed:\\n" + (r.stderr or r.stdout)[-800:]) # v2.3: the DIAMOND file is built at the end of "checkv download_database"; an interrupted download (or a manual # one) leaves it missing -> build it from checkv_reps.faa, or download the database again gdb = os.path.join(dbdir, "genome_db") dmnd = os.path.join(gdb, "checkv_reps.dmnd") if not os.path.isfile(dmnd) or os.path.getsize(dmnd) == 0: faa = next((os.path.join(gdb, f) for f in ("checkv_reps.faa", "checkv_reps.faa.gz") if os.path.isfile(os.path.join(gdb, f))), "") if faa and shutil.which("diamond", path=env["PATH"]): print(f"The CheckV database is missing {os.path.basename(dmnd)}; building it with DIAMOND (once; a few minutes)…", flush=True) r = subprocess.run(["diamond", "makedb", "--in", faa, "--db", os.path.join(gdb, "checkv_reps"), "--threads", str(threads)], env=env, capture_output=True, text=True) if r.returncode != 0 or not os.path.isfile(dmnd): raise RuntimeError("could not build the CheckV DIAMOND database:\\n" + (r.stderr or r.stdout)[-600:] + f"\\nBuild it by hand: cd {gdb} && diamond makedb --in {os.path.basename(faa)} --db checkv_reps") elif download: print(f"The CheckV database in {dbdir} is incomplete; downloading it again (once; a few GB)…", flush=True) bad = dbdir.rstrip("/") + ".incomplete" if os.path.exists(bad): shutil.rmtree(bad, ignore_errors=True) os.replace(dbdir, bad) r = subprocess.run(["checkv", "download_database", db], env=env, capture_output=True, text=True) dbdir = find_checkv_db(db) if r.returncode != 0 or not dbdir or not os.path.isfile(os.path.join(dbdir, "genome_db", "checkv_reps.dmnd")): raise RuntimeError("CheckV database download failed:\\n" + (r.stderr or r.stdout)[-800:]) shutil.rmtree(bad, ignore_errors=True) else: raise RuntimeError(f"the CheckV database in {dbdir} is incomplete ({os.path.basename(dmnd)} and checkv_reps.faa are " "missing); tick 'Download the database if it is missing' or delete the folder and download it again") work = os.path.join(out_dir, "checkv") print(f"Running CheckV (database {os.path.basename(dbdir)})…", flush=True) r = subprocess.run(["checkv", "end_to_end", qpath, work, "-d", dbdir, "-t", str(threads)], env=env, capture_output=True, text=True) summ = os.path.join(work, "quality_summary.tsv") if r.returncode != 0 or not os.path.isfile(summ): raise RuntimeError("CheckV failed:\\n" + (r.stderr or r.stdout)[-800:]) res = {} with open(summ) as fh: cols = next(fh).rstrip("\\n").split("\\t") for line in fh: d = dict(zip(cols, line.rstrip("\\n").split("\\t"))) res[d.get("contig_id", "")] = d return res def similarity_matrix(qpath, info, threads, qoff=None): """VIRIDIC-style intergenomic similarity between all pairs of genomes: SIM = 2 x identical bases in the alignments / (length A + length B) x 100, from minimap2 (asm20) alignments (overlapping alignments counted once).""" alns = {} for f in paf_lines(["minimap2", "-c", "-x", "asm20", "-X", "-t", str(threads), qpath, qpath], "similarity"): (g1, o1), (g2, o2) = gpos(qoff, f[0]), gpos(qoff, f[5]) if g1 == g2: continue a, b = sorted((g1, g2)) qs, qe, ts, te = int(f[2]) + o1, int(f[3]) + o1, int(f[7]) + o2, int(f[8]) + o2 if a != g1: qs, qe, ts, te = ts, te, qs, qe alns.setdefault((a, b), []).append((int(f[9]), qs, qe, ts, te, int(f[10]))) sim, pair = {}, {} for (a, b), lst in alns.items(): lst.sort(key=lambda x: -x[0]) used_a, used_b, total, blk = [], [], 0, 0 for m, qs, qe, ts, te, bl in lst: ov_a = sum(max(0, min(qe, e) - max(qs, s)) for s, e in used_a) ov_b = sum(max(0, min(te, e) - max(ts, s)) for s, e in used_b) if ov_a > 0.5 * (qe - qs) or ov_b > 0.5 * (te - ts): continue used_a.append((qs, qe)) used_b.append((ts, te)) total += m blk += bl la, lb = info[a]["length"], info[b]["length"] sim[(a, b)] = round(min(100.0, 200.0 * total / (la + lb)), 1) if la + lb else 0.0 ca, cb = sum(e - s for s, e in merge(used_a)), sum(e - s for s, e in merge(used_b)) short = 100.0 * (ca / la if la <= lb else cb / lb) if la and lb else 0.0 pair[(a, b)] = (round(100.0 * total / blk, 2) if blk else 0.0, round(min(100.0, short), 1)) return sim, pair def groups_at(qids, sim, cut): """Connected groups of genomes with similarity >= cut (single linkage).""" parent = {q: q for q in qids} def find(x): while parent[x] != x: parent[x] = parent[parent[x]] x = parent[x] return x for (a, b), s in sim.items(): if s >= cut: parent[find(a)] = find(b) comp = {} for q in qids: comp.setdefault(find(q), []).append(q) return [sorted(v) for v in comp.values()] # ---------------------------------------------------------------- input genomes # folders in a pipeline output folder that are not phages (reports, uploads, indexes, earlier runs of this tool) NOT_PHAGE_DIR = re.compile(r"^(\\.|closest_phages|assembly_comparison|nanocomp_|sharepoint_upload|phagesdb_index|" r"sample_logs|timings_|upload)", re.I) def pick_fasta(folder, name, pattern): files = sorted(f for f in os.listdir(folder) if f.lower().endswith(FASTA_EXT) and not f.startswith(".") and os.path.isfile(os.path.join(folder, f))) if not files: return None if pattern: rx = re.compile("^" + re.escape(pattern).replace(r"\\*", ".*").replace(r"\\{phage\\}", re.escape(name)) + "$", re.I) hit = [f for f in files if rx.match(f)] return os.path.join(folder, hit[0]) if hit else None low = {f.lower(): f for f in files} for cand in (f"{name}.fasta", f"{name}.fa", f"{name}.fna", f"{name}.longest_contig.fasta"): if cand.lower() in low: return os.path.join(folder, low[cand.lower()]) skip = re.compile(r"(\\.flye|\\.autocycler|unrotated|_reoriented|\\.ids)\\.f", re.I) rest = [f for f in files if not skip.search(f)] or files return os.path.join(folder, max(rest, key=lambda f: os.path.getsize(os.path.join(folder, f)))) def find_genomes(root, pattern, skip_dirs=()): out, skipped = [], [] skip = {os.path.realpath(d) for d in skip_dirs if d} for entry in sorted(os.listdir(root), key=str.lower): p = os.path.join(root, entry) if os.path.isdir(p) and (NOT_PHAGE_DIR.match(entry) or os.path.realpath(p) in skip): if not entry.startswith("."): skipped.append(entry) continue if os.path.isdir(p): f = pick_fasta(p, entry, pattern) if f: out.append((entry, f)) elif os.path.isfile(p) and entry.lower().endswith(FASTA_EXT) and not pattern and not entry.startswith("."): stem = re.sub(r"\\.(fasta|fa|fna|fas)(\\.gz)?$", "", entry, flags=re.I) out.append((stem, p)) return out, skipped def genome_query(name, path, recs, min_len): """One query per genome: its contigs >= min_len joined with N spacers (the longest contig if none is that long).""" recs = sorted(recs, key=lambda r: -len(r[2])) big = [r for r in recs if len(r[2]) > MAX_CONTIG] use = [r for r in recs if min_len <= len(r[2]) <= MAX_CONTIG] or [r for r in recs if len(r[2]) <= MAX_CONTIG][:1] or recs[:1] offs, parts, pos = [], [], 0 for r in use: if parts: parts.append(SPACER) pos += len(SPACER) offs.append((pos, pos + len(r[2]), r[0], len(r[2]))) parts.append(r[2]) pos += len(r[2]) L = sum(o[3] for o in offs) gcn = sum(r[2].upper().count("G") + r[2].upper().count("C") for r in use) if big: print(f" {name}: {len(big)} contig(s) over {MAX_CONTIG // 1000:,} kb not searched (host DNA, not phage)") return {"sample": name, "file": path, "contigs": len(recs), "searched": len(use), "length": L, "longest": len(recs[0][2]), "circ_hdr": header_topology(recs[0][1]), "offs": offs, "gc": round(100.0 * gcn / L, 1) if L else 0, "seq": "".join(parts)} def header_topology(desc): """'yes'/'no' from a FASTA header tag (circular=true, Autocycler/Flye style), else ''.""" m = re.search(r"\\bcirc(?:ular)?\\s*[=:]\\s*(true|yes|y|false|no|n)\\b", str(desc), re.I) return "" if not m else ("yes" if m.group(1).lower() in ("true", "yes", "y") else "no") def norm_key(name): return re.sub(r"[^a-z0-9]+", "", str(name).lower()) def read_isolation_hosts(paths): """phage -> isolation host from pasted lists or CSV/TSV exports (e.g. PhageTracker): 'PheterStaphylococcus epidermidis', 'Pheter, S. epidermidis' or a table with phage/name and host columns.""" import csv out = {} for p in paths: if not p: continue if not os.path.isfile(p): print(f"⚠️ Isolation host list not found: {p}") continue lines = [ln for ln in open_text(p).read().splitlines() if ln.strip() and not ln.lstrip().startswith("#")] if not lines: continue def split(ln): # separator per line: tab, comma, semicolon, colon, else two spaces / "name Genus species" d = next((c for c in ("\\t", ",", ";", ":") if c in ln), None) if d: return next(csv.reader([ln], delimiter=d)) return re.split(r"\\s{2,}|\\s+(?=[A-Z][a-z]+\\s)", ln.strip(), maxsplit=1) table = [split(ln) for ln in lines] head = [c.strip().lower() for c in table[0]] pc = hc = None for i, c in enumerate(head): if pc is None and re.fullmatch(r"(phage|phage[ _]?name|name|sample|phage[ _]?id)", c): pc = i if hc is None and re.search(r"isolation[ _]?host|^host([ _]?(species|strain|name|organism|genus))?$", c): hc = i if pc is not None and hc is not None: body = table[1:] else: pc, hc, body = 0, 1, table for r in body: if len(r) > max(pc, hc) and r[pc].strip() and r[hc].strip(): out[norm_key(r[pc])] = r[hc].strip() return out def host_genus(h): h = str(h or "").strip() if not h: return "" w = re.sub(r"[^A-Za-z-]", "", h.split()[0]) w = GENUS_SYNONYMS.get(w.lower(), w) return w if len(w) > 2 else h # genera split from Mycobacterium in 2018 (PhagesDB lists M. smegmatis as Mycolicibacterium smegmatis) GENUS_SYNONYMS = {"mycolicibacterium": "Mycobacterium", "mycobacteroides": "Mycobacterium", "mycolicibacter": "Mycobacterium", "mycolicibacillus": "Mycobacterium"} def read_kv(path): d = {} if os.path.isfile(path): for line in open(path, errors="replace"): k, _, v = line.rstrip("\\n").partition("\\t") d[k] = v return d def pipeline_info(name, fasta, out_dir, in_dir=""): """coverage.html link (relative to the report, and relative to the input folder for a SharePoint base URL), warnings and circularity (Flye/Autocycler topology of the longest contig) from a Nanopore phage pipeline output folder.""" d = os.path.dirname(os.path.abspath(fasta)) link, rel, flags = "", "", [] for cand in (f"{name}.coverage.html", os.path.basename(d) + ".coverage.html"): p = os.path.join(d, cand) if os.path.isfile(p): link = os.path.relpath(p, os.path.abspath(out_dir)).replace(os.sep, "/") rel = os.path.relpath(p, os.path.abspath(in_dir)).replace(os.sep, "/") if in_dir else cand break circ = {"Y": "yes", "N": "no"}.get(read_kv(os.path.join(d, "qc", "longest_contig.tsv")).get("flye_circular", "").strip(), "") f1 = read_kv(os.path.join(d, "qc", "sample_flags.txt")).get("flags", "") for x in re.split(r";\\s*", f1): if x.strip() and x.strip().lower() not in ("none", "ok", "-"): flags.append(x.strip()) f2 = read_kv(os.path.join(d, "qc", "input_check.txt")).get("flag", "") if f2.strip() and f2.strip().lower() != "none": flags.append("input check: " + f2.strip()) return link, flags, circ, rel def pipeline_reads(fasta): """Bases sequenced (deduplicated; host reads removed when that was done) and read depth along the genome, from a Nanopore phage pipeline output folder: qc/readstats_*.json, qc/coverage_stats.txt, qc/coverage_dips.tsv.""" qc = os.path.join(os.path.dirname(os.path.abspath(fasta)), "qc") out = {"bases": "", "reads_label": "", "med_depth": "", "pct_low": "", "dips": ""} for fn, label in (("readstats_host_depleted.json", "phage reads (host reads removed)"), ("readstats_raw.json", "deduplicated reads")): p = os.path.join(qc, fn) if os.path.isfile(p): try: tb = json.load(open(p)).get("total_bases") if tb: out["bases"], out["reads_label"] = int(tb), label break except (ValueError, OSError): pass cs = read_kv(os.path.join(qc, "coverage_stats.txt")) out["med_depth"], out["pct_low"] = cs.get("median_depth", ""), cs.get("pct_below_lowcov", "") out["dips"] = read_kv(os.path.join(qc, "coverage_dips.tsv")).get("regions", "") return out def _ranks(v): o = sorted(range(len(v)), key=lambda i: v[i]) r = [0.0] * len(v) i = 0 while i < len(o): j = i while j + 1 < len(o) and v[o[j + 1]] == v[o[i]]: j += 1 for k in range(i, j + 1): r[o[k]] = (i + j) / 2.0 i = j + 1 return r def spearman(x, y): rx, ry = _ranks(x), _ranks(y) n = len(x) m = (n - 1) / 2.0 sxy = sum((a - m) * (b - m) for a, b in zip(rx, ry)) sxx = sum((a - m) ** 2 for a in rx) syy = sum((b - m) ** 2 for b in ry) return sxy / (sxx * syy) ** 0.5 if sxx and syy else 0.0 def gc_bias(fasta, sample, win=500, min_contig=5000): """v2.4: read depth vs GC content in 500 bp windows, from the pipeline's per-base coverage table (qc/.coverage.tsv.gz: contig, pos, consensus base, depth). Contigs of 5 kb or more; the first and last window of each contig are left out (ends have lower depth anyway). Depth is relative to the phage's median. Returns rho (Spearman, GC vs depth), slope (log2 depth change per +10% GC), and per-phage medians in 1% bins of GC minus the phage's mean GC, so phages with different GC content can be compared.""" import math qc = os.path.join(os.path.dirname(os.path.abspath(fasta)), "qc") p = os.path.join(qc, f"{sample}.coverage.tsv.gz") if not os.path.isfile(p): cand = [f for f in os.listdir(qc) if f.endswith(".coverage.tsv.gz")] if os.path.isdir(qc) else [] if len(cand) != 1: return None p = os.path.join(qc, cand[0]) wins, cur, acc = [], None, [] def flush(): if len(acc) >= max(3, min_contig // win): wins.extend(acc[1:-1]) gc = dep = n = 0 try: with gzip.open(p, "rt") as fh: next(fh, None) for line in fh: f = line.split("\\t", 5) if len(f) < 4: continue if f[0] != cur: flush() cur, acc, gc, dep, n = f[0], [], 0, 0, 0 b = f[2][:1].upper() gc += b in "GC" dep += int(f[3]) n += 1 if n == win: acc.append((100.0 * gc / win, dep / win)) gc = dep = n = 0 flush() except (OSError, ValueError, EOFError): return None if len(wins) < 20: return None ds = sorted(w[1] for w in wins) med = ds[len(ds) // 2] if med < 10: return None x = [w[0] for w in wins] y = [w[1] / med for w in wins] rho = spearman(x, y) ly = [math.log2(max(v, 0.01)) for v in y] mx, my = sum(x) / len(x), sum(ly) / len(ly) sxx = sum((a - mx) ** 2 for a in x) slope = 10 * sum((a - mx) * (b - my) for a, b in zip(x, ly)) / sxx if sxx else 0.0 bins = {} for a, v in zip(x, y): k = max(-15, min(15, int(round(a - mx)))) bins.setdefault(k, []).append(v) bout = {str(k): [round(sorted(v)[len(v) // 2], 3), len(v)] for k, v in bins.items() if len(v) >= 3} low = [v for a, v in zip(x, y) if a <= mx - 5] return {"rho": round(rho, 3), "slope": round(slope, 3), "n": len(wins), "gc": round(mx, 1), "med": round(med, 1), "bins": bout, "low5": round(sorted(low)[len(low) // 2], 3) if len(low) >= 5 else None} def check_mode(args, genomes, skipped): """--check: list the genomes and FASTA files that would be searched, without searching.""" iso = read_isolation_hosts(args.isolation_hosts) print(f"\\n{len(genomes)} genome(s) in {args.input} (contigs >= {args.min_contig:,} bp are searched):\\n") print(f" {'phage':<26} {'contigs':>7} {'searched':>8} {'searched bp':>12} {'longest bp':>11} FASTA") total = 0 for name, path in genomes: recs = read_fasta(path) if not recs: print(f" {name:<26} {'—':>7} ⚠️ no sequence in {path}") continue q = genome_query(name, path, recs, args.min_contig) total += 1 print(f" {name:<26} {q['contigs']:>7} {q['searched']:>8} {q['length']:>12,} {q['longest']:>11,} {path}") extra = [] if iso: extra.append("isolation host: " + (iso.get(norm_key(name)) or "— (not in the list)")) link, flags, circ, _ = pipeline_info(name, path, args.out, args.input) circ = circ or q["circ_hdr"] extra.append("circular: " + (circ or "unknown")) if link: extra.append("coverage.html found") extra += [f"⚠️ {f}" for f in flags] if extra: print(f" {'':<26} " + "; ".join(extra)) if skipped: print(f"\\n not phage folders, skipped: {', '.join(skipped)}") if iso: names = {norm_key(n) for n, _ in genomes} unused = [k for k in iso if k not in names] print(f"\\nIsolation hosts: {len(iso)} listed" + (f"; {len(unused)} not matched to a genome folder" if unused else "")) print(f"\\nCheck only: nothing was searched. {total} genome(s) would be searched.") # ---------------------------------------------------------------- report def url_pdb(name): return "https://phagesdb.org/phages/" + urllib.parse.quote(str(name)) + "/" HTML = r""" Closest PhagesDB phages

Closest PhagesDB phages

Overview

Phages

Host Cluster Rows visible

How to read this table

Closest phage is the better match of PhagesDB and GenBank (ranked by identity × share of your genome covered, minimap2 asm20). Identity is over the aligned parts and covered is the share of your genome that aligns (like BLAST's Per. ident and Query cover, but not identical to BLASTn's numbers). Covers of closest is the share of the closest phage covered by your contigs; status: complete = one contig covers ≥90% of it, or is about as long as it (±10%) with ≥80% of yours aligned (a complete genome of a different phage; marked *), fragmented = the same with several contigs, partial = less, no relative = under 50% of your genome aligns. ICTV is an approximate intergenomic similarity to the closest phage (≥95% same species, ≥70% same genus). +N similar = other phages within 0.5% of the best match. All contigs of at least bp are searched. "Likely cluster" is tentative (PhagesDB assigns clusters; "?" = under 50% aligned). Click ▸ for the top hits and where your contigs align; click a bar or number in the Overview to show only those phages; click a column heading to sort. All hits and numbers: closest_phages.tsv, closest_phages_top_hits.tsv, closest_genbank_top_hits.tsv.

""" def main(): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--input", required=True, help="folder with one subfolder (or FASTA file) per phage") ap.add_argument("--phagesdb", required=True, help="PhagesDB all-genomes FASTA (Actinobacteriophages-All.fasta)") ap.add_argument("--table", default="", help="PhagesDB data table (tab-delimited), optional") ap.add_argument("--out", required=True, help="output folder") ap.add_argument("--pattern", default="", help="FASTA file name in each subfolder, e.g. {phage}.fasta or *.fasta (default: automatic)") ap.add_argument("--threads", type=int, default=4) ap.add_argument("--top", type=int, default=5) ap.add_argument("--ncbi", choices=["none", "hosts", "all"], default="none", help="also search complete phage genomes from NCBI: by host genus, or all Caudoviricetes") ap.add_argument("--ncbi-hosts", default="Mycobacterium Gordonia Microbacterium Arthrobacter Staphylococcus", help="host genera for --ncbi hosts") ap.add_argument("--ncbi-fasta", default="", help="use this FASTA of NCBI phage genomes instead of downloading") ap.add_argument("--ncbi-dir", default="", help="where NCBI genomes are kept (default: NCBI_phages next to the PhagesDB FASTA)") ap.add_argument("--ncbi-refresh", action="store_true", help="download the NCBI genomes again") ap.add_argument("--host-genome", action="append", default=[], help="host genome FASTA (repeat for several hosts)") ap.add_argument("--checkv-db", default="", help="CheckV database folder (run CheckV when given)") ap.add_argument("--checkv-bin", default="", help="bin folder of the CheckV conda environment") ap.add_argument("--checkv-download", action="store_true", help="download the CheckV database if it is missing") ap.add_argument("--no-similarity", action="store_true", help="skip the similarity matrix") ap.add_argument("--min-contig", type=int, default=5000, help="search every contig at least this long (bp); the longest contig is used when none is") ap.add_argument("--isolation-hosts", action="append", default=[], help="phage -> isolation host list (TSV/CSV, or a PhageTracker export); repeat for several") ap.add_argument("--coverage-url", default="", help="web folder (e.g. SharePoint) that holds the phage folders with the coverage.html files; " "can also be changed in the report") ap.add_argument("--no-gc-bias", action="store_true", help="skip the read depth vs GC content analysis") ap.add_argument("--check", action="store_true", help="only list the genomes and FASTA files; no search") args = ap.parse_args() t0 = time.time() print(TOOL_VERSION) if not os.path.isdir(args.input): sys.exit(f"Input folder not found: {args.input}") if args.check: genomes, skipped = find_genomes(args.input, args.pattern, skip_dirs=[args.out]) if not genomes: sys.exit(f"No FASTA files found in the subfolders of {args.input}") check_mode(args, genomes, skipped) if not os.path.isfile(args.phagesdb): print(f"⚠️ PhagesDB FASTA not found: {args.phagesdb}") return if not os.path.isfile(args.phagesdb): sys.exit(f"PhagesDB FASTA not found: {args.phagesdb}") table = "" if args.table: for cand in (args.table, args.table + ".txt", args.table + ".tsv", args.table + ".tab"): if os.path.isfile(cand): table = cand break if not table: print(f"⚠️ PhagesDB data table not found: {args.table} (also tried .txt/.tsv/.tab); using the FASTA headers only.") os.makedirs(args.out, exist_ok=True) genomes, skipped = find_genomes(args.input, args.pattern, skip_dirs=[args.out]) if not genomes: sys.exit(f"No FASTA files found in the subfolders of {args.input}") print(f"{len(genomes)} genome(s) found in {args.input}") if skipped: print(f" not phage folders, skipped: {', '.join(skipped)}") index, meta = prepare_phagesdb(args.phagesdb, table, args.threads) by_phage = {} for pid, m in meta.items(): by_phage.setdefault(str(m.get("phage", "")).lower(), pid) # one query per genome: all its contigs >= --min-contig, joined with N spacers import tempfile qdir = tempfile.mkdtemp(prefix="closest_phages_") qpath = os.path.join(qdir, "queries.fasta") jpath = os.path.join(qdir, "genomes_joined.fasta") # for CheckV: one sequence per genome info, qoff = {}, {} with open(qpath, "w") as fh, open(jpath, "w") as fj: for i, (name, path) in enumerate(genomes): recs = read_fasta(path) if not recs: print(f"⚠️ {name}: {path} has no sequence; skipped") continue qid = f"q{i}" info[qid] = genome_query(name, path, recs, args.min_contig) write_fasta(fj, qid, info[qid].pop("seq")) byname = {r[0]: r[2] for r in recs} for k, o in enumerate(info[qid]["offs"]): write_fasta(fh, f"{qid}__c{k}", byname[o[2]]) qoff[f"{qid}__c{k}"] = (qid, o[0]) nmulti = sum(1 for q in info.values() if q["searched"] > 1) print(f"Contigs >= {args.min_contig:,} bp are searched" + (f" ({nmulti} genome(s) have more than one)" if nmulti else "")) iso = read_isolation_hosts(args.isolation_hosts) if args.isolation_hosts: print(f"Isolation hosts: {len(iso)} listed, " f"{sum(1 for q in info.values() if norm_key(q['sample']) in iso)} matched to these genomes") print(f"Searching PhagesDB with minimap2 ({args.threads} threads)…" + mem_note(), flush=True) try: hits = search(index, qpath, args.threads, qoff) except RuntimeError as e: sys.exit(str(e)) ncbi_hits, ncbi_meta, ncbi_info, ncbi_label = None, {}, {}, "" if args.ncbi != "none" or args.ncbi_fasta: hosts = [h for h in re.split(r"[,;\\s]+", args.ncbi_hosts) if h] ncbi_dir = args.ncbi_dir or os.path.join(os.path.dirname(os.path.abspath(args.phagesdb)), "NCBI_phages") try: nidx, ncbi_meta, ncbi_info = prepare_ncbi(ncbi_dir, args.ncbi, hosts, args.ncbi_fasta, args.ncbi_refresh, args.threads) ncbi_label = (os.path.basename(args.ncbi_fasta) if args.ncbi_fasta else ("all Caudoviricetes" if args.ncbi == "all" else "phages of " + ", ".join(hosts))) print(f"Searching {len(ncbi_meta):,} NCBI genomes ({ncbi_label}" f"{'; downloaded ' + ncbi_info['downloaded'] if ncbi_info.get('downloaded') else ''})…" + mem_note(), flush=True) ncbi_hits = search(nidx, qpath, args.threads, qoff) except Exception as e: # keep the PhagesDB results even if NCBI fails print(f"⚠️ NCBI search skipped: {e}") ncbi_hits = None # ---- genome checks hostres, checkv, sim, pair, checks = {}, {}, {}, {}, {"host": False, "checkv": False, "sim": False, "host_names": []} hosts_ok = [h for h in args.host_genome if h and os.path.isfile(h)] for h in args.host_genome: if h and not os.path.isfile(h): print(f"⚠️ Host genome not found: {h}") if hosts_ok: print(f"Checking for host DNA ({len(hosts_ok)} host genome(s))…" + mem_note(), flush=True) try: hostres = host_check(qpath, hosts_ok, args.threads, qoff, {k: v["length"] for k, v in info.items()}) checks["host"] = True checks["host_names"] = [re.sub(r"\\.(fasta|fa|fna|fas)(\\.gz)?$", "", os.path.basename(h), flags=re.I) for h in hosts_ok] except RuntimeError as e: print(f"⚠️ Host check skipped: {e}") if args.checkv_db: try: checkv = run_checkv(jpath, args.out, args.checkv_bin, args.checkv_db, args.checkv_download, args.threads) checks["checkv"] = True except Exception as e: print(f"⚠️ CheckV skipped: {e}") if not args.no_similarity and len(info) > 1: print("Comparing all genomes with each other (intergenomic similarity)…" + mem_note(), flush=True) try: sim, pair = similarity_matrix(qpath, info, args.threads, qoff) checks["sim"] = True except RuntimeError as e: print(f"⚠️ Similarity matrix skipped: {e}") species = {q: g for g in groups_at(list(info), sim, 95.0) for q in g} if checks["sim"] else {} genus = {q: g for g in groups_at(list(info), sim, 70.0) for q in g} if checks["sim"] else {} gnum, snum = {}, {} for q in sorted(info, key=lambda x: info[x]["sample"].lower()): if genus: gnum.setdefault(tuple(genus[q]), len(gnum) + 1) snum.setdefault(tuple(species[q]), len(snum) + 1) import shutil shutil.rmtree(qdir, ignore_errors=True) # phages in this run that are the same phage: >= 98% similarity, or >= 98% identity over >= 80% of the shorter # genome (a fragmented assembly can't reach 98% similarity); the value shown is the identity dups = {} for (a, b), v in sim.items(): idn, cs = pair.get((a, b), (0.0, 0.0)) if v >= 98 or (idn >= 98 and cs >= 80): dups.setdefault(a, []).append([info[b]["sample"], idn if v < 98 else v]) dups.setdefault(b, []).append([info[a]["sample"], idn if v < 98 else v]) if not args.no_gc_bias: print("Read depth vs GC content (from the pipeline's coverage tables)…", flush=True) rows, long_rows, ncbi_long = [], [], [] for qid, q in info.items(): L = q["length"] pr = rank_hits(hits.get(qid), L) nr = rank_hits(ncbi_hits.get(qid), L) if ncbi_hits is not None else [] row = {"sample": q["sample"], "file": q["file"], "contigs": q["contigs"], "searched": q["searched"], "longest": q["longest"], "contig_list": [[o[2], o[3]] for o in q["offs"]], "length": L, "gc": q["gc"], "hit": "", "hit_url": "", "self": False, "cluster": "", "identity": "", "aligned": "", "len_diff": "", "host": "", "temperate": "", "morphotype": "", "ends": "", "accession": "", "gc_hit": "", "other": "", "other_url": "", "other_cluster": "", "other_identity": "", "other_aligned": "", "next": [], "note": "", "ncbi": None, "ncbi_next": [], "host_pct": "", "host_name": "", "cv_quality": "", "cv_completeness": "", "cv_method": "", "cv_contamination": "", "cv_viral_genes": "", "cv_host_genes": "", "cv_provirus": "", "cv_warnings": "", "species": "", "species_with": [], "genus": "", "genus_with": [], "best": {}, "status": "", "ties": [], "dups": sorted(dups.get(qid, []), key=lambda x: -x[1]), "iso_host": iso.get(norm_key(q["sample"]), ""), "host_mismatch": "", "cov_html": "", "pipe_flags": [], "top_pdb": [], "top_gb": []} row["cov_html"], row["pipe_flags"], row["circular"], row["cov_rel"] = pipeline_info(q["sample"], q["file"], args.out, args.input) row["circular"] = row["circular"] or q["circ_hdr"] row.update(pipeline_reads(q["file"])) row["gcb"] = gc_bias(q["file"], q["sample"]) if not args.no_gc_bias else None if checks["host"]: hp = hostres.get(qid) row["host_pct"], row["host_name"] = (hp[0], hp[1]) if hp else (0.0, "") if checks["checkv"] and qid in checkv: c = checkv[qid] row.update({"cv_quality": c.get("checkv_quality", ""), "cv_completeness": c.get("completeness", "NA"), "cv_method": c.get("completeness_method", ""), "cv_contamination": c.get("contamination", ""), "cv_viral_genes": c.get("viral_genes", ""), "cv_host_genes": c.get("host_genes", ""), "cv_provirus": c.get("provirus", ""), "cv_warnings": c.get("warnings", "")}) if checks["sim"]: row["species"] = f"S{snum[tuple(species[qid])]}" row["genus"] = f"G{gnum[tuple(genus[qid])]}" row["species_with"] = [info[x]["sample"] for x in species[qid] if x != qid] row["genus_with"] = [info[x]["sample"] for x in genus[qid] if x != qid] if ncbi_hits is not None: for rank, x in enumerate(nr[:args.top], 1): m = ncbi_meta.get(x["id"], {}) ncbi_long.append([q["sample"], rank, x["id"], m.get("description", ""), m.get("host", ""), x["ident"], x["qcov"], x["aln"], x["tlen"], x["scov"], L - x["tlen"], url_ncbi(x["id"])]) row["top_gb"].append([x["id"], m.get("description", ""), m.get("host", ""), x["ident"], x["qcov"], x["scov"], x["tlen"], url_ncbi(x["id"])]) if nr: x = nr[0] m = ncbi_meta.get(x["id"], {}) row["ncbi"] = {"acc": x["id"], "desc": m.get("description", ""), "host": m.get("host", ""), "ident": x["ident"], "qcov": x["qcov"], "aln": x["aln"], "slen": x["tlen"], "scov": x["scov"], "len_diff": f"{L - x['tlen']:+d}", "url": url_ncbi(x["id"])} row["ncbi_next"] = [[y["id"], ncbi_meta.get(y["id"], {}).get("description", ""), y["ident"], y["qcov"], url_ncbi(y["id"])] for y in nr[1:4]] else: row["ncbi"] = {} for rank, x in enumerate(pr[:args.top], 1): m = meta.get(x["id"], {}) nm = m.get("phage") or x["id"] long_rows.append([q["sample"], rank, nm, m.get("cluster", ""), x["ident"], x["qcov"], x["tlen"], L - x["tlen"], m.get("host", ""), m.get("temperate", ""), m.get("morphotype", ""), m.get("ends", ""), m.get("accession", ""), url_pdb(nm)]) row["top_pdb"].append([nm, m.get("cluster", ""), m.get("host", ""), x["ident"], x["qcov"], x["scov"], x["tlen"], url_pdb(nm)]) if pr: x = pr[0] m = meta.get(x["id"], {}) name = m.get("phage") or x["id"] row.update({"hit": name, "hit_url": url_pdb(name), "self": name.lower() == q["sample"].lower(), "cluster": m.get("cluster", ""), "identity": x["ident"], "aligned": x["qcov"], "len_diff": f"{L - x['tlen']:+d}", "host": m.get("host", ""), "temperate": m.get("temperate", ""), "morphotype": m.get("morphotype", ""), "ends": m.get("ends", ""), "accession": m.get("accession", ""), "gc_hit": m.get("gc_pct", "")}) for y in pr: n2 = meta.get(y["id"], {}).get("phage") or y["id"] if n2.lower() != q["sample"].lower(): row.update({"other": n2, "other_url": url_pdb(n2), "other_cluster": meta.get(y["id"], {}).get("cluster", ""), "other_identity": y["ident"], "other_aligned": y["qcov"]}) break row["next"] = [[meta.get(y["id"], {}).get("phage") or y["id"], meta.get(y["id"], {}).get("cluster", ""), y["ident"], y["qcov"], url_pdb(meta.get(y["id"], {}).get("phage") or y["id"])] for y in pr[1:4]] if x["qcov"] < 50: row["note"] = "no close relative (<50% aligned)" elif row["self"]: row["note"] = "same name as the PhagesDB phage" else: row["note"] = "no PhagesDB phage aligns" # closest phage overall: the better of the PhagesDB and GenBank top hits (identity x query cover) cands = [] if pr: x = pr[0] cands.append({"score": float(x["ident"]) * float(x["qcov"]), "db": "PhagesDB", "name": row["hit"], "url": row["hit_url"], "extra": row["cluster"], "host": row["host"], "acc": row["accession"], "x": x, "d": hits[qid][x["id"]]}) if nr: x, nb = nr[0], row["ncbi"] short = re.sub(r",?\\s*(complete|partial)\\s+(genome|sequence|cds).*$", "", nb["desc"] or nb["acc"], flags=re.I).strip() cands.append({"score": float(x["ident"]) * float(x["qcov"]), "db": "GenBank", "name": short or nb["acc"], "url": nb["url"], "extra": nb["acc"], "host": nb["host"], "acc": nb["acc"], "x": x, "d": ncbi_hits[qid][x["id"]]}) if cands: b = max(cands, key=lambda c: c["score"]) x = b["x"] sv = ictv_sim(x["ident"], x["aln"], x["taln"], L, x["tlen"]) row["best"] = {"name": b["name"], "url": b["url"], "db": b["db"], "ident": x["ident"], "qcov": x["qcov"], "extra": b["extra"], "len_diff": f"{L - x['tlen']:+d}", "host": b["host"], "acc": b["acc"], "slen": x["tlen"], "scov": x["scov"], "sim": sv, "rank": ictv_rank(sv), "rank_note": ("" if float(x["scov"]) >= 90 or L >= 0.9 * x["tlen"] else "your contigs cover under 90% of the closest phage, so the similarity is underestimated"), "blocks": aln_blocks(b["d"], q["offs"])} if b["db"] == "GenBank" and row["note"].startswith("no "): row["note"] = "closest relative is in GenBank, not PhagesDB" # near-equal matches: identity x cover within 0.5% of the best, in either database thr, seen, ties = b["score"] * 0.995, {b["name"].lower(), str(b["acc"]).split(".")[0].lower()}, [] for db, lst in (("PhagesDB", pr), ("GenBank", nr)): for y in lst: sc = float(y["ident"]) * float(y["qcov"]) if sc < thr: break if db == "PhagesDB": m = meta.get(y["id"], {}) nm, acc, url = m.get("phage") or y["id"], m.get("accession", ""), url_pdb(m.get("phage") or y["id"]) else: m = ncbi_meta.get(y["id"], {}) nm = re.sub(r",?\\s*(complete|partial)\\s+(genome|sequence|cds).*$", "", m.get("description", "") or y["id"], flags=re.I).strip() or y["id"] acc, url = y["id"], url_ncbi(y["id"]) keys = {nm.lower(), str(acc).split(".")[0].lower()} - {""} if keys & seen: continue seen |= keys ties.append([nm, db, y["ident"], y["qcov"], url]) row["ties"] = ties[:20] n_ties = len(ties) else: row["best"] = {} n_ties = 0 bst = row["best"] row["status_note"] = "" ratio = L / float(bst["slen"]) if bst and bst.get("slen") else 0 if not bst or float(bst["qcov"]) < 50: row["status"] = "no relative" elif float(bst["scov"]) >= 90: row["status"] = "complete" if q["searched"] == 1 else "fragmented" elif 0.9 <= ratio <= 1.1 and float(bst["qcov"]) >= 80: # v2.2: about as long as the closest phage and mostly aligned, but only part of the relative matches: # a complete genome of a different phage, not a missing piece row["status"] = "complete" if q["searched"] == 1 else "fragmented" row["status_note"] = (f"{L:,} bp, about as long as the closest phage ({int(bst['slen']):,} bp), but only " f"{bst['scov']}% of it matches: likely complete, and a different phage") else: row["status"] = "partial" ig, bg = host_genus(row["iso_host"]), host_genus(bst.get("host", "")) if ig and bg and ig.lower() != bg.lower() and row["status"] != "no relative": row["host_mismatch"] = f"isolated on {ig}, closest phage infects {bg}" flags = [] if row["host_pct"] != "" and float(row["host_pct"]) >= 50: flags.append(f"mostly host DNA ({row['host_pct']}% matches {row['host_name']})") elif row["host_pct"] != "" and float(row["host_pct"]) >= 5: flags.append(f"{row['host_pct']}% host DNA") if row["cv_quality"] in ("Low-quality", "Not-determined"): flags.append(f"CheckV: {row['cv_quality']}") if row["dups"]: flags.append("same phage as " + ", ".join(f"{n} ({v}%)" for n, v in row["dups"][:3])) if row["host_mismatch"]: flags.append(row["host_mismatch"]) if flags: row["note"] = "; ".join([row["note"]] + flags if row["note"] else flags) rows.append(row) status = (f"{bst['name']} [{bst['db']}] {bst['ident']}% identity, {bst['qcov']}% of yours covered, " f"covers {bst['scov']}% of it ({bst['slen']:,} bp) — {row['status']}, {bst['rank']} (~{bst['sim']}%)" if bst else "none") print(f" {q['sample']:<28} {status}{' — ' + row['note'] if row['note'] else ''}") if q["searched"] > 1: print(f" {'':<28} {q['searched']} contigs searched ({L:,} bp)") if n_ties: print(f" {'':<28} {n_ties} other phage(s) within 0.5% of the best: " + ", ".join(t[0] for t in row["ties"][:4])) if checks["host"] or checks["checkv"]: bits = [] if checks["host"]: bits.append(f"host DNA {row['host_pct']}%") if row["cv_quality"]: bits.append(f"CheckV {row['cv_quality']}, completeness {row['cv_completeness']}%") print(f" {'':<28} " + "; ".join(bits)) if row["ncbi"]: nb = row["ncbi"] print(f" {'':<28} GenBank: {nb['acc']} {nb['desc'][:60]} — {nb['ident']}% ident, {nb['qcov']}% query cover") elif row["ncbi"] == {}: print(f" {'':<28} GenBank: no phage aligns") rows.sort(key=lambda x: x["sample"].lower()) cols = ["phage", "genome_file", "contigs", "length_bp", "gc_pct", "closest_phage", "closest_source", "closest_identity_pct", "closest_query_cover_pct", "closest_link", "closest_phagesdb", "likely_cluster", "identity_pct", "aligned_pct", "length_difference_bp", "host", "temperate", "morphotype", "ends", "accession", "phagesdb_gc_pct", "closest_other_phage", "other_cluster", "other_identity_pct", "other_aligned_pct", "note", "phagesdb_link", "genbank_accession", "genbank_description", "genbank_host", "genbank_per_ident", "genbank_query_cover", "genbank_aligned_bp", "genbank_subject_length", "genbank_subject_cover", "genbank_length_difference_bp", "genbank_link", "host_dna_pct", "host_matched", "checkv_quality", "checkv_completeness_pct", "checkv_completeness_method", "checkv_contamination_pct", "checkv_viral_genes", "checkv_host_genes", "checkv_provirus", "checkv_warnings", "species_group_95pct", "same_species_as", "genus_group_70pct", "same_genus_as", "contigs_searched", "closest_length_bp", "closest_covered_pct", "status", "similar_matches", "ictv_similarity_to_closest_pct", "ictv_rank", "same_phage_in_run", "isolation_host", "host_mismatch", "coverage_html", "pipeline_flags", "circular", "status_note", "bases_sequenced", "median_read_depth", "pct_bases_low_depth", "low_depth_regions", "gc_bias_rho", "gc_bias_log2_depth_per_10pct_gc", "depth_in_windows_5pct_below_mean_gc"] with open(os.path.join(args.out, "closest_phages.tsv"), "w") as out: out.write("\\t".join(cols) + "\\n") for x in rows: out.write("\\t".join(str(v) for v in [ x["sample"], x["file"], x["contigs"], x["length"], x["gc"]] + [x["best"].get(k, "") for k in ("name", "db", "ident", "qcov", "url")] + [x["hit"], x["cluster"], x["identity"], x["aligned"], x["len_diff"], x["host"], x["temperate"], x["morphotype"], x["ends"], x["accession"], x["gc_hit"], x["other"], x["other_cluster"], x["other_identity"], x["other_aligned"], x["note"], x["hit_url"]] + [(x["ncbi"] or {}).get(k, "") for k in ("acc", "desc", "host", "ident", "qcov", "aln", "slen", "scov", "len_diff", "url")] + [x["host_pct"], x["host_name"], x["cv_quality"], x["cv_completeness"], x["cv_method"], x["cv_contamination"], x["cv_viral_genes"], x["cv_host_genes"], x["cv_provirus"], x["cv_warnings"], x["species"], ", ".join(x["species_with"]), x["genus"], ", ".join(x["genus_with"])] + [x["searched"], x["best"].get("slen", ""), x["best"].get("scov", ""), x["status"], ", ".join(t[0] for t in x["ties"]), x["best"].get("sim", ""), x["best"].get("rank", "") if x["best"] else "", ", ".join(f"{n} ({v}%)" for n, v in x["dups"]), x["iso_host"], x["host_mismatch"], x["cov_html"], "; ".join(x["pipe_flags"]), x["circular"], x["status_note"], x["bases"], x["med_depth"], x["pct_low"], x["dips"]] + ([x["gcb"]["rho"], x["gcb"]["slope"], x["gcb"]["low5"] if x["gcb"]["low5"] is not None else ""] if x.get("gcb") else ["", "", ""])) + "\\n") with open(os.path.join(args.out, "closest_phages_top_hits.tsv"), "w") as out: out.write("phage\\trank\\tphagesdb_phage\\tcluster\\tidentity_pct\\taligned_pct\\tphagesdb_length_bp\\tlength_difference_bp" "\\thost\\ttemperate\\tmorphotype\\tends\\taccession\\tphagesdb_link\\n") for lr in long_rows: out.write("\\t".join(str(v) for v in lr) + "\\n") if ncbi_hits is not None: with open(os.path.join(args.out, "closest_genbank_top_hits.tsv"), "w") as out: out.write("phage\\trank\\taccession\\tdescription\\thost\\tper_ident\\tquery_cover\\taligned_bp\\tsubject_length" "\\tsubject_cover\\tlength_difference_bp\\tgenbank_link\\n") for lr in ncbi_long: out.write("\\t".join(str(v) for v in lr) + "\\n") matrix = None if checks["sim"]: order = sorted(info, key=lambda q: (gnum[tuple(genus[q])], snum[tuple(species[q])], info[q]["sample"].lower())) names = [info[q]["sample"] for q in order] vals = [[100.0 if a == b else sim.get(tuple(sorted((a, b))), 0.0) for b in order] for a in order] with open(os.path.join(args.out, "similarity_matrix.tsv"), "w") as out: out.write("phage\\t" + "\\t".join(names) + "\\n") for n, row_v in zip(names, vals): out.write(n + "\\t" + "\\t".join(str(v) for v in row_v) + "\\n") matrix = {"names": names, "vals": vals, "species": [f"S{snum[tuple(species[q])]}" for q in order], "genus": [f"G{gnum[tuple(genus[q])]}" for q in order]} data = {"rows": rows, "input": args.input, "pdb_file": os.path.basename(args.phagesdb), "pdb_n": len(meta), "checks": checks, "matrix": matrix, "when": time.strftime("%Y-%m-%d %H:%M"), "ncbi": ncbi_hits is not None, "ncbi_n": len(ncbi_meta), "ncbi_label": ncbi_label, "ncbi_date": ncbi_info.get("downloaded", ""), "ncbi_max_age": NCBI_MAX_AGE_DAYS, "min_contig": args.min_contig, "cov_base": args.coverage_url, "has_iso": bool(iso), "version": TOOL_VERSION} page = HTML.replace("__DATA__", json.dumps(data, separators=(",", ":")).replace("= 50) print(f"\\n{len(rows)} genomes: {n_close} with a close PhagesDB relative, {len(rows) - n_close} without " f"({time.time() - t0:.0f} s)") if checks["host"]: nh = sum(1 for x in rows if x["host_pct"] != "" and float(x["host_pct"]) >= 50) print(f"Host DNA: {nh} genome(s) are mostly host DNA (≥50% matches a host genome)") if checks["checkv"]: qual = {} for x in rows: qual[x["cv_quality"] or "not run"] = qual.get(x["cv_quality"] or "not run", 0) + 1 print("CheckV: " + ", ".join(f"{k} {v}" for k, v in sorted(qual.items()))) if checks["sim"]: multi = [x for x in rows if x["species_with"]] print(f"Similarity: {len(snum)} species group(s) (≥95%), {len(gnum)} genus group(s) (≥70%); " f"{len(multi)} genome(s) share a species group with another") print(f"Matrix: {os.path.join(args.out, 'similarity_matrix.tsv')}") st = {} for x in rows: st[x["status"]] = st.get(x["status"], 0) + 1 print("Status: " + ", ".join(f"{k} {v}" for k, v in sorted(st.items()))) gb = [x for x in rows if x.get("gcb")] if gb: neg = sum(1 for x in gb if x["gcb"]["rho"] >= 0.3) print(f"GC bias: {len(gb)} phage(s) with coverage tables; {neg} with clearly lower depth in AT-rich DNA (rho >= 0.3), " f"{sum(1 for x in gb if x['gcb']['rho'] <= -0.3)} with lower depth in GC-rich DNA (rho <= -0.3)") nd = sum(1 for x in rows if x["dups"]) if nd: print(f"Same phage in this run (>= 98% identity over >= 80% of the shorter genome, or >= 98% similar): {nd} genome(s)") print(f"Table: {os.path.join(args.out, 'closest_phages.tsv')}") print(f"Report: {os.path.join(args.out, 'closest_phages.html')}") if __name__ == "__main__": main() `; // v24: one shared tool — when the "Closest phages" generator cell is on the same page and carries a newer copy of // closest_phages.py, that copy is used, so the tool is edited in one place const CLOSEST_PY_VERSION = 346; const closestPy = () => { try { const x = window.__closestPhagesPy; if (x && x.code && x.version >= CLOSEST_PY_VERSION) return x.code; } catch (_) {} return CLOSEST_PY; }; // Static body of the generated Bash script. Configuration is prepended. const PIPELINE_BODY = `MAIN_TRAP='st=$?; echo; echo "❌ Pipeline failed at line $LINENO (exit $st)" >&2; exit $st' trap "$MAIN_TRAP" ERR export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH" STAMP="$(date +%Y%m%d_%H%M%S)" echo "Generated by: \${GENERATOR_VERSION:-nanopore-phage-pipeline (unknown version)}" # v20: bash phage_pipeline.sh --check → list the phages and what a run would do, then stop (nothing is assembled) CHECK_ONLY=false for _arg in "$@"; do if [[ "$_arg" == "--check" ]]; then CHECK_ONLY=true; fi; done if [[ "$CHECK_ONLY" == "true" ]]; then echo "🔎 CHECK ONLY: nothing will be assembled."; fi # v26: reports only — keep every existing assembly whatever the settings; rebuild the reports and the genome checks if [[ "\${REPORTS_ONLY:-false}" == "true" ]]; then OVERWRITE=false echo "📄 REPORTS ONLY: existing assemblies are kept as they are (assembly settings are ignored); coverage reports," echo " annotation checks and the genome checks are rebuilt. Phages without an assembly are skipped." fi # ============================================================== # Small helpers # ============================================================== fail() { echo "❌ $*" >&2; exit 1; } warn() { echo "⚠️ $*" >&2; } # step KEY "message": prints the message with the time, and records the time of the step for the timing report step() { local key="$1" shift echo "[$(date +%H:%M:%S)] $*" if [[ -n "\${STEP_FILE:-}" ]]; then printf '%s\\t%s\\n' "$(date +%s)" "$key" >> "$STEP_FILE" 2>/dev/null || true; fi } is_true() { [[ "$1" == "true" ]]; } expand_user_path() { case "$1" in "~") printf '%s' "$HOME" ;; "~/"*) printf '%s/%s' "$HOME" "\${1:2}" ;; *) printf '%s' "$1" ;; esac } strip_trailing_slash() { local x="$1" while [[ "$x" == */ && "$x" != "/" ]]; do x="\${x%/}"; done printf '%s' "$x" } strip_fastq_ext() { local x="$1" shopt -s nocasematch case "$x" in *.fastq.gz) x="\${x:0:\${#x}-9}" ;; *.fq.gz) x="\${x:0:\${#x}-6}" ;; *.fastq) x="\${x:0:\${#x}-6}" ;; *.fq) x="\${x:0:\${#x}-3}" ;; esac shopt -u nocasematch printf '%s' "$x" } sanitize_sample() { local x="$1" x="\${x//\\//__}" x="$(printf '%s' "$x" | LC_ALL=C tr -c 'A-Za-z0-9._-' '_')" while [[ "$x" == _* ]]; do x="\${x#_}"; done while [[ "$x" == *_ ]]; do x="\${x%_}"; done [[ -n "$x" ]] || x="sample" printf '%s' "$x" } # Run a command with a time limit; returns 124 if it had to be stopped. Works without GNU "timeout". run_limited() { # run_limited SECONDS command... local secs="$1"; shift "$@" & local pid=$! ( kill_tree() { # stop a process and everything it started (e.g. the kaleido renderer) local sig="$1" p child for p in "\${@:2}"; do for child in $(pgrep -P "$p" 2>/dev/null); do kill_tree "$sig" "$child"; done kill "-$sig" "$p" 2>/dev/null done } waited=0 while kill -0 "$pid" 2>/dev/null; do if (( waited >= secs )); then kill_tree TERM "$pid"; sleep 5; kill_tree KILL "$pid" exit 0 fi sleep 5; waited=$((waited + 5)) done ) & local watcher=$! local st=0 wait "$pid" || st=$? wait "$watcher" 2>/dev/null || true if (( st == 143 || st == 137 )); then return 124; fi return "$st" } # Conda's activation scripts are not always safe under "set -u". use_env() { set +u conda activate "$1" set -u } # ============================================================== # Validate paths # ============================================================== INPUT_PATH="$(strip_trailing_slash "$INPUT_PATH")" OUT_ROOT="$(strip_trailing_slash "$OUT_ROOT")" PHAROKKA_DB="$(strip_trailing_slash "$(expand_user_path "$PHAROKKA_DB_INPUT")")" PHOLD_DB="$(strip_trailing_slash "$(expand_user_path "$PHOLD_DB_INPUT")")" [[ -n "$INPUT_PATH" ]] || fail "No input path supplied." [[ -n "$OUT_ROOT" ]] || fail "No output path supplied." if [[ "$INPUT_MODE" == "single" ]]; then [[ -f "$INPUT_PATH" ]] || fail "FASTQ file does not exist: $INPUT_PATH" else [[ -d "$INPUT_PATH" ]] || fail "Input folder does not exist: $INPUT_PATH" fi if is_true "$RUN_PHOLD" && ! is_true "$RUN_PHAROKKA"; then fail "Phold needs Pharokka output. Enable Pharokka as well." fi mkdir -p "$OUT_ROOT" [[ -w "$OUT_ROOT" ]] || fail "Output folder is not writable: $OUT_ROOT" TOOLS_DIR="$OUT_ROOT/.phage_pipeline" mkdir -p "$TOOLS_DIR" LOG_FILE="$OUT_ROOT/pipeline_\${STAMP}.log" SUMMARY_FILE="$OUT_ROOT/pipeline_summary_\${STAMP}.tsv" # Copy all console output into a log file. exec > >(tee -a "$LOG_FILE") 2>&1 echo echo "==============================================================" if [[ "$ASSEMBLY_MODE" == "accuracy" ]]; then echo "Nanopore phage pipeline (Accuracy mode): dedup → Filtlong → Autocycler consensus → Medaka" echo " → genome ends / rotation" else echo "Nanopore phage pipeline: dedup → Filtlong → Flye → Medaka" fi if is_true "$RUN_PHAROKKA"; then echo " → longest contig → Pharokka"; fi if is_true "$RUN_PHOLD"; then echo " → Phold"; fi echo "==============================================================" echo "Input mode: $INPUT_MODE" echo "Input path: $INPUT_PATH" echo "Output root: $OUT_ROOT" echo "Log file: $LOG_FILE" echo # ============================================================== # Locate a Conda installation that holds every needed environment # ============================================================== case "$ASSEMBLY_MODE" in standard|accuracy) ;; *) fail "Unknown assembly mode: $ASSEMBLY_MODE" ;; esac REQUIRED_ENVS=("$CORE_ENV" "$MEDAKA_ENV") if [[ "$ASSEMBLY_MODE" == "accuracy" ]]; then REQUIRED_ENVS+=("$AUTOCYCLER_ENV"); fi if is_true "$RUN_PHAROKKA"; then REQUIRED_ENVS+=("$PHAROKKA_ENV"); fi if is_true "$RUN_PHOLD"; then REQUIRED_ENVS+=("$PHOLD_ENV"); fi CONDA_CANDIDATES=() add_conda_candidate() { local c="$1" existing [[ -n "$c" && -f "$c/etc/profile.d/conda.sh" ]] || return 0 for existing in "\${CONDA_CANDIDATES[@]}"; do if [[ "$existing" == "$c" ]]; then return 0; fi done CONDA_CANDIDATES+=("$c") } env_present() { local base="$1" env="$2" if [[ "$env" == /* ]]; then [[ -d "$env/conda-meta" ]] else [[ -d "$base/envs/$env/conda-meta" ]] fi } if [[ -n "$CONDA_BASE_INPUT" ]]; then requested_base="$(strip_trailing_slash "$(expand_user_path "$CONDA_BASE_INPUT")")" [[ -f "$requested_base/etc/profile.d/conda.sh" ]] || fail "Not a Conda base folder: $requested_base" add_conda_candidate "$requested_base" else if [[ -n "\${CONDA_EXE:-}" ]]; then add_conda_candidate "$(dirname "$(dirname "$CONDA_EXE")")" fi for c in "$HOME/miniforge3" "$HOME/mambaforge" "$HOME/miniconda3" "$HOME/miniconda" \\ "$HOME/anaconda3" "/opt/conda" "/opt/miniforge3"; do add_conda_candidate "$c" done fi CONDA_BASE="" for c in "\${CONDA_CANDIDATES[@]}"; do all_found=true for e in "\${REQUIRED_ENVS[@]}"; do if ! env_present "$c" "$e"; then all_found=false; fi done if [[ "$all_found" == "true" ]]; then CONDA_BASE="$c"; break; fi done if [[ -z "$CONDA_BASE" ]]; then echo "❌ No single Conda installation contains all required environments:" >&2 printf ' %s\\n' "\${REQUIRED_ENVS[@]}" >&2 echo " Conda installations checked:" >&2 for c in "\${CONDA_CANDIDATES[@]}"; do missing=() for e in "\${REQUIRED_ENVS[@]}"; do if ! env_present "$c" "$e"; then missing+=("$e"); fi done echo " $c (missing: \${missing[*]})" >&2 done if [[ \${#CONDA_CANDIDATES[@]} -eq 0 ]]; then echo " (none found)" >&2; fi exit 1 fi set +u # shellcheck disable=SC1091 . "$CONDA_BASE/etc/profile.d/conda.sh" set -u echo "Conda: $CONDA_BASE" # ============================================================== # Preflight: check every tool before any long job starts # ============================================================== need() { command -v "$1" >/dev/null 2>&1 || fail "$1 not found in environment $2" } # phold_clean CMD...: run without library/Python paths left over from other environments (e.g. Dorado's # bundled libraries in LD_LIBRARY_PATH can clash with PyTorch's CUDA libraries) phold_clean() { env -u LD_LIBRARY_PATH -u PYTHONPATH -u PYTHONHOME "$@" } use_env "$CORE_ENV" need python3 "$CORE_ENV" need filtlong "$CORE_ENV" need flye "$CORE_ENV" need gzip "$CORE_ENV" declare -A TOOL_VERSION=() first_line() { "$@" 2>&1 | head -n 1 | tr -d '\\r' || true; } TOOL_VERSION[filtlong]="$(first_line filtlong --version | sed 's/^Filtlong v*//')" TOOL_VERSION[flye]="$(first_line flye --version)" echo "Filtlong: $(command -v filtlong)" echo "Flye: $(command -v flye) ($(flye --version 2>/dev/null || echo '?'))" use_env "$MEDAKA_ENV" need medaka_consensus "$MEDAKA_ENV" TOOL_VERSION[medaka]="$(first_line medaka --version | sed 's/^medaka *//')" echo "Medaka: $(command -v medaka_consensus) (\${TOOL_VERSION[medaka]})" AC_USE=() DNAAPLER_OK=false if [[ "$ASSEMBLY_MODE" == "accuracy" ]]; then use_env "$AUTOCYCLER_ENV" need autocycler "$AUTOCYCLER_ENV" TOOL_VERSION[autocycler]="$(first_line autocycler --version | grep -Eo '[0-9]+\\.[0-9]+(\\.[0-9]+)?' | head -n 1 || true)" echo "Autocycler: $(command -v autocycler) (\${TOOL_VERSION[autocycler]})" # Each Autocycler helper runs a separate assembler; keep only the ones that are installed. for asm in $AC_ASSEMBLERS; do case "$asm" in flye) bins=(flye) ;; raven) bins=(raven) ;; miniasm) bins=(miniasm minipolish any2fasta) ;; canu) bins=(canu) ;; redbean) bins=(wtdbg2 wtpoa-cns) ;; metamdbg) bins=(metaMDBG) ;; myloasm) bins=(myloasm) ;; lja) bins=(lja) ;; plassembler|hifiasm|necat|nextdenovo) bins=("$asm") ;; *) warn "Unknown Autocycler assembler '$asm'; skipped."; continue ;; esac missing=() for b in "\${bins[@]}"; do command -v "$b" >/dev/null 2>&1 || missing+=("$b"); done if (( \${#missing[@]} == 0 )); then AC_USE+=("$asm") else warn "Autocycler assembler '$asm' skipped: \${missing[*]} not found in $AUTOCYCLER_ENV." fi done if [[ " $AC_ASSEMBLERS " == *" canu "* && " \${AC_USE[*]} " != *" canu "* && " \${AC_USE[*]} " != *" miniasm "* ]]; then if command -v miniasm >/dev/null 2>&1 && command -v minipolish >/dev/null 2>&1 && command -v any2fasta >/dev/null 2>&1; then AC_USE+=(miniasm) warn "Canu is not installed in $AUTOCYCLER_ENV; using miniasm instead. To add Canu: mamba install -n $AUTOCYCLER_ENV -c conda-forge -c bioconda canu" fi fi (( \${#AC_USE[@]} >= 2 )) || fail "Accuracy mode needs at least two working assemblers in $AUTOCYCLER_ENV (found: \${AC_USE[*]:-none}). Install with: mamba install -n $AUTOCYCLER_ENV -c bioconda flye raven-assembler miniasm minipolish any2fasta" if [[ "$AC_GENOME_SIZE" == "auto" || "$AC_GENOME_SIZE" == "flye" ]]; then command -v raven >/dev/null 2>&1 || fail "Automatic genome size needs Raven in $AUTOCYCLER_ENV (or enter a genome size)." fi echo "Autocycler assemblers: \${AC_USE[*]} × \${AC_COUNT} read subsets = $(( \${#AC_USE[@]} * AC_COUNT )) assemblies per sample$( is_true "$AC_FLYE_VOTE" && echo " + Flye votes at \${AC_FLYE_TIERS// /x, }x of the best reads (weight $AC_FLYE_WEIGHT each)")" echo "Autocycler genome size: $( case "$AC_GENOME_SIZE" in flye) echo "from a first Flye assembly of the best reads (up to $FLYE_FIRST_MB Mb; high-depth contigs)";; auto) echo "Autocycler's estimate from the reads";; *) echo "$AC_GENOME_SIZE";; esac )" TOOL_VERSION[autocycler_assemblers]="\${AC_USE[*]}" if [[ "$ROTATE" == "ends_terl" || "$ROTATE" == "terl" ]]; then if [[ -n "$DNAAPLER_ENV" ]] && env_present "$CONDA_BASE" "$DNAAPLER_ENV"; then use_env "$DNAAPLER_ENV" if command -v dnaapler >/dev/null 2>&1; then DNAAPLER_OK=true TOOL_VERSION[dnaapler]="$(first_line dnaapler --version | grep -Eo '[0-9]+\\.[0-9]+(\\.[0-9]+)?' | head -n 1 || true)" echo "dnaapler: $(command -v dnaapler) (\${TOOL_VERSION[dnaapler]})" fi fi if [[ "$DNAAPLER_OK" != "true" ]]; then warn "dnaapler was not found (environment '$DNAAPLER_ENV'); circular genomes without clear read ends will not be rotated to terL." warn "To add it: mamba create -n dnaapler -c conda-forge -c bioconda dnaapler" fi fi fi # NanoPlot / NanoComp are optional: a missing tool turns that report off instead of stopping the run. use_env "$CORE_ENV" NANOPLOT_EXTRA=() NANOCOMP_EXTRA=() NANOPLOT_TIME_LIMIT="\${NANOPLOT_TIME_LIMIT:-1800}" # seconds per sample; a stuck report is stopped and the sample continues NANOCOMP_TIME_LIMIT="\${NANOCOMP_TIME_LIMIT:-3600}" if is_true "$RUN_NANOPLOT"; then if command -v NanoPlot >/dev/null 2>&1; then TOOL_VERSION[nanoplot]="$(first_line NanoPlot --version | grep -Eo '[0-9]+\\.[0-9]+(\\.[0-9]+)?' | head -n 1 || true)" echo "NanoPlot: $(command -v NanoPlot) (\${TOOL_VERSION[nanoplot]})" # Static PNG export goes through Plotly's "kaleido", which can hang forever (often on WSL). # The interactive HTML report and NanoStats.txt do not need it. if NanoPlot --help 2>&1 | grep -q -- '--no_static'; then NANOPLOT_EXTRA+=(--no_static); fi else warn "NanoPlot was not found in $CORE_ENV; per-sample NanoPlot reports are turned off." RUN_NANOPLOT=false fi fi if is_true "$RUN_NANOCOMP"; then if command -v NanoComp >/dev/null 2>&1; then TOOL_VERSION[nanocomp]="$(first_line NanoComp --version | grep -Eo '[0-9]+\\.[0-9]+(\\.[0-9]+)?' | head -n 1 || true)" echo "NanoComp: $(command -v NanoComp) (\${TOOL_VERSION[nanocomp]})" if NanoComp --help 2>&1 | grep -q -- '--no_static'; then NANOCOMP_EXTRA+=(--no_static); fi else warn "NanoComp was not found in $CORE_ENV; the run-level NanoComp report is turned off." RUN_NANOCOMP=false fi fi COMPARE_ENV="" if is_true "$COMPARE"; then use_env "$MEDAKA_ENV" if command -v minimap2 >/dev/null 2>&1; then COMPARE_ENV="$MEDAKA_ENV" else use_env "$CORE_ENV" if command -v minimap2 >/dev/null 2>&1; then COMPARE_ENV="$CORE_ENV"; fi fi if [[ -z "$COMPARE_ENV" ]]; then warn "minimap2 was not found in $MEDAKA_ENV or $CORE_ENV; the Flye vs Medaka comparison is disabled." COMPARE=false else echo "minimap2: found in $COMPARE_ENV" fi fi COVERAGE_ENV="" if is_true "$RUN_COVERAGE"; then for e in "$MEDAKA_ENV" "$CORE_ENV"; do use_env "$e" if command -v minimap2 >/dev/null 2>&1 && command -v samtools >/dev/null 2>&1; then COVERAGE_ENV="$e" break fi done if [[ -z "$COVERAGE_ENV" ]]; then warn "minimap2 + samtools were not found together in $MEDAKA_ENV or $CORE_ENV; the coverage report is disabled." RUN_COVERAGE=false else TOOL_VERSION[samtools]="$(first_line samtools --version | sed 's/^samtools *//')" TOOL_VERSION[minimap2]="$(first_line minimap2 --version)" echo "samtools: found in $COVERAGE_ENV (\${TOOL_VERSION[samtools]})" fi fi # Genome-end detection maps reads to the longest contig (minimap2 + samtools). ENDS_ENV="" if is_true "$FIND_ENDS"; then for e in "$MEDAKA_ENV" "$CORE_ENV" "\${AUTOCYCLER_ENV:-}"; do [[ -n "$e" ]] && env_present "$CONDA_BASE" "$e" || continue use_env "$e" if command -v minimap2 >/dev/null 2>&1 && command -v samtools >/dev/null 2>&1; then ENDS_ENV="$e"; break; fi done if [[ -z "$ENDS_ENV" ]]; then warn "minimap2 + samtools were not found together; genome-end detection is turned off." FIND_ENDS=false fi fi # Host-read removal for every sample: only with host genome(s); handled as a picked-plaque list that matches all. if [[ "$HOST_REMOVAL" == "all" ]]; then if [[ "$HOST_METHOD" == "ref" && -n "$HOST_GENOME_1$HOST_GENOME_2$HOST_GENOME_3" ]]; then HOST_REMOVAL="plaque"; PLAQUE_SAMPLES="__ALL__"; HOST_ALL=true else warn "Host-read removal for all samples needs the host genome method and a host genome; it is turned off." HOST_REMOVAL="off" fi fi HOST_ALL="\${HOST_ALL:-false}" # Host-read removal (picked-plaque samples): minimap2 against the draft phage contigs or the host genome. HOST_ENV="" HOST_INDEX="" HOST_LABELS="-" HOST_LIST="" if [[ "$HOST_REMOVAL" == "plaque" && -n "$PLAQUE_SAMPLES" ]]; then for e in "$CORE_ENV" "$MEDAKA_ENV" "\${AUTOCYCLER_ENV:-}"; do [[ -n "$e" ]] && env_present "$CONDA_BASE" "$e" || continue use_env "$e" if command -v minimap2 >/dev/null 2>&1; then HOST_ENV="$e"; break; fi done if [[ -z "$HOST_ENV" ]]; then warn "minimap2 was not found in $CORE_ENV or $MEDAKA_ENV; host-read removal is turned off (all reads are used)." HOST_REMOVAL="off" else TOOL_VERSION[minimap2]="$(first_line minimap2 --version)" if is_true "$HOST_ALL"; then echo "Host-read removal: on for all samples (host genome); minimap2 from $HOST_ENV" else echo "Host-read removal: on for picked-plaque samples ($PLAQUE_SAMPLES); minimap2 from $HOST_ENV"; fi if [[ "$HOST_METHOD" == "ref" ]]; then # All listed hosts go into one reference; each sequence name is prefixed H__ so matches can be # counted per host. A sample's reads are mapped to every host at once. mkdir -p "$TOOLS_DIR" HOST_FASTA="$TOOLS_DIR/host_genomes.fasta" HOST_LABELS="$TOOLS_DIR/host_genomes.tsv" HOST_INDEX="$TOOLS_DIR/host_genomes.map-ont.mmi" : > "$HOST_FASTA" printf 'tag\\tname\\tprophage\\tfasta\\n' > "$HOST_LABELS" HOST_LIST="" for i in 1 2 3; do hg_var="HOST_GENOME_$i"; hn_var="HOST_NAME_$i"; hp_var="HOST_PROPHAGE_$i" hg="\${!hg_var:-}"; hn="\${!hn_var:-host$i}"; hp="\${!hp_var:-unsure}" [[ -n "$hg" ]] || continue [[ -s "$hg" ]] || fail "Host $i ($hn) genome FASTA not found: $hg" gzip -cdf -- "$hg" | sed "s/^>/>H\${i}__/" >> "$HOST_FASTA" printf 'H%s\\t%s\\t%s\\t%s\\n' "$i" "$hn" "$hp" "$hg" >> "$HOST_LABELS" HOST_LIST+="\${HOST_LIST:+; }$hn" echo " host $i: $hn ($( [[ "$hp" == "no" ]] && echo "no prophage" || echo "prophage possible" )) — $hg" done [[ -s "$HOST_FASTA" ]] || fail "No host genome sequences were read." echo "Indexing the host genome(s) for minimap2" minimap2 -x map-ont -t "$FLYE_THREADS" -d "$HOST_INDEX" "$HOST_FASTA" > "$TOOLS_DIR/host_index.log" 2>&1 \\ || fail "minimap2 could not index the host genome(s) (see $TOOLS_DIR/host_index.log)" rm -f -- "$HOST_FASTA" if [[ "$HOST_PROPHAGE" == "no" ]]; then echo " method: discard reads matching any host genome (no host has a prophage)" else echo " method: discard reads matching any host genome, except reads that also match the phage draft (possible prophage)" fi else echo " method: no host genome; keep reads matching the draft phage contigs" fi fi else HOST_REMOVAL="off" fi PHAROKKA_RUN=() PHAROKKA_INSTALL=() if is_true "$RUN_PHAROKKA"; then use_env "$PHAROKKA_ENV" if command -v pharokka >/dev/null 2>&1 && pharokka run --help >/dev/null 2>&1; then PHAROKKA_RUN=(pharokka run) PHAROKKA_INSTALL=(pharokka install -o) elif command -v pharokka.py >/dev/null 2>&1; then PHAROKKA_RUN=(pharokka.py) PHAROKKA_INSTALL=(install_databases.py -o) else fail "Pharokka was not found in environment $PHAROKKA_ENV" fi TOOL_VERSION[pharokka]="$(first_line "\${PHAROKKA_RUN[0]}" --version | grep -Eo '[0-9]+\\.[0-9]+(\\.[0-9]+)?' | head -n 1 || true)" echo "Pharokka: \${PHAROKKA_RUN[*]} (\${TOOL_VERSION[pharokka]})" if [[ ! -d "$PHAROKKA_DB" ]] || [[ -z "$(find "$PHAROKKA_DB" -mindepth 1 -maxdepth 1 -print -quit 2>/dev/null)" ]]; then if is_true "$INSTALL_DBS"; then echo "📚 Installing Pharokka databases into $PHAROKKA_DB" mkdir -p "$PHAROKKA_DB" "\${PHAROKKA_INSTALL[@]}" "$PHAROKKA_DB" else fail "Pharokka database not found: $PHAROKKA_DB (tick 'Install missing databases' or fix the path)" fi fi echo "Pharokka database: $PHAROKKA_DB" fi PHOLD_ARGS=() if is_true "$RUN_PHOLD"; then use_env "$PHOLD_ENV" need phold "$PHOLD_ENV" need foldseek "$PHOLD_ENV" TOOL_VERSION[phold]="$(first_line phold --version | grep -Eo '[0-9]+\\.[0-9]+(\\.[0-9]+)?' | head -n 1 || true)" echo "Phold: $(command -v phold) (\${TOOL_VERSION[phold]})" # Test with the Python that phold itself runs with (its #! line), not whichever python3 is first on PATH. PHOLD_PY="$(head -n 1 "$(command -v phold)" 2>/dev/null | sed -n 's/^#![[:space:]]*//p' | awk '{print $1}')" if [[ "$PHOLD_PY" == */env ]]; then PHOLD_PY="$(head -n 1 "$(command -v phold)" | awk '{print $2}')"; PHOLD_PY="$(command -v "$PHOLD_PY" || true)"; fi [[ -n "$PHOLD_PY" && -x "$PHOLD_PY" ]] || PHOLD_PY="\${CONDA_PREFIX:-}/bin/python" [[ -x "$PHOLD_PY" ]] || PHOLD_PY="$(command -v python3)" echo "Phold Python: $PHOLD_PY" if [[ -n "\${LD_LIBRARY_PATH:-}\${PYTHONPATH:-}\${PYTHONHOME:-}" ]]; then echo " (Phold runs without LD_LIBRARY_PATH/PYTHONPATH/PYTHONHOME set by other environments: \${LD_LIBRARY_PATH:-}\${PYTHONPATH:+ PYTHONPATH=$PYTHONPATH}\${PYTHONHOME:+ PYTHONHOME=$PYTHONHOME})" fi if ! TORCH_INFO="$(phold_clean "$PHOLD_PY" -c 'import torch; print(torch.__version__)' 2>&1)"; then warn "PyTorch is missing or broken in the $PHOLD_ENV environment, so Phold cannot run. Phold is turned off;" warn "assembly, polishing and Pharokka still run. To fix: conda activate $PHOLD_ENV then pip install torch --index-url https://download.pytorch.org/whl/cu128 (CUDA 12.8 build; needed for RTX 50-series GPUs)" printf '%s\\n' "$TORCH_INFO" | tail -n 1 | sed 's/^/ /' >&2 echo " (checked with $PHOLD_PY; test it yourself: $PHOLD_PY -c 'import torch')" >&2 echo " Details for troubleshooting (working folder: $PWD):" >&2 phold_clean "$PHOLD_PY" -c 'import sys; print(" executable:", sys.executable); print(" search path:", sys.path)' >&2 2>&1 || true RUN_PHOLD=false fi fi if is_true "$RUN_PHOLD"; then if [[ ! -s "$PHOLD_DB/acrs_plddt_over_70_metadata.tsv" ]]; then if is_true "$INSTALL_DBS"; then echo "📚 Installing Phold database into $PHOLD_DB" mkdir -p "$PHOLD_DB" phold_clean phold install -d "$PHOLD_DB" -t "$PHOLD_THREADS" else fail "Phold database not found or incomplete: $PHOLD_DB (tick 'Install missing databases' or fix the path)" fi fi echo "Phold database: $PHOLD_DB" PHOLD_ARGS=(-d "$PHOLD_DB" -t "$PHOLD_THREADS" --batch_size "$PHOLD_BATCH" -f) if [[ "$PHOLD_COMPUTE" == "gpu" ]]; then # Run a real calculation on the GPU: a PyTorch build without kernels for the card # (e.g. an older build on an RTX 50-series GPU) reports CUDA as available but then fails. GPU_TEST='import torch assert torch.cuda.is_available(), "PyTorch was built without CUDA or cannot see the GPU" x = torch.ones(256, 256, device="cuda") assert float((x @ x).sum()) == 256.0 ** 3 print(torch.cuda.get_device_name(0), "| PyTorch", torch.__version__, "| CUDA", torch.version.cuda)' if GPU_INFO="$(phold_clean "$PHOLD_PY" -c "$GPU_TEST" 2>&1)"; then echo "Phold GPU: $(printf '%s' "$GPU_INFO" | tail -n 1)" else warn "Phold cannot use the GPU, so it will run on the CPU (slower). Reason:" printf '%s\\n' "$GPU_INFO" | tail -n 2 | sed 's/^/ /' >&2 PHOLD_ARGS+=(--cpu) fi else PHOLD_ARGS+=(--cpu) fi fi use_env "$CORE_ENV" echo "✅ Preflight checks passed." # ============================================================== # Helper Python tools (standard library only) # ============================================================== TOOLS="$TOOLS_DIR/phage_tools.py" cat > "$TOOLS" <<'PHAGE_TOOLS_PY' """Helper tools for the Nanopore phage pipeline (standard library only). Subcommands: dedup IN.fastq[.gz] OUT.fastq.gz REPORT.txt stats ASSEMBLY.fasta -> prints total length (bp) compare FLYE.fasta MEDAKA.fasta ALN.paf PREFIX finalize MEDAKA.fasta OUT.fasta SAMPLE RENAME(true|false) longest FINAL.fasta OUT.fasta OUT.tsv SAMPLE FLYE_ASSEMBLY_INFO gbkstats FILE.gbk -> prints "CDShypothetical" pileupstats FINAL.fasta DEPTH.txt PILEUP.txt[.gz] OUT.tsv.gz report FINAL.fasta COVERAGE.tsv.gz SAMPLE OUT.html STATS.txt CHANGES.tsv|- GBK|- LONGEST.tsv|- AGREE_PCT LOWCOV_X READS_LABEL readstats FASTQ OUT.json bundle OUT_ROOT SAMPLES.tsv META.tsv OUT.json sheetmatch SHEET.csv INPUT_ROOT FILES.nul OUT.tsv acinfo CONSENSUS.fasta CONSENSUS.gfa|- OUT_INFO.txt (Autocycler contigs in Flye assembly_info format) readends CONTIG.fasta SAM|- OUT.txt (read start/stop pile-ups that mark physical genome ends) rotate CONTIG.fasta POSITION OUT.fasta (circular contig: start the sequence at POSITION, 1-based) shiftchanges CHANGES.tsv CONTIG LENGTH POSITION OUT.tsv (changes table moved onto a rotated contig) relabel IN.fasta NAME OUT.fasta (first record renamed NAME, description kept) replacecontig FINAL.fasta CONTIG_NAME NEW.fasta (replace one contig of FINAL with the sequence in NEW) codingqc GBK GENOME.fasta OUT.txt (coding density / short-CDS check for leftover indel errors) sampleflags FINAL.fasta LONGEST.fasta ANNOTATION_QC.txt|- AUTOCYCLER_DIR|- OUT.txt """ import gzip import json import math import os import re import sys # ---------------------------------------------------------------- utilities def open_text(path, mode="rt"): if path.lower().endswith(".gz"): return gzip.open(path, mode) return open(path, mode) def read_fasta(path): records = [] name, desc, chunks = None, "", [] with open_text(path) as fh: for line in fh: line = line.rstrip("\\r\\n") if line.startswith(">"): if name is not None: records.append((name, desc, "".join(chunks))) parts = line[1:].strip().split(None, 1) name = parts[0] if parts else "contig" desc = parts[1] if len(parts) > 1 else "" chunks = [] elif line.strip(): chunks.append(line.strip()) if name is not None: records.append((name, desc, "".join(chunks))) return records def write_fasta(fh, name, seq, desc="", width=80): fh.write(">" + name + ((" " + desc) if desc else "") + "\\n") for i in range(0, len(seq), width): fh.write(seq[i:i + width] + "\\n") def atomic_path(path): return path + ".tmp" # --------------------------------------------------------------------- dedup def cmd_dedup(input_path, output_path, report_path): seen = set() duplicate_ids = [] total = 0 removed = 0 tmp = atomic_path(output_path) with open_text(input_path) as fin, gzip.open(tmp, "wt") as fout: while True: header = fin.readline() if not header: break if not header.strip(): continue sequence = fin.readline() plus = fin.readline() quality = fin.readline() if not sequence or not plus or not quality: raise RuntimeError("Incomplete FASTQ record at end of file") if not header.startswith("@"): raise RuntimeError(f"Invalid FASTQ header: {header.rstrip()}") if not plus.startswith("+"): raise RuntimeError(f"Invalid FASTQ plus line for {header.rstrip()}") read_id = header[1:].split()[0] total += 1 if read_id in seen: removed += 1 if len(duplicate_ids) < 25: duplicate_ids.append(read_id) continue seen.add(read_id) fout.write(header) fout.write(sequence) fout.write(plus) fout.write(quality) os.replace(tmp, output_path) with open(report_path, "wt") as report: report.write(f"Input FASTQ: {input_path}\\n") report.write(f"Total records: {total}\\n") report.write(f"Unique read IDs: {len(seen)}\\n") report.write(f"Duplicate records removed: {removed}\\n") if duplicate_ids: report.write("\\nFirst duplicate read IDs encountered:\\n") for read_id in duplicate_ids: report.write(read_id + "\\n") print(f"FASTQ records: {total}") print(f"Unique read IDs: {len(seen)}") print(f"Duplicates removed: {removed}") # --------------------------------------------------------------------- stats def cmd_stats(fasta): print(sum(len(seq) for _, _, seq in read_fasta(fasta))) # ------------------------------------------------------------------- compare CS_TOKEN = re.compile(r":\\d+|\\*[a-z]{2}|[+\\-][a-z]+|~[a-z]{2}\\d+[a-z]{2}|=[A-Za-z]+") def homopolymer_info(ref, start, bases, kind): """Return (is_homopolymer, flye_run_length) for an indel.""" if not bases or len(set(bases)) != 1: return False, 0 base = bases[0] # Deletion removes ref[start:start+len]; insertion sits between ref[start-1] and ref[start]. lo = start hi = start + len(bases) if kind == "deletion" else start while lo > 0 and ref[lo - 1] == base: lo -= 1 while hi < len(ref) and ref[hi] == base: hi += 1 run = hi - lo # length of the Flye homopolymer run that the event touches if kind == "deletion": return run > len(bases), run return run >= 1, run def context(ref, start, end, shown): left = ref[max(0, start - 10):start].lower() right = ref[end:end + 10].lower() return f"{left}[{shown}]{right}" def cmd_compare(flye_fasta, medaka_fasta, paf_path, prefix): flye = {n: s.upper() for n, _, s in read_fasta(flye_fasta)} medaka_records = read_fasta(medaka_fasta) medaka = {n: s.upper() for n, _, s in medaka_records} alignments = {} with open(paf_path) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) < 12: continue tags = dict((t[:2], t[5:]) for t in f[12:] if len(t) > 5) if tags.get("tp", "P") != "P" or "cs" not in tags: continue alignments.setdefault(f[0], []).append({ "qname": f[0], "qlen": int(f[1]), "qs": int(f[2]), "qe": int(f[3]), "strand": f[4], "tname": f[5], "tlen": int(f[6]), "ts": int(f[7]), "te": int(f[8]), "cs": tags["cs"], }) changes = [] summary = [] seen_targets = set() totals = dict(changes=0, substitutions=0, insertions=0, deletions=0, homopolymer=0) for qname, _, _ in medaka_records: qseq = medaka[qname] alns = sorted(alignments.get(qname, []), key=lambda a: a["ts"]) if not alns: summary.append([qname, flye_name_guess(qname, flye), str(len(flye.get(qname, ""))) or "NA", str(len(qseq)), "NA", "0"] + ["NA"] * 11 + ["unaligned"]) continue stats = dict(matches=0, subs=0, ins_ev=0, ins_bp=0, del_ev=0, del_bp=0, hp=0, t_aln=0) tnames = [] unal_t = unal_q = 0 for a in alns: tname = a["tname"] ref = flye.get(tname, "") seen_targets.add(tname) if tname not in tnames: tnames.append(tname) stats["t_aln"] += a["te"] - a["ts"] t = a["ts"] q_off = 0 def qpos(off): return a["qs"] + off + 1 if a["strand"] == "+" else a["qe"] - off for m in CS_TOKEN.finditer(a["cs"]): tok = m.group(0) op = tok[0] if op == ":": n = int(tok[1:]) t += n q_off += n stats["matches"] += n elif op == "=": n = len(tok) - 1 t += n q_off += n stats["matches"] += n elif op == "*": r, alt = tok[1].upper(), tok[2].upper() stats["subs"] += 1 changes.append([tname, t + 1, qname, qpos(q_off), "substitution", r, alt, 1, "no", "", context(ref, t, t + 1, f"{r}>{alt}")]) t += 1 q_off += 1 elif op == "+": ins = tok[1:].upper() is_hp, run = homopolymer_info(ref, t, ins, "insertion") stats["ins_ev"] += 1 stats["ins_bp"] += len(ins) stats["hp"] += int(is_hp) changes.append([tname, t, qname, qpos(q_off), "insertion", "-", ins, len(ins), "yes" if is_hp else "no", run if is_hp else "", context(ref, t, t, "+" + ins)]) q_off += len(ins) elif op == "-": dele = tok[1:].upper() is_hp, run = homopolymer_info(ref, t, dele, "deletion") stats["del_ev"] += 1 stats["del_bp"] += len(dele) stats["hp"] += int(is_hp) changes.append([tname, t + 1, qname, qpos(q_off), "deletion", dele, "-", len(dele), "yes" if is_hp else "no", run if is_hp else "", context(ref, t, t + len(dele), "-" + dele)]) t += len(dele) unal_t += a["ts"] + (a["tlen"] - a["te"]) unal_q += a["qs"] + (a["qlen"] - a["qe"]) n_changes = stats["subs"] + stats["ins_ev"] + stats["del_ev"] columns = stats["matches"] + stats["subs"] + stats["ins_bp"] + stats["del_bp"] identity = 100.0 * stats["matches"] / columns if columns else 0.0 flye_len = sum(len(flye.get(t, "")) for t in tnames) per10kb = 10000.0 * n_changes / stats["t_aln"] if stats["t_aln"] else 0.0 status = "ok" if len(alns) == 1 else f"split_into_{len(alns)}_alignments" summary.append([ qname, ",".join(tnames), str(flye_len), str(len(qseq)), str(len(qseq) - flye_len), str(len(alns)), str(stats["t_aln"]), f"{identity:.4f}", str(stats["subs"]), str(stats["ins_ev"]), str(stats["ins_bp"]), str(stats["del_ev"]), str(stats["del_bp"]), str(stats["hp"]), str(n_changes), f"{per10kb:.2f}", f"{unal_t}/{unal_q}", status, ]) totals["changes"] += n_changes totals["substitutions"] += stats["subs"] totals["insertions"] += stats["ins_ev"] totals["deletions"] += stats["del_ev"] totals["homopolymer"] += stats["hp"] for tname, tseq in flye.items(): if tname not in seen_targets and tname not in medaka: summary.append([tname + " (Flye only)", tname, str(len(tseq)), "0", str(-len(tseq)), "0"] + ["NA"] * 11 + ["missing_in_medaka"]) header = ["medaka_contig", "flye_contig", "flye_length", "medaka_length", "length_change", "alignments", "flye_bp_aligned", "identity_pct", "substitutions", "insertion_events", "inserted_bp", "deletion_events", "deleted_bp", "homopolymer_indels", "total_changes", "changes_per_10kb", "unaligned_ends_flye/medaka", "status"] with open(prefix + ".summary.tsv", "w") as out: out.write("\\t".join(header) + "\\n") for row in summary: out.write("\\t".join(row) + "\\n") with open(prefix + ".changes.tsv", "w") as out: out.write("\\t".join(["flye_contig", "flye_pos", "medaka_contig", "medaka_pos", "type", "flye_bases", "medaka_bases", "length", "homopolymer", "flye_homopolymer_run", "flye_context"]) + "\\n") for row in changes: out.write("\\t".join(str(x) for x in row) + "\\n") with open(prefix + ".totals.txt", "w") as out: for k, v in totals.items(): out.write(f"{k}\\t{v}\\n") # Console report print("Flye → Medaka differences") print(f" {'contig':<22}{'flye_bp':>10}{'medaka_bp':>11}{'subs':>7}{'ins':>6}{'del':>6}{'HP-indel':>10}{'identity%':>11}") for row in summary: print(f" {row[0][:21]:<22}{row[2]:>10}{row[3]:>11}{row[8]:>7}{row[9]:>6}{row[11]:>6}{row[13]:>10}{row[7]:>11}") print(f" Total changes: {totals['changes']} " f"({totals['substitutions']} substitutions, {totals['insertions']} insertions, " f"{totals['deletions']} deletions; {totals['homopolymer']} indels in homopolymers)") def flye_name_guess(qname, flye): return qname if qname in flye else "NA" # ------------------------------------------------------------------ finalize def cmd_finalize(medaka_fasta, out_fasta, sample, rename): rename = rename.lower() == "true" records = read_fasta(medaka_fasta) if not records: raise RuntimeError(f"No sequences found in {medaka_fasta}") tmp = atomic_path(out_fasta) with open(tmp, "w") as out: for name, desc, seq in records: new = name if rename and not name.startswith(sample + "_"): new = f"{sample}_{name}" write_fasta(out, new, seq, desc) os.replace(tmp, out_fasta) print(f"Polished FASTA: {out_fasta} ({len(records)} contig(s))") # ------------------------------------------------------------------- longest def read_flye_info(path): info = {} if not path or not os.path.isfile(path): return info with open(path) as fh: for line in fh: if line.startswith("#") or not line.strip(): continue f = line.rstrip("\\n").split("\\t") if len(f) >= 4: info[f[0]] = {"coverage": f[2], "circular": f[3]} return info def cmd_longest(final_fasta, out_fasta, out_tsv, sample, flye_info_path): records = read_fasta(final_fasta) if not records: raise RuntimeError(f"No sequences found in {final_fasta}") name, desc, seq = max(records, key=lambda r: len(r[2])) total = sum(len(r[2]) for r in records) original = name[len(sample) + 1:] if name.startswith(sample + "_") else name info = read_flye_info(flye_info_path).get(original, {}) tmp = atomic_path(out_fasta) with open(tmp, "w") as out: write_fasta(out, sample, seq, f"source_contig={name} length={len(seq)}") os.replace(tmp, out_fasta) values = [ ("sample", sample), ("longest_contig", name), ("longest_length", len(seq)), ("n_contigs", len(records)), ("total_length", total), ("longest_fraction", f"{len(seq) / total:.3f}"), ("flye_coverage", info.get("coverage", "NA")), ("flye_circular", info.get("circular", "NA")), ] with open(out_tsv, "w") as out: for k, v in values: out.write(f"{k}\\t{v}\\n") print(f"Longest contig: {name} ({len(seq):,} bp; assembler depth {info.get('coverage', 'NA')}x; " f"circular {info.get('circular', 'NA')})") if len(records) > 1: others = ", ".join(f"{r[0]} ({len(r[2]):,} bp)" for r in records if r[0] != name) print(f"⚠️ {len(records)} contigs assembled. Others: {others}") if info.get("circular") == "N": print("ℹ️ The assembler reported the longest contig as linear; check for terminal repeats or an incomplete genome.") # ------------------------------------------------------------------ annotpick (v16) def _ctg_num(name, fallback): m = re.search(r"(\\d+)$", name) return m.group(1) if m else str(fallback) def cmd_annotpick(final_fasta, info_path, sample, longest_tsv, out_fasta, out_tsv, large_fasta, min_len, max_n): """Which contigs to annotate besides the longest one (annotated as before): a circular contig of min_len or more -> only circular contigs (that is the genome); no circular contig -> every contig of min_len or more (at most max_n, largest first). Writes the extra contigs (short ids for Pharokka) and all large contigs (for the overlap check).""" min_len, max_n = int(min_len), int(max_n) records = read_fasta(final_fasta) info = read_flye_info(info_path if info_path != "-" else "") longest = read_kv(longest_tsv).get("longest_contig") if longest_tsv != "-" else None longest = longest or (max(records, key=lambda r: len(r[2]))[0] if records else None) def circular(name): orig = name[len(sample) + 1:] if name.startswith(sample + "_") else name return (info.get(orig) or info.get(name) or {}).get("circular") == "Y" big = sorted([r for r in records if len(r[2]) >= min_len], key=lambda r: -len(r[2])) circ = [r for r in big if circular(r[0])] if circ: chosen, rule = circ[:max_n], f"circular contig(s) of {min_len // 1000} kb or more: only those (the genome)" elif big: chosen, rule = big[:max_n], f"no circular contig: every contig of {min_len // 1000} kb or more (at most {max_n})" else: chosen, rule = [], f"no contig of {min_len // 1000} kb or more" chosen_names = {r[0] for r in chosen} extras = [r for r in chosen if r[0] != longest] text = "".join(f">ctg{_ctg_num(r[0], i)} source_contig={r[0]}\\n{r[2]}\\n" for i, r in enumerate(extras, 1)) old = open(out_fasta).read() if os.path.isfile(out_fasta) else None if not extras: if os.path.exists(out_fasta): os.remove(out_fasta) elif text != old: # unchanged -> keep the file date (no re-annotation) tmp = atomic_path(out_fasta) with open(tmp, "w") as out: out.write(text) os.replace(tmp, out_fasta) tmp = atomic_path(large_fasta) with open(tmp, "w") as out: for r in big: out.write(f">{r[0]}\\n{r[2]}\\n") os.replace(tmp, large_fasta) with open(out_tsv, "w") as out: out.write("short_id\\tcontig\\tlength\\tcircular\\tannotation\\n") for i, r in enumerate(big, 1): role = "main (longest)" if r[0] == longest else ("extra" if r[0] in chosen_names else "not annotated") sid = f"ctg{_ctg_num(r[0], i)}" if role == "extra" else "" out.write(f"{sid}\\t{r[0]}\\t{len(r[2])}\\t{'Y' if circular(r[0]) else 'N'}\\t{role}\\n") out.write(f"#rule\\t{rule}\\n") if extras: print(f"Also annotating {len(extras)} other contig(s) ({rule}): " + ", ".join(f"{r[0]} ({len(r[2]):,} bp)" for r in extras)) # ------------------------------------------------------------------ gbkstats def cmd_gbkstats(gbk): cds = hyp = 0 with open(gbk) as fh: for line in fh: if re.match(r"^\\s{5}CDS\\s", line): cds += 1 elif '/product="hypothetical protein"' in line: hyp += 1 print(f"{cds}\\t{hyp}") # --------------------------------------------------------------- coverage INDEL_RE = re.compile(r"[+-](\\d+)") START_RE = re.compile(r"\\^.") def pileup_counts(bases): """Return (match, mismatch, deletion, insertion_events) from a samtools pileup column.""" s = START_RE.sub("", bases).replace("$", "") ins = 0 if "+" in s or "-" in s: pieces = [] i = 0 for m in INDEL_RE.finditer(s): if m.start() < i: continue pieces.append(s[i:m.start()]) if s[m.start()] == "+": ins += 1 i = m.end() + int(m.group(1)) pieces.append(s[i:]) s = "".join(pieces) match = s.count(".") + s.count(",") dele = s.count("*") + s.count("#") skip = s.count("<") + s.count(">") mismatch = len(s) - match - dele - skip return match, mismatch, dele, ins def cmd_pileupstats(final_fasta, depth_path, pileup_path, out_tsv): contigs = read_fasta(final_fasta) depth = {} with open_text(depth_path) as fh: for line in fh: f = line.split("\\t") if len(f) >= 3: depth[(f[0], int(f[1]))] = int(f[2]) counts = {} with open_text(pileup_path) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 5 and f[3] != "0": counts[(f[0], int(f[1]))] = pileup_counts(f[4]) tmp = atomic_path(out_tsv) with gzip.open(tmp, "wt") as out: out.write("contig\\tpos\\tconsensus_base\\tdepth\\tagreement_pct\\tmismatch_pct\\tdeletion_pct\\tinsertion_pct\\n") for name, _, seq in contigs: for i, base in enumerate(seq, start=1): d = depth.get((name, i), 0) m, mm, de, ins = counts.get((name, i), (0, 0, 0, 0)) tot = m + mm + de if tot: out.write(f"{name}\\t{i}\\t{base}\\t{d}\\t{100 * m / tot:.1f}\\t{100 * mm / tot:.1f}" f"\\t{100 * de / tot:.1f}\\t{100 * ins / tot:.1f}\\n") else: out.write(f"{name}\\t{i}\\t{base}\\t{d}\\tNA\\tNA\\tNA\\tNA\\n") os.replace(tmp, out_tsv) print(f"Per-base coverage table: {out_tsv}") def parse_genbank_features(path): features = [] current = None last_key = None in_features = False with open(path) as fh: for raw in fh: line = raw.rstrip("\\n") if line.startswith("FEATURES"): in_features = True continue if line.startswith("ORIGIN") or line.startswith("//") or line.startswith("LOCUS"): in_features = False if current: features.append(current) current = None continue if in_features is False or len(line) < 22: continue if line[:5] == " " and line[5] != " ": if current: features.append(current) current = {"type": line[5:21].strip(), "loc": line[21:].strip(), "q": {}} last_key = None elif current is not None and line[:21].strip() == "": content = line[21:].strip() if content.startswith("/"): key, _, value = content[1:].partition("=") current["q"][key] = value last_key = key elif last_key: current["q"][last_key] += " " + content else: current["loc"] += content if current: features.append(current) genes = [] for f in features: if f["type"] not in ("CDS", "tRNA", "tmRNA", "repeat_region"): continue nums = [int(x) for x in re.findall(r"\\d+", f["loc"])] if not nums: continue strand = -1 if "complement" in f["loc"] else 1 q = {k: v.strip().strip('"') for k, v in f["q"].items()} genes.append({ "s": min(nums), "e": max(nums), "st": strand, "t": f["type"], "p": q.get("product", f["type"]), "f": q.get("function", "unknown function" if f["type"] == "CDS" else f["type"]), "id": q.get("locus_tag", q.get("ID", "")), }) return genes def median(values): v = sorted(values) if not v: return 0 n = len(v) return v[n // 2] if n % 2 else (v[n // 2 - 1] + v[n // 2]) / 2 def _closest_report(qc, genes, L): """Closest PhagesDB phage + genome comparison for coverage.html (None if there is no search result).""" import urllib.parse tsv = os.path.join(qc, "closest_phagesdb.tsv") if not os.path.isfile(tsv): return None with open(tsv) as fh: cols = next(fh, "").rstrip("\\n").split("\\t") rows = [dict(zip(cols, line.rstrip("\\n").split("\\t"))) for line in fh if line.strip()] if not rows: return {"none": True} top = rows[0] meta = read_kv(os.path.join(qc, "closest_top.tsv")) rs = read_kv(os.path.join(qc, "reference_start.txt")) def url(name): return "https://phagesdb.org/phages/" + urllib.parse.quote(str(name)) + "/" blocks, qlen, tlen = [], L, int(top.get("length_bp") or 0) paf = os.path.join(qc, "closest_alignment.paf") if os.path.isfile(paf): with open(paf) as fh: for line in fh: f = line.split("\\t") if len(f) < 12: continue qlen, tlen = int(f[1]), int(f[6]) blk = int(f[10]) blocks.append([int(f[2]), int(f[3]), int(f[7]), int(f[8]), f[4], round(100.0 * int(f[9]) / blk, 1) if blk else 0]) blocks.sort(key=lambda b: -(b[1] - b[0])) blocks = blocks[:400] def gaps(iv, total): out, pos = [], 0 for s, e in _merge(iv): if s - pos >= 200: out.append([pos, s]) pos = max(pos, e) if total - pos >= 200: out.append([pos, total]) return out uniq_this = [] for s, e in gaps([(b[0], b[1]) for b in blocks], qlen)[:100]: hit = [g for g in genes if g.get("t") == "CDS" and g["e"] >= s + 1 and g["s"] <= e] names = [g.get("p") or "CDS" for g in hit] uniq_this.append([s + 1, e, e - s, len(hit), "; ".join(dict.fromkeys(names))[:300]]) uniq_ref = [[s + 1, e, e - s] for s, e in gaps([(b[2], b[3]) for b in blocks], tlen)[:100]] win = max(500, qlen // 300) if qlen else 500 idwin = [] for s in range(0, qlen, win): e = min(qlen, s + win) tot = num = 0 for b in blocks: ov = min(e, b[1]) - max(s, b[0]) if ov > 0: tot += ov num += ov * b[5] idwin.append(round(num / tot, 1) if tot else 0) name = top.get("phagesdb_phage", "") acc = meta.get("accession") or top.get("accession") or "" return { "name": name, "url": url(name), "accession": acc, "ncbi": ("https://www.ncbi.nlm.nih.gov/nuccore/" + urllib.parse.quote(acc)) if acc else "", "cluster": top.get("cluster", ""), "host": top.get("host", ""), "ends": top.get("ends", ""), "temperate": top.get("temperate", ""), "morphotype": top.get("morphotype", ""), "gc_ref": top.get("gc_pct", ""), "gc_this": top.get("gc_pct_this_genome", ""), "orfs": top.get("orfs", ""), "trnas": top.get("trnas", ""), "term_rpt": top.get("terminal_repeat_bp", ""), "cds_this": sum(1 for g in genes if g.get("t") == "CDS"), "identity": top.get("identity_pct", ""), "aligned": top.get("aligned_pct_of_genome", ""), "length_ref": top.get("length_bp", ""), "length_diff": top.get("length_difference_bp", ""), "start_check": rs.get("overhang_check", ""), "rotation": rs.get("note", ""), "qlen": qlen, "tlen": tlen, "blocks": blocks, "uniq_this": uniq_this, "uniq_ref": uniq_ref, "idwin": idwin, "win": win, "others": [[r.get("phagesdb_phage", ""), r.get("cluster", ""), r.get("identity_pct", ""), r.get("aligned_pct_of_genome", ""), url(r.get("phagesdb_phage", ""))] for r in rows[1:6]], } def cmd_readsets(list_tsv, out_tsv): """Reads, bases, mean Q and N50 of each read set listed in list_tsv (rolepath), for coverage.html.""" err = [10 ** (-q / 10) for q in range(94)] rows = [] all_len, all_q, all_nq = [], 0.0, 0 # every input FASTQ together ("All reads") with open(list_tsv) as fh: items = [l.rstrip("\\n").split("\\t", 1) for l in fh if "\\t" in l] for role, path in items: if not os.path.isfile(path): continue lengths, q_sum, nq = [], 0.0, 0 with open_text(path) as fq: while True: h = fq.readline() if not h: break s = fq.readline().rstrip("\\n") fq.readline() q = fq.readline().rstrip("\\n") lengths.append(len(s)) if q: me = sum(err[min(93, ord(c) - 33)] for c in q) / len(q) q_sum += -10 * math.log10(max(me, 1e-10)) nq += 1 rows.append([role, path, len(lengths), sum(lengths), round(q_sum / nq, 2) if nq else "", n50(lengths) or "", os.path.getsize(path)]) if role == "input": all_len.extend(lengths) all_q += q_sum all_nq += nq if sum(1 for r in rows if r[0] == "input") > 1: rows.insert(0, ["all_inputs", "all input FASTQ files", len(all_len), sum(all_len), round(all_q / all_nq, 2) if all_nq else "", n50(all_len) or "", ""]) tmp = atomic_path(out_tsv) with open(tmp, "w") as out: out.write("role\\tfile\\treads\\tbases\\tmean_q\\tn50\\tbytes\\n") for r in rows: out.write("\\t".join(str(x) for x in r) + "\\n") os.replace(tmp, out_tsv) def _read_length_hist(qc_dir): """v25: bases per read-length bin of the deduplicated reads (all input FASTQs when there is no deduplicated set), for the read-length figure in coverage.html. Cached in qc/read_lengths.json.""" import bisect rs = os.path.join(qc_dir, "read_sets.tsv") if not os.path.isfile(rs): return None with open(rs) as fh: next(fh, None) rows = [line.rstrip("\\n").split("\\t") for line in fh] src = [r[1] for r in rows if len(r) > 1 and r[0] == "deduplicated" and os.path.isfile(r[1])][:1] or \\ [r[1] for r in rows if len(r) > 1 and r[0] == "input" and os.path.isfile(r[1])] if not src: return None key = "|".join(f"{p}:{os.path.getsize(p)}:{int(os.path.getmtime(p))}" for p in src) cache = os.path.join(qc_dir, "read_lengths.json") try: with open(cache) as fh: c = json.load(fh) if c.get("key") == key: return c["hist"] except (OSError, ValueError, KeyError): pass edges = [0, 500, 1000, 1500, 2000, 2500, 3000, 4000, 5000, 6000, 8000, 10000, 15000, 20000, 30000, 50000] reads, bases, lens = [0] * len(edges), [0] * len(edges), [] for p in src: with open_text(p) as fh: for i, line in enumerate(fh): if i % 4 == 1: n = len(line.rstrip("\\n")) k = bisect.bisect_right(edges, n) - 1 reads[k] += 1 bases[k] += n lens.append(n) total = sum(lens) n50, acc = 0, 0 for n in sorted(lens, reverse=True): acc += n if acc >= total / 2: n50 = n break hist = {"edges": edges, "reads": reads, "bases": bases, "n50": n50, "total": total, "n": len(lens), "source": "deduplicated reads" if len(src) == 1 and any(r[0] == "deduplicated" for r in rows) else "all input reads"} try: with open(cache, "w") as out: json.dump({"key": key, "hist": hist}, out) except OSError: pass return hist def _read_sets(qc_dir): p = os.path.join(qc_dir, "read_sets.tsv") if not os.path.isfile(p): return None out = [] with open(p) as fh: next(fh, None) for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 6: out.append({"role": f[0], "file": os.path.basename(f[1]), "reads": int(f[2]), "bases": int(f[3]), "mean_q": f[4], "n50": f[5]}) return out or None # ---- v16: PCR primers for low-coverage regions ------------------------------------------------------- # Nearest-neighbour Tm (SantaLucia 1998), 50 mM Na+, 250 nM primer (Primer3-style defaults). _NN = {"AA": (-7.9, -22.2), "TT": (-7.9, -22.2), "AT": (-7.2, -20.4), "TA": (-7.2, -21.3), "CA": (-8.5, -22.7), "TG": (-8.5, -22.7), "GT": (-8.4, -22.4), "AC": (-8.4, -22.4), "CT": (-7.8, -21.0), "AG": (-7.8, -21.0), "GA": (-8.2, -22.2), "TC": (-8.2, -22.2), "CG": (-10.6, -27.2), "GC": (-9.8, -24.4), "GG": (-8.0, -19.9), "CC": (-8.0, -19.9)} _COMP = str.maketrans("ACGTacgtN", "TGCAtgcaN") def _rc(s): return s.translate(_COMP)[::-1] def primer_tm(seq, na=0.05, ct=250e-9): seq = seq.upper() dh = ds = 0.0 for end in (seq[0], seq[-1]): if end in "GC": dh += 0.1; ds += -2.8 else: dh += 2.3; ds += 4.1 for i in range(len(seq) - 1): a, b = _NN.get(seq[i:i + 2], (-8.0, -21.0)) dh += a; ds += b ds += 0.368 * (len(seq) - 1) * math.log(na) return 1000 * dh / (ds + 1.987 * math.log(ct / 4)) - 273.15 def _primer_score(seq, kmer_count): """lower is better; None if unusable""" s = seq.upper() if "N" in s or len(s) < 18: return None, 0, 0 gc = 100 * sum(c in "GC" for c in s) / len(s) tm = primer_tm(s) if not (55 <= tm <= 65) or not (35 <= gc <= 65): return None, tm, gc if re.search(r"(A{5,}|C{5,}|G{5,}|T{5,})", s) or re.search(r"((..)\\2{3,})", s): return None, tm, gc tail = s[-5:] if sum(c in "GC" for c in tail) > 3 or s[-1] not in "GC" and s[-2] not in "GC": return None, tm, gc if _rc(s[-5:]) in s: # 3' end can fold back / pair with itself return None, tm, gc if kmer_count.get(s[-15:], 0) + kmer_count.get(_rc(s[-15:]), 0) > 1: # 3' end must be unique return None, tm, gc score = abs(tm - 60) + abs(gc - 50) / 10 + (0 if s[-1] in "GC" else 1) return score, tm, gc def _best_primer(seq, depth, lo, hi, ok_depth, kmer_count, reverse): """best primer whose binding site lies within [lo, hi) (0-based, top strand)""" best = None lo, hi = max(0, lo), min(len(seq), hi) for start in range(lo, hi - 18 + 1, 3): for ln in range(18, 27): end = start + ln if end > hi: break if min(depth[start:end]) < ok_depth: continue site = seq[start:end] p = _rc(site) if reverse else site sc, tm, gc = _primer_score(p, kmer_count) if sc is not None and (best is None or sc < best[0]): best = (sc, p.upper(), start + 1, end, round(tm, 1), round(gc)) return best def design_lowcov_primers(seqs, panels, lowcov_x, max_regions=20, min_contig=3000): """For each low-depth region inside a contig, one primer pair in well-covered sequence on both sides.""" kmer_count = {} for s in seqs.values(): s = s.upper() for i in range(len(s) - 14): k = s[i:i + 15] kmer_count[k] = kmer_count.get(k, 0) + 1 out = [] for p in panels: n, d = p["name"], p["depth"] seq = seqs[n] if len(seq) < min_contig or not d: continue thr = max(lowcov_x, 0.25 * (p["median"] or 0)) # low = below 20x or below 25% of the median depth regions, start = [], None for i, x in enumerate(d + [thr + 1]): if x < thr and start is None: start = i elif x >= thr and start is not None: if regions and start - regions[-1][1] <= 100: regions[-1][1] = i else: regions.append([start, i]) start = None regions = [r for r in regions if r[1] - r[0] >= 100][:max_regions] for a, b in regions: rdepth = median(d[a:b]) row = {"contig": n, "start": a + 1, "end": b, "bp": b - a, "depth": rdepth} if a < 150 or b > len(seq) - 150: row["note"] = "at the contig end: no room for a primer on both sides (use an outward primer from the contig)" out.append(row) continue ok = max(thr, lowcov_x) f = r = None for flank in (450, 900): f = f or _best_primer(seq, d, a - flank, a - 60, ok, kmer_count, False) r = r or _best_primer(seq, d, b + 60, b + flank, ok, kmer_count, True) if f and r: break if not (f and r): row["note"] = "no suitable primer found in the well-covered flanks" out.append(row) continue product = r[3] - f[2] + 1 row.update({"fwd": f[1], "fwd_pos": f[2], "fwd_tm": f[4], "fwd_gc": f[5], "rev": r[1], "rev_pos": r[3], "rev_tm": r[4], "rev_gc": r[5], "product": product, "note": "long-range PCR (> 10 kb)" if product > 10000 else ("Tm differ by > 3 °C" if abs(f[4] - r[4]) > 3 else "")}) out.append(row) return out def design_gap_primers(seqs, panels, lowcov_x, read_sets, min_len=5000, max_contigs=8): """v16: when the genome is in two or more large contigs: one outward-facing primer at each end of each large contig (test the pairs by PCR to find which ends join), plus clues to why the assembly is split.""" large = sorted([p for p in panels if p["len"] >= min_len and p["depth"]], key=lambda p: -p["len"])[:max_contigs] if len(large) < 2: return None kmer_count = {} for sq in seqs.values(): sq = sq.upper() for i in range(len(sq) - 14): k = sq[i:i + 15] kmer_count[k] = kmer_count.get(k, 0) + 1 primers, weak_ends = [], [] for idx, p in enumerate(large, 1): seq, d, n = seqs[p["name"]], p["depth"], len(seqs[p["name"]]) ok = lowcov_x # v16: contig ends are often weak for kilobases; a primer only needs a reliable consensus med = p["median"] or 0 for side in ("start", "end"): ends = d[:300] if side == "start" else d[-300:] end_depth = median(ends) if ends else 0 if med and end_depth < 0.3 * med: weak_ends.append([p["name"], side, round(end_depth)]) best = None for win in ((60, 600), (60, 1500), (20, 3000), (20, 5000)): if side == "start": # primer on the bottom strand, 3' end pointing out of the contig start best = _best_primer(seq, d, win[0], win[1], ok, kmer_count, True) else: # primer on the top strand, 3' end pointing out of the contig end best = _best_primer(seq, d, n - win[1], n - win[0], ok, kmer_count, False) if best: break num = re.search(r"(\\d+)$", p["name"]) # same number as "contig N" in the report row = {"id": f"C{num.group(1) if num else idx}{'L' if side == 'start' else 'R'}", "contig": p["name"], "side": side, "contig_len": n, "depth": round(med)} if best: dist = best[2] - 1 if side == "start" else n - best[3] row.update({"primer": best[1], "pos": best[2] if side == "end" else best[3], "tm": best[4], "gc": best[5], "to_end": dist}) else: row["note"] = "no suitable primer near this end" primers.append(row) usable = [r for r in primers if r.get("primer")] pairs = sum(1 for i in range(len(usable)) for j in range(i + 1, len(usable)) if usable[i]["contig"] != usable[j]["contig"] or len(large) == 1) meds = [p["median"] for p in large if p["median"]] spread = round(max(meds) / min(meds), 2) if meds and min(meds) else None samples = set() for r in read_sets or []: if r.get("role") != "input": continue m = re.search(r"(SampleSet-?\\d+)?.*?-(\\d+)-Barcode0*(\\d+)", r.get("file", ""), re.I) if m: samples.add(f"-{m.group(2)} (barcode {m.group(3)})") return {"n": len(large), "contigs": [[p["name"], p["len"], p["median"]] for p in large], "depth_spread": spread, "samples": sorted(samples), "weak_ends": weak_ends, "primers": primers, "n_pairs": pairs} def parse_genbank_records(path): """{LOCUS name: (length, genes)} for a multi-record GenBank file (v16)""" out, chunk, name, rec_len = {}, [], None, 0 with open(path) as fh: for line in fh: if line.startswith("LOCUS"): parts = line.split() name = parts[1] if len(parts) > 1 else None m = re.search(r"(\\d+) bp", line) rec_len = int(m.group(1)) if m else 0 chunk = [line] else: chunk.append(line) if line.startswith("//") and name: tmp = atomic_path(path + ".rec") with open(tmp, "w") as o: o.writelines(chunk) try: out[name] = (rec_len, parse_genbank_features(tmp)) finally: os.remove(tmp) name, chunk = None, [] return out _MODULES = [("structure", ("head and packaging", "connector", "tail")), ("lysis", ("lysis",)), ("integration / regulation", ("integration and excision", "transcription regulation")), ("DNA metabolism", ("dna, rna and nucleotide metabolism",)), ("moron / auxiliary", ("moron, auxiliary metabolic gene and host takeover",))] _TYPICAL_ORDER = ["structure", "lysis", "integration / regulation", "DNA metabolism"] def _contig_annotation(panels, annot_rows, rule): """per large contig: genes, how many have a known function, functional modules; plus a suggested order""" if not annot_rows: return None by = {p["name"]: p for p in panels} rows = [] for f in annot_rows: name, length, circ, role = f[1], int(f[2]), f[3], f[4] p = by.get(name) genes = [g for g in (p or {}).get("genes", []) if g.get("t") == "CDS"] mods = {} for g in genes: fn = (g.get("f") or "").lower() for mod, keys in _MODULES: if fn in keys: mods[mod] = mods.get(mod, 0) + 1 known = sum(1 for g in genes if (g.get("f") or "").lower() not in ("unknown function", "other", "")) dominant = max(mods, key=mods.get) if mods else "" rows.append({"name": name, "len": length, "circular": circ, "role": role, "cds": len(genes), "known": known, "modules": mods, "dominant": dominant, "phage_like": (known >= 3) if genes else None}) annotated = [r for r in rows if r["cds"]] order = None if len(annotated) >= 2 and all(r["dominant"] in _TYPICAL_ORDER for r in annotated): order = [r["name"] for r in sorted(annotated, key=lambda r: (_TYPICAL_ORDER.index(r["dominant"]), -r["len"]))] return {"rule": rule, "contigs": rows, "order": order} def _contig_overlaps(paf, min_len=300, min_id=0.85, end_tol=300): """shared sequence between different large contigs: an overlap at the ends, or a repeat inside (v16)""" if not os.path.isfile(paf): return None hits, seen = [], set() with open(paf) as fh: for line in fh: f = line.split("\\t") if len(f) < 12 or f[0] == f[5]: continue qn, ql, qs, qe, strand, tn, tl, ts, te, match, alen = (f[0], int(f[1]), int(f[2]), int(f[3]), f[4], f[5], int(f[6]), int(f[7]), int(f[8]), int(f[9]), int(f[10])) if alen < min_len or match / max(alen, 1) < min_id: continue key = tuple(sorted([(qn, qs // 100, qe // 100), (tn, ts // 100, te // 100)])) if key in seen: continue seen.add(key) q_end = qs <= end_tol or qe >= ql - end_tol t_end = ts <= end_tol or te >= tl - end_tol kind = "overlap at the contig ends" if q_end and t_end else "shared region (repeat or duplication)" def side(s, e, l): return "start" if s <= end_tol else ("end" if e >= l - end_tol else "middle") hits.append({"a": qn, "a_from": qs + 1, "a_to": qe, "a_side": side(qs, qe, ql), "b": tn, "b_from": ts + 1, "b_to": te, "b_side": side(ts, te, tl), "strand": strand, "bp": alen, "identity": round(100 * match / alen, 1), "kind": kind}) hits.sort(key=lambda h: -h["bp"]) return hits[:20] def _phagesdb_databases(cache, worker): """[[id, label], ...] of the BLAST databases PhagesDB's form offers (the real DATALIB values), read from https://phagesdb.org/blast/ once a day (cached); falls back to the Worker's /phagesdb-version. [] if neither works.""" import time import urllib.request if cache and os.path.isfile(cache) and time.time() - os.path.getmtime(cache) < 86400: try: with open(cache) as fh: dbs = json.load(fh) if dbs: return dbs except (OSError, ValueError): pass dbs = [] try: req = urllib.request.Request("https://phagesdb.org/blast/", headers={"User-Agent": "Mozilla/5.0 (phage pipeline)"}) page = urllib.request.urlopen(req, timeout=15).read().decode("utf-8", "replace") m = re.search(r"]*name=[\\"']?DATALIB[\\"']?[^>]*>(.*?)", page, re.S | re.I) if m: for v, lab in re.findall(r"]*value=[\\"']?([^\\"'>\\s]+)[\\"']?[^>]*>([^<]*)", m.group(1), re.I): if v: dbs.append([v, lab.strip() or v]) except Exception: dbs = [] if not dbs and worker: try: j = json.loads(urllib.request.urlopen(worker + "/phagesdb-version", timeout=15).read().decode()) d = j.get("database") or {} if d.get("value"): dbs = [[d["value"], d.get("label") or d["value"]]] except Exception: dbs = [] if dbs and cache: try: with open(cache, "w") as fh: json.dump(dbs, fh) except OSError: pass return dbs # ------------------------------------------------------------------ v17: input read-set check def cmd_inputcheck(final_fasta, list_txt, out_tsv, flag_txt, threads, max_reads): """v17: map a sample of each input read set (one barcode in one run; -partN files together) to this assembly and flag read sets whose reads mostly don't fit it: a different phage, a mixed or a mislabelled sample.""" import tempfile threads, max_reads = int(threads), int(max_reads) recs = read_fasta(final_fasta) refs = [r for r in recs if len(r[2]) >= 10000] or [r for r in recs if len(r[2]) >= 2000] groups = {} with open(list_txt) as fh: for line in fh: p = line.strip() if p and os.path.isfile(p): label = re.sub(r"(-part\\d+)?\\.(fastq|fq)(\\.gz)?$", "", os.path.basename(p), flags=re.I) groups.setdefault(label, []).append(p) rows = [] if refs and groups: with tempfile.TemporaryDirectory() as tmp: ref = os.path.join(tmp, "ref.fa") with open(ref, "w") as fh: for name, _, seq in refs: write_fasta(fh, name, seq) for label, files in sorted(groups.items()): q = os.path.join(tmp, "q.fastq") n = 0 with open(q, "w") as out: for f in files: with open_text(f) as fin: while n < max_reads: h = fin.readline() if not h: break sq, pl, ql = fin.readline(), fin.readline(), fin.readline() if len(sq.strip()) < 500: continue out.write(f"@r{n}\\n" + sq + "+\\n" + ql) # numbered: names may repeat across files n += 1 if n >= max_reads: break if n == 0: rows.append([label, len(files), 0, "", ""]) continue fit = set() for a in _mm2(["-x", "map-ont", "--secondary=no", "-t", str(threads), ref, q]): if len(a) >= 12 and (int(a[3]) - int(a[2])) >= 0.5 * int(a[1]): fit.add(a[0]) rows.append([label, len(files), n, round(100 * len(fit) / n, 1), ""]) vals = sorted(r[3] for r in rows if r[2] >= 100) med = vals[len(vals) // 2] if vals else None bad = [] for r in rows: if r[2] < 100 or med is None: continue if len(vals) >= 2 and med >= 40 and (r[3] < 0.5 * med or r[3] < 25): r[4] = f"only {r[3]:g}% of reads fit this assembly (other read sets ~{med:g}%)" bad.append(r[0]) elif len(vals) == 1 and r[3] < 30: r[4] = f"only {r[3]:g}% of reads fit this assembly" bad.append(r[0]) with open(out_tsv, "w") as out: out.write("read_set\\tfiles\\treads_checked\\tpct_fit_assembly\\tflag\\n") for r in rows: out.write("\\t".join(str(x) for x in r) + "\\n") flag = "none" if bad: flag = (f"{len(bad)} input read set(s) mostly don't fit this assembly - a different phage, a mixed or a " f"mislabelled sample (see qc/input_check.tsv): " + ", ".join(bad[:3]) + (" …" if len(bad) > 3 else "")) with open(flag_txt, "w") as out: out.write(f"read_sets\\t{len(rows)}\\nmedian_pct_fit\\t{med if med is not None else 'NA'}\\nflag\\t{flag}\\n") if rows: rng = f"{vals[0]:g}-{vals[-1]:g}%" if vals else "NA" print(f"Input read sets: {len(rows)}; reads that fit the assembly: median {med if med is not None else 'NA'}% " f"(range {rng})") for r in rows: if r[4]: print(f"⚠️ {r[0]}: {r[4]} — a different phage, a mixed or a mislabelled sample; don't combine it " "with the others until checked") def _filtlong_label(): """v18: the Filtlong settings, e.g. '>= 200 bp, >= Q15' (MIN_MEAN_Q is Filtlong's mean accuracy in %)""" parts = [] try: if int(os.environ.get("MIN_LENGTH", "0") or 0) > 0: parts.append(f">= {int(os.environ['MIN_LENGTH'])} bp") except ValueError: pass try: acc = float(os.environ.get("MIN_MEAN_Q", "") or 0) if 0 < acc < 100: import math as _m parts.append(f">= Q{round(-10 * _m.log10(1 - acc / 100))}") except ValueError: pass return ", ".join(parts) def _input_check(qc_dir): p = os.path.join(qc_dir, "input_check.tsv") if not os.path.isfile(p): return None rows = [] with open(p) as fh: next(fh, None) for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 5 and f[2].isdigit(): rows.append({"set": f[0], "files": int(f[1]), "reads": int(f[2]), "pct": f[3], "flag": f[4]}) return rows or None # ------------------------------------------------------------------ v18: report for a phage that could not be assembled FAIL_HTML = r""" __TITLE__ – not assembled
""" _STEP_LABELS = { "start": "the start (reading the input files)", "dedup": "removing duplicate reads", "filtlong": "the length / quality filter (Filtlong)", "host": "removing host reads", "flye": "the Flye assembly", "flye_tiers": "the Flye assemblies at several depths", "ac_genome_size": "estimating the genome size", "ac_subsets": "making Autocycler's read subsets", "ac_assemblies": "the Autocycler assemblies", "ac_compress": "Autocycler (combining the assemblies)", "ac_cluster": "Autocycler (clustering)", "ac_trim": "Autocycler (trimming)", "ac_resolve": "Autocycler (resolving)", "ac_combine": "Autocycler (combining)", "medaka_reads": "selecting reads for Medaka", "medaka": "polishing (Medaka)", } def cmd_failreport(cfg_path): """v18: .coverage.html for a phage the pipeline could not assemble""" import tempfile import datetime c = read_kv(cfg_path) sample, qc_dir = c.get("sample", "?"), c.get("qc_dir", ".") threads = c.get("threads", "4") files = [l.strip() for l in open(c["fastq_list"]) if l.strip() and os.path.isfile(l.strip())] if c.get("fastq_list") else [] read_sets = _read_sets(qc_dir) or [] inputs = [r for r in read_sets if r["role"] == "input"] allr = next((r for r in read_sets if r["role"] == "all_inputs"), None) total_reads = allr["reads"] if allr else sum(r["reads"] for r in inputs) total_bases = allr["bases"] if allr else sum(r["bases"] for r in inputs) # expected genome size: the larger of the Flye genome size setting and the smallest genome-size estimate def parse_size(v): v = str(v or "").strip().lower().rstrip("b") try: return int(float(v[:-1]) * {"k": 1e3, "m": 1e6, "g": 1e9}[v[-1]]) if v and v[-1] in "kmg" else int(float(v)) except (ValueError, KeyError): return 0 expected = max(parse_size(c.get("genome_size")), parse_size(c.get("min_gsize")), 0) or 50000 depth_x = round(total_bases / expected) if expected else None # read lengths (share of bases per length class) and a sample of reads for the host / PhagesDB check bins = [(0, 500, "< 0.5 kb"), (500, 1000, "0.5-1 kb"), (1000, 2000, "1-2 kb"), (2000, 3000, "2-3 kb"), (3000, 5000, "3-5 kb"), (5000, 10000, "5-10 kb"), (10000, 20000, "10-20 kb"), (20000, 10 ** 12, "> 20 kb")] acc = [0] * len(bins) max_sample = int(c.get("sample_reads", "2000")) per_file = max(50, -(-max_sample // max(1, len(files)))) tmp = tempfile.mkdtemp() sfq = os.path.join(tmp, "sample.fastq") n_sample, n_len = 0, 0 with open(sfq, "w") as out: for f in files: taken = 0 with open_text(f) as fh: while n_len < 300000: h = fh.readline() if not h: break sq, pl, ql = fh.readline().strip(), fh.readline(), fh.readline() L = len(sq) n_len += 1 for i, (lo, hi, _) in enumerate(bins): if lo <= L < hi: acc[i] += L break if L >= 500 and taken < per_file and n_sample < max_sample: out.write(f"@s{n_sample}\\n{sq}\\n+\\n{ql.strip()}\\n") taken += 1 n_sample += 1 tot = sum(acc) length_bins = [[b[2], round(100 * a / tot)] for b, a in zip(bins, acc)] if tot else [] def mapped(index): hits = {} try: for a in _mm2(["-x", "map-ont", "--secondary=no", "-t", str(threads), index, sfq]): if len(a) >= 12 and (int(a[3]) - int(a[2])) >= 0.5 * int(a[1]): hits[a[0]] = a[5] except Exception as exc: print(f" (mapping to {os.path.basename(index)} skipped: {exc})") return None return hits host_pct = pdb_hits = unmatched_pct = None host_ids, pdb_checked = set(), False if n_sample and c.get("host_index") not in (None, "", "-") and os.path.isfile(c["host_index"]): h = mapped(c["host_index"]) if h is not None: host_ids = set(h) host_pct = round(100 * len(h) / n_sample, 1) if n_sample and c.get("pdb_index") not in (None, "", "-") and os.path.isfile(c["pdb_index"]): h = mapped(c["pdb_index"]) if h is not None: pdb_checked = True meta = _pdb_meta(c["pdb_meta"]) if os.path.isfile(c.get("pdb_meta", "")) else {} cnt = {} for rid, tgt in h.items(): if rid not in host_ids: cnt[tgt] = cnt.get(tgt, 0) + 1 pdb_hits = [] for tgt, n in sorted(cnt.items(), key=lambda kv: -kv[1])[:6]: m = meta.get(tgt, {}) pdb_hits.append({"phage": m.get("phage", tgt), "cluster": m.get("cluster", ""), "length": m.get("length_bp", ""), "pct": round(100 * n / n_sample, 1)}) unmatched_pct = round(100 * (n_sample - len(host_ids) - sum(cnt.values())) / n_sample, 1) import shutil as _sh _sh.rmtree(tmp, ignore_errors=True) # partial assembly contigs, contigs_from = [], "" pf = c.get("partial_fasta", "-") if pf not in ("", "-") and os.path.isfile(pf): info = {} pi = c.get("partial_info", "-") if pi not in ("", "-") and os.path.isfile(pi): for line in open(pi): f = line.rstrip("\\n").split("\\t") if len(f) >= 4 and not line.startswith("#"): info[f[0]] = (f[2], f[3]) for name, _, seq in sorted(read_fasta(pf), key=lambda r: -len(r[2]))[:10]: if len(seq) >= 1000: d, circ = info.get(name, ("", "")) contigs.append({"name": name, "len": len(seq), "depth": d, "circ": circ, "seq": seq}) contigs_from = c.get("partial_label", "") # verdict step = c.get("failed_step", "") step_label = _STEP_LABELS.get(step, step or "unknown") multi = [x for x in (pdb_hits or []) if x["pct"] >= 15] clusters = {x["cluster"] for x in multi if x["cluster"]} if total_bases == 0: verdict, cls, advice = "no reads in the input files", "bad", "Check the barcode-to-phage map and the input folder." elif depth_x is not None and depth_x < 20: verdict, cls = f"too little data: about {depth_x}× for a {expected:,} bp genome", "bad" advice = "Sequence this phage again (or add another run's reads); 40× or more usually assembles." elif host_pct is not None and host_pct >= 50: verdict, cls = f"mostly host DNA ({host_pct}% of reads)", "bad" advice = "Turn on host-read removal for this sample, or prepare the DNA again with less host DNA (e.g. DNase before extraction)." elif len(clusters) >= 2: verdict, cls = ("the reads match phages from different clusters (" + ", ".join(sorted(clusters)) + "): a mixed sample"), "bad" advice = "Plaque-purify and sequence single plaques separately." elif depth_x is not None and depth_x < 40: verdict, cls = f"borderline amount of data (about {depth_x}×) and the assembler did not finish", "warn" advice = "More reads would help; you can also try Standard mode (Flye only), which needs less data." else: verdict, cls = f"enough data (about {depth_x}×), but {step_label} failed", "warn" advice = ("See the log below. Very uneven depth or a large genome can stop the assemblers: check the genome size " "settings, or try Standard mode.") log_tail = "" lp = c.get("log_path", "") if lp and os.path.isfile(lp): lines = [l.rstrip("\\n") for l in open(lp, errors="replace")] mine = [l for l in lines if sample in l] if c.get("log_shared") == "yes" else lines log_tail = "\\n".join((mine or lines)[-25:]) data = {"sample": sample, "version": c.get("version", ""), "date": datetime.datetime.now().strftime("%Y-%m-%d %H:%M"), "failed_step_label": step_label, "verdict": verdict, "verdict_cls": cls, "advice": advice, "log_tail": log_tail, "total_reads": total_reads, "total_bases": total_bases, "n_files": len(inputs), "expected_bp": expected, "depth_x": depth_x, "read_sets": read_sets, "filtlong": c.get("filtlong", ""), "length_bins": length_bins, "sampled": n_sample, "host_pct": host_pct, "pdb_hits": pdb_hits, "pdb_checked": pdb_checked, "unmatched_pct": unmatched_pct, "contigs": contigs, "contigs_from": contigs_from, "pdb_url": os.environ.get("PHAGESDB_BLAST_URL", "") or "https://phagesdb.org/blast/results/blast.cgi", "pdb_field": os.environ.get("PHAGESDB_SEQ_FIELD", "") or "SEQUENCE", "pdb_datalib": os.environ.get("PHAGESDB_DATALIB", "") or next(iter(_phagesdb_databases( os.environ.get("PHAGESDB_DB_CACHE", ""), os.environ.get("NCBI_BLAST_PROXY", "").rstrip("/")) or []), "")} if isinstance(data["pdb_datalib"], (list, tuple)): data["pdb_datalib"] = data["pdb_datalib"][0] if isinstance(data["pdb_datalib"], dict): data["pdb_datalib"] = data["pdb_datalib"].get("id") or data["pdb_datalib"].get("value") or "" html_out = FAIL_HTML.replace("__TITLE__", sample.replace("&", "&").replace("<", "<")).replace( "__DATA__", json.dumps(data, separators=(",", ":")).replace(" len(depth[name]): continue depth[name][pos - 1] = int(f[3]) if f[4] != "NA": agree[name][pos - 1] = int(round(float(f[4]) * 10)) if float(f[4]) < agree_pct and int(f[3]) >= lowcov_x: detail[(name, pos)] = (f[2], f[3], f[4], f[5], f[6], f[7]) def resolve(name): if name in seqs: return name if f"{sample}_{name}" in seqs: return f"{sample}_{name}" return None edits = {n: [] for n in order} if changes_tsv != "-" and os.path.isfile(changes_tsv): with open(changes_tsv) as fh: next(fh) for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) < 9: continue name = resolve(f[2]) if name: label = {"substitution": f"{f[5]}>{f[6]}", "insertion": f"+{f[6]}", "deletion": f"-{f[5]}"}.get(f[4], f[4]) edits[name].append([int(f[3]), f[4][0], label, f[8] == "yes"]) longest_name = None info = {} if longest_tsv != "-" and os.path.isfile(longest_tsv): with open(longest_tsv) as fh: for line in fh: k, _, v = line.rstrip("\\n").partition("\\t") info[k] = v longest_name = info.get("longest_contig") genes = {n: [] for n in order} gbk_label = "" if gbk != "-" and os.path.isfile(gbk) and longest_name in genes: genes[longest_name] = parse_genbank_features(gbk) gbk_label = os.path.basename(gbk) # v16: genes on the other large contigs (separate Pharokka/Phold run, see annotpick) sample_dir = os.path.dirname(os.path.abspath(out_html)) qc_dir0 = os.path.dirname(os.path.abspath(stats_txt)) annotated = {longest_name: "main"} if gbk_label else {} extra_gbk = next((g for g in (os.path.join(sample_dir, f"{sample}.other_contigs.phold.gbk"), os.path.join(sample_dir, f"{sample}.other_contigs.pharokka.gbk")) if os.path.isfile(g)), None) id_map, annot_rows, annot_rule = {}, [], "" at = os.path.join(qc_dir0, "annotated_contigs.tsv") if os.path.isfile(at): with open(at) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") if f[0] == "#rule": annot_rule = f[1] if len(f) > 1 else "" elif f[0] != "short_id" and len(f) >= 5: annot_rows.append(f) if f[0]: id_map[f[0]] = f[1] if extra_gbk and id_map: extra_len = {f[1]: int(f[2]) for f in annot_rows if f[0]} for locus, (rec_len, feats) in parse_genbank_records(extra_gbk).items(): name = id_map.get(locus) if name is None: # renamed by Pharokka/Phold: match by the short id inside the name, then by length name = next((v for k, v in id_map.items() if k in locus), None) if name is None and rec_len: same = [n for n, ln_ in extra_len.items() if ln_ == rec_len] name = same[0] if len(same) == 1 else None if name in genes: genes[name] = feats annotated[name] = "extra" missing = [n for n in extra_len if annotated.get(n) != "extra"] if missing: print(f"⚠️ No genes found for {', '.join(missing)} in {os.path.basename(extra_gbk)}", file=sys.stderr) panels = [] stats_out = {} all_depth = [] low_agree_total = 0 for n in order: d = depth[n] a = agree[n] all_depth.extend(d) low_cov = [] start = None for i, x in enumerate(d + [lowcov_x + 1]): if x < lowcov_x and start is None: start = i elif x >= lowcov_x and start is not None: if low_cov and start + 1 - low_cov[-1][1] <= 50: low_cov[-1][1] = i # merge regions separated by short gaps else: low_cov.append([start + 1, i]) start = None low_agree = sorted( ([pos, *detail[(n, pos)]] for (c, pos) in detail if c == n), key=lambda r: float(r[3])) low_agree_total += len(low_agree) covered = [x for x in a if x >= 0] panels.append({ "name": n, "len": len(d), "depth": d, "agree": a, "edits": edits[n], "genes": genes[n], "lowcov": low_cov[:200], "lowagree": low_agree[:200], "lowagree_n": len(low_agree), "mean": round(sum(d) / len(d), 1) if d else 0, "median": median(d), "min": min(d) if d else 0, "max": max(d) if d else 0, "pct_low": round(100 * sum(1 for x in d if x < lowcov_x) / len(d), 2) if d else 0, "mean_agree": round(sum(covered) / len(covered) / 10, 2) if covered else None, "longest": n == longest_name, "annotated": annotated.get(n, ""), "gbk": os.path.basename(extra_gbk) if annotated.get(n) == "extra" and extra_gbk else "", "seq": seqs[n] if len(seqs[n]) >= 200 else "", }) ln = longest_name if longest_name in depth else (order[0] if order else None) if ln: p = next(x for x in panels if x["name"] == ln) stats_out = { "contig": ln, "median_depth": p["median"], "mean_depth": p["mean"], "min_depth": p["min"], "pct_below_lowcov": p["pct_low"], "mean_agreement_pct": p["mean_agree"], "low_agreement_positions": p["lowagree_n"], } with open(stats_txt, "w") as out: for k, v in stats_out.items(): out.write(f"{k}\\t{v}\\n") # v16: contigs with much higher depth than the main contig (a repeat present twice, or DNA that the MuA + PCR # library over-represents because it is less modified) main_med = next((x["median"] for x in panels if x["name"] == ln), 0) if ln else 0 high_depth = [] for x in panels: if x["name"] != ln and main_med and x["len"] >= 1000 and (x["median"] or 0) >= 1.7 * main_med: x["high_depth_ratio"] = round(x["median"] / main_med, 1) high_depth.append([x["name"], x["len"], x["high_depth_ratio"]]) # v16: PCR primers across low-coverage regions qc_dir = os.path.dirname(os.path.abspath(stats_txt)) try: primers = design_lowcov_primers(seqs, panels, lowcov_x) except Exception as exc: # never let primer design break the report print(f"Primer design skipped: {exc}", file=sys.stderr) primers = [] pcols = ["contig", "start", "end", "bp", "depth", "fwd", "fwd_pos", "fwd_tm", "fwd_gc", "rev", "rev_pos", "rev_tm", "rev_gc", "product", "note"] with open(os.path.join(qc_dir, f"{sample}.lowcov_primers.tsv"), "w") as out: out.write("\\t".join(pcols) + "\\n") for r in primers: out.write("\\t".join(str(r.get(c, "")) for c in pcols) + "\\n") for x in panels: x["primers"] = [r for r in primers if r["contig"] == x["name"]] # v16: genome split into two or more large contigs: outward primers at each end + clues read_sets = _read_sets(qc_dir) try: fragmented = design_gap_primers(seqs, panels, lowcov_x, read_sets) except Exception as exc: print(f"Gap-closing primer design skipped: {exc}", file=sys.stderr) fragmented = None if fragmented: gcols = ["id", "contig", "side", "contig_len", "depth", "primer", "pos", "tm", "gc", "to_end", "note"] with open(os.path.join(qc_dir, f"{sample}.gap_primers.tsv"), "w") as out: out.write("\\t".join(gcols) + "\\n") for r in fragmented["primers"]: out.write("\\t".join(str(r.get(c, "")) for c in gcols) + "\\n") host = read_kv(os.path.join(os.path.dirname(os.path.abspath(stats_txt)), "host_removal.txt")) data = { "sample": sample, "panels": panels, "agree": agree_pct, "lowcov": lowcov_x, "reads": reads_label, "gbk": gbk_label, "circular": info.get("flye_circular", "NA"), "longest": ln, "host": host if host.get("status") == "ok" else None, "high_depth": high_depth, "read_sets": read_sets, "fragmented": fragmented, "input_check": _input_check(qc_dir), "filtlong": _filtlong_label(), "contig_annotation": _contig_annotation(panels, annot_rows, annot_rule), "blast": {"ncbi_proxy": os.environ.get("NCBI_BLAST_PROXY", "").rstrip("/"), "phagesdb_url": os.environ.get("PHAGESDB_BLAST_URL", "") or "https://phagesdb.org/blast/results/blast.cgi", "phagesdb_field": os.environ.get("PHAGESDB_SEQ_FIELD", "") or "SEQUENCE", "phagesdb_datalib": os.environ.get("PHAGESDB_DATALIB", ""), "phagesdb_dbs": _phagesdb_databases(os.environ.get("PHAGESDB_DB_CACHE", ""), os.environ.get("NCBI_BLAST_PROXY", "").rstrip("/"))}, "overlaps": _contig_overlaps(os.path.join(qc_dir, "contig_overlaps.paf")), "read_hist": _read_length_hist(qc_dir), "closest": _closest_report(os.path.dirname(os.path.abspath(stats_txt)), genes.get(ln) or [] if ln else [], len(seqs.get(ln, "")) if ln else 0), } html = REPORT_HTML.replace("__BANG__", chr(33)).replace("__TITLE__", sample).replace( "__DATA__", json.dumps(data, separators=(",", ":")).replace(" __TITLE__ coverage and consensus

Reads

BLAST settings

How to read the coverage plots

Coverage is the number of reads aligned at each base (minimap2 → samtools depth). Read agreement is the percentage of reads whose base matches the polished consensus at that position (mismatches and deletions count as disagreement; up to 2,000 reads sampled per position). Bases are flagged only where coverage is at least the low-coverage threshold, because agreement is unreliable from a handful of reads. Nanopore reads are typically 95–99% accurate, so isolated dips are expected; clusters of low agreement, or dips well below the rest, can indicate a homopolymer, a mixed population, or a possible consensus error worth checking in IGV with the BAM file in the qc folder. Coverage often falls at the ends of a circular contig because reads crossing the junction are clipped. Drag across a plot to zoom; double-click or press Reset to zoom out. GC content is the share of G and C bases in a sliding window; depth that falls in GC-rich or GC-poor DNA points to PCR amplification bias (see the coverage-vs-GC figure). Log scale shows low-coverage stretches next to high peaks.

""" # ------------------------------------------------------------ PhageTracker BARCODE_RE = re.compile(r"barcode(\\d{1,3})", re.IGNORECASE) HEADER_FIELD_RE = re.compile(r"(\\w+)=(\\S+)") def n50(lengths): total = sum(lengths) acc = 0 for x in sorted(lengths, reverse=True): acc += x if acc * 2 >= total: return x return 0 CP_KEYS = ["closest_phage", "closest_source", "closest_identity_pct", "closest_query_cover_pct", "closest_link", "closest_length_bp", "closest_covered_pct", "status", "status_note", "ictv_similarity_to_closest_pct", "ictv_rank", "same_phage_in_run", "likely_cluster", "genbank_accession", "genbank_description", "genbank_host", "genbank_per_ident", "genbank_query_cover", "genbank_subject_length", "genbank_subject_cover", "genbank_link", "host_dna_pct", "checkv_quality", "checkv_completeness_pct", "isolation_host", "host_mismatch"] def cmd_cpinject(tsv, out_root): """v25: write each phage's genome-check results (closest PhagesDB/GenBank phage, status, ICTV similarity, duplicates; closest_phages.tsv) into its coverage.html, which reads them from ', lambda m: '", page, count=1, flags=re.S) if not k: continue # a report made before v25 (no place for the results) tmp = p + ".tmp" with open(tmp, "w", encoding="utf-8") as out: out.write(new) os.replace(tmp, p) n += 1 print(f"Genome-check results added to {n} coverage report(s)") def cmd_readstats(fastq, out_json): """Read-level statistics for one FASTQ, plus barcode/run info from the first Dorado header.""" err = [10 ** (-q / 10) for q in range(94)] lengths = [] q_sum = 0.0 q20 = 0 header_info = {} with open_text(fastq) as fh: while True: header = fh.readline() if not header: break seq = fh.readline() fh.readline() qual = fh.readline().rstrip("\\n") n = len(seq.rstrip("\\n")) if not header_info: header_info = dict(HEADER_FIELD_RE.findall(header)) header_info["_raw"] = header.strip()[:300] lengths.append(n) if qual: mean_err = sum(err[min(93, ord(c) - 33)] for c in qual) / len(qual) q = -10 * math.log10(max(mean_err, 1e-10)) q_sum += q if q >= 20: q20 += 1 count = len(lengths) srt = sorted(lengths) med = (srt[count // 2] if count % 2 else (srt[count // 2 - 1] + srt[count // 2]) / 2) if count else None barcode = None for text in (header_info.get("barcode", ""), header_info.get("_raw", ""), fastq): m = BARCODE_RE.search(text or "") if m: barcode = int(m.group(1)) break stats = { "fastq_file": fastq, "read_count": count, "total_bases": sum(lengths), "mean_read_length": round(sum(lengths) / count, 1) if count else None, "median_read_length": med, "read_n50": n50(lengths), "mean_q_score": round(q_sum / count, 2) if count else None, "min_read_length": srt[0] if count else None, "longest_read_length": srt[-1] if count else None, "percent_reads_q20_or_higher": round(100 * q20 / count, 2) if count else None, "reads_1kb_or_longer": sum(1 for x in lengths if x >= 1000), "reads_5kb_or_longer": sum(1 for x in lengths if x >= 5000), "reads_10kb_or_longer": sum(1 for x in lengths if x >= 10000), "barcode_number": barcode, "dorado_run_id": header_info.get("runid") or header_info.get("RG"), "flow_cell_id": header_info.get("flow_cell_id"), "basecall_model": header_info.get("basecall_model_version_id") or header_info.get("model_version_id"), } with open(out_json, "w") as out: json.dump(stats, out, indent=1) print(f"Reads: {count:,} | {stats['total_bases']:,} bases | N50 {stats['read_n50']:,} | " f"mean Q {stats['mean_q_score']} | barcode {barcode if barcode is not None else '?'}") def read_kv(path): out = {} if os.path.isfile(path): with open(path) as fh: for line in fh: k, sep, v = line.rstrip("\\n").partition("\\t") if sep: out[k] = v return out def read_json(path): if os.path.isfile(path): with open(path) as fh: return json.load(fh) return None def sha256_file(path): import hashlib h = hashlib.sha256() with open(path, "rb") as fh: for chunk in iter(lambda: fh.read(1 << 20), b""): h.update(chunk) return h.hexdigest() def gc_percent(seq): """GC content (%) over A/C/G/T only, rounded to 2 decimals; None for an empty sequence.""" s = seq.upper() gc = s.count("G") + s.count("C") at = s.count("A") + s.count("T") return round(100.0 * gc / (gc + at), 2) if gc + at else None def assembly_record(fasta, label, assembler, version, parameters, flye_info, sample, selected_name): recs = read_fasta(fasta) lengths = [len(s) for _, _, s in recs] contigs = [] for name, _, seq in recs: original = name[len(sample) + 1:] if name.startswith(sample + "_") else name info = flye_info.get(original, {}) cov = info.get("coverage") contigs.append({ "contig_name": name, "length_bp": len(seq), "gc_percent": gc_percent(seq), "coverage": float(cov) if cov not in (None, "", "NA") else None, "circular": info.get("circular"), "keep_contig": name == selected_name or original == selected_name, }) longest = max(contigs, key=lambda c: c["length_bp"]) if contigs else None circ = longest.get("circular") if longest else None return { "assembly_label": label, "assembler": assembler, "assembler_version": version, "parameters": parameters, "assembly_fasta": fasta, "assembly_sha256": sha256_file(fasta), "number_of_contigs": len(recs), "total_assembly_length_bp": sum(lengths), "assembly_n50": n50(lengths), "longest_contig_length_bp": max(lengths) if lengths else None, "gc_percent": longest.get("gc_percent") if longest else None, "topology": {"Y": "circular", "N": "linear"}.get(circ), "circularized": True if circ == "Y" else (False if circ == "N" else None), "contigs": contigs, } def parse_gbk_genes(gbk): """Genes from a Pharokka/Phold GenBank file, keeping translations and all qualifiers.""" genes = [] features = [] current = None last_key = None in_features = False with open(gbk) as fh: for raw in fh: line = raw.rstrip("\\n") if line.startswith("FEATURES"): in_features = True continue if line.startswith(("ORIGIN", "//", "LOCUS")): in_features = False if current: features.append(current) current = None continue if in_features is False or len(line) < 22: continue if line[:5] == " " and line[5] != " ": if current: features.append(current) current = {"type": line[5:21].strip(), "loc": line[21:].strip(), "q": {}} last_key = None elif current is not None and line[:21].strip() == "": content = line[21:].strip() if content.startswith("/"): key, _, value = content[1:].partition("=") current["q"][key] = value last_key = key elif last_key: joiner = "" if last_key == "translation" else " " current["q"][last_key] += joiner + content else: current["loc"] += content if current: features.append(current) n = 0 for f in features: if f["type"] not in ("CDS", "tRNA", "tmRNA"): continue nums = [int(x) for x in re.findall(r"\\d+", f["loc"])] if not nums: continue n += 1 q = {k: v.strip().strip('"') for k, v in f["q"].items()} genes.append({ "gene_number": n, "feature_type": f["type"], "locus_tag": q.get("locus_tag") or q.get("ID"), "gene_name": q.get("gene"), "start_bp": min(nums), "end_bp": max(nums), "strand": "-" if "complement" in f["loc"] else "+", "product": q.get("product"), "function": q.get("function"), "protein_id": q.get("protein_id"), "translation": q.get("translation", "").replace(" ", "") or None, "qualifiers": {k: v for k, v in q.items() if k not in ("translation",)}, }) return genes def cmd_bundle(out_root, samples_tsv, meta_json, out_json): """Collect everything one pipeline run produced into a single import file.""" flat = read_kv(meta_json) # keyvalue lines; "tool." and "setting." prefixes group values meta = {k: v for k, v in flat.items() if "." not in k} meta["tools"] = {k[5:]: v for k, v in flat.items() if k.startswith("tool.")} meta["settings"] = {k[8:]: v for k, v in flat.items() if k.startswith("setting.")} sample_rows = [] with open(samples_tsv) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 3: sample_rows.append((f[0], f[1], f[2:])) settings = meta.get("settings", {}) tools = meta.get("tools", {}) sheet_barcodes = {} sheet_meta = {} if meta.get("sample_sheet_matches") and os.path.isfile(meta["sample_sheet_matches"]): for line in open(meta["sample_sheet_matches"]): f = line.rstrip("\\n").split("\\t") if line.startswith("#") and len(f) == 2: sheet_meta[f[0][1:]] = f[1] elif len(f) == 3 and f[1] and f[2].isdigit(): sheet_barcodes[f[1]] = int(f[2]) def _norm(x): return re.sub(r"[^a-z0-9]+", "", str(x or "").lower()) def _num(x): try: return float(x) except (TypeError, ValueError): return None pmap = [] if meta.get("phage_map") and os.path.isfile(meta["phage_map"]): with open(meta["phage_map"]) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 5 and f[0] != "barcode" and f[3]: pmap.append({"barcode": int(f[0]) if f[0].isdigit() else None, "name": f[1], "linked": f[2], "phage_id": f[3], "lab_phage_id": f[4]}) gchecks = {} if meta.get("closest_tsv") and os.path.isfile(meta["closest_tsv"]): import csv with open(meta["closest_tsv"], newline="", encoding="utf-8") as fh: for r in csv.DictReader(fh, delimiter="\\t"): gchecks[r.get("phage", "")] = r samples = [] for sample, status, fastqs in sample_rows: d = os.path.join(out_root, sample) qc = os.path.join(d, "qc") rec = { "sample": sample, "phage_name_from_folder": sample, "pipeline_status": status, "input_fastqs": fastqs, "output_folder": d, "files": {}, } for key, fname in [("polished_fasta", f"{sample}.fasta"), ("flye_fasta", f"{sample}.flye.fasta"), ("autocycler_fasta", f"{sample}.autocycler.fasta"), ("genome_ends", "qc/genome_ends.txt"), ("rotation", "qc/rotation.txt"), ("annotation_qc", "qc/annotation_qc.txt"), ("sample_flags", "qc/sample_flags.txt"), ("longest_contig_fasta", f"{sample}.longest_contig.fasta"), ("pharokka_gbk", f"{sample}.pharokka.gbk"), ("phold_gbk", f"{sample}.phold.gbk"), ("coverage_html", f"{sample}.coverage.html"), ("nanoplot_report", f"nanoplot/{sample}_NanoPlot-report.html"), ("nanoplot_stats", f"nanoplot/{sample}_NanoStats.txt"), ("coverage_bam", f"qc/{sample}.reads_vs_polished.bam"), ("flye_vs_medaka_changes", f"qc/{sample}.flye_vs_medaka.changes.tsv")]: p = os.path.join(d, fname) if os.path.isfile(p): rec["files"][key] = p dedup = {} dedup_report = os.path.join(d, "input", f"{sample}.deduplication.txt") if os.path.isfile(dedup_report): for line in open(dedup_report): k, _, v = line.partition(":") if v.strip().isdigit(): dedup[k.strip().lower().replace(" ", "_")] = int(v.strip()) rec["deduplication"] = dedup raw = read_json(os.path.join(qc, "readstats_raw.json")) filt = read_json(os.path.join(qc, "readstats_filtered.json")) rec["barcode_number"] = sheet_barcodes.get(sample) rec["barcode_source"] = "sample_sheet" if rec["barcode_number"] is not None else None if rec["barcode_number"] is None: rec["barcode_number"] = (raw or {}).get("barcode_number") or (filt or {}).get("barcode_number") rec["barcode_source"] = "read_headers" if rec["barcode_number"] is not None else None if rec["barcode_number"] is None: for p in fastqs: m = BARCODE_RE.search(os.path.basename(p)) if m: rec["barcode_number"] = int(m.group(1)) # phage identity: the PhageTracker sample set (by folder name, then barcode), else an earlier phage.json ident = None byname = [m for m in pmap if _norm(m["name"]) == _norm(sample) or _norm(m["linked"]) == _norm(sample)] bybc = [m for m in pmap if rec["barcode_number"] is not None and m["barcode"] == rec["barcode_number"]] if byname and bybc and byname[0]["phage_id"] != bybc[0]["phage_id"]: rec["phage_map_conflict"] = (f"folder name matches {byname[0]['linked']}, but barcode " f"{rec['barcode_number']} is {bybc[0]['linked']}") print(f"⚠️ {sample}: {rec['phage_map_conflict']} — not linked by ID; check the folder name or the sample sheet") else: m = (byname or bybc or [None])[0] if m: ident = {"phage_id": m["phage_id"], "lab_phage_id": m["lab_phage_id"], "phage_name": m["linked"] or m["name"], "barcode": m["barcode"], "sample_set": meta.get("sample_set_identifier"), "matched_by": "folder name" if byname else "barcode", "written": meta.get("generated_at")} pj = os.path.join(d, "phage.json") if ident and os.path.isdir(d): tmp = atomic_path(pj) with open(tmp, "w") as fh: json.dump(ident, fh, indent=1) os.replace(tmp, pj) elif not ident and "phage_map_conflict" not in rec: old = read_json(pj) if old and old.get("phage_id"): ident = old if ident: rec.update({"phage_id": ident.get("phage_id"), "lab_phage_id": ident.get("lab_phage_id"), "phage_name": ident.get("phage_name")}) gc = gchecks.get(sample) if gc: src = gc.get("closest_source") or None rec["genome_checks"] = { "closest_phage": gc.get("closest_phage") or None, "closest_source": src, "closest_identity_pct": _num(gc.get("closest_identity_pct")), "closest_query_cover_pct": _num(gc.get("closest_query_cover_pct")), "closest_link": gc.get("closest_link") or None, "closest_cluster": (gc.get("likely_cluster") or None) if src == "PhagesDB" else None, "phagesdb_cluster": gc.get("likely_cluster") or None, "host_dna_pct": _num(gc.get("host_dna_pct")), "checkv_quality": gc.get("checkv_quality") or None, "checkv_completeness_pct": _num(gc.get("checkv_completeness_pct")), "checkv_contamination_pct": _num(gc.get("checkv_contamination_pct")), "genus_group": gc.get("genus_group_70pct") or None, "same_genus_as": gc.get("same_genus_as") or None, } read_sets = [] if raw: raw = dict(raw) raw.update({"stage": "raw", "reads_before": dedup.get("total_records"), "reads_after": raw["read_count"], "notes": f"Deduplicated by read ID ({dedup.get('duplicate_records_removed', 0)} duplicate records removed)."}) read_sets.append(raw) host = read_kv(os.path.join(qc, "host_removal.txt")) rec["host_removal"] = host or None rec["dna_source_hint"] = "Plaque" if host else None hostdep = read_json(os.path.join(qc, "readstats_host_depleted.json")) parent = raw if hostdep and host.get("status") == "ok": hostdep = dict(hostdep) by_host = host.get("mode") == "host_genome" hostdep.update({ "stage": "host_depleted", "host_removal_program": ("minimap2 vs host genome" + (" (phage-matching reads protected)" if host.get("host_prophage") not in (None, "no") else "")) if by_host else "Flye --meta draft + minimap2 (reference-free)", "host_removal_version": (f"minimap2 {tools.get('minimap2') or '?'}" if by_host else f"Flye {tools.get('flye') or '?'}; minimap2 {tools.get('minimap2') or '?'}"), "host_reference": (host.get("host_genome", "") if by_host else "Phage contigs from a draft assembly of this sample: " + host.get("phage_contigs", "")), "host_reference_accession": None, "host_removal_parameters": { "min_aligned_pct": settings.get("HOST_MIN_ALIGNED_PCT"), "contig_depth_fraction": None if by_host else settings.get("HOST_CONTIG_FRACTION"), "host_has_prophage": host.get("host_prophage") if by_host else None, "reads_matching_host": host.get("reads_matching_host"), "reads_per_host": host.get("reads_per_host"), "host_genomes": [{"name": settings.get(f"HOST_NAME_{i}"), "fasta": settings.get(f"HOST_GENOME_{i}"), "prophage": settings.get(f"HOST_PROPHAGE_{i}")} for i in (1, 2, 3) if settings.get(f"HOST_GENOME_{i}")] if by_host else None, "method": host.get("method"), }, "reads_before": raw["read_count"] if raw else None, "reads_after": hostdep["read_count"], "bases_before": raw["total_bases"] if raw else None, "bases_after": hostdep["total_bases"], "percent_reads_retained": round(100 * hostdep["read_count"] / raw["read_count"], 2) if raw and raw["read_count"] else None, "notes": (f"Picked-plaque DNA: reads matching the host genome were removed; {host.get('percent_reads_phage')}% of reads " f"({host.get('percent_bases_phage')}% of bases) were kept for assembly." if by_host else f"Picked-plaque DNA: {host.get('percent_reads_phage')}% of reads " f"({host.get('percent_bases_phage')}% of bases) aligned to the phage contigs; " "the other reads (host) were removed before assembly."), }) read_sets.append(hostdep) parent = hostdep if filt: filt = dict(filt) filt.update({ "stage": "filtered", "filter_program": "Filtlong", "filter_version": tools.get("filtlong"), "filter_parameters": {k: settings.get(k) for k in ("MIN_LENGTH", "MIN_MEAN_Q", "TARGET_BASES", "KEEP_PERCENT")}, "reads_before": parent["read_count"] if parent else None, "reads_after": filt["read_count"], "bases_before": parent["total_bases"] if parent else None, "bases_after": filt["total_bases"], "percent_reads_retained": round(100 * filt["read_count"] / parent["read_count"], 2) if parent and parent["read_count"] else None, }) read_sets.append(filt) rec["read_sets"] = read_sets accuracy = (settings.get("ASSEMBLY_MODE") == "accuracy" and not str(read_kv(os.path.join(qc, "assembly_method.txt")).get("method") or "").startswith("flye")) rec["assembly_method"] = read_kv(os.path.join(qc, "assembly_method.txt")).get("method") if accuracy: flye_info = read_flye_info(os.path.join(qc, "contig_info.txt")) else: flye_info = read_flye_info(os.path.join(d, "flye", "assembly_info.txt")) ends = read_kv(os.path.join(qc, "genome_ends.txt")) rotation = read_kv(os.path.join(qc, "rotation.txt")) annot_qc = read_kv(os.path.join(qc, "annotation_qc.txt")) flags = read_kv(os.path.join(qc, "sample_flags.txt")) longest = read_kv(os.path.join(qc, "longest_contig.tsv")) cov = read_kv(os.path.join(qc, "coverage_stats.txt")) totals = read_kv(os.path.join(qc, f"{sample}.flye_vs_medaka.totals.txt")) selected = longest.get("longest_contig") flye_params = {k: settings.get(k) for k in ("FLYE_READ_TYPE", "GENOME_SIZE", "FLYE_ITERATIONS", "MIN_OVERLAP", "ASM_COVERAGE", "FLYE_META", "FLYE_SCAFFOLD")} assemblies = [] if accuracy: flye_params = {k: settings.get(k) for k in ("AC_ASSEMBLERS", "AC_COUNT", "AC_GENOME_SIZE", "AC_MAX_DEPTH", "AC_FLYE_VOTE", "AC_FLYE_WEIGHT", "AC_FLYE_TIERS", "FLYE_FIRST_MB", "FLYE_READ_TYPE", "ROTATE")} draft_key = "autocycler_fasta" if accuracy else "flye_fasta" draft_name = "Autocycler consensus" if accuracy else "Flye" draft_ver = tools.get("autocycler") if accuracy else tools.get("flye") polished_name = "Autocycler + Medaka" if accuracy else "Flye + Medaka" polished_ver = (f"Autocycler {tools.get('autocycler')}; Medaka {tools.get('medaka')}" if accuracy else f"Flye {tools.get('flye')}; Medaka {tools.get('medaka')}") if draft_key in rec["files"]: a = assembly_record(rec["files"][draft_key], f"{sample} {draft_name}", draft_name, draft_ver, flye_params, flye_info, sample, None) a["assembly_status"] = "draft" a["role"] = "unpolished" a["mean_coverage"] = None assemblies.append(a) if "polished_fasta" in rec["files"]: params = dict(flye_params) params.update({k: settings.get(k) for k in ("MEDAKA_READS", "MEDAKA_DEPTH", "MEDAKA_MODEL", "MEDAKA_BACTERIA")}) a = assembly_record(rec["files"]["polished_fasta"], f"{sample} {polished_name}", polished_name, polished_ver, params, flye_info, sample, selected) a["assembly_status"] = "needs_review" a["role"] = "polished" md = cov.get("median_depth") a["mean_coverage"] = float(cov["mean_depth"]) if cov.get("mean_depth") not in (None, "", "None") else None a["qc"] = { "median_read_depth": float(md) if md not in (None, "", "None") else None, "min_read_depth": cov.get("min_depth"), "pct_bases_below_lowcov": cov.get("pct_below_lowcov"), "mean_read_agreement_pct": cov.get("mean_agreement_pct"), "low_agreement_positions": cov.get("low_agreement_positions"), "flye_to_medaka_changes": {k: int(v) for k, v in totals.items() if v.isdigit()}, "longest_contig_fraction": longest.get("longest_fraction"), "assembly_mode": "accuracy" if accuracy else "standard", "genome_ends": ends or None, "rotation": rotation or None, "annotation_check": annot_qc or None, "flags": (flags.get("flags") if flags.get("flags") not in (None, "", "none") else None), } assemblies.append(a) rec["assemblies"] = assemblies gbk = rec["files"].get("phold_gbk") or rec["files"].get("pharokka_gbk") if gbk: genes = parse_gbk_genes(gbk) rec["annotation"] = { "source_tool": "Pharokka + Phold" if gbk.endswith(".phold.gbk") else "Pharokka", "source_file_name": os.path.basename(gbk), "pharokka_version": tools.get("pharokka"), "phold_version": tools.get("phold") if gbk.endswith(".phold.gbk") else None, "contig_name": selected, "parameters": {k: settings.get(k) for k in ("PHAROKKA_GENE_CALLER", "PHAROKKA_FAST", "PHOLD_COMPUTE")}, "gene_count": sum(1 for g in genes if g["feature_type"] == "CDS"), "hypothetical_count": sum(1 for g in genes if (g["product"] or "").lower() == "hypothetical protein"), "genes": genes, } samples.append(rec) bundle = { "format": "phage-pipeline-export/1", "generated_at": meta.get("generated_at"), "generator_version": meta.get("generator_version"), "phagetracker": { "sample_set_identifier": meta.get("sample_set_identifier") or None, "sequencing_run_id": meta.get("sequencing_run_id") or None, "sharepoint_base_url": meta.get("sharepoint_base_url") or None, }, "dorado_sample_sheet": ({"file": meta.get("sample_sheet"), **sheet_meta} if meta.get("sample_sheet") else None), "output_root": out_root, "output_root_windows": meta.get("output_root_windows"), "tools": tools, "settings": settings, "samples": samples, } tmp = atomic_path(out_json) with open(tmp, "w") as out: json.dump(bundle, out, indent=1) os.replace(tmp, out_json) n_ann = sum(1 for s in samples if s.get("annotation")) print(f"PhageTracker import file: {out_json} ({len(samples)} sample(s), {n_ann} annotated)") # ------------------------------------------------------------ sample sheet def cmd_sheetmatch(sheet_csv, input_root, files_list, out_tsv): """Assign FASTQ files to Dorado sample-sheet aliases (barcode -> alias, as written by the Nanopore Workflow app). A file matches an alias when a folder or the file name equals the alias, ends with "_alias" (Dorado prefixes the kit name), or names the alias's barcode (barcodeNN). Output: filealiasbarcode (blank if none). """ import csv aliases = {} # lower alias -> (alias, barcode) by_barcode = {} # barcode -> (alias, barcode) meta = {} with open(sheet_csv, newline="", encoding="utf-8-sig") as fh: for row in csv.DictReader(fh): row = {(k or "").strip().lower(): (v or "").strip() for k, v in row.items()} m = BARCODE_RE.search(row.get("barcode", "")) alias = row.get("alias", "") if not m or not alias: continue bc = int(m.group(1)) aliases[alias.lower()] = (alias, bc) by_barcode[bc] = (alias, bc) for k in ("experiment_id", "kit", "flow_cell_id"): if row.get(k): meta[k] = row[k] if not aliases: raise SystemExit(f"No barcode,alias rows found in {sheet_csv}") def strip_ext(x): return re.sub(r"\\.(fastq|fq)(\\.gz)?$", "", x, flags=re.IGNORECASE) files = [f for f in open(files_list, "rb").read().decode("utf-8", "replace").split("\\0") if f] root = input_root.rstrip("/") + "/" out = [] for f in files: rel = f[len(root):] if f.startswith(root) else os.path.basename(f) parts = [strip_ext(p) for p in rel.split("/") if p] hit = None for part in reversed(parts): # exact folder/file name if part.lower() in aliases: hit = aliases[part.lower()] break if hit is None: for part in reversed(parts): # kit-prefixed names such as SQK-RPB114-24_Akaza low = part.lower() cands = [a for a in aliases if low.endswith("_" + a) or low.endswith("-" + a)] if cands: hit = aliases[max(cands, key=len)] break if hit is None: for part in reversed(parts): # barcodeNN folders or files m = BARCODE_RE.search(part) if m and int(m.group(1)) in by_barcode: hit = by_barcode[int(m.group(1))] break out.append((f, hit[0] if hit else "", str(hit[1]) if hit else "")) with open(out_tsv, "w") as fh: for k, v in meta.items(): fh.write(f"#{k}\\t{v}\\n") for row in out: fh.write("\\t".join(row) + "\\n") matched = sum(1 for r in out if r[1]) print(f"Sample sheet: {len(aliases)} barcode(s); {matched} of {len(out)} FASTQ file(s) matched") # ------------------------------------------------------------- accuracy mode def cmd_acinfo(consensus_fasta, gfa_path, out_info): """Write Autocycler consensus contigs in Flye's assembly_info.txt layout (name, length, depth, circular).""" circular = set() if gfa_path != "-" and os.path.isfile(gfa_path): with open(gfa_path) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") # a segment linked to itself (same orientation) is a circular sequence if len(f) >= 5 and f[0] == "L" and f[1] == f[3] and f[2] == f[4]: circular.add(f[1]) rows = [] for name, desc, seq in read_fasta(consensus_fasta): tags = dict(t.split("=", 1) for t in desc.split() if "=" in t) circ = tags.get("circular", "").lower() is_circ = circ in ("true", "yes", "y") or name in circular depth = tags.get("depth", "NA") if depth in ("", "unavailable"): depth = "NA" rows.append((name, len(seq), depth, "Y" if is_circ else "N")) with open(out_info, "w") as out: out.write("#seq_name\\tlength\\tcov.\\tcirc.\\n") for r in rows: out.write("\\t".join(str(x) for x in r) + "\\n") print(f"Autocycler consensus: {len(rows)} sequence(s): " + ", ".join(f"{r[0]} {r[1]:,} bp{' circular' if r[3] == 'Y' else ''}" for r in rows)) CIGAR_RE = re.compile(r"(\\d+)([MIDNSHP=X])") def cmd_readends(contig_fasta, sam_path, out_txt): """Find positions where many reads start and stop: the physical ends of a phage genome (cos site, terminal repeat or pac site). Reads are counted only when the alignment reaches the read's own end (<= 15 bases clipped), so reads crossing an end are not mistaken for ends. Native (ligation) libraries show sharp peaks; transposase-based kits (Rapid, Rapid PCR) cut DNA at random and usually show none.""" recs = read_fasta(contig_fasta) if not recs: raise RuntimeError("no contig") name, _, seq = recs[0] L = len(seq) starts = [0] * (L + 2) stops = [0] * (L + 2) total = 0 fh = sys.stdin if sam_path == "-" else open_text(sam_path) for line in fh: if line.startswith("@"): continue f = line.split("\\t") if len(f) < 11: continue flag = int(f[1]) if flag & 0x904 or f[2] == "*": continue pos = int(f[3]) ops = CIGAR_RE.findall(f[5]) if not ops: continue lead = sum(int(n) for n, op in ops[:2] if op in "SH") if ops[0][1] in "SH" else 0 trail = sum(int(n) for n, op in ops[-2:] if op in "SH") if ops[-1][1] in "SH" else 0 ref_len = sum(int(n) for n, op in ops if op in "MDN=X") end = pos + ref_len - 1 total += 1 if lead <= 15 and 1 <= pos <= L: starts[pos] += 1 if trail <= 15 and 1 <= end <= L: stops[end] += 1 bg = max(total / max(L, 1), 0.05) # strongest start pile-up, paired with the strongest stop pile-up just before it (allowing a small # overhang / repeat offset; positions wrap around so an end at the contig start is found too). best = None win = 25 for x in range(1, L + 1): if starts[x] < 5: continue y_best, s_val = None, 0 for d in range(-win, win + 1): y = ((x - 2 + d) % L) + 1 if stops[y] > s_val: y_best, s_val = y, stops[y] score = starts[x] + s_val if best is None or score > best[0]: best = (score, x, starts[x], y_best, s_val) top = sorted(((starts[i] + stops[i], i) for i in range(1, L + 1)), reverse=True)[:5] detected = False values = {"contig": name, "length": L, "reads_counted": total, "background_per_bp": f"{bg:.2f}"} if best: _, x, sx, y, sy = best fold_s, fold_e = sx / bg, (sy / bg if sy else 0) edge = x <= 30 or x >= L - 30 detected = sx >= 10 and sy >= 5 and fold_s >= 20 and fold_e >= 10 gap = "NA" if y: gap = x - 1 - y if gap > L / 2: gap -= L elif gap < -L / 2: gap += L values.update({"best_start_position": x, "reads_starting": sx, "start_fold": f"{fold_s:.1f}", "best_stop_position": y if y else "NA", "reads_stopping": sy, "stop_fold": f"{fold_e:.1f}", "gap_or_overlap_bp": gap, "at_contig_edge": "yes" if edge else "no"}) values["end_detected"] = "yes" if detected else "no" if detected: gap = values["gap_or_overlap_bp"] values["end_type_guess"] = ("cohesive end (cos) or exact end" if isinstance(gap, int) and abs(gap) <= 20 else "terminal repeat or offset ends") # an end already at the contig start (or a linear contig's edge) needs no rotation values["end_position"] = 1 if values["at_contig_edge"] == "yes" else values["best_start_position"] else: values["end_type_guess"] = "no clear physical end (circularly permuted, or library kit fragments DNA at random)" values["end_position"] = "NA" values["top_positions"] = ";".join(f"{i}:{s}" for s, i in top) with open(out_txt, "w") as out: for k, v in values.items(): out.write(f"{k}\\t{v}\\n") if detected: print(f"Genome end: reads start at {values['best_start_position']:,} ({values['reads_starting']} reads, " f"{values['start_fold']}x background) and stop at {values['best_stop_position']:,} — {values['end_type_guess']}") else: print("Genome end: no clear read start/stop pile-up (expected for circularly permuted phages and Rapid kits)") def cmd_rotate(contig_fasta, position, out_fasta): recs = read_fasta(contig_fasta) name, desc, seq = recs[0] p = int(position) if not 1 <= p <= len(seq): raise RuntimeError(f"rotation position {p} is outside the contig (1-{len(seq)})") rotated = seq[p - 1:] + seq[:p - 1] tmp = atomic_path(out_fasta) with open(tmp, "w") as out: write_fasta(out, name, rotated, (desc + f" rotated_to={p}").strip()) os.replace(tmp, out_fasta) print(f"Rotated {name} to start at position {p:,}") def cmd_shiftchanges(changes_tsv, contig_name, length, position, out_tsv): """Move Medaka-coordinate positions of one contig in a changes table onto that contig after it was rotated to start at POSITION (names match with or without the sample prefix).""" p, L = int(position), int(length) with open(changes_tsv) as fh: header = next(fh) rows = [line.rstrip("\\n").split("\\t") for line in fh] tmp = atomic_path(out_tsv) with open(tmp, "w") as out: out.write(header) for f in rows: if len(f) > 3 and f[3].isdigit() and (f[2] == contig_name or contig_name.endswith("_" + f[2])): f[3] = str((int(f[3]) - p) % L + 1) out.write("\\t".join(f) + "\\n") os.replace(tmp, out_tsv) def cmd_relabel(in_fasta, new_name, out_fasta): recs = read_fasta(in_fasta) if not recs: raise RuntimeError(f"No sequence in {in_fasta}") name, desc, seq = max(recs, key=lambda r: len(r[2])) tmp = atomic_path(out_fasta) with open(tmp, "w") as out: write_fasta(out, new_name, seq, desc) os.replace(tmp, out_fasta) def cmd_replacecontig(final_fasta, contig_name, new_fasta): new = read_fasta(new_fasta) if not new: raise RuntimeError(f"No sequence in {new_fasta}") new_seq = max(new, key=lambda r: len(r[2]))[2] recs = read_fasta(final_fasta) found = False tmp = atomic_path(final_fasta) with open(tmp, "w") as out: for name, desc, seq in recs: if name == contig_name: seq, found = new_seq, True write_fasta(out, name, seq, desc) if not found: os.remove(tmp) raise RuntimeError(f"{contig_name} not found in {final_fasta}") os.replace(tmp, final_fasta) def cmd_codingqc(gbk, genome_fasta, out_txt): """Leftover Nanopore indel errors split genes: coding density drops and many short CDS appear. Tailed phages are typically ~90-95% coding with few CDS under 40 aa.""" recs = read_fasta(genome_fasta) L = len(recs[0][2]) if recs else 0 covered = bytearray(L + 1) cds = short = 0 aa_total = 0 loc_re = re.compile(r"\\d+") with open(gbk) as fh: for line in fh: m = re.match(r"^\\s{5}CDS\\s+(\\S+)", line) if not m: continue nums = [int(n) for n in loc_re.findall(m.group(1))] if len(nums) < 2: continue a, b = min(nums), max(nums) if L and b - a > L / 2 and "join" in m.group(1): # CDS wrapping the contig start spans = [(max(nums[i], 1), nums[i + 1]) for i in range(0, len(nums) - 1, 2)] else: spans = [(a, b)] length = sum(e - s + 1 for s, e in spans) cds += 1 aa = max(length // 3 - 1, 0) aa_total += aa if aa < 40: short += 1 for s_, e_ in spans: for i in range(max(s_, 1), min(e_, L) + 1): covered[i] = 1 density = 100 * sum(covered) / L if L else 0 per_kb = 1000 * cds / L if L else 0 flags = [] if L and density < 85: flags.append(f"low coding density {density:.1f}%") if cds and short / cds > 0.15: flags.append(f"{short} of {cds} CDS are shorter than 40 aa") with open(out_txt, "w") as out: for k, v in [("genome_length", L), ("cds_count", cds), ("cds_per_kb", f"{per_kb:.2f}"), ("coding_density_pct", f"{density:.1f}"), ("short_cds_under_40aa", short), ("mean_cds_aa", f"{aa_total / cds:.0f}" if cds else "NA"), ("possible_indel_errors", "yes" if flags else "no"), ("flags", "; ".join(flags))]: out.write(f"{k}\\t{v}\\n") print(f"Annotation check: {cds} CDS, coding density {density:.1f}%, {short} CDS < 40 aa" + (f" — ⚠️ {'; '.join(flags)} (possible leftover indel errors)" if flags else "")) def cmd_sampleflags(final_fasta, longest_fasta, annotation_qc, ac_dir, out_txt): """Warnings worth a look before accepting a genome.""" flags = [] recs = read_fasta(final_fasta) if os.path.isfile(final_fasta) else [] big = [r for r in recs if len(r[2]) >= 10000] if len(big) >= 2: flags.append(f"{len(big)} sequences of 10 kb or more — possible mixed sample or contamination") doubled = "no" lrec = read_fasta(longest_fasta) if os.path.isfile(longest_fasta) else [] if lrec: seq = lrec[0][2].upper() n = len(seq) if n >= 4000: # a genome assembled twice end-to-end: k-mers from the first half all occur again in the second k, hits, tried = 31, 0, 0 for i in range(0, n // 2 - k, max((n // 2) // 60, 1)): kmer = seq[i:i + k] if "N" in kmer: continue tried += 1 if seq.find(kmer, i + n // 4) != -1: hits += 1 if tried >= 20 and hits / tried >= 0.8: doubled = "yes" flags.append("the genome sequence appears twice end-to-end (doubled assembly) — check the length") clusters_pass = clusters_fail = "NA" if ac_dir != "-" and os.path.isdir(ac_dir): cdir = os.path.join(ac_dir, "clustering") clusters_pass = len([d for d in os.listdir(os.path.join(cdir, "qc_pass"))]) if os.path.isdir(os.path.join(cdir, "qc_pass")) else 0 clusters_fail = len([d for d in os.listdir(os.path.join(cdir, "qc_fail"))]) if os.path.isdir(os.path.join(cdir, "qc_fail")) else 0 if annotation_qc != "-" and os.path.isfile(annotation_qc): q = read_kv(annotation_qc) if q.get("flags"): flags.append(q["flags"] + " (possible leftover indel errors)") am = read_kv(os.path.join(os.path.dirname(out_txt), "assembly_method.txt")) words = {"low_depth_region": "part of the kept assembly has low read depth (see qc/assembly_method.txt)", "host_end_trimmed": "host DNA trimmed from a contig end", "assemblies_disagree": "Autocycler and Flye assembled different sequences (possible mixed sample)"} for fl in str(am.get("flags") or "").split(","): if fl in words: flags.append(words[fl]) ic = read_kv(os.path.join(os.path.dirname(out_txt), "input_check.txt")) # v17 if ic.get("flag") and ic["flag"] != "none": flags.append(ic["flag"]) with open(out_txt, "w") as out: for k, v in [("phage_sized_sequences", len(big)), ("doubled_genome", doubled), ("autocycler_clusters_pass", clusters_pass), ("autocycler_clusters_fail", clusters_fail), ("flags", "; ".join(flags) if flags else "none")]: out.write(f"{k}\\t{v}\\n") if flags: print("⚠️ Check: " + "; ".join(flags)) # ------------------------------------------------------------ host removal def cmd_phagecontigs(info_path, fasta_path, out_fasta, out_tsv, min_frac): """Pick the phage contigs in a draft Flye --meta assembly of picked-plaque reads. Phage DNA is many copies per infected cell, so phage contigs are much deeper than host fragments. The reference is the contig holding the most read bases (length x depth) among 5-600 kb contigs; every contig >= 1 kb whose depth is >= min_frac of it is kept.""" min_frac = float(min_frac) seqs = {n: s for n, _, s in read_fasta(fasta_path)} rows = [] if os.path.isfile(info_path): with open(info_path) as fh: for line in fh: if line.startswith("#") or not line.strip(): continue f = line.rstrip("\\n").split("\\t") if len(f) < 4 or f[0] not in seqs: continue try: rows.append({"name": f[0], "len": int(f[1]), "depth": float(f[2]), "circ": f[3]}) except ValueError: continue if not rows: # no assembly_info.txt: fall back to lengths only rows = [{"name": n, "len": len(s), "depth": 1.0, "circ": "NA"} for n, s in seqs.items()] cand = [r for r in rows if 5000 <= r["len"] <= 600000] or [r for r in rows if r["len"] >= 1000] if not cand: print("No draft contig of at least 1 kb; the phage could not be identified.") sys.exit(2) ref = max(cand, key=lambda r: r["len"] * r["depth"]) for r in rows: r["keep"] = 1000 <= r["len"] <= 600000 and r["depth"] >= min_frac * ref["depth"] kept = [r for r in rows if r["keep"]] tmp = atomic_path(out_fasta) with open(tmp, "w") as out: for r in kept: write_fasta(out, r["name"], seqs[r["name"]], f"length={r['len']} depth={r['depth']:g}") os.replace(tmp, out_fasta) with open(out_tsv, "w") as out: out.write("contig\\tlength\\tdepth\\tcircular\\tphage\\n") for r in sorted(rows, key=lambda r: -r["len"] * r["depth"]): out.write(f"{r['name']}\\t{r['len']}\\t{r['depth']:g}\\t{r['circ']}\\t{'yes' if r['keep'] else 'no'}\\n") total = sum(r["len"] * r["depth"] for r in rows) or 1 share = sum(r["len"] * r["depth"] for r in kept) / total print(f" draft: {len(rows)} contig(s); phage contig(s): " + ", ".join(f"{r['name']} ({r['len']:,} bp, {r['depth']:g}x)" for r in kept) + f"; about {100 * share:.0f}% of assembled read bases") def cmd_hostfilter(paf_path, reads_in, reads_out, report_path, min_pct, contigs_tsv): """Keep reads whose aligned part (union of PAF query intervals) covers >= min_pct of the read.""" min_pct = float(min_pct) spans, qlen = {}, {} with open_text(paf_path) as fh: for line in fh: f = line.split("\\t", 5) if len(f) < 5: continue spans.setdefault(f[0], []).append((int(f[2]), int(f[3]))) qlen[f[0]] = int(f[1]) phage = set() for q, iv in spans.items(): iv.sort() covered, (cs, ce) = 0, iv[0] for s, e in iv[1:]: if s > ce: covered += ce - cs cs, ce = s, e else: ce = max(ce, e) covered += ce - cs if qlen[q] and 100.0 * covered / qlen[q] >= min_pct: phage.add(q) contigs = [] if os.path.isfile(contigs_tsv): with open(contigs_tsv) as fh: next(fh, None) for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 5 and f[4] == "yes": contigs.append(f"{f[0]} ({f[1]} bp, {f[2]}x)") n_in = n_out = b_in = b_out = 0 tmp = atomic_path(reads_out) with open_text(reads_in) as fin, gzip.open(tmp, "wt") as fout: while True: header = fin.readline() if not header: break if not header.strip(): continue seq, plus, qual = fin.readline(), fin.readline(), fin.readline() n = len(seq.rstrip("\\n")) n_in += 1 b_in += n if header[1:].split()[0] in phage: fout.write(header) fout.write(seq) fout.write(plus) fout.write(qual) n_out += 1 b_out += n pct_r = round(100 * n_out / n_in, 2) if n_in else 0 pct_b = round(100 * b_out / b_in, 2) if b_in else 0 values = [ ("status", "ok" if n_out else "no_phage_reads"), ("method", "reference-free: Flye --meta draft, high-depth contigs, minimap2 map-ont"), ("mode", "reference_free"), ("phage_contigs", "; ".join(contigs)), ("min_aligned_pct", f"{min_pct:g}"), ("reads_before", n_in), ("reads_after", n_out), ("reads_removed", n_in - n_out), ("bases_before", b_in), ("bases_after", b_out), ("percent_reads_phage", pct_r), ("percent_bases_phage", pct_b), ] with open(report_path, "w") as out: for k, v in values: out.write(f"{k}\\t{v}\\n") if not n_out: os.remove(tmp) print(" no read aligned to the phage contigs") sys.exit(2) os.replace(tmp, reads_out) print(f" phage reads: {n_out:,} of {n_in:,} ({pct_r}%), {b_out:,} of {b_in:,} bases ({pct_b}%); " f"{n_in - n_out:,} host/other reads removed") if pct_b < 20: print("⚠️ Less than 20% of the bases were phage: check the plaque pick or sequence more deeply.") def aligned_reads(paf_path, min_pct): """Read IDs whose aligned part (union of PAF query intervals) covers >= min_pct of the read.""" spans, qlen = {}, {} with open_text(paf_path) as fh: for line in fh: f = line.split("\\t", 5) if len(f) < 5: continue spans.setdefault(f[0], []).append((int(f[2]), int(f[3]))) qlen[f[0]] = int(f[1]) hits = set() for q, iv in spans.items(): iv.sort() covered, (cs, ce) = 0, iv[0] for s, e in iv[1:]: if s > ce: covered += ce - cs cs, ce = s, e else: ce = max(ce, e) covered += ce - cs if qlen[q] and 100.0 * covered / qlen[q] >= min_pct: hits.add(q) return hits def best_targets(paf_path): """read ID -> reference name of its longest alignment.""" best = {} with open_text(paf_path) as fh: for line in fh: f = line.split("\\t", 7) if len(f) < 7: continue span = int(f[3]) - int(f[2]) if f[0] not in best or span > best[f[0]][0]: best[f[0]] = (span, f[5]) return {q: v[1] for q, v in best.items()} def cmd_hostremove(host_paf, reads_in, reads_out, report_path, min_pct, protect_paf, labels_tsv): """Discard reads that match a host genome; reads that also match the phage draft are kept.""" min_pct = float(min_pct) host = aligned_reads(host_paf, min_pct) labels = {} if labels_tsv != "-" and os.path.isfile(labels_tsv): with open(labels_tsv) as fh: next(fh, None) for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 3: labels[f[0]] = (f[1], f[2]) per_host = {} if labels: target = best_targets(host_paf) for q in host: tag = target.get(q, "").split("__", 1)[0] per_host[tag] = per_host.get(tag, 0) + 1 protect = aligned_reads(protect_paf, min_pct) if protect_paf != "-" and os.path.isfile(protect_paf) else set() drop = host - protect n_in = n_out = b_in = b_out = 0 tmp = atomic_path(reads_out) with open_text(reads_in) as fin, gzip.open(tmp, "wt") as fout: while True: header = fin.readline() if not header: break if not header.strip(): continue seq, plus, qual = fin.readline(), fin.readline(), fin.readline() n = len(seq.rstrip("\\n")) n_in += 1 b_in += n if header[1:].split()[0] not in drop: fout.write(header) fout.write(seq) fout.write(plus) fout.write(qual) n_out += 1 b_out += n pct_r = round(100 * n_out / n_in, 2) if n_in else 0 pct_b = round(100 * b_out / b_in, 2) if b_in else 0 values = [ ("status", "ok" if n_out else "no_reads_left"), ("method", "host genome: minimap2 map-ont, matching reads discarded" + ("; reads also matching the phage draft kept (possible prophage)" if protect_paf != "-" else "")), ("mode", "host_genome"), ("min_aligned_pct", f"{min_pct:g}"), ("reads_before", n_in), ("reads_after", n_out), ("reads_removed", n_in - n_out), ("reads_matching_host", len(host)), ("host_matches_kept_as_phage", len(host & protect)), ("reads_per_host", "; ".join(f"{labels[k][0]}: {v}" for k, v in sorted(per_host.items(), key=lambda kv: -kv[1]) if k in labels) or "-"), ("bases_before", b_in), ("bases_after", b_out), ("percent_reads_phage", pct_r), ("percent_bases_phage", pct_b), ] with open(report_path, "w") as out: for k, v in values: out.write(f"{k}\\t{v}\\n") if not n_out: os.remove(tmp) print(" every read matched the host genome") sys.exit(2) os.replace(tmp, reads_out) if per_host: print(" host reads by genome: " + ", ".join( f"{labels[k][0]} {v:,}" for k, v in sorted(per_host.items(), key=lambda kv: -kv[1]) if k in labels)) if len(per_host) > 1: top = max(per_host.values()) others = [labels[k][0] for k, v in per_host.items() if k in labels and v < top and v > 0.2 * top] if others: print("⚠️ Many reads match more than one host genome (" + ", ".join(others) + "): check that the right hosts are listed, or for DNA from a second host.") print(f" kept {n_out:,} of {n_in:,} reads ({pct_r}%), {b_out:,} of {b_in:,} bases ({pct_b}%); " f"{len(drop):,} host reads removed" + (f"; {len(host & protect):,} reads matched both host and phage and were kept" if protect else "")) if protect and len(host & protect) > 0.05 * max(1, len(host)): print("⚠️ Many reads match both the host genome and the phage: the host may carry a related prophage.") if pct_b < 20: print("⚠️ Less than 20% of the bases were left: check the plaque pick or sequence more deeply.") def cmd_acverdict(info_path, settings_path, min_pct): """Print a reason (and exit 0) when Autocycler's consensus looks fragmented; print nothing when it is fine.""" min_pct = float(min_pct) rows = [] if os.path.isfile(info_path): with open(info_path) as fh: for line in fh: if line.startswith("#") or not line.strip(): continue f = line.rstrip("\\n").split("\\t") if len(f) >= 4: try: rows.append((int(f[1]), f[3] == "Y")) except ValueError: pass gsize = 0 for k, v in read_kv(settings_path).items(): if k == "genome_size": try: gsize = int(float(v)) except ValueError: pass if not rows: print("Autocycler produced no consensus contig") return length, circular = max(rows) if circular or not gsize: return if length < min_pct / 100 * gsize: print(f"Autocycler's consensus is fragmented ({len(rows)} pieces; longest {length:,} bp, not circular, " f"is {100 * length / gsize:.0f}% of the estimated {gsize:,} bp genome)") def _flye_info(info_path): rows = [] if info_path != "-" and os.path.isfile(info_path): with open(info_path) as fh: for line in fh: if line.startswith("#") or not line.strip(): continue f = line.rstrip("\\n").split("\\t") if len(f) >= 4 and f[1].isdigit(): try: cov = float(f[2]) except ValueError: cov = 0.0 rows.append((f[0], int(f[1]), cov, f[3] == "Y")) return rows def cmd_flyegsize(info_path, fasta): """Genome size from a Flye assembly of all the reads: total length of the contigs with at least half the depth of the deepest large contig (the phage), so host and other low-depth contigs don't count.""" rows = _flye_info(info_path) if not rows: recs = read_fasta(fasta) if not recs: sys.exit(1) print(max(len(r[2]) for r in recs)) return big = [r for r in rows if r[1] >= 5000] or rows top = max(r[2] for r in big) keep = [r for r in rows if r[1] >= 1000 and r[2] >= 0.5 * top] size = sum(r[1] for r in keep) or max(r[1] for r in rows) if size < 2000 or size > 1_000_000: sys.exit(1) print(size) def _pdb_header(header): """Fields of a PhagesDB FASTA header, e.g. 'Mycobacterium phage 20ES complete sequence, 53124 bp including 10-base 3' overhang (CGGTCGGTTA), Cluster A2'.""" h = header.strip() out = {"host": "", "phage": "", "length": "", "overhang": "", "ends": "", "cluster": ""} m = re.match(r"(.+?)\\s+phage\\s+([^\\s,;]+)", h, re.I) if m: out["host"], out["phage"] = m.group(1).strip(), m.group(2).strip() else: out["phage"] = h.split(None, 1)[0] if h else "" m = re.search(r"([\\d,]+)\\s*bp", h) if m: out["length"] = m.group(1).replace(",", "") m = re.search(r"(\\d+)[- ]base\\s+([35])['’′]?\\s*overhang\\s*\$([A-Za-z]+)\$", h, re.I) if m: out["overhang"] = m.group(3).upper() out["ends"] = f"{m.group(1)}-base {m.group(2)}' overhang ({m.group(3).upper()})" elif re.search(r"circularly permuted", h, re.I): out["ends"] = "circularly permuted" else: m = re.search(r"(\\d[\\d,]*)\\s*bp\\s+direct terminal repeats?", h, re.I) if m: out["ends"] = f"{m.group(1)} bp direct terminal repeats" elif re.search(r"terminal repeat", h, re.I): out["ends"] = "direct terminal repeats" m = re.search(r"Cluster\\s+([A-Za-z0-9]+)", h) if m: out["cluster"] = m.group(1) elif re.search(r"\\bSingleton\\b", h, re.I): out["cluster"] = "Singleton" return out PDB_TABLE_COLS = ["in_table", "subcluster", "temperate", "gc_pct", "end_type", "overhang_len", "term_rpt_len", "orfs", "trnas", "morphotype", "host_strain", "accession", "genome_length_table"] def _pdb_table(path): """PhagesDB data table (tab-delimited, header row) -> {lower-case phage name: row dict}.""" rows = {} if path == "-" or not os.path.isfile(path): return rows raw = open(path, "rb").read() for enc in ("utf-8-sig", "cp1252", "latin-1"): try: text = raw.decode(enc) break except UnicodeDecodeError: continue lines = text.splitlines() if not lines: return rows cols = [c.strip() for c in lines[0].split("\\t")] for line in lines[1:]: f = line.split("\\t") if not f or not f[0].strip(): continue d = {c: (f[i].strip() if i < len(f) else "") for i, c in enumerate(cols)} rows[d.get("Phage Name", f[0]).strip().lower()] = d return rows def _none(v): v = str(v or "").strip() return "" if v.lower() in ("none", "null", "nan", "n/a") else v def cmd_pdbprep(fasta, table_path, out_fasta, out_meta): """Copy the PhagesDB FASTA with a unique ID per genome (the phage name) and a table of its header fields, joined to the PhagesDB data table when one is given.""" used, n, matched = set(), 0, 0 table = _pdb_table(table_path) with open_text(fasta) as fin, open(out_fasta, "w") as fo, open(out_meta, "w") as fm: fm.write("id\\tphage\\thost\\tlength_bp\\tends\\toverhang\\tcluster\\theader\\t" + "\\t".join(PDB_TABLE_COLS) + "\\n") for line in fin: if line.startswith(">"): header = line[1:].strip() f = _pdb_header(header) base = re.sub(r"[^A-Za-z0-9._-]+", "_", f["phage"] or "phage").strip("_") or "phage" pid, k = base, 1 while pid in used: k += 1 pid = f"{base}_{k}" used.add(pid) n += 1 fo.write(f">{pid}\\n") clean = header.replace("\\t", " ") row = table.get(f["phage"].lower()) extra = {c: "" for c in PDB_TABLE_COLS} if row: matched += 1 sub, clu = _none(row.get("Subcluster")), _none(row.get("Cluster")) f["cluster"] = sub or clu or f["cluster"] end_type = _none(row.get("End Type")) oh_len, oh_seq = _none(row.get("Overhang Length (bp)")), _none(row.get("Overhang Sequence")).upper() trl = _none(row.get("Term Rpt Length")) if oh_seq and re.fullmatch(r"[ACGT]+", oh_seq): f["overhang"] = oh_seq if end_type: bits = [end_type] if oh_len: bits.append(f"{oh_len}-base") if f["overhang"]: bits.append(f"({f['overhang']})") if trl: bits.append(f"terminal repeat {trl} bp") f["ends"] = " ".join(bits) if _none(row.get("Host")): f["host"] = _none(row.get("Host")) extra.update({"in_table": "yes", "subcluster": sub or clu, "temperate": _none(row.get("Temperate?")), "gc_pct": _none(row.get("GC%")), "end_type": end_type, "overhang_len": oh_len, "term_rpt_len": trl, "orfs": _none(row.get("# ORFs")), "trnas": _none(row.get("# tRNAs")), "morphotype": _none(row.get("Morphotype")), "host_strain": _none(row.get("Host")), "accession": _none(row.get("Accession #")), "genome_length_table": _none(row.get("Genome Length(bp)"))}) fm.write(f"{pid}\\t{f['phage']}\\t{f['host']}\\t{f['length']}\\t{f['ends']}\\t{f['overhang']}\\t{f['cluster']}" f"\\t{clean}\\t" + "\\t".join(extra[c].replace("\\t", " ") for c in PDB_TABLE_COLS) + "\\n") elif line.strip(): fo.write(line.strip() + "\\n") if n == 0: sys.exit("no sequences in the PhagesDB FASTA") print(f" {n} PhagesDB genomes read" + (f"; {matched} matched to the data table ({len(table)} rows)" if table else "")) def _pdb_meta(path): meta = {} with open(path) as fh: cols = next(fh).rstrip("\\n").split("\\t") for line in fh: f = line.rstrip("\\n").split("\\t") meta[f[0]] = dict(zip(cols, f)) return meta def _fasta_record(path, wanted): """One record (name, header, seq) from a large FASTA by its first word, streaming.""" name, header, chunks, hit = None, "", [], None with open_text(path) as fh: for line in fh: if line.startswith(">"): if hit: break name = line[1:].split(None, 1)[0] if line[1:].strip() else "" if name == wanted: hit = True header = line[1:].strip() chunks = [] elif hit: chunks.append(line.strip()) if not hit: return None return name, header, "".join(chunks) def cmd_closestphage(genome_fasta, index, pdb_fasta, meta_path, out_tsv, out_ref, threads): """Closest PhagesDB phages to a genome: share of the genome aligned and identity (minimap2 asm20).""" import subprocess recs = read_fasta(genome_fasta) if not recs: raise RuntimeError("empty genome") name, _, seq = max(recs, key=lambda r: len(r[2])) L = len(seq) r = subprocess.run(["minimap2", "-c", "-x", "asm20", "-N", "100", "-p", "0.2", "-t", str(threads), index, genome_fasta], capture_output=True, text=True) if r.returncode != 0: raise RuntimeError("minimap2 failed: " + r.stderr[-300:]) per = {} for line in r.stdout.splitlines(): f = line.split("\\t") if len(f) < 12: continue d = per.setdefault(f[5], {"tlen": int(f[6]), "iv": [], "match": 0, "block": 0}) d["iv"].append((int(f[2]), int(f[3]))) d["match"] += int(f[9]) d["block"] += int(f[10]) rows = [] for tname, d in per.items(): covered = sum(e - s for s, e in _merge(d["iv"])) ident = 100.0 * d["match"] / d["block"] if d["block"] else 0.0 rows.append((covered * ident, tname, d["tlen"], round(ident, 2), round(100.0 * covered / L, 1))) rows.sort(reverse=True) meta = _pdb_meta(meta_path) tmp = atomic_path(out_tsv) with open(tmp, "w") as out: gc = round(100.0 * sum(seq.upper().count(b) for b in "GC") / L, 1) if L else 0 out.write("phagesdb_phage\\tlength_bp\\tlength_difference_bp\\tidentity_pct\\taligned_pct_of_genome" "\\tcluster\\thost\\tends\\tphagesdb_header\\ttemperate\\tgc_pct\\torfs\\ttrnas\\tterminal_repeat_bp" "\\tmorphotype\\taccession\\tgc_pct_this_genome\\n") for _, tid, tlen, ident, apct in rows[:10]: m = meta.get(tid, {}) out.write(f"{m.get('phage') or tid}\\t{tlen}\\t{L - tlen:+d}\\t{ident}\\t{apct}\\t{m.get('cluster', '')}" f"\\t{m.get('host', '')}\\t{m.get('ends', '')}\\t{m.get('header', '')}\\t{m.get('temperate', '')}" f"\\t{m.get('gc_pct', '')}\\t{m.get('orfs', '')}\\t{m.get('trnas', '')}\\t{m.get('term_rpt_len', '')}" f"\\t{m.get('morphotype', '')}\\t{m.get('accession', '')}\\t{gc}\\n") os.replace(tmp, out_tsv) ref_tsv = os.path.join(os.path.dirname(out_ref), "closest_ref.tsv") top_fa = os.path.join(os.path.dirname(out_ref), "closest_top.fasta") top_tsv = os.path.join(os.path.dirname(out_ref), "closest_top.tsv") for p in (out_ref, ref_tsv, top_fa, top_tsv, os.path.join(os.path.dirname(out_ref), "closest_alignment.paf")): if os.path.exists(p): os.remove(p) if not rows: print(" closest PhagesDB phage: none found") return _, best, blen, bident, bapct = rows[0] m = meta.get(best, {}) label = m.get("phage") or best extra = ", ".join(x for x in (f"Cluster {m['cluster']}" if m.get("cluster") else "", m.get("host", ""), m.get("ends", "")) if x) note = (f"{label} ({blen:,} bp{'; ' + extra if extra else ''}) — {bident}% identity over {bapct}% of the genome; " f"length difference {L - blen:+,} bp") print(f" closest PhagesDB phage: {note}") if len(rows) > 1: print(" next: " + "; ".join(f"{meta.get(x[1], {}).get('phage') or x[1]}" f"{' (' + meta[x[1]]['cluster'] + ')' if meta.get(x[1], {}).get('cluster') else ''}" f" {x[3]}%/{x[4]}%" for x in rows[1:4])) if m.get("cluster") and bapct >= 50: print(f" likely the same cluster as {label}: {m['cluster']} (tentative; PhagesDB assigns clusters)") facts = [] if m.get("temperate"): facts.append("temperate" if m["temperate"].lower() == "true" else "not temperate (lytic)" if m["temperate"].lower() == "false" else f"temperate: {m['temperate']}") if m.get("morphotype"): facts.append(m["morphotype"]) if m.get("gc_pct"): gc = round(100.0 * sum(seq.upper().count(b) for b in "GC") / L, 1) try: dgc = gc - float(m["gc_pct"]) facts.append(f"GC {m['gc_pct']}% (this genome {gc}%{'; differs by ' + format(dgc, '+.1f') + ' points — check' if abs(dgc) > 2 else ''})") except ValueError: pass if m.get("orfs"): facts.append(f"{m['orfs']} ORFs, {m.get('trnas') or 0} tRNAs") if m.get("term_rpt_len") and m["term_rpt_len"].isdigit() and int(m["term_rpt_len"]) > 0: facts.append(f"direct terminal repeats {int(m['term_rpt_len']):,} bp — this genome may need the repeat at both ends") if m.get("accession"): facts.append(f"GenBank {m['accession']}") if facts: print(f" {label}: " + "; ".join(facts)) rec = _fasta_record(pdb_fasta, best) if rec: targets = [(top_fa, top_tsv)] + ([(out_ref, ref_tsv)] if bapct >= 50 else []) for fa, kvp in targets: with open(fa, "w") as fh: write_fasta(fh, rec[0], rec[2]) with open(kvp, "w") as fh: for k in ("id", "phage", "host", "length_bp", "ends", "overhang", "cluster", "header", *PDB_TABLE_COLS): fh.write(f"{k}\\t{m.get(k, best if k == 'id' else '')}\\n") fh.write(f"identity_pct\\t{bident}\\naligned_pct\\t{bapct}\\n") def cmd_refstart(genome_fasta, ref_fasta, ref_meta, out_fasta, report_path, threads): """Rotate (and reverse-complement if needed) a circular genome to start where the reference starts. Exit 1 (with a note in the report) if the reference start can't be placed confidently.""" import subprocess import tempfile recs = read_fasta(genome_fasta) name, desc, seq = max(recs, key=lambda r: len(r[2])) ref = read_fasta(ref_fasta) rid, _, rseq = ref[0] rmeta = read_kv(ref_meta) if os.path.isfile(ref_meta) else {} rname = rmeta.get("phage") or rid overhang = str(rmeta.get("overhang") or "").upper() L = len(seq) anchor_len = min(3000, len(rseq)) def place(s): """Best placement of the reference's first bases on the circular sequence s (forward strand only).""" with tempfile.TemporaryDirectory() as tmp: tf, qf = os.path.join(tmp, "t.fa"), os.path.join(tmp, "q.fa") with open(tf, "w") as fh: write_fasta(fh, "t", s + s) with open(qf, "w") as fh: write_fasta(fh, "anchor", rseq[:anchor_len]) r = subprocess.run(["minimap2", "-c", "-x", "asm20", "-t", str(threads), tf, qf], capture_output=True, text=True) hits = [] for line in r.stdout.splitlines(): f = line.split("\\t") if len(f) >= 12: hits.append({"qs": int(f[2]), "qe": int(f[3]), "strand": f[4], "ts": int(f[7]), "te": int(f[8]), "match": int(f[9]), "block": int(f[10])}) return sorted(hits, key=lambda h: -h["match"]) def write_report(fields): with open(report_path, "w") as out: for k, v in fields.items(): out.write(f"{k}\\t{v}\\n") hits = place(seq) flipped = False if hits and hits[0]["strand"] == "-": seq = seq.translate(str.maketrans("ACGTacgtNn", "TGCAtgcaNn"))[::-1] flipped = True hits = place(seq) base = {"reference": rname, "reference_cluster": rmeta.get("cluster") or "NA", "reference_host": rmeta.get("host") or "NA", "reference_ends": rmeta.get("ends") or "NA", "reference_length": len(rseq), "genome_length": L, "anchor_bp": anchor_len} if not hits or hits[0]["strand"] != "+": write_report({**base, "note": f"the start of {rname} could not be found in the genome"}) return sys.exit(1) h = hits[0] ident = h["match"] / h["block"] if h["block"] else 0 # a second placement almost as good elsewhere = the start sits in a repeat; don't trust it others = [x for x in hits[1:] if x["strand"] == "+" and x["match"] >= 0.9 * h["match"] and min(abs((x["ts"] - h["ts"]) % L), abs((h["ts"] - x["ts"]) % L)) > 50] if h["block"] < 1000 or ident < 0.85 or h["qs"] > 200 or others: why = ("its start lies in a repeat" if others else f"its start aligns too weakly ({h['block']} bp at {100 * ident:.1f}% identity, offset {h['qs']} bp)") write_report({**base, "note": f"start of {rname} not placed confidently: {why}"}) return sys.exit(1) start0 = (h["ts"] - h["qs"]) % L # cos phages: PhagesDB genomes begin with the 3' overhang; put the start exactly on it when it is close by oh_check, oh_shift = "NA (the relative has no 3' overhang listed)", 0 if overhang and 4 <= len(overhang) <= 30: if rseq.upper().endswith(overhang) and not rseq.upper().startswith(overhang): # this PhagesDB entry carries the overhang at its right end: check our genome ends the same way tail = (seq[start0:] + seq[:start0]).upper() oh_check = ("overhang at the end, as in " + rname) if tail.endswith(overhang) else \\ f"not found: {rname} ends with {overhang}, this genome does not" elif not rseq.upper().startswith(overhang): oh_check = f"NA (the {rname} sequence does not begin or end with its overhang {overhang})" else: ring = (seq + seq).upper() near = sorted({((p - start0 + L // 2) % L) - L // 2 for p in range(len(seq)) if ring.startswith(overhang, p) and min((p - start0) % L, (start0 - p) % L) <= 100}) if len(near) == 1: oh_shift = near[0] oh_check = "overhang at the start" if oh_shift == 0 else f"start moved {oh_shift:+d} bp onto the overhang" elif len(near) > 1: oh_check = f"not found uniquely: {overhang} occurs {len(near)} times within 100 bp of the start" else: oh_check = f"not found: {overhang} is not within 100 bp of the start" start0 = (start0 + oh_shift) % L rotated = seq[start0:] + seq[:start0] orig_pos = (L - start0) if flipped else (start0 + 1) tmp = atomic_path(out_fasta) with open(tmp, "w") as out: write_fasta(out, name, rotated, (desc + f" start_matched_to={rname}" + (" reverse_complemented" if flipped else "")).strip()) os.replace(tmp, out_fasta) note = (f"rotated{' and reverse-complemented' if flipped else ''} to start where {rname}" f"{' (Cluster ' + rmeta['cluster'] + ')' if rmeta.get('cluster') else ''} starts " f"(anchor {h['block']:,} bp at {100 * ident:.1f}% identity" + (f", reference start {h['qs']} bp before the first aligned base" if h["qs"] else "") + ")" + (f"; 3' overhang {overhang}: {oh_check}" if overhang else "")) write_report({**base, "start_position": orig_pos, "strand": "minus" if flipped else "plus", "anchor_identity_pct": round(100 * ident, 2), "anchor_aligned_bp": h["block"], "reference_offset_bp": h["qs"], "length_difference_bp": f"{L - len(rseq):+d}", "overhang": overhang or "NA", "overhang_check": oh_check, "note": note}) print(f" {note}") def cmd_flyesummary(info_path): rows = sorted(_flye_info(info_path), key=lambda r: -r[1]) if not rows: print("assembled") return top = rows[0] print(f"{len(rows)} contig(s); longest {top[1]:,} bp{' circular' if top[3] else ''} at {top[2]:g}x") def cmd_flyeplan(total, gsize, tiers, done): """Read sets (label, bases) for the Flye votes: best reads at each depth; a depth above what's available becomes 'all'; under 30x is skipped; sets within 15% of one already planned or done are not repeated.""" total, gsize = int(total), int(gsize) have = [] for item in done.split(): lab, _, b = item.rpartition(":") if b.isdigit(): have.append((lab, int(b))) plan = [] for d in [int(x) for x in tiers.split() if x.isdigit()]: bases = d * gsize label = f"d{d}x" if bases >= total: bases, label = total, "all" if label != "all" and bases < 30 * gsize: continue near = [lab for lab, b in have + plan if abs(b - bases) <= 0.15 * max(b, bases)] if near: if near[0] not in [p[0] for p in plan]: plan.append((near[0], [b for lab, b in have if lab == near[0]][0])) continue plan.append((label, bases)) for lab, b in plan: print(f"{lab}\\t{b}") def cmd_pickflye(runs): """Choose a Flye run for the cross-check: the run with the most reads that has a circular contig of 5 kb or more; otherwise the run with the longest contig.""" best = None for name in os.listdir(runs): info = os.path.join(runs, name, "assembly_info.txt") if not os.path.isfile(os.path.join(runs, name, "assembly.fasta")): continue rows = _flye_info(info) if not rows: continue circ = any(r[3] and r[1] >= 5000 for r in rows) longest = max(r[1] for r in rows) try: with open(os.path.join(runs, name, "read_bases.txt")) as fh: nb = int(fh.read().strip() or 0) except (OSError, ValueError): nb = 0 key = (circ, nb, longest) if circ else (circ, longest, nb) if best is None or key > best[0]: best = (key, name) if not best: sys.exit(1) print(best[1]) def cmd_acvote(fasta, info_path, weight, out_fasta): """Copy a Flye assembly into Autocycler's assemblies folder with cluster/consensus weights in the headers.""" circ = {r[0]: r[3] for r in _flye_info(info_path)} recs = read_fasta(fasta) if not recs: sys.exit(1) w = int(weight) tmp = atomic_path(out_fasta) with open(tmp, "w") as out: for name, desc, seq in recs: tags = f"length={len(seq)}" + (" circular=true" if circ.get(name) else "") tags += f" Autocycler_cluster_weight={w} Autocycler_consensus_weight={w}" write_fasta(out, name, seq, tags) os.replace(tmp, out_fasta) def cmd_accircular(info_path): """Print yes if the longest contig in a Flye-style info table is circular.""" best = None if os.path.isfile(info_path): with open(info_path) as fh: for line in fh: if line.startswith("#") or not line.strip(): continue f = line.rstrip("\\n").split("\\t") if len(f) >= 4 and f[1].isdigit(): if best is None or int(f[1]) > best[0]: best = (int(f[1]), f[3] == "Y") print("yes" if best and best[1] else "no") def _mm2(args): import subprocess r = subprocess.run(["minimap2"] + args, capture_output=True, text=True) if r.returncode != 0: raise RuntimeError("minimap2 failed: " + r.stderr[-400:]) return [l.split("\\t") for l in r.stdout.splitlines() if l.strip()] def _merge(iv): out = [] for s, e in sorted(iv): if out and s <= out[-1][1]: out[-1][1] = max(out[-1][1], e) else: out.append([s, e]) return out def _depth_on(seq, reads, threads, tmp): """Per-base read depth on one sequence (reads mapped with minimap2 map-ont, primary alignments).""" ref = os.path.join(tmp, "dref.fa") with open(ref, "w") as fh: write_fasta(fh, "ref", seq) diff = [0] * (len(seq) + 1) for f in _mm2(["-x", "map-ont", "--secondary=no", "-t", str(threads), ref, reads]): if len(f) < 12: continue s, e = int(f[7]), int(f[8]) diff[s] += 1 diff[e] -= 1 out, run = [], 0 for i in range(len(seq)): run += diff[i] out.append(run) return out def cmd_pickdraft(ac_fasta, flye_fasta, reads, host_index, out_fasta, report_path, threads): """Choose between the Autocycler and Flye drafts of one phage. The longer genome is kept (it is more complete) unless its extra sequence matches the host genome. Read depth is reported and flagged but never removes sequence on its own: PCR and DNA modifications can lower depth over real phage DNA. Host-matching sequence at a contig end is trimmed.""" import tempfile ac = read_fasta(ac_fasta) fl = read_fasta(flye_fasta) if not ac or not fl: raise RuntimeError("an assembly is empty") a = max(ac, key=lambda r: len(r[2])) f = max(fl, key=lambda r: len(r[2])) flye_longer = len(f[2]) > len(a[2]) L, S = (f, a) if flye_longer else (a, f) Llab, Slab = ("Flye", "Autocycler") if flye_longer else ("Autocycler", "Flye") info = {"ac_length": len(a[2]), "flye_length": len(f[2])} flags = [] with tempfile.TemporaryDirectory() as tmp: ll = os.path.join(tmp, "ll.fa") sq = os.path.join(tmp, "s.fa") with open(ll, "w") as fh: write_fasta(fh, "L", L[2] + L[2]) with open(sq, "w") as fh: write_fasta(fh, "S", S[2]) n = len(L[2]) cov = [] q_al = 0 for h in _mm2(["-c", "-x", "asm5", "-t", str(threads), ll, sq]): if len(h) < 12 or not int(h[10]) or int(h[9]) / int(h[10]) < 0.9: continue q_al = max(q_al, int(h[3]) - int(h[2])) s, e = int(h[7]), int(h[8]) for x0, x1 in ((s, min(e, n)), (max(s, n) - n, e - n)): if x1 > x0: cov.append((x0, x1)) cov = _merge(cov) covered = [False] * n for s, e in cov: for i in range(s, min(e, n)): covered[i] = True runs, st = [], None for i in range(n + 1): u = i < n and not covered[i] if u and st is None: st = i elif not u and st is not None: if i - st >= 200: runs.append((st, i)) st = None extra = sum(e - s for s, e in runs) info["extra_bp"] = extra info["shorter_aligned_pct"] = round(100 * q_al / len(S[2]), 1) if S[2] else 0 def med_depth(depth, mask_val): vals = [depth[i] for i in range(n) if covered[i] == mask_val] return median(vals) if vals else None if info["shorter_aligned_pct"] < 50: # different sequences: keep the one with more read depth (usually the phage) dL = median(_depth_on(L[2], reads, threads, tmp)) dS = median(_depth_on(S[2], reads, threads, tmp)) keep_long = dL >= dS info["reason"] = (f"the two assemblies are different sequences; kept {Llab if keep_long else Slab} " f"(median depth {max(dL, dS):g}x vs {min(dL, dS):g}x) — check this sample, it may be mixed") flags.append("assemblies_disagree") else: host_bp = 0 if extra and host_index != "-" and os.path.isfile(host_index): ex = os.path.join(tmp, "extra.fa") with open(ex, "w") as fh: for k, (s, e) in enumerate(runs): write_fasta(fh, f"x{k}", L[2][s:e]) spans = {} for h in _mm2(["-x", "asm5", "-t", str(threads), host_index, ex]): if len(h) >= 12: spans.setdefault(h[0], []).append((int(h[2]), int(h[3]))) host_bp = sum(e - s for iv in spans.values() for s, e in _merge(iv)) info["extra_host_bp"] = host_bp if extra < 300: keep_long = not flye_longer # practically the same genome: prefer Autocycler info["reason"] = f"Flye agrees with Autocycler (difference {extra} bp)" elif host_bp >= 0.5 * extra: keep_long = False info["reason"] = f"{Llab}'s extra {extra:,} bp matches the host genome ({host_bp:,} bp); kept {Slab}" else: keep_long = True depth = _depth_on(L[2], reads, threads, tmp) de, ds = med_depth(depth, False), med_depth(depth, True) info["extra_depth"], info["shared_depth"] = de, ds info["reason"] = f"{Llab} is more complete ({extra:,} bp more); kept {Llab}" if de is not None and ds and de < 0.25 * ds: flags.append("low_depth_region") info["reason"] += (f"; the extra part has low read depth ({de:g}x vs {ds:g}x): PCR or DNA-modification " f"bias, a second element, or host DNA — check") chosen_is_flye = (keep_long and flye_longer) or (not keep_long and not flye_longer) recs = fl if chosen_is_flye else ac main = f if chosen_is_flye else a seq = main[2] trimmed = 0 if host_index != "-" and os.path.isfile(host_index): cq = os.path.join(tmp, "chosen.fa") with open(cq, "w") as fh: write_fasta(fh, "C", seq) iv = _merge([(int(h[2]), int(h[3])) for h in _mm2(["-x", "asm5", "-t", str(threads), host_index, cq]) if len(h) >= 12]) start, end = 0, len(seq) for s, e in iv: if e - s >= 1000 and s <= 500: start = max(start, e) if e - s >= 1000 and e >= len(seq) - 500: end = min(end, s) if end - start >= 1000 and (start > 0 or end < len(seq)): trimmed = len(seq) - (end - start) seq = seq[start:end] flags.append("host_end_trimmed") info["reason"] += f"; trimmed {trimmed:,} bp of host DNA from the contig end(s)" info["trimmed_bp"] = trimmed tmpf = atomic_path(out_fasta) with open(tmpf, "w") as out: for name, desc, s in recs: if name == main[0]: write_fasta(out, name, seq, desc + (f" trimmed_host_bp={trimmed}" if trimmed else "")) else: write_fasta(out, name, s, desc) os.replace(tmpf, out_fasta) method = "flye_chosen" if chosen_is_flye else "autocycler" with open(report_path, "w") as out: out.write(f"method\\t{method}\\nreason\\t{info['reason']}\\nflags\\t{','.join(flags) or 'none'}\\n") for k, v in info.items(): if k != "reason": out.write(f"{k}\\t{v}\\n") print(f" kept {'Flye' if chosen_is_flye else 'Autocycler'}: {info['reason']}") def cmd_dips(cov_tsv, contig, out_tsv, frac): """Regions of the longest contig whose read depth is below frac x the contig's median (1 kb windows, ignoring the first and last 2 kb).""" frac = float(frac) depth = [] with open_text(cov_tsv) as fh: next(fh, None) for line in fh: f = line.split("\\t", 4) if f[0] == contig: depth.append(int(f[3])) elif depth: break with open(out_tsv, "w") as out: if len(depth) < 6000: out.write("regions\\t0\\n") return gmed = median(depth) win, step_ = 1000, 500 low = [] edge = 2000 # depth falls near the ends of a linearised contig because reads can't span the cut for s in range(edge, len(depth) - win - edge + 1, step_): m = median(depth[s:s + win]) if gmed and m < frac * gmed: low.append([s, s + win, m]) merged = [] for s, e, m in low: if merged and s <= merged[-1][1]: merged[-1][1] = e merged[-1][2] = min(merged[-1][2], m) else: merged.append([s, e, m]) out.write(f"regions\\t{len(merged)}\\ngenome_median_depth\\t{gmed:g}\\n") for s, e, m in merged: out.write(f"region\\t{s + 1}-{e} (min window median {m:g}x)\\n") if merged: print(f" {len(merged)} region(s) below {frac:.0%} of the median depth ({gmed:g}x): " + ", ".join(f"{s + 1:,}-{e:,}" for s, e, _ in merged[:6]) + (" …" if len(merged) > 6 else "")) def cmd_crosscheck(list_tsv, out_tsv, threads): """v17: compares the assemblies of this run's samples (all contigs of 2 kb or more): (1) samples whose large contigs (10 kb+) are near-identical (>= 98%): the same phage under two names or a mix-up (2) small contigs (< 20 kb) found in two or more samples: host plasmids, other shared elements or contamination""" import tempfile items = [] with open(list_tsv) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) == 2 and os.path.isfile(f[1]): for name, _, seq in read_fasta(f[1]): if len(seq) >= 2000: items.append((f[0], name, seq)) samples = sorted({x[0] for x in items}) if len(samples) < 2: return length = {(x[0], x[1]): len(x[2]) for x in items} big_total = {sm: sum(v for (a, _), v in length.items() if a == sm and v >= 10000) for sm in samples} cover, ident = {}, {} with tempfile.TemporaryDirectory() as tmp: allfa = os.path.join(tmp, "all.fa") idx = {} with open(allfa, "w") as fh: for i, (sm, c, q) in enumerate(items): idx[f"s{i}"] = (sm, c) write_fasta(fh, f"s{i}", q) for h in _mm2(["-x", "asm5", "-c", "-X", "-t", str(threads), allfa, allfa]): if len(h) < 12: continue a, b = idx.get(h[0]), idx.get(h[5]) if not a or not b or a[0] == b[0]: continue blk, mat = int(h[10]), int(h[9]) if blk < 1000 or mat / blk < 0.98: continue cover.setdefault((a, b[0]), []).append([int(h[2]), int(h[3])]) cover.setdefault((b, a[0]), []).append([int(h[7]), int(h[8])]) if mat / blk >= 0.995: # shared elements (plasmid copies) are essentially identical; related phages are not ident.setdefault((a, b[0]), []).append([int(h[2]), int(h[3])]) ident.setdefault((b, a[0]), []).append([int(h[7]), int(h[8])]) def covered(iv): return sum(e - st for st, e in _merge(iv)) def stem(x): return re.split(r"[-_]", x.lower())[0] rows, same, shared = [], [], {} for i, x in enumerate(samples): for y in samples[i + 1:]: small, other = (x, y) if big_total[x] <= big_total[y] else (y, x) if big_total[small] < 10000: continue bp = sum(covered(iv) for (a, o), iv in cover.items() if a[0] == small and o == other and length[a] >= 10000) frac = bp / big_total[small] if frac >= 0.5: expected = stem(x) == stem(y) and x.lower() != y.lower() note = ("same phage, different prep (expected)" if expected else "CHECK: the same phage under two names, or a sample mix-up") same.append((x, y, small, bp, frac, expected)) rows.append(["same_phage", x, "-", big_total[small], y, bp, round(100 * frac), note]) same_pairs = {frozenset((t[0], t[1])) for t in same} for (a, o), iv in ident.items(): # parts of two samples that are the same phage (already reported as a pair) are not "elements" if length[a] < 20000 and covered(iv) >= 0.9 * length[a] and frozenset((a[0], o)) not in same_pairs: shared.setdefault(a, set()).add(o) for a, others in sorted(shared.items()): rows.append(["shared_element", a[0], a[1], length[a], ",".join(sorted(others)), length[a], 100, "likely a host plasmid, another shared element or contamination - not this phage's genome"]) with open(out_tsv, "w") as out: out.write("kind\\tsample\\tcontig\\tlength_bp\\tother_samples\\tshared_bp\\tshared_pct\\tnote\\n") for r in rows: out.write("\\t".join(str(v) for v in r) + "\\n") odd = [t for t in same if not t[5]] if odd: print(f"\\n⚠️ {len(odd)} pair(s) of samples have near-identical genomes (>= 98% identity over at least half of " f"the large contigs): the same phage under two names, or a sample mix-up. Details: {out_tsv}") for x, y, small, bp, frac, _ in odd[:15]: print(f" {x} ↔ {y}: {round(100 * frac)}% of {small}'s large contigs ({bp:,} bp)") if shared: print(f"\\nℹ️ {len(shared)} small contig(s) also occur in other samples (likely host plasmids or other shared " f"elements, not phage genomes; check with BLAST). Details: {out_tsv}") for a, others in sorted(shared.items())[:15]: print(f" {a[0]} {a[1]} ({length[a]:,} bp) also in: {', '.join(sorted(others))}") def cmd_timings(folder, out_tsv): """Seconds spent per step for every sample, from the per-sample step files; prints the median per step.""" per = {} order = [] for fn in sorted(os.listdir(folder)): if not fn.endswith(".tsv"): continue marks = [] with open(os.path.join(folder, fn)) as fh: for line in fh: f = line.rstrip("\\n").split("\\t") if len(f) >= 2 and f[0].isdigit(): marks.append((int(f[0]), f[1])) if len(marks) < 2: continue sample = fn[:-4] d = {} for (t0, k), (t1, _) in zip(marks, marks[1:]): d[k] = d.get(k, 0) + (t1 - t0) if k not in order: order.append(k) d["total"] = marks[-1][0] - marks[0][0] per[sample] = d if not per: return cols = [k for k in order if k not in ("done",)] + ["total"] with open(out_tsv, "w") as out: out.write("sample\\t" + "\\t".join(c + "_s" for c in cols) + "\\n") for s, d in per.items(): out.write(s + "\\t" + "\\t".join(str(d.get(c, "")) for c in cols) + "\\n") print(f"\\n⏱ Time per step, median over {len(per)} sample(s) (full table: {out_tsv}):") for c in cols: vals = [d[c] for d in per.values() if c in d] if vals: m = median(vals) fmt = lambda x: f"{x:4.0f} s " if x < 120 else f"{x / 60:4.1f} min" print(f" {c:<16} {fmt(m)} (max {fmt(max(vals)).strip()}, {len(vals)} sample(s))") COMMANDS = { "dedup": (cmd_dedup, 3), "stats": (cmd_stats, 1), "compare": (cmd_compare, 4), "finalize": (cmd_finalize, 4), "longest": (cmd_longest, 5), "gbkstats": (cmd_gbkstats, 1), "pileupstats": (cmd_pileupstats, 4), "report": (cmd_report, 11), "readstats": (cmd_readstats, 2), "cpinject": (cmd_cpinject, 2), "readsets": (cmd_readsets, 2), "bundle": (cmd_bundle, 4), "sheetmatch": (cmd_sheetmatch, 4), "acinfo": (cmd_acinfo, 3), "readends": (cmd_readends, 3), "rotate": (cmd_rotate, 3), "relabel": (cmd_relabel, 3), "shiftchanges": (cmd_shiftchanges, 5), "replacecontig": (cmd_replacecontig, 3), "codingqc": (cmd_codingqc, 3), "sampleflags": (cmd_sampleflags, 5), "phagecontigs": (cmd_phagecontigs, 5), "hostfilter": (cmd_hostfilter, 6), "hostremove": (cmd_hostremove, 7), "acverdict": (cmd_acverdict, 3), "timings": (cmd_timings, 2), "accircular": (cmd_accircular, 1), "flyegsize": (cmd_flyegsize, 2), "annotpick": (cmd_annotpick, 9), "acvote": (cmd_acvote, 4), "flyesummary": (cmd_flyesummary, 1), "closestphage": (cmd_closestphage, 7), "refstart": (cmd_refstart, 6), "pdbprep": (cmd_pdbprep, 4), "flyeplan": (cmd_flyeplan, 4), "pickflye": (cmd_pickflye, 1), "pickdraft": (cmd_pickdraft, 7), "dips": (cmd_dips, 4), "crosscheck": (cmd_crosscheck, 3), "inputcheck": (cmd_inputcheck, 6), "failreport": (cmd_failreport, 1), } if __name__ == "__main__": if len(sys.argv) < 2 or sys.argv[1] not in COMMANDS: sys.exit(__doc__) func, nargs = COMMANDS[sys.argv[1]] args = sys.argv[2:] if len(args) != nargs: sys.exit(f"{sys.argv[1]} expects {nargs} arguments, got {len(args)}") func(*args) PHAGE_TOOLS_PY # ============================================================== # Build read-filter and assembler arguments # ============================================================== # ---- library kit: transposase kits cut at random, so read ends don't mark genome ends ---- case "\${LIBRARY_KIT:-ligation}" in rapid_pcr|rapid) if is_true "$FIND_ENDS"; then echo "ℹ️ \${LIBRARY_KIT} library (transposase): genome-end detection from read pile-ups is turned off."; fi FIND_ENDS=false if [[ "$ROTATE" == "ends_terl" ]]; then ROTATE="terl"; fi ;; esac # ---- minimap2 for the assembly checks (Autocycler vs Flye, host match, shared sequences) ---- MM2_ENV="" for e in "$MEDAKA_ENV" "$CORE_ENV" "\${AUTOCYCLER_ENV:-}"; do [[ -n "$e" ]] && env_present "$CONDA_BASE" "$e" || continue use_env "$e" if command -v minimap2 >/dev/null 2>&1; then MM2_ENV="$e"; break; fi done use_env "$CORE_ENV" if [[ -z "$MM2_ENV" ]]; then warn "minimap2 was not found; Autocycler vs Flye cross-checks and the shared-sequence check are off." AC_CROSSCHECK=false fi # Host genome(s) are also used to recognise host DNA joined to a contig, even when host-read removal is off. if [[ -z "$HOST_INDEX" && "$HOST_METHOD" == "ref" && -n "$MM2_ENV" ]]; then mkdir -p "$TOOLS_DIR" HOST_FASTA="$TOOLS_DIR/host_genomes.fasta" HOST_LABELS="$TOOLS_DIR/host_genomes.tsv" : > "$HOST_FASTA" printf 'tag\\tname\\tprophage\\tfasta\\n' > "$HOST_LABELS" for i in 1 2 3; do hg_var="HOST_GENOME_$i"; hn_var="HOST_NAME_$i"; hp_var="HOST_PROPHAGE_$i" hg="\${!hg_var:-}"; hn="\${!hn_var:-host$i}"; hp="\${!hp_var:-unsure}" [[ -n "$hg" && -s "$hg" ]] || continue gzip -cdf -- "$hg" | sed "s/^>/>H\${i}__/" >> "$HOST_FASTA" printf 'H%s\\t%s\\t%s\\t%s\\n' "$i" "$hn" "$hp" "$hg" >> "$HOST_LABELS" done if [[ -s "$HOST_FASTA" ]]; then use_env "$MM2_ENV" if minimap2 -x asm5 -d "$TOOLS_DIR/host_genomes.asm5.mmi" "$HOST_FASTA" > "$TOOLS_DIR/host_index.log" 2>&1; then HOST_INDEX="$TOOLS_DIR/host_genomes.asm5.mmi" echo "Host genome(s) indexed for the contig checks." fi use_env "$CORE_ENV" fi rm -f -- "$HOST_FASTA" fi # ---- PhagesDB genomes: closest phage for each genome, and the reference start ---- PDB_INDEX="" if [[ -n "\${PHAGESDB_FASTA:-}" ]] && [[ "$CHECK_ONLY" != "true" ]]; then if [[ ! -s "$PHAGESDB_FASTA" ]]; then warn "PhagesDB FASTA not found: $PHAGESDB_FASTA — no closest-phage search; genome start falls back to terL." elif [[ -z "$MM2_ENV" ]]; then warn "minimap2 was not found, so the PhagesDB search is off; genome start falls back to terL." else pdb_dir="$(dirname "$PHAGESDB_FASTA")/phagesdb_index" pdb_sig="$(stat -c '%s_%Y' "$PHAGESDB_FASTA")" pdb_stem="$pdb_dir/$(basename "$PHAGESDB_FASTA").\${pdb_sig}.v9" PDB_INDEX="$pdb_stem.asm20.mmi" PDB_IDS="$pdb_stem.ids.fasta" # same genomes, each named by a unique ID (the phage name) # the PhagesDB data table (optional): exact path, or the same name with .txt / .tsv / .tab PDB_TABLE="-" for cand in "\${PHAGESDB_TABLE:-}" "\${PHAGESDB_TABLE:-}.txt" "\${PHAGESDB_TABLE:-}.tsv" "\${PHAGESDB_TABLE:-}.tab"; do [[ -n "\${PHAGESDB_TABLE:-}" && -f "$cand" && -s "$cand" ]] && { PDB_TABLE="$cand"; break; } done if [[ -n "\${PHAGESDB_TABLE:-}" && "$PDB_TABLE" == "-" ]]; then warn "PhagesDB data table not found: $PHAGESDB_TABLE (also tried .txt/.tsv/.tab); using the FASTA headers only." fi tbl_sig="none"; [[ "$PDB_TABLE" != "-" ]] && tbl_sig="$(stat -c '%s_%Y' "$PDB_TABLE")" PDB_META="$pdb_stem.table_\${tbl_sig}.meta.tsv" # ID, phage, host, length, ends, overhang, cluster, header, + table fields mkdir -p "$pdb_dir" if [[ ! -s "$PDB_INDEX" || ! -s "$PDB_IDS" || ! -s "$PDB_META" ]]; then echo "Preparing the PhagesDB genomes$( [[ "$PDB_TABLE" != "-" ]] && echo " and data table") (once per downloaded file; a few minutes)…" use_env "$CORE_ENV" if python3 "$TOOLS" pdbprep "$PHAGESDB_FASTA" "$PDB_TABLE" "$PDB_IDS.part" "$PDB_META.part" && mv -- "$PDB_IDS.part" "$PDB_IDS" \\ && mv -- "$PDB_META.part" "$PDB_META"; then use_env "$MM2_ENV" if [[ -s "$PDB_INDEX" ]] || { minimap2 -x asm20 -t "$FLYE_THREADS" -d "$PDB_INDEX.part" "$PDB_IDS" > "$pdb_dir/index.log" 2>&1 \\ && mv -- "$PDB_INDEX.part" "$PDB_INDEX"; }; then find "$pdb_dir" -maxdepth 1 -type f ! -name "$(basename "$pdb_stem").asm20.mmi" ! -name "$(basename "$PDB_IDS")" \\ ! -name "$(basename "$PDB_META")" ! -name index.log -delete 2>/dev/null || true else rm -f -- "$PDB_INDEX.part" warn "Indexing the PhagesDB FASTA failed (see $pdb_dir/index.log); no closest-phage search." PDB_INDEX="" fi else rm -f -- "$PDB_IDS.part" "$PDB_META.part" warn "Could not read the PhagesDB FASTA $PHAGESDB_FASTA; no closest-phage search." PDB_INDEX="" fi use_env "$CORE_ENV" fi if [[ -n "$PDB_INDEX" ]]; then echo "PhagesDB genomes: $(( $(wc -l < "$PDB_META") - 1 )) from $PHAGESDB_FASTA" echo " with a cluster: $(awk -F'\\t' 'NR > 1 && $7 != ""' "$PDB_META" | wc -l); with a 3' overhang sequence: $(awk -F'\\t' 'NR > 1 && $6 != ""' "$PDB_META" | wc -l)$( [[ "$PDB_TABLE" != "-" ]] && echo "; matched to the data table: $(awk -F'\\t' 'NR > 1 && $9 == "yes"' "$PDB_META" | wc -l)")" fi fi fi if [[ "$ROTATE" == "reference" && -z "$PDB_INDEX" ]]; then ROTATE="terl"; fi # ---- several phages at once: share the threads between them ---- PARALLEL_SAMPLES="\${PARALLEL_SAMPLES:-1}" (( PARALLEL_SAMPLES >= 1 )) || PARALLEL_SAMPLES=1 if (( PARALLEL_SAMPLES > 1 )); then per_job=$(( FLYE_THREADS / PARALLEL_SAMPLES )) (( per_job >= 1 )) || per_job=1 FLYE_THREADS="$per_job" (( PHAROKKA_THREADS > per_job )) && PHAROKKA_THREADS="$per_job" (( NANO_THREADS > per_job )) && NANO_THREADS="$per_job" if ! command -v flock >/dev/null 2>&1; then warn "flock was not found; Medaka/Phold may use the GPU at the same time for several phages." fi echo "Running $PARALLEL_SAMPLES phages at once with $per_job thread(s) each (Pharokka $PHAROKKA_THREADS, NanoPlot $NANO_THREADS)." fi # gpu_run CMD...: one phage at a time on the GPU when several phages run at once # time_limited CMD...: run CMD with the per-assembly time limit (the whole process group is stopped; 124 = timed out) time_limited() { if (( \${AC_TIMEOUT_MIN:-0} > 0 )) && command -v timeout >/dev/null 2>&1; then timeout --kill-after=60 "$(( AC_TIMEOUT_MIN * 60 ))" "$@" else "$@" fi } gpu_run() { if (( PARALLEL_SAMPLES > 1 )) && command -v flock >/dev/null 2>&1; then flock "$TOOLS_DIR/gpu.lock" "$@" else "$@" fi } FILTLONG_ARGS=() if (( MIN_LENGTH > 0 )); then FILTLONG_ARGS+=(--min_length "$MIN_LENGTH"); fi if [[ -n "$MIN_MEAN_Q" ]]; then FILTLONG_ARGS+=(--min_mean_q "$MIN_MEAN_Q"); fi if [[ "$ASSEMBLY_MODE" == "accuracy" ]]; then # Autocycler makes its own read subsets, so only drop short and low-quality reads here. if (( TARGET_BASES > 0 )) || [[ -n "$KEEP_PERCENT" ]]; then echo "ℹ️ Accuracy mode: Filtlong target bases / keep percent are not used (read depth is set by 'Maximum read depth for Autocycler')." fi else if (( TARGET_BASES > 0 )); then FILTLONG_ARGS+=(--target_bases "$TARGET_BASES"); fi if [[ -n "$KEEP_PERCENT" ]]; then FILTLONG_ARGS+=(--keep_percent "$KEEP_PERCENT"); fi fi FLYE_ARGS=(--threads "$FLYE_THREADS" --iterations "$FLYE_ITERATIONS") if [[ -n "$GENOME_SIZE" ]]; then FLYE_ARGS+=(--genome-size "$GENOME_SIZE"); fi if (( MIN_OVERLAP > 0 )); then FLYE_ARGS+=(--min-overlap "$MIN_OVERLAP"); fi if (( ASM_COVERAGE > 0 )); then FLYE_ARGS+=(--asm-coverage "$ASM_COVERAGE"); fi if is_true "$FLYE_META"; then FLYE_ARGS+=(--meta); fi if is_true "$FLYE_SCAFFOLD"; then FLYE_ARGS+=(--scaffold); fi MEDAKA_ARGS=(-t "$MEDAKA_THREADS") if [[ -n "$MEDAKA_MODEL" ]]; then MEDAKA_ARGS+=(-m "$MEDAKA_MODEL"); fi if is_true "$MEDAKA_BACTERIA"; then MEDAKA_ARGS+=(--bacteria); fi # ============================================================== # Accuracy mode: Autocycler consensus of several assemblers # autocycler_assemble SAMPLE READS AC_DIR QC_DIR # ============================================================== autocycler_assemble() { local sample="$1" reads="$2" acdir="$3" qcdir="$4" mkdir -p "$acdir/assemblies" "$acdir/logs" use_env "$AUTOCYCLER_ENV" local gsize="$AC_GENOME_SIZE" if [[ "$gsize" == "flye" ]]; then if [[ -n "\${SAMPLE_FLYE_GSIZE:-}" ]]; then gsize="$SAMPLE_FLYE_GSIZE" echo " genome size from the first Flye assembly: $gsize bp" else echo " no usable Flye assembly for the genome size; using Autocycler's estimate" gsize="auto" fi fi if [[ "$gsize" == "auto" ]]; then step ac_genome_size "[3/5] Autocycler: estimating genome size" # v24: estimate from the best 15 Mb only. The estimate assembles ALL the reads it is given (Raven), and a small phage # with very deep data (e.g. 180 Mb for a 16 kb genome, >10,000x) can use up all of WSL's memory and stop the run. local est_reads="$reads" est_have est_have="$(gzip -cd -- "$reads" | awk 'NR % 4 == 2 { n += length($0) } END { print n + 0 }')" if (( est_have > 15000000 )); then use_env "$CORE_ENV" est_reads="$acdir/reads_for_genome_size.fastq.gz" filtlong --target_bases 15000000 "$reads" 2> "$acdir/logs/filtlong_genome_size.log" | gzip -c > "$est_reads" use_env "$AUTOCYCLER_ENV" echo " estimating from the best 15 Mb of $(( est_have / 1000000 )) Mb" fi gsize="$(autocycler helper genome_size --reads "$est_reads" --threads "$FLYE_THREADS" 2> "$acdir/logs/genome_size.log" | tail -n 1 | tr -d '[:space:]')" || gsize="" if ! [[ "$gsize" =~ ^[0-9.]+[kKmMgG]?[bB]?$ ]]; then if [[ -n "$GENOME_SIZE" ]]; then warn "Autocycler could not estimate the genome size (see $acdir/logs/genome_size.log); using $GENOME_SIZE from the settings." gsize="$GENOME_SIZE" else fail "Autocycler could not estimate the genome size (see $acdir/logs/genome_size.log)." fi fi fi # accept 50k / 0.05m style sizes gsize="$(python3 -c 'import sys v=sys.argv[1].lower().rstrip("b"); m={"k":1e3,"m":1e6,"g":1e9} print(int(float(v[:-1])*m[v[-1]]) if v and v[-1] in m else int(float(v)))' "$gsize" 2>/dev/null)" \\ || fail "Could not work out the genome size (see $acdir/logs/genome_size.log)." [[ "$gsize" =~ ^[0-9]+$ ]] && (( gsize > 1000 )) || fail "Genome size estimate looks wrong: '$gsize' (see $acdir/logs/genome_size.log)" # v16: an estimate (not a typed number) below AC_MIN_GSIZE is raised to it local gsize_note="" if [[ ! "$AC_GENOME_SIZE" =~ ^[0-9.]+[kKmM]?$ ]] && (( \${AC_MIN_GSIZE:-0} > 0 && gsize < \${AC_MIN_GSIZE:-0} )); then gsize_note=" (estimate $gsize bp raised to the minimum)" echo " genome size estimate $gsize bp is below \${AC_MIN_GSIZE} bp (a high-depth repeat or over-amplified region?); using \${AC_MIN_GSIZE} bp" gsize="\${AC_MIN_GSIZE}" fi echo " genome size: $gsize bp$gsize_note" printf 'genome_size\\t%s\\ngenome_size_source\\t%s\\n' "$gsize" "$( [[ "$AC_GENOME_SIZE" == flye && -n "\${SAMPLE_FLYE_GSIZE:-}" ]] && echo flye || echo "$AC_GENOME_SIZE")" > "$qcdir/autocycler_settings.txt" # Very deep phage data slows assemblers and adds nothing. v16: keep the best reads up to a read budget # (AC_MAX_MB, default 15 Mb) — independent of the genome-size estimate; or, with AC_MAX_MB 0, up to AC_MAX_DEPTH. local ac_reads="$reads" cap=0 cap_label="" have have="$(gzip -cd -- "$reads" | awk 'NR % 4 == 2 { n += length($0) } END { print n + 0 }')" local ac_mb_bp ac_mb_bp="$(awk -v m="\${AC_MAX_MB:-0}" 'BEGIN { printf "%d", m * 1000000 }')" if (( ac_mb_bp > 0 )); then cap=$ac_mb_bp; cap_label="\${AC_MAX_MB} Mb" elif (( AC_MAX_DEPTH > 0 )); then cap=$(( gsize * AC_MAX_DEPTH )); cap_label="\${AC_MAX_DEPTH}x" fi if (( cap > 0 && have > cap )); then echo " reads: $have bases ($(( have / gsize ))x of $gsize bp); keeping the best $cap_label for Autocycler" use_env "$CORE_ENV" ac_reads="$acdir/reads_capped.fastq.gz" filtlong --target_bases "$cap" "$reads" 2> "$acdir/logs/filtlong_cap.log" | gzip -c > "$ac_reads" use_env "$AUTOCYCLER_ENV" else echo " reads: $have bases ($(( have / gsize ))x of $gsize bp); all used" fi printf 'max_mb\\t%s\\nmax_depth\\t%s\\nreads_available_bp\\t%s\\nread_cap_bp\\t%s\\nassemblers\\t%s\\nsubsets\\t%s\\n' \\ "\${AC_MAX_MB:-0}" "$AC_MAX_DEPTH" "$have" "$cap" "\${AC_USE[*]}" "$AC_COUNT" >> "$qcdir/autocycler_settings.txt" step ac_subsets "[3/5] Autocycler: making $AC_COUNT read subsets" autocycler subsample --reads "$ac_reads" --out_dir "$acdir/subsampled_reads" --genome_size "$gsize" \\ --count "$AC_COUNT" > "$acdir/logs/subsample.log" 2>&1 \\ || { tail -n 20 "$acdir/logs/subsample.log" >&2; fail "autocycler subsample failed (see $acdir/logs/subsample.log)"; } local i asm n_ok=0 n_try=0 sub step ac_assemblies "[3/5] Autocycler: assembling the subsets" for asm in "\${AC_USE[@]}"; do for sub in "$acdir"/subsampled_reads/sample_*.fastq; do [[ -s "$sub" ]] || continue i="$(basename "$sub" .fastq)"; i="\${i#sample_}" n_try=$((n_try + 1)) echo " assembling subset $i with $asm" local h_status=0 time_limited autocycler helper "$asm" --reads "$sub" --out_prefix "$acdir/assemblies/\${asm}_\${i}" \\ --threads "$FLYE_THREADS" --genome_size "$gsize" --read_type "$AC_READ_TYPE" \\ > "$acdir/logs/\${asm}_\${i}.log" 2>&1 || h_status=$? if (( h_status == 0 )) && [[ -s "$acdir/assemblies/\${asm}_\${i}.fasta" ]]; then n_ok=$((n_ok + 1)) elif (( h_status == 124 || h_status == 137 )); then warn "$asm timed out on subset $i after \${AC_TIMEOUT_MIN} min and was stopped; continuing without it." rm -f -- "$acdir/assemblies/\${asm}_\${i}".* else warn "$asm failed on subset $i (see $acdir/logs/\${asm}_\${i}.log); continuing without it." rm -f -- "$acdir/assemblies/\${asm}_\${i}".* fi done done (( n_ok >= 2 )) || fail "Only $n_ok of $n_try assemblies finished; Autocycler needs at least 2." echo " $n_ok of $n_try assemblies finished" if is_true "$AC_FLYE_VOTE" && (( \${#SAMPLE_FLYE_VOTES[@]} > 0 )); then use_env "$CORE_ENV" local v vl vf vi nv=0 for v in "\${SAMPLE_FLYE_VOTES[@]}"; do IFS='|' read -r vl vf vi <<< "$v" [[ -s "$vf" ]] || continue if python3 "$TOOLS" acvote "$vf" "$vi" "$AC_FLYE_WEIGHT" "$acdir/assemblies/flyevote_\${vl}.fasta"; then nv=$((nv + 1)) else warn "Could not add the Flye $vl assembly to Autocycler; continuing without it."; fi done (( nv > 0 )) && echo " + $nv Flye vote(s) ($(printf '%s ' "\${SAMPLE_FLYE_VOTES[@]%%|*}"| sed 's/ $//'); weight $AC_FLYE_WEIGHT each)" use_env "$AUTOCYCLER_ENV" fi rm -rf -- "$acdir/subsampled_reads" step ac_consensus "[3/5] Autocycler: compress → cluster → trim → resolve → combine" autocycler compress -i "$acdir/assemblies" -a "$acdir/autocycler_out" --threads "$FLYE_THREADS" \\ > "$acdir/logs/compress.log" 2>&1 || { tail -n 20 "$acdir/logs/compress.log" >&2; fail "autocycler compress failed"; } autocycler cluster -a "$acdir/autocycler_out" > "$acdir/logs/cluster.log" 2>&1 \\ || { tail -n 20 "$acdir/logs/cluster.log" >&2; fail "autocycler cluster failed"; } local c finals=() for c in "$acdir"/autocycler_out/clustering/qc_pass/cluster_*; do [[ -d "$c" ]] || continue autocycler trim -c "$c" --threads "$FLYE_THREADS" > "$acdir/logs/trim_$(basename "$c").log" 2>&1 \\ || { warn "autocycler trim failed for $(basename "$c")"; continue; } autocycler resolve -c "$c" > "$acdir/logs/resolve_$(basename "$c").log" 2>&1 \\ || { warn "autocycler resolve failed for $(basename "$c")"; continue; } [[ -s "$c/5_final.gfa" ]] && finals+=("$c/5_final.gfa") done (( \${#finals[@]} > 0 )) || fail "Autocycler found no cluster that passed QC (assemblies disagree). See $acdir/logs/cluster.log; Standard mode may still give a draft." autocycler combine -a "$acdir/autocycler_out" -i "\${finals[@]}" > "$acdir/logs/combine.log" 2>&1 \\ || { tail -n 20 "$acdir/logs/combine.log" >&2; fail "autocycler combine failed"; } [[ -s "$acdir/autocycler_out/consensus_assembly.fasta" ]] || fail "Autocycler did not produce consensus_assembly.fasta." use_env "$CORE_ENV" local gfa="$acdir/autocycler_out/consensus_assembly.gfa" [[ -s "$gfa" ]] || gfa="-" python3 "$TOOLS" acinfo "$acdir/autocycler_out/consensus_assembly.fasta" "$gfa" "$qcdir/contig_info.txt" # the per-assembly copies are large and no longer needed; keep the logs and consensus if ! is_true "$AC_KEEP_ASSEMBLIES"; then rm -rf -- "$acdir/assemblies"; fi } # ============================================================== # Genome ends (read start/stop pile-ups) and rotation of circular genomes # finish_genome SAMPLE SAMPLE_DIR QC_DIR FINAL LONGEST FILTERED_READS UNIQUE_READS # ============================================================== finish_genome() { local sample="$1" sdir="$2" qcdir="$3" final="$4" longest="$5" freads="$6" ureads="$7" [[ -s "$longest" ]] || return 0 local reads="$freads" [[ -s "$reads" ]] || reads="$ureads" if is_true "$FIND_ENDS" && [[ ! -s "$qcdir/genome_ends.txt" && -s "$reads" ]]; then step genome_ends "🔚 Looking for physical genome ends (read start/stop pile-ups)" use_env "$ENDS_ENV" minimap2 -ax map-ont --secondary=no -t "$FLYE_THREADS" "$longest" "$reads" 2> "$qcdir/ends_minimap2.log" \\ | samtools view -F 0x904 - 2>> "$qcdir/ends_minimap2.log" \\ | python3 "$TOOLS" readends "$longest" - "$qcdir/genome_ends.txt" \\ || warn "Genome-end detection failed for $sample; continuing." use_env "$CORE_ENV" fi # closest PhagesDB phages (every genome, circular or not); one search at a time keeps memory low if [[ -n "$PDB_INDEX" && ! -s "$qcdir/closest_phagesdb.tsv" ]]; then step closest_phage "🔎 Closest PhagesDB phage" use_env "$MM2_ENV" if flock "$TOOLS_DIR/phagesdb.lock" python3 "$TOOLS" closestphage "$longest" "$PDB_INDEX" "$PDB_IDS" "$PDB_META" \\ "$qcdir/closest_phagesdb.tsv" "$qcdir/closest_ref.fasta" "$FLYE_THREADS"; then :; else warn "PhagesDB search failed for $sample; continuing." fi use_env "$CORE_ENV" fi [[ "$ASSEMBLY_MODE" == "accuracy" && "$ROTATE" != "none" ]] || return 0 [[ -s "$qcdir/rotation.txt" ]] && return 0 local circ end_found end_pos contig circ="$(kv "$qcdir/longest_contig.tsv" flye_circular)" contig="$(kv "$qcdir/longest_contig.tsv" longest_contig)" end_found="$(kv "$qcdir/genome_ends.txt" end_detected)" end_pos="$(kv "$qcdir/genome_ends.txt" end_position)" local by="none" pos="NA" note="" cp -- "$longest" "$qcdir/longest_unrotated.fasta" # start where the closest PhagesDB phage starts local ref_ok=false ref_note="" if [[ "$circ" == "Y" && "$ROTATE" == "reference" ]]; then if [[ -s "$qcdir/closest_ref.fasta" ]]; then step rotate "🔄 Starting the genome where its closest PhagesDB phage starts" use_env "$MM2_ENV" if python3 "$TOOLS" refstart "$longest" "$qcdir/closest_ref.fasta" "$qcdir/closest_ref.tsv" \\ "$qcdir/longest_refstart.fasta" "$qcdir/reference_start.txt" "$FLYE_THREADS"; then ref_ok=true fi use_env "$CORE_ENV" ref_note="$(kv "$qcdir/reference_start.txt" note)" else ref_note="no PhagesDB phage covers at least half of the genome" fi fi if [[ "$circ" != "Y" ]]; then note="contig is linear; left as assembled" elif $ref_ok; then mv -- "$qcdir/longest_refstart.fasta" "$longest" by="reference"; pos="$(kv "$qcdir/reference_start.txt" start_position)" note="$ref_note" elif [[ "$ROTATE" != "terl" && "$end_found" == "yes" && "$end_pos" =~ ^[0-9]+$ ]]; then if (( end_pos <= 1 )); then by="read_ends"; pos=1; note="already starts at the physical end found from read pile-ups" else use_env "$CORE_ENV" python3 "$TOOLS" rotate "$longest" "$end_pos" "$longest" by="read_ends"; pos="$end_pos"; note="rotated to the physical end found from read start/stop pile-ups" fi elif [[ "$DNAAPLER_OK" == "true" ]]; then step rotate "🔄 dnaapler: starting the circular genome at the large terminase gene (terL)" use_env "$DNAAPLER_ENV" rm -rf -- "$sdir/dnaapler" if dnaapler phage -i "$longest" -o "$sdir/dnaapler" -p "$sample" -t "$FLYE_THREADS" -f \\ > "$sdir/dnaapler.log" 2>&1; then local out out="$(find "$sdir/dnaapler" -maxdepth 1 -name '*_reoriented.fasta' -print -quit)" if [[ -n "$out" && -s "$out" ]]; then use_env "$CORE_ENV" python3 "$TOOLS" relabel "$out" "$sample" "$longest" by="dnaapler_terL"; note="$( [[ -n "$ref_note" ]] && echo "$ref_note; " )rotated to start at terL (PhagesDB-style start may differ)" else note="dnaapler found no terL; left as assembled" fi else warn "dnaapler failed for $sample (see $sdir/dnaapler.log); genome left as assembled." note="dnaapler failed; left as assembled" fi else note="$( [[ -n "$ref_note" ]] && echo "$ref_note; " )dnaapler unavailable; left as assembled" fi use_env "$CORE_ENV" if [[ "$by" != "none" && "$pos" != "1" ]]; then python3 "$TOOLS" replacecontig "$final" "$contig" "$longest" fi printf 'rotated_by\\t%s\\nrotated_to_position\\t%s\\ncircular\\t%s\\nnote\\t%s\\n' "$by" "$pos" "$circ" "$note" > "$qcdir/rotation.txt" echo " rotation: $note" } # ============================================================== # Host-read removal for picked-plaque samples (reference-free) # ============================================================== sample_barcode_number() { # sample_barcode_number SAMPLE FASTQ... local sample="$1" shift local re='[Bb][Aa][Rr][Cc][Oo][Dd][Ee]0*([0-9]+)' local bc="\${ALIAS_BARCODE[$sample]:-}" if [[ "$bc" =~ ^0*([0-9]+)$ ]]; then echo "\${BASH_REMATCH[1]}"; return 0; fi local x for x in "$sample" "$@"; do if [[ "$x" =~ $re ]]; then echo "\${BASH_REMATCH[1]}"; return 0; fi done echo "" } is_plaque_sample() { # is_plaque_sample SAMPLE FASTQ... (true if listed in PLAQUE_SAMPLES) [[ "$HOST_REMOVAL" == "plaque" && -n "$PLAQUE_SAMPLES" ]] || return 1 [[ "$PLAQUE_SAMPLES" == "__ALL__" ]] && return 0 local sample="$1" shift local key bc tok key="$(printf '%s' "$sample" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]')" bc="$(sample_barcode_number "$sample" "$@")" for tok in $PLAQUE_SAMPLES; do case "$tok" in bc:*) if [[ -n "$bc" && "\${tok#bc:}" == "$bc" ]]; then return 0; fi ;; nm:*) if [[ "\${tok#nm:}" == "$key" ]]; then return 0; fi ;; esac done return 1 } host_draft_phage() { # host_draft_phage SAMPLE READS WORK_DIR REPORT -> phage_contigs.fasta + reads_vs_phage.paf local sample="$1" reads="$2" work="$3" report="$4" echo " quick draft assembly to find the phage contigs" use_env "$CORE_ENV" if ! flye "$FLYE_READ_TYPE" "$reads" --out-dir "$work/draft" --meta --iterations 1 --threads "$FLYE_THREADS" \\ > "$work/flye.log" 2>&1 || [[ ! -s "$work/draft/assembly.fasta" ]]; then warn "Host-read removal: the draft assembly failed for $sample (see $work/flye.log)." printf 'status\\tdraft_failed\\n' > "$report" return 1 fi if ! python3 "$TOOLS" phagecontigs "$work/draft/assembly_info.txt" "$work/draft/assembly.fasta" \\ "$work/phage_contigs.fasta" "$work/phage_contigs.tsv" "$HOST_CONTIG_FRACTION"; then warn "Host-read removal: no phage contig was identified for $sample." printf 'status\\tno_phage_contig\\n' > "$report" return 1 fi use_env "$HOST_ENV" if ! minimap2 -x map-ont --secondary=no -t "$FLYE_THREADS" "$work/phage_contigs.fasta" "$reads" \\ > "$work/reads_vs_phage.paf" 2> "$work/minimap2.log"; then use_env "$CORE_ENV" warn "Host-read removal: minimap2 failed against the phage draft for $sample (see $work/minimap2.log)." printf 'status\\tmapping_failed\\n' > "$report" return 1 fi use_env "$CORE_ENV" find "$work/draft" -mindepth 1 -maxdepth 1 -type d -exec rm -rf {} + 2>/dev/null || true return 0 } host_deplete() { # host_deplete SAMPLE READS OUT_READS WORK_DIR QC_DIR local sample="$1" reads="$2" out="$3" work="$4" qcdir="$5" local report="$qcdir/host_removal.txt" rm -rf -- "$work" rm -f -- "$out" mkdir -p "$work" if [[ "$HOST_METHOD" == "ref" ]]; then step host_removal "🧹 Host-read removal: mapping reads to the host genome" local protect="-" if [[ "$HOST_PROPHAGE" != "no" ]]; then # the host may carry a prophage: reads that also match this sample's phage draft are kept if host_draft_phage "$sample" "$reads" "$work" "$report"; then protect="$work/reads_vs_phage.paf" else warn "Host-read removal: continuing without protecting phage reads for $sample." fi fi use_env "$HOST_ENV" # --secondary=no: each read is assigned to the host it matches best if ! minimap2 -x map-ont --secondary=no -t "$FLYE_THREADS" "$HOST_INDEX" "$reads" \\ > "$work/reads_vs_host.paf" 2> "$work/minimap2_host.log"; then use_env "$CORE_ENV" warn "Host-read removal: minimap2 failed against the host genome for $sample (see $work/minimap2_host.log); using all reads." printf 'status\\tmapping_failed\\n' > "$report" return 0 fi use_env "$CORE_ENV" if ! python3 "$TOOLS" hostremove "$work/reads_vs_host.paf" "$reads" "$out" "$report" \\ "$HOST_MIN_ALIGNED_PCT" "$protect" "$HOST_LABELS"; then rm -f -- "$out" warn "Host-read removal: no reads were left for $sample; using all reads." return 0 fi printf 'host_genome\\t%s\\nhost_prophage\\t%s\\n' "$HOST_LIST" "$HOST_PROPHAGE" >> "$report" rm -f -- "$work/reads_vs_host.paf" "$work/reads_vs_phage.paf" host_keep_check "$sample" "$out" "$report" return 0 fi step host_removal "🧹 Host-read removal (picked-plaque sample): no host genome, keeping reads that match the phage" host_draft_phage "$sample" "$reads" "$work" "$report" || { warn "Using all reads for $sample."; return 0; } if ! python3 "$TOOLS" hostfilter "$work/reads_vs_phage.paf" "$reads" "$out" "$report" \\ "$HOST_MIN_ALIGNED_PCT" "$work/phage_contigs.tsv"; then rm -f -- "$out" warn "Host-read removal: no reads were kept for $sample; using all reads." return 0 fi rm -f -- "$work/reads_vs_phage.paf" host_keep_check "$sample" "$out" "$report" } # flye_tiers SAMPLE READS RUNS_DIR QC_DIR # Sets process_sample's SAMPLE_FLYE_GSIZE and SAMPLE_FLYE_VOTES ("label|fasta|info") and copies the best run to # FLYE_DIR (cross-check / fallback). Flye only ever gets a capped set of the best reads. flye_tiers() { local sample="$1" reads="$2" runs="$3" qcdir="$4" rm -rf -- "$runs" "$FLYE_DIR" mkdir -p "$runs" local total first_cap total="$(gzip -cd -- "$reads" | awk 'NR % 4 == 2 { n += length($0) } END { print n + 0 }')" first_cap=$(( FLYE_FIRST_MB * 1000000 )) (( total > 0 )) || return 1 # run_tier LABEL BASES: Flye on the best BASES of the reads (all of them when BASES >= total) run_tier() { local label="$1" bases="$2" in="$reads" dir="$runs/$1" if (( bases < total )); then in="$runs/$label.reads.fastq.gz" filtlong --target_bases "$bases" "$reads" 2> "$runs/$label.filtlong.log" | gzip -c > "$in" fi if time_limited flye "$FLYE_READ_TYPE" "$in" --out-dir "$dir" "\${fb_args[@]}" > "$runs/$label.flye.log" 2>&1 \\ && [[ -s "$dir/assembly.fasta" ]]; then rm -f -- "$runs/$label.reads.fastq.gz" find "$dir" -mindepth 1 -maxdepth 1 -type d -exec rm -rf {} + # Flye's working folders echo "$bases" > "$dir/read_bases.txt" echo " $label: $(python3 "$TOOLS" flyesummary "$dir/assembly_info.txt")" return 0 fi rm -f -- "$runs/$label.reads.fastq.gz" echo " $label: Flye failed (see $runs/$label.flye.log)" return 1 } local first_bases=$(( total < first_cap ? total : first_cap )) local first_label="first_$(( first_bases / 1000000 ))Mb" (( first_bases >= total )) && first_label="first_all" step flye "[3/5] Flye on the best reads ($(( first_bases / 1000000 )) of $(( total / 1000000 )) Mb) for the genome size \${fb_args[*]}" local done_runs=() if run_tier "$first_label" "$first_bases"; then done_runs+=("$first_label:$first_bases") SAMPLE_FLYE_GSIZE="$(python3 "$TOOLS" flyegsize "$runs/$first_label/assembly_info.txt" "$runs/$first_label/assembly.fasta")" || SAMPLE_FLYE_GSIZE="" [[ "$SAMPLE_FLYE_GSIZE" =~ ^[0-9]+$ ]] || SAMPLE_FLYE_GSIZE="" # v16: the minimum genome size also applies to the Flye vote read sets (200x/100x/50x of the size) if [[ -n "$SAMPLE_FLYE_GSIZE" ]] && (( \${AC_MIN_GSIZE:-0} > 0 && SAMPLE_FLYE_GSIZE < \${AC_MIN_GSIZE:-0} )); then echo " genome size from Flye: $SAMPLE_FLYE_GSIZE bp, below the minimum; using \${AC_MIN_GSIZE} bp" SAMPLE_FLYE_GSIZE="\${AC_MIN_GSIZE}" fi [[ -n "$SAMPLE_FLYE_GSIZE" ]] && echo " genome size from Flye: $SAMPLE_FLYE_GSIZE bp ($(( total / SAMPLE_FLYE_GSIZE ))x of reads in total)" fi if is_true "$AC_FLYE_VOTE" && [[ -n "$SAMPLE_FLYE_GSIZE" ]]; then local plan label bases plan="$(python3 "$TOOLS" flyeplan "$total" "$SAMPLE_FLYE_GSIZE" "$AC_FLYE_TIERS" "\${done_runs[*]}")" step flye_tiers "[3/5] Flye votes for Autocycler: $(printf '%s' "$plan" | awk '{ printf "%s%s", (NR > 1 ? ", " : ""), $1 }')" while read -r label bases; do [[ -n "$label" ]] || continue if [[ -d "$runs/$label" ]]; then :; elif run_tier "$label" "$bases"; then :; else continue; fi SAMPLE_FLYE_VOTES+=("$label|$runs/$label/assembly.fasta|$runs/$label/assembly_info.txt") done <<< "$plan" fi # the run used for the cross-check / fallback: deepest one with a circular contig, else the one with the longest contig local best best="$(python3 "$TOOLS" pickflye "$runs")" || best="" if [[ -n "$best" && -s "$runs/$best/assembly.fasta" ]]; then cp -r -- "$runs/$best" "$FLYE_DIR" printf 'flye_run_used\\t%s\\ntotal_bases\\t%s\\ngenome_size_flye\\t%s\\nvotes\\t%s\\n' "$best" "$total" "\${SAMPLE_FLYE_GSIZE:-NA}" \\ "$(printf '%s ' "\${SAMPLE_FLYE_VOTES[@]%%|*}")" > "$qcdir/flye_runs.txt" echo " Flye assembly used for the cross-check: $best" return 0 fi warn "Flye failed on every read set for $sample (see $runs); Autocycler runs without it." return 1 } # meta_args ARGS...: META_ARGS = ARGS without --asm-coverage (Flye refuses it with --meta) plus --meta (v16) meta_args() { META_ARGS=() while (( $# )); do if [[ "$1" == --asm-coverage ]]; then shift; (( $# )) && shift; continue; fi META_ARGS+=("$1"); shift done META_ARGS+=(--meta) } # apply_assembly_method: point draft/contig_info (process_sample's locals) at the assembly that was kept apply_assembly_method() { case "$(kv "$QC_DIR/assembly_method.txt" method)" in flye_fallback) draft="$FLYE_DIR/assembly.fasta"; contig_info="$FLYE_DIR/assembly_info.txt" ;; flye_chosen) contig_info="$FLYE_DIR/assembly_info.txt" ;; esac if [[ -s "$QC_DIR/chosen_draft.fasta" ]]; then draft="$QC_DIR/chosen_draft.fasta"; fi } # host_keep_check SAMPLE OUT_READS REPORT: if too little would be left, keep every read instead host_keep_check() { local sample="$1" out="$2" report="$3" pct pct="$(kv "$report" percent_bases_phage)" [[ "$pct" =~ ^[0-9.]+$ ]] || return 0 if awk -v p="$pct" -v m="$HOST_MIN_KEEP_PCT" 'BEGIN { exit !(p < m) }'; then warn "$sample: only \${pct}% of bases would be left after host-read removal (below \${HOST_MIN_KEEP_PCT}%); using ALL reads instead. Check that $sample really is a picked-plaque sample and that the host genome FASTA contains no phage sequence." rm -f -- "$out" sed -i 's/^status\\tok$/status\\tnot_applied_low_yield/' "$report" printf 'note\\tremoval would have left %s%% of bases; all reads used\\n' "$pct" >> "$report" fi } # ============================================================== # Process one sample (runs in its own subshell) # $1 = sample name, remaining arguments = FASTQ file(s) # ============================================================== # v17: what the assembly depends on (settings + input FASTQ names and sizes); a change means "reassemble" assembly_signature() { local k f for k in ASSEMBLY_MODE MIN_LENGTH MIN_MEAN_Q TARGET_BASES KEEP_PERCENT HOST_REMOVAL HOST_METHOD PLAQUE_SAMPLES \\ FLYE_READ_TYPE GENOME_SIZE FLYE_ITERATIONS MIN_OVERLAP ASM_COVERAGE FLYE_META FLYE_SCAFFOLD \\ AC_ASSEMBLERS AC_COUNT AC_GENOME_SIZE AC_FLYE_VOTE AC_FLYE_WEIGHT AC_FLYE_TIERS FLYE_FIRST_MB \\ AC_MAX_DEPTH AC_MAX_MB AC_MIN_GSIZE AC_READ_TYPE AC_CROSSCHECK MEDAKA_READS MEDAKA_DEPTH MEDAKA_MODEL \\ MEDAKA_BACTERIA; do printf 'setting %s=%s\\n' "$k" "\${!k-}" done for f in "$@"; do printf 'reads %s %s\\n' "$(basename "$f")" "$(stat -c %s "$f" 2>/dev/null || echo NA)"; done | sort } process_sample() { local sample="$1" shift local fastqs=("$@") local SAMPLE_DIR="$OUT_ROOT/$sample" local INPUT_DIR="$SAMPLE_DIR/input" local FILTER_DIR="$SAMPLE_DIR/filtered" local FLYE_DIR="$SAMPLE_DIR/flye" local MEDAKA_DIR="$SAMPLE_DIR/medaka" local QC_DIR="$SAMPLE_DIR/qc" local PHAROKKA_DIR="$SAMPLE_DIR/pharokka" local PHOLD_DIR="$SAMPLE_DIR/phold" local unique_reads="$INPUT_DIR/\${sample}.unique.fastq.gz" local dedup_report="$INPUT_DIR/\${sample}.deduplication.txt" local filtered_reads="$FILTER_DIR/\${sample}.filtered.fastq.gz" local medaka_reads="$unique_reads" # picked-plaque samples: reads that align to the phage contigs (host reads removed) local phage_reads="$INPUT_DIR/\${sample}.phage_reads.fastq.gz" local HOST_DIR="$SAMPLE_DIR/host_removal" local asm_reads="$unique_reads" local flye_copy="$SAMPLE_DIR/\${sample}.flye.fasta" local final_fasta="$SAMPLE_DIR/\${sample}.fasta" local longest_fasta="$SAMPLE_DIR/\${sample}.longest_contig.fasta" local compare_prefix="$QC_DIR/\${sample}.flye_vs_medaka" local AC_DIR="$SAMPLE_DIR/autocycler" local ac_copy="$SAMPLE_DIR/\${sample}.autocycler.fasta" # draft = the unpolished assembly Medaka polishes; contig_info = Flye-style table of its contigs local draft="$FLYE_DIR/assembly.fasta" local contig_info="$FLYE_DIR/assembly_info.txt" if [[ "$ASSEMBLY_MODE" == "accuracy" ]]; then draft="$AC_DIR/autocycler_out/consensus_assembly.fasta" contig_info="$QC_DIR/contig_info.txt" fi local pharokka_gbk="$SAMPLE_DIR/\${sample}.pharokka.gbk" local STEP_FILE="$TOOLS_DIR/timings_\${STAMP}/\${sample}.tsv" mkdir -p "$TOOLS_DIR/timings_\${STAMP}" : > "$STEP_FILE" step start "started $sample" local phold_gbk="$SAMPLE_DIR/\${sample}.phold.gbk" echo echo "==============================================================" echo "Sample: $sample" printf ' reads: %s\\n' "\${fastqs[@]}" echo " output: $SAMPLE_DIR" echo "==============================================================" # v17: reassemble automatically when the assembly settings or the input reads changed since the last run # (the previous genome, report and qc text files are kept in /previous_run/) local asm_sig asm_sig="$(assembly_signature "\${fastqs[@]}")" if [[ -s "$final_fasta" ]] && ! is_true "$OVERWRITE" && ! is_true "\${REPORTS_ONLY:-false}"; then local old_sig="$QC_DIR/assembly_signature.txt" if [[ -s "$old_sig" ]] && [[ "$(cat "$old_sig")" != "$asm_sig" ]]; then echo "🔁 The assembly settings or the input reads changed since $sample was assembled; reassembling:" diff <(cat "$old_sig") <(printf '%s\\n' "$asm_sig") | grep '^[<>]' | head -n 12 \\ | sed -e 's/^< / was: /' -e 's/^> / now: /' local prev_tmp="\${SAMPLE_DIR}.previous_run.tmp" rm -rf -- "$prev_tmp" mkdir -p "$prev_tmp/qc" find "$SAMPLE_DIR" -maxdepth 1 -type f -exec cp -p {} "$prev_tmp/" \\; 2>/dev/null find "$QC_DIR" -maxdepth 1 -type f \$ -name '*.txt' -o -name '*.tsv' \$ -exec cp -p {} "$prev_tmp/qc/" \\; 2>/dev/null rm -rf -- "$SAMPLE_DIR" mkdir -p "$SAMPLE_DIR" mv -- "$prev_tmp" "$SAMPLE_DIR/previous_run" elif [[ ! -s "$old_sig" ]]; then echo "ℹ️ $sample was assembled by an earlier pipeline version (its settings were not recorded): keeping that" echo " assembly. Tick 'Overwrite completed samples' to reassemble it with the current settings." fi fi if [[ -d "$SAMPLE_DIR" ]] && is_true "$OVERWRITE"; then echo "Overwrite is on: removing previous output." rm -rf -- "$SAMPLE_DIR" fi # ---------------- Assembly + polishing ---------------- if [[ -s "$final_fasta" ]]; then echo "✅ Polished assembly already exists; skipping assembly and polishing." else if [[ -d "$SAMPLE_DIR" ]]; then echo "⚠️ Incomplete previous run found; restarting this sample." rm -rf -- "$SAMPLE_DIR" fi mkdir -p "$INPUT_DIR" "$FILTER_DIR" "$FLYE_DIR" "$MEDAKA_DIR" "$QC_DIR" use_env "$CORE_ENV" local reads_in="\${fastqs[0]}" if (( \${#fastqs[@]} > 1 )); then reads_in="$INPUT_DIR/\${sample}.combined.fastq.gz" step combine "Combining \${#fastqs[@]} FASTQ files" local f for f in "\${fastqs[@]}"; do case "\${f,,}" in *.gz) gzip -cd -- "$f" ;; *) cat -- "$f" ;; esac done | gzip -c > "$reads_in" fi step dedup "[1/5] Deduplicating read IDs" python3 "$TOOLS" dedup "$reads_in" "$unique_reads" "$dedup_report" if (( \${#fastqs[@]} > 1 )); then rm -f -- "$reads_in"; fi [[ -s "$unique_reads" ]] || fail "Deduplication produced an empty FASTQ." if is_plaque_sample "$sample" "\${fastqs[@]}"; then host_deplete "$sample" "$unique_reads" "$phage_reads" "$HOST_DIR" "$QC_DIR" fi if [[ -s "$phage_reads" ]]; then asm_reads="$phage_reads" echo " assembling from the phage reads only: $(kv "$QC_DIR/host_removal.txt" percent_reads_phage)% of reads, $(kv "$QC_DIR/host_removal.txt" percent_bases_phage)% of bases" fi step filtlong "[2/5] Filtlong \${FILTLONG_ARGS[*]}" filtlong "\${FILTLONG_ARGS[@]}" "$asm_reads" \\ 2> >(tee "$FILTER_DIR/filtlong.log" >&2) \\ | gzip -c > "$filtered_reads" [[ "$(gzip -cd -- "$filtered_reads" | wc -c)" -gt 0 ]] \\ || fail "Filtlong kept no reads. See $FILTER_DIR/filtlong.log" if [[ "$ASSEMBLY_MODE" == "accuracy" ]]; then # Flye on capped sets of the best reads: genome size for Autocycler, weighted votes, cross-check and fallback. local fb_args=("\${FLYE_ARGS[@]}") if [[ -n "$GENOME_SIZE" ]] && (( ASM_COVERAGE == 0 )); then fb_args+=(--asm-coverage 150); fi local SAMPLE_FLYE_GSIZE="" flye_done=false local SAMPLE_FLYE_VOTES=() if [[ "$AC_GENOME_SIZE" == "flye" ]] || is_true "$AC_FLYE_VOTE" || is_true "$AC_CROSSCHECK"; then flye_tiers "$sample" "$filtered_reads" "$SAMPLE_DIR/flye_runs" "$QC_DIR" && flye_done=true fi # Autocycler runs in its own process so that a failure can fall back to Flye instead of stopping the phage. local ac_status=0 need_flye="" ac_circ="" ( trap - ERR; set -e; autocycler_assemble "$sample" "$filtered_reads" "$AC_DIR" "$QC_DIR" ) & wait $! || ac_status=$? use_env "$CORE_ENV" if (( ac_status != 0 )); then need_flye="Autocycler did not finish (see $AC_DIR/logs)" warn "$sample: $need_flye; assembling with Flye from the same reads instead." else cp -- "$draft" "$ac_copy" ac_circ="$(python3 "$TOOLS" accircular "$QC_DIR/contig_info.txt")" if [[ "$ac_circ" != "yes" ]] && is_true "$AC_CROSSCHECK"; then need_flye="check"; fi fi if [[ -n "$need_flye" ]] && ! $flye_done; then rm -rf -- "$FLYE_DIR" step flye "[3/5] Flye $( [[ "$need_flye" == check ]] && echo "(cross-check: Autocycler's genome is not circular)" || echo "(fallback)") \${fb_args[*]}" flye "$FLYE_READ_TYPE" "$filtered_reads" --out-dir "$FLYE_DIR" "\${fb_args[@]}" > "$QC_DIR/flye_crosscheck.log" 2>&1 || true if [[ ! -s "$FLYE_DIR/assembly.fasta" && " \${fb_args[*]} " != *" --meta "* ]]; then # v16: Flye sometimes assembles nothing ("No disjointigs") on very deep read sets; --meta copes with that warn "$sample: Flye assembled nothing; retrying with --meta." rm -rf -- "$FLYE_DIR" meta_args "\${fb_args[@]}" flye "$FLYE_READ_TYPE" "$filtered_reads" --out-dir "$FLYE_DIR" "\${META_ARGS[@]}" >> "$QC_DIR/flye_crosscheck.log" 2>&1 || true fi if [[ ! -s "$FLYE_DIR/assembly.fasta" ]]; then tail -n 20 "$QC_DIR/flye_crosscheck.log" >&2 (( ac_status == 0 )) || fail "Flye failed too (see $QC_DIR/flye_crosscheck.log)." warn "Flye failed for $sample (see $QC_DIR/flye_crosscheck.log); keeping Autocycler." fi fi if (( ac_status != 0 )); then [[ -s "$FLYE_DIR/assembly.fasta" ]] || fail "Flye did not produce assembly.fasta." cp -- "$FLYE_DIR/assembly.fasta" "$flye_copy" printf 'method\\tflye_fallback\\nreason\\t%s\\n' "$need_flye" > "$QC_DIR/assembly_method.txt" elif [[ "$need_flye" == "check" && -s "$FLYE_DIR/assembly.fasta" ]]; then cp -- "$FLYE_DIR/assembly.fasta" "$flye_copy" step pick_draft "Choosing between Autocycler and Flye (alignment, host match, read depth)" use_env "$MM2_ENV" if ! python3 "$TOOLS" pickdraft "$draft" "$FLYE_DIR/assembly.fasta" "$filtered_reads" "\${HOST_INDEX:--}" \\ "$QC_DIR/chosen_draft.fasta" "$QC_DIR/assembly_method.txt" "$FLYE_THREADS"; then warn "Could not compare the Autocycler and Flye assemblies for $sample; keeping Autocycler." rm -f -- "$QC_DIR/chosen_draft.fasta" printf 'method\\tautocycler\\nreason\\tcomparison with Flye failed\\n' > "$QC_DIR/assembly_method.txt" fi use_env "$CORE_ENV" else if [[ -s "$FLYE_DIR/assembly.fasta" ]]; then cp -- "$FLYE_DIR/assembly.fasta" "$flye_copy"; fi rmdir -- "$FLYE_DIR" 2>/dev/null || true printf 'method\\tautocycler\\nreason\\t%s\\n' "$( [[ "$ac_circ" == yes ]] && echo "circular consensus" || echo "not cross-checked")" > "$QC_DIR/assembly_method.txt" fi else step flye "[3/5] Flye \${FLYE_READ_TYPE} \${FLYE_ARGS[*]}" local flye_note="" if ! flye "$FLYE_READ_TYPE" "$filtered_reads" --out-dir "$FLYE_DIR" "\${FLYE_ARGS[@]}" || [[ ! -s "$FLYE_DIR/assembly.fasta" ]]; then # v16: Flye sometimes assembles nothing ("No disjointigs") on very deep phage read sets; --meta copes with that is_true "$FLYE_META" && fail "Flye did not produce assembly.fasta." warn "$sample: Flye assembled nothing (this happens with very deep coverage); retrying with --meta." rm -rf -- "\${FLYE_DIR}_failed"; mv -- "$FLYE_DIR" "\${FLYE_DIR}_failed" 2>/dev/null || true meta_args "\${FLYE_ARGS[@]}" step flye_meta_retry "[3/5] Flye \${FLYE_READ_TYPE} \${META_ARGS[*]} (retry)" flye "$FLYE_READ_TYPE" "$filtered_reads" --out-dir "$FLYE_DIR" "\${META_ARGS[@]}" || true [[ -s "$FLYE_DIR/assembly.fasta" ]] || fail "Flye did not produce assembly.fasta (also not with --meta; see \${FLYE_DIR}_failed/flye.log)." flye_note="retried with --meta (normal Flye assembled nothing)" fi cp -- "$FLYE_DIR/assembly.fasta" "$flye_copy" printf 'method\\tflye\\n' > "$QC_DIR/assembly_method.txt" if [[ -n "$flye_note" ]]; then printf 'reason\\t%s\\n' "$flye_note" >> "$QC_DIR/assembly_method.txt"; fi fi # the draft Medaka polishes (Flye's when Autocycler fell back; the chosen one after a cross-check) apply_assembly_method case "$MEDAKA_READS" in filtered) medaka_reads="$filtered_reads" ;; capped) local asm_len target asm_len="$(python3 "$TOOLS" stats "$draft")" target=$(( asm_len * MEDAKA_DEPTH )) medaka_reads="$FILTER_DIR/\${sample}.medaka_\${MEDAKA_DEPTH}x.fastq.gz" step medaka_reads "Selecting best reads for Medaka: \${MEDAKA_DEPTH}x of \${asm_len} bp = \${target} bases" local cap_args=(--target_bases "$target") if (( MIN_LENGTH > 0 )); then cap_args+=(--min_length "$MIN_LENGTH"); fi filtlong "\${cap_args[@]}" "$asm_reads" 2>>"$FILTER_DIR/filtlong.log" \\ | gzip -c > "$medaka_reads" ;; *) medaka_reads="$asm_reads" ;; esac use_env "$MEDAKA_ENV" step medaka "[4/5] Medaka (\${MEDAKA_READS} reads) \${MEDAKA_ARGS[*]}" gpu_run medaka_consensus -i "$medaka_reads" -d "$draft" -o "$MEDAKA_DIR" "\${MEDAKA_ARGS[@]}" if [[ ! -s "$MEDAKA_DIR/consensus.fasta" ]]; then # v24: the GPU run can fail on some drafts (e.g. "cuDNN error: CUDNN_STATUS_NOT_SUPPORTED" on many tiny contigs); # polish again on the CPU (slower, but phage genomes are small) warn "Medaka failed on the GPU for $sample; trying again on the CPU." rm -rf -- "$MEDAKA_DIR" CUDA_VISIBLE_DEVICES="" medaka_consensus -i "$medaka_reads" -d "$draft" -o "$MEDAKA_DIR" "\${MEDAKA_ARGS[@]}" || true fi [[ -s "$MEDAKA_DIR/consensus.fasta" ]] || fail "Medaka did not produce consensus.fasta (GPU and CPU)." use_env "$CORE_ENV" python3 "$TOOLS" finalize "$MEDAKA_DIR/consensus.fasta" "$final_fasta" "$sample" "$RENAME_HEADERS" mkdir -p "$QC_DIR" printf '%s\\n' "$asm_sig" > "$QC_DIR/assembly_signature.txt" # v17 fi mkdir -p "$QC_DIR" if [[ -s "$phage_reads" ]]; then asm_reads="$phage_reads"; fi apply_assembly_method # ---------------- NanoPlot read-QC report ---------------- if is_true "$RUN_NANOPLOT" && [[ -s "$unique_reads" && ! -s "$SAMPLE_DIR/nanoplot/\${sample}_NanoStats.txt" ]]; then step nanoplot "📊 NanoPlot (deduplicated reads, before Filtlong)" use_env "$CORE_ENV" mkdir -p "$SAMPLE_DIR/nanoplot" local np_status=0 run_limited "$NANOPLOT_TIME_LIMIT" NanoPlot --fastq "$unique_reads" --threads "$NANO_THREADS" --loglength --N50 \\ "\${NANOPLOT_EXTRA[@]}" --outdir "$SAMPLE_DIR/nanoplot" --prefix "\${sample}_" > "$SAMPLE_DIR/nanoplot.log" 2>&1 \\ || np_status=$? if (( np_status == 0 )); then echo "NanoPlot report: $SAMPLE_DIR/nanoplot/\${sample}_NanoPlot-report.html" elif (( np_status == 124 )); then warn "NanoPlot was still running after $((NANOPLOT_TIME_LIMIT / 60)) min for $sample and was stopped; continuing." else warn "NanoPlot failed for $sample (see $SAMPLE_DIR/nanoplot.log); continuing." fi fi # ---------------- Read statistics (for PhageTracker) ---------------- if is_true "$EXPORT_PHAGETRACKER"; then use_env "$CORE_ENV" if [[ ! -s "$QC_DIR/readstats_raw.json" && -s "$unique_reads" ]]; then step readstats "📊 Read statistics: deduplicated reads" python3 "$TOOLS" readstats "$unique_reads" "$QC_DIR/readstats_raw.json" fi if [[ ! -s "$QC_DIR/readstats_filtered.json" && -s "$filtered_reads" ]]; then echo "📊 Read statistics: Filtlong-filtered reads" python3 "$TOOLS" readstats "$filtered_reads" "$QC_DIR/readstats_filtered.json" fi if [[ ! -s "$QC_DIR/readstats_host_depleted.json" && -s "$phage_reads" ]]; then echo "📊 Read statistics: phage reads (host reads removed)" python3 "$TOOLS" readstats "$phage_reads" "$QC_DIR/readstats_host_depleted.json" fi fi # ---------------- Flye vs Medaka ---------------- if is_true "$COMPARE" && [[ ! -s "\${compare_prefix}.summary.tsv" ]]; then if [[ -s "$draft" && -s "$MEDAKA_DIR/consensus.fasta" ]]; then step compare "[5/5] Comparing the draft ($( [[ "$ASSEMBLY_MODE" == "accuracy" ]] && echo Autocycler || echo Flye )) and Medaka assemblies" use_env "$COMPARE_ENV" minimap2 -c --cs -x asm5 --secondary=no -t 2 \\ "$draft" "$MEDAKA_DIR/consensus.fasta" \\ > "\${compare_prefix}.paf" 2> "$QC_DIR/minimap2.log" python3 "$TOOLS" compare "$draft" "$MEDAKA_DIR/consensus.fasta" \\ "\${compare_prefix}.paf" "$compare_prefix" else warn "Draft/Medaka intermediate files are missing; comparison skipped." fi fi use_env "$CORE_ENV" # ---------------- Longest contig ---------------- if [[ ! -s "$longest_fasta" ]]; then step longest "Selecting the longest contig" python3 "$TOOLS" longest "$final_fasta" "$longest_fasta" "$QC_DIR/longest_contig.tsv" \\ "$sample" "$contig_info" fi # ---------------- Genome ends + rotation ---------------- finish_genome "$sample" "$SAMPLE_DIR" "$QC_DIR" "$final_fasta" "$longest_fasta" "$filtered_reads" "$asm_reads" # genome vs its closest PhagesDB phage (for the comparison in coverage.html); redone when the genome changes if [[ -n "$MM2_ENV" && -s "$QC_DIR/closest_top.fasta" && -s "$longest_fasta" ]] \\ && { [[ ! -s "$QC_DIR/closest_alignment.paf" ]] || [[ "$longest_fasta" -nt "$QC_DIR/closest_alignment.paf" ]]; }; then use_env "$MM2_ENV" minimap2 -c -x asm20 --secondary=no -t "$FLYE_THREADS" "$QC_DIR/closest_top.fasta" "$longest_fasta" \\ > "$QC_DIR/closest_alignment.paf" 2> "$QC_DIR/closest_alignment.log" \\ || warn "Could not align $sample to its closest PhagesDB phage; the comparison is left out of the report." use_env "$CORE_ENV" fi # ---------------- Pharokka ---------------- if is_true "$RUN_PHAROKKA"; then if [[ -s "$pharokka_gbk" ]]; then echo "✅ Pharokka output already exists." else step pharokka "🧬 Pharokka on the longest contig" use_env "$PHAROKKA_ENV" local pharokka_args=(-i "$longest_fasta" -o "$PHAROKKA_DIR" -d "$PHAROKKA_DB" -t "$PHAROKKA_THREADS" -g "$PHAROKKA_GENE_CALLER" -f) if is_true "$PHAROKKA_FAST"; then pharokka_args+=(--fast); fi "\${PHAROKKA_RUN[@]}" "\${pharokka_args[@]}" > "$SAMPLE_DIR/pharokka.log" 2>&1 \\ || { tail -n 30 "$SAMPLE_DIR/pharokka.log" >&2; fail "Pharokka failed. See $SAMPLE_DIR/pharokka.log"; } [[ -s "$PHAROKKA_DIR/pharokka.gbk" ]] || fail "Pharokka finished without pharokka.gbk." cp -- "$PHAROKKA_DIR/pharokka.gbk" "$pharokka_gbk" echo "Pharokka GenBank: $pharokka_gbk" fi fi # ---------------- Phold ---------------- if is_true "$RUN_PHOLD"; then if [[ -s "$phold_gbk" ]]; then echo "✅ Phold output already exists." else step phold "🧬 Phold on the Pharokka annotation" use_env "$PHOLD_ENV" local ph_limit=() ph_status=0 if (( \${PHOLD_TIMEOUT_MIN:-0} > 0 )) && command -v timeout >/dev/null 2>&1; then ph_limit=(timeout --kill-after=60 "$(( PHOLD_TIMEOUT_MIN * 60 ))") fi gpu_run "\${ph_limit[@]}" env -u LD_LIBRARY_PATH -u PYTHONPATH -u PYTHONHOME phold run -i "$PHAROKKA_DIR/pharokka.gbk" -o "$PHOLD_DIR" "\${PHOLD_ARGS[@]}" \\ > "$SAMPLE_DIR/phold.log" 2>&1 || ph_status=$? if (( ph_status == 124 || ph_status == 137 )) && [[ " \${PHOLD_ARGS[*]} " != *" --cpu "* ]]; then # GPU run hung: try once more on the CPU (slower, no GPU lock needed) warn "Phold on the GPU was still running after \${PHOLD_TIMEOUT_MIN} min for $sample; stopped it and trying once on the CPU." mv -f -- "$SAMPLE_DIR/phold.log" "$SAMPLE_DIR/phold_gpu_timeout.log" ph_status=0 "\${ph_limit[@]}" env -u LD_LIBRARY_PATH -u PYTHONPATH -u PYTHONHOME phold run -i "$PHAROKKA_DIR/pharokka.gbk" -o "$PHOLD_DIR" \\ "\${PHOLD_ARGS[@]}" --cpu > "$SAMPLE_DIR/phold.log" 2>&1 || ph_status=$? (( ph_status == 0 )) && echo " Phold finished on the CPU." fi if (( ph_status == 0 )) && [[ -s "$PHOLD_DIR/phold.gbk" ]]; then cp -- "$PHOLD_DIR/phold.gbk" "$phold_gbk" echo "Phold GenBank: $phold_gbk" elif (( ph_status == 124 || ph_status == 137 )); then warn "Phold was still running after \${PHOLD_TIMEOUT_MIN} min for $sample and was stopped; keeping the Pharokka annotation (see $SAMPLE_DIR/phold.log)." else tail -n 15 "$SAMPLE_DIR/phold.log" >&2 warn "Phold failed for $sample (see $SAMPLE_DIR/phold.log); keeping the Pharokka annotation." fi fi fi # ---------------- v16: other large contigs (annotated separately; the main annotation stays the longest contig) ---- use_env "$CORE_ENV" local other_fa="$QC_DIR/other_contigs.fasta" large_fa="$QC_DIR/large_contigs.fasta" local other_pk="$SAMPLE_DIR/\${sample}.other_contigs.pharokka.gbk" other_ph="$SAMPLE_DIR/\${sample}.other_contigs.phold.gbk" python3 "$TOOLS" annotpick "$final_fasta" "\${contig_info:--}" "$sample" "$QC_DIR/longest_contig.tsv" "$other_fa" \\ "$QC_DIR/annotated_contigs.tsv" "$large_fa" "\${ANNOT_MIN_KB:-10}000" "\${ANNOT_MAX_CONTIGS:-6}" \\ || warn "Could not choose the other contigs to annotate for $sample; continuing." if is_true "$RUN_PHAROKKA" && [[ -s "$other_fa" ]]; then if [[ -s "$other_pk" && "$other_pk" -nt "$other_fa" ]]; then echo "✅ Pharokka output for the other contigs already exists." else step pharokka_other "🧬 Pharokka on the other large contigs" use_env "$PHAROKKA_ENV" rm -f -- "$other_ph" local pk_other_args=(-i "$other_fa" -o "$SAMPLE_DIR/pharokka_other" -d "$PHAROKKA_DB" -t "$PHAROKKA_THREADS" -g "$PHAROKKA_GENE_CALLER" -f) if is_true "$PHAROKKA_FAST"; then pk_other_args+=(--fast); fi if "\${PHAROKKA_RUN[@]}" "\${pk_other_args[@]}" > "$SAMPLE_DIR/pharokka_other.log" 2>&1 \\ && [[ -s "$SAMPLE_DIR/pharokka_other/pharokka.gbk" ]]; then cp -- "$SAMPLE_DIR/pharokka_other/pharokka.gbk" "$other_pk" else warn "Pharokka on the other contigs failed for $sample (see $SAMPLE_DIR/pharokka_other.log); continuing." fi use_env "$CORE_ENV" fi if is_true "$RUN_PHOLD" && [[ -s "$other_pk" ]] && ! [[ -s "$other_ph" && "$other_ph" -nt "$other_pk" ]]; then step phold_other "🧬 Phold on the other large contigs" use_env "$PHOLD_ENV" local ph2_limit=() ph2_status=0 if (( \${PHOLD_TIMEOUT_MIN:-0} > 0 )) && command -v timeout >/dev/null 2>&1; then ph2_limit=(timeout --kill-after=60 "$(( PHOLD_TIMEOUT_MIN * 60 ))") fi gpu_run "\${ph2_limit[@]}" env -u LD_LIBRARY_PATH -u PYTHONPATH -u PYTHONHOME phold run -i "$other_pk" \\ -o "$SAMPLE_DIR/phold_other" "\${PHOLD_ARGS[@]}" > "$SAMPLE_DIR/phold_other.log" 2>&1 || ph2_status=$? if (( ph2_status == 0 )) && [[ -s "$SAMPLE_DIR/phold_other/phold.gbk" ]]; then cp -- "$SAMPLE_DIR/phold_other/phold.gbk" "$other_ph" else warn "Phold on the other contigs did not finish for $sample (see $SAMPLE_DIR/phold_other.log); keeping Pharokka's annotation." fi use_env "$CORE_ENV" fi fi # contig-to-contig alignment (overlaps / shared regions) when there are two or more large contigs rm -f -- "$QC_DIR/contig_overlaps.paf" if [[ -n "$MM2_ENV" && -s "$large_fa" ]] && (( $(grep -c '^>' "$large_fa") >= 2 )); then use_env "$MM2_ENV" minimap2 -c -x asm20 -X -t "$FLYE_THREADS" "$large_fa" "$large_fa" > "$QC_DIR/contig_overlaps.paf" 2>/dev/null \\ || rm -f -- "$QC_DIR/contig_overlaps.paf" use_env "$CORE_ENV" fi # ---------------- v17: does every input read set (barcode / run) contain this phage? ---------------- if [[ -n "$MM2_ENV" && -s "$final_fasta" ]]; then mkdir -p "$QC_DIR" { printf '%s\\n' "\${fastqs[@]}"; md5sum "$final_fasta" | cut -d' ' -f1; } > "$QC_DIR/input_check.list.new" if [[ -s "$QC_DIR/input_check.tsv" ]] && cmp -s "$QC_DIR/input_check.list.new" "$QC_DIR/input_check.list"; then rm -f "$QC_DIR/input_check.list.new" else mv -f "$QC_DIR/input_check.list.new" "$QC_DIR/input_check.list" step input_check "🔎 Checking that every input read set contains this phage" use_env "$MM2_ENV" python3 "$TOOLS" inputcheck "$final_fasta" "$QC_DIR/input_check.list" "$QC_DIR/input_check.tsv" \\ "$QC_DIR/input_check.txt" "$FLYE_THREADS" 2000 || warn "Input read check failed for $sample; continuing." use_env "$CORE_ENV" fi fi # ---------------- Annotation check + warnings ---------------- use_env "$CORE_ENV" local gbk_for_qc="-" if [[ -s "$phold_gbk" ]]; then gbk_for_qc="$phold_gbk" elif [[ -s "$pharokka_gbk" ]]; then gbk_for_qc="$pharokka_gbk"; fi if [[ "$gbk_for_qc" != "-" ]]; then step annotation_qc "Checking the annotation (coding density, split genes)" python3 "$TOOLS" codingqc "$gbk_for_qc" "$longest_fasta" "$QC_DIR/annotation_qc.txt" \\ || warn "Annotation check failed for $sample; continuing." fi local ac_for_flags="-" if [[ -d "$AC_DIR/autocycler_out" ]]; then ac_for_flags="$AC_DIR/autocycler_out"; fi local aqc="-" if [[ -s "$QC_DIR/annotation_qc.txt" ]]; then aqc="$QC_DIR/annotation_qc.txt"; fi python3 "$TOOLS" sampleflags "$final_fasta" "$longest_fasta" "$aqc" "$ac_for_flags" "$QC_DIR/sample_flags.txt" \\ || warn "Sample checks failed for $sample; continuing." # ---------------- Coverage + consensus report ---------------- if is_true "$RUN_COVERAGE"; then local cov_tsv="$QC_DIR/\${sample}.coverage.tsv.gz" local cov_bam="$QC_DIR/\${sample}.reads_vs_polished.bam" local cov_html="$SAMPLE_DIR/\${sample}.coverage.html" local cov_reads="$asm_reads" local cov_label="all deduplicated reads" if [[ "$asm_reads" == "$phage_reads" ]]; then cov_label="phage reads (host reads removed)"; fi if [[ "\${COVERAGE_READS:-medaka}" == "all" ]]; then cov_label="all reads (deduplicated$( [[ "$asm_reads" == "$phage_reads" ]] && echo ", host reads removed"))" elif [[ "$MEDAKA_READS" == "capped" && -s "$FILTER_DIR/\${sample}.medaka_\${MEDAKA_DEPTH}x.fastq.gz" ]]; then cov_reads="$FILTER_DIR/\${sample}.medaka_\${MEDAKA_DEPTH}x.fastq.gz" cov_label="best reads capped at \${MEDAKA_DEPTH}x (Medaka input)" elif [[ "$MEDAKA_READS" == "filtered" && -s "$filtered_reads" ]]; then cov_reads="$filtered_reads" cov_label="Filtlong-filtered reads (Medaka input)" fi # v18: map again when the reads for the plots changed (e.g. switched to all reads) if [[ -s "$cov_tsv" ]] && [[ "$(cat "$QC_DIR/coverage_reads.txt" 2>/dev/null)" != "$cov_reads" ]] && ! is_true "\${REPORTS_ONLY:-false}"; then rm -f -- "$cov_tsv" fi if [[ ! -s "$cov_tsv" ]]; then [[ -s "$cov_reads" ]] || fail "Reads for the coverage report are missing: $cov_reads" printf '%s\\n' "$cov_reads" > "$QC_DIR/coverage_reads.txt" step coverage_map "📈 Mapping reads to the polished assembly ($cov_label)" use_env "$COVERAGE_ENV" cp -- "$final_fasta" "$QC_DIR/polished_ref.fasta" samtools faidx "$QC_DIR/polished_ref.fasta" minimap2 -ax map-ont --secondary=no -t "$FLYE_THREADS" "$QC_DIR/polished_ref.fasta" "$cov_reads" \\ 2> "$QC_DIR/coverage_minimap2.log" \\ | samtools sort -@ 2 -o "$cov_bam" - 2> "$QC_DIR/coverage_samtools.log" samtools index "$cov_bam" samtools depth -aa "$cov_bam" > "$QC_DIR/depth.txt" samtools mpileup -aa -B -Q 0 -q 0 -d 2000 -f "$QC_DIR/polished_ref.fasta" "$cov_bam" \\ 2>> "$QC_DIR/coverage_samtools.log" | gzip -c > "$QC_DIR/pileup.txt.gz" use_env "$CORE_ENV" python3 "$TOOLS" pileupstats "$QC_DIR/polished_ref.fasta" "$QC_DIR/depth.txt" "$QC_DIR/pileup.txt.gz" "$cov_tsv" rm -f -- "$QC_DIR/depth.txt" "$QC_DIR/pileup.txt.gz" fi # The HTML is rebuilt every run so newly added annotations appear on it. use_env "$CORE_ENV" local gbk_for_plot="-" if [[ -s "$phold_gbk" ]]; then gbk_for_plot="$phold_gbk" elif [[ -s "$pharokka_gbk" ]]; then gbk_for_plot="$pharokka_gbk"; fi local changes_for_plot="-" if [[ -s "\${compare_prefix}.changes.tsv" ]]; then changes_for_plot="\${compare_prefix}.changes.tsv"; fi local rot_by rot_pos rot_by="$(kv "$QC_DIR/rotation.txt" rotated_by)" rot_pos="$(kv "$QC_DIR/rotation.txt" rotated_to_position)" if [[ "$changes_for_plot" != "-" && "$rot_by" == "read_ends" && "$rot_pos" =~ ^[0-9]+$ ]]; then python3 "$TOOLS" shiftchanges "$changes_for_plot" "$(kv "$QC_DIR/longest_contig.tsv" longest_contig)" \\ "$(kv "$QC_DIR/longest_contig.tsv" longest_length)" "$rot_pos" "$QC_DIR/changes_rotated.tsv" changes_for_plot="$QC_DIR/changes_rotated.tsv" elif [[ "$changes_for_plot" != "-" && ( "$rot_by" == "dnaapler_terL" || "$rot_by" == "reference" ) ]]; then changes_for_plot="-" # dnaapler may also reverse the strand, so old positions no longer line up fi # v16: reads behind this genome (input FASTQs, deduplicated, filtered, coverage set) for the top of coverage.html { for f in "\${fastqs[@]}"; do printf 'input\\t%s\\n' "$f"; done [[ -s "$unique_reads" ]] && printf 'deduplicated\\t%s\\n' "$unique_reads" [[ -s "$filtered_reads" ]] && printf 'filtered\\t%s\\n' "$filtered_reads" [[ -n "\${phage_reads:-}" && -s "\${phage_reads:-}" && "$phage_reads" != "$unique_reads" ]] && printf 'host_removed\\t%s\\n' "$phage_reads" # reads the assembler actually used: Autocycler's best-reads cap (Accuracy mode) or the Filtlong output asm_in="$filtered_reads" if [[ "$ASSEMBLY_MODE" == "accuracy" ]] && ! grep -qs "flye_fallback" "$QC_DIR/assembly_method.txt"; then for cand in "$AC_DIR/reads_capped.fastq.gz" "$AC_DIR/reads_\${AC_MAX_DEPTH}x.fastq.gz"; do if [[ -s "$cand" ]]; then asm_in="$cand"; break; fi done fi [[ -s "$asm_in" ]] && printf 'assembly\\t%s\\n' "$asm_in" [[ -s "$cov_reads" ]] && printf 'coverage\\t%s\\n' "$cov_reads" : } > "$QC_DIR/read_sets.list.new" if [[ -s "$QC_DIR/read_sets.tsv" ]] && cmp -s "$QC_DIR/read_sets.list.new" "$QC_DIR/read_sets.list"; then rm -f "$QC_DIR/read_sets.list.new" else mv -f "$QC_DIR/read_sets.list.new" "$QC_DIR/read_sets.list" step readsets "📊 Reads used (input FASTQs and read sets)" python3 "$TOOLS" readsets "$QC_DIR/read_sets.list" "$QC_DIR/read_sets.tsv" \\ || warn "Could not summarise the read sets for $sample; continuing." fi step coverage_report "Building the coverage report" NCBI_BLAST_PROXY="\${NCBI_BLAST_PROXY:-}" PHAGESDB_BLAST_URL="\${PHAGESDB_BLAST_URL:-}" \\ PHAGESDB_SEQ_FIELD="\${PHAGESDB_SEQ_FIELD:-SEQUENCE}" PHAGESDB_DATALIB="\${PHAGESDB_DATALIB:-}" \\ PHAGESDB_DB_CACHE="$TOOLS_DIR/phagesdb_blast_databases.json" MIN_LENGTH="$MIN_LENGTH" MIN_MEAN_Q="$MIN_MEAN_Q" \\ python3 "$TOOLS" report "$final_fasta" "$cov_tsv" "$sample" "$cov_html" "$QC_DIR/coverage_stats.txt" \\ "$changes_for_plot" "$gbk_for_plot" "$QC_DIR/longest_contig.tsv" "$AGREE_PCT" "$LOWCOV_X" "$cov_label" python3 "$TOOLS" dips "$cov_tsv" "$(kv "$QC_DIR/longest_contig.tsv" longest_contig)" "$QC_DIR/coverage_dips.tsv" 0.25 \\ || warn "Coverage-dip report failed for $sample; continuing." fi step done "✅ Finished $sample" } # ============================================================== # Run one sample in isolation so a failure does not stop the batch # ============================================================== kv() { # kv FILE KEY if [[ -s "$1" ]]; then awk -F'\\t' -v k="$2" '$1 == k { print $2 }' "$1"; else echo "NA"; fi } gbk_counts() { # prints "cdshypothetical" if [[ -s "$1" ]]; then python3 "$TOOLS" gbkstats "$1"; else printf 'NA\\tNA'; fi } printf 'sample\\tstatus\\tcontigs\\tlongest_bp\\tcircular\\tflye_coverage\\tmedian_read_depth\\tlow_agreement_bases\\tflye_to_medaka_changes\\thomopolymer_indels\\tpharokka_CDS\\tpharokka_hypothetical\\tphold_hypothetical\\tpolished_fasta\\treads\\tassembly_mode\\tphage_sized_seqs\\tgenome_end\\trotated_by\\tcoding_density_pct\\tflags\\tphage_reads_pct\\tassembly_method\\tlow_depth_regions\\tclosest_phagesdb\\tclosest_identity_pct\\tclosest_aligned_pct\\tclosest_cluster\\tclosest_ends\\tstart_check\\tclosest_temperate\\tclosest_morphotype\\tgc_pct\\tclosest_gc_pct\\n' \\ > "$SUMMARY_FILE" OK_COUNT=0 FAIL_COUNT=0 # mv_retry SRC DEST: rename, retrying for up to ~30 s (Windows can lock a freshly copied folder for a while, # e.g. while Defender scans it) (v16) mv_retry() { local i for i in 1 2 3 4 5 6 7 8 9 10; do mv -- "$1" "$2" 2>/dev/null && return 0 sleep 3 done return 1 } # local_copy_back SAMPLE WORKDIR: move a phage assembled on the WSL drive into the output folder local_copy_back() { local sample="$1" work="$2" real="$OUT_ROOT/$1" f [[ -d "$work" ]] || return 0 echo "Copying $sample from the WSL drive to $real …" # file paths written inside reports and tables point to the WSL folder: point them at the output folder grep -rlIF --exclude='*.gz' --exclude='*.bam' --exclude='*.bai' --exclude='*.fastq' --exclude='*.fq' \\ -- "$LOCAL_RUN_DIR" "$work" 2>/dev/null | while IFS= read -r f; do sed -i "s|$LOCAL_RUN_DIR|$OUT_ROOT|g" "$f" 2>/dev/null || true done || true rm -rf -- "\${real}.copying" if ! cp -r --preserve=timestamps -- "$work" "\${real}.copying" 2>/dev/null; then rm -rf -- "\${real}.copying" warn "$sample: could not copy the results to $real (disk full?). They are still in $work." return 0 fi # swap in safely: old folder aside first, new one in place, then delete the old one; the WSL copy is # deleted only when the new folder is in place (a folder open in Windows Explorer can block a rename) rm -rf -- "\${real}.old" if [[ -e "$real" ]] && ! mv_retry "$real" "\${real}.old"; then rm -rf -- "\${real}.copying" warn "$sample: could not replace $real (is it open in Windows Explorer?). The new results are in $work." return 0 fi if mv_retry "\${real}.copying" "$real"; then rm -rf -- "\${real}.old" "$work" elif cp -r --preserve=timestamps -- "$work" "$real" 2>/dev/null && [[ -d "$real" ]]; then # Windows kept blocking the rename: copied straight into place instead rm -rf -- "\${real}.copying" "\${real}.old" "$work" 2>/dev/null || true else [[ -e "\${real}.old" && ! -e "$real" ]] && mv_retry "\${real}.old" "$real" warn "$sample: could not move the new results into $real; they are in \${real}.copying and $work." fi } # local_recover SAMPLE: put back a phage folder left half-swapped by an interrupted copy-back local_recover() { local real="$OUT_ROOT/$1" [[ -e "$real" ]] && { rm -rf -- "\${real}.copying" "\${real}.old"; return 0; } # with the folder itself gone, a .copying folder is always complete (it is renamed only after a full copy) if [[ -d "\${real}.copying" ]]; then mv -- "\${real}.copying" "$real" 2>/dev/null && echo "Recovered $1 from an interrupted copy (\${real}.copying)." elif [[ -d "\${real}.old" ]]; then mv -- "\${real}.old" "$real" 2>/dev/null && echo "Recovered $1 from an interrupted copy (\${real}.old)." fi rm -rf -- "\${real}.copying" "\${real}.old" return 0 } # v18: coverage.html for a phage that could not be assembled: why, its reads, what they are, any partial contigs failed_report() { # failed_report SAMPLE ROOT FASTQ... local sample="$1" root="$2" shift 2 local d="$root/$sample" qc="$root/$sample/qc" cand pf="-" pi="-" plabel="" step="" logp="" shared="no" mkdir -p "$qc" || return 0 use_env "$CORE_ENV" { for cand in "$@"; do printf 'input\\t%s\\n' "$cand"; done [[ -s "$d/input/\${sample}.unique.fastq.gz" ]] && printf 'deduplicated\\t%s\\n' "$d/input/\${sample}.unique.fastq.gz" [[ -s "$d/filtered/\${sample}.filtered.fastq.gz" ]] && printf 'filtered\\t%s\\n' "$d/filtered/\${sample}.filtered.fastq.gz" [[ -s "$d/input/\${sample}.phage_reads.fastq.gz" ]] && printf 'host_removed\\t%s\\n' "$d/input/\${sample}.phage_reads.fastq.gz" if [[ "$ASSEMBLY_MODE" == "accuracy" && -s "$d/autocycler/reads_capped.fastq.gz" ]]; then printf 'assembly\\t%s\\n' "$d/autocycler/reads_capped.fastq.gz" elif [[ -s "$d/filtered/\${sample}.filtered.fastq.gz" && -d "$d/flye$( [[ "$ASSEMBLY_MODE" == "accuracy" ]] && echo _runs)" ]]; then printf 'assembly\\t%s\\n' "$d/filtered/\${sample}.filtered.fastq.gz" fi : } > "$qc/read_sets.list" python3 "$TOOLS" readsets "$qc/read_sets.list" "$qc/read_sets.tsv" > /dev/null 2>&1 || true printf '%s\\n' "$@" > "$qc/fail_fastqs.list" # any partial assembly, best first for cand in "$d/medaka/consensus.fasta|-|draft before polishing failed" \\ "$d/autocycler/autocycler_out/consensus_assembly.fasta|-|Autocycler consensus" \\ "$d/\${sample}.flye.fasta|$d/flye/assembly_info.txt|Flye" "$d/flye/assembly.fasta|$d/flye/assembly_info.txt|Flye"; do if [[ -s "\${cand%%|*}" ]]; then pf="\${cand%%|*}"; pi="\${cand#*|}"; pi="\${pi%%|*}"; plabel="\${cand##*|}" break fi done if [[ "$pf" == "-" && -d "$d/flye_runs" ]]; then cand="$(find "$d/flye_runs" -name assembly.fasta -size +0 -printf '%s %p\\n' 2>/dev/null | sort -rn | head -n 1 | cut -d' ' -f2-)" if [[ -n "$cand" ]]; then pf="$cand"; pi="$(dirname "$cand")/assembly_info.txt"; plabel="Flye ($(basename "$(dirname "$cand")"))"; fi fi step="$(tail -n 1 "$TOOLS_DIR/timings_\${STAMP}/\${sample}.tsv" 2>/dev/null | cut -f2)" if [[ -n "\${SLOG_DIR:-}" && -s "$SLOG_DIR/\${sample}.log" ]]; then logp="$SLOG_DIR/\${sample}.log" else logp="$LOG_FILE"; shared="yes"; fi { printf 'sample\\t%s\\nqc_dir\\t%s\\nout_html\\t%s\\nfastq_list\\t%s\\n' "$sample" "$qc" "$d/\${sample}.coverage.html" "$qc/fail_fastqs.list" printf 'failed_step\\t%s\\nlog_path\\t%s\\nlog_shared\\t%s\\n' "$step" "$logp" "$shared" printf 'genome_size\\t%s\\nmin_gsize\\t%s\\nthreads\\t%s\\nsample_reads\\t2000\\n' "$GENOME_SIZE" "\${AC_MIN_GSIZE:-0}" "$FLYE_THREADS" printf 'host_index\\t%s\\npdb_index\\t%s\\npdb_meta\\t%s\\n' "\${HOST_INDEX:--}" "\${PDB_INDEX:--}" "\${PDB_META:--}" printf 'partial_fasta\\t%s\\npartial_info\\t%s\\npartial_label\\t%s\\n' "$pf" "$pi" "$plabel" printf 'version\\t%s\\nfiltlong\\t%s\\n' "$GENERATOR_VERSION" "$(MIN_LENGTH="$MIN_LENGTH" MIN_MEAN_Q="$MIN_MEAN_Q" python3 -c ' import math, os p = [] if int(os.environ.get("MIN_LENGTH") or 0) > 0: p.append(">= %s bp" % os.environ["MIN_LENGTH"]) a = float(os.environ.get("MIN_MEAN_Q") or 0) if 0 < a < 100: p.append(">= Q%d" % round(-10 * math.log10(1 - a / 100))) print(", ".join(p))' 2>/dev/null)" } > "$qc/fail_report.cfg" [[ -n "$MM2_ENV" ]] && use_env "$MM2_ENV" PHAGESDB_BLAST_URL="\${PHAGESDB_BLAST_URL:-}" PHAGESDB_SEQ_FIELD="\${PHAGESDB_SEQ_FIELD:-SEQUENCE}" \\ PHAGESDB_DATALIB="\${PHAGESDB_DATALIB:-}" PHAGESDB_DB_CACHE="$TOOLS_DIR/phagesdb_blast_databases.json" \\ NCBI_BLAST_PROXY="\${NCBI_BLAST_PROXY:-}" python3 "$TOOLS" failreport "$qc/fail_report.cfg" \\ || echo "⚠️ Could not write the report for $sample" use_env "$CORE_ENV" } run_sample() { local sample="$1" shift local status if [[ "\${REPORTS_ONLY:-false}" == "true" && ! -s "$OUT_ROOT/$sample/\${sample}.fasta" ]]; then echo "⏭ $sample: no assembly in the output folder; skipped (reports-only run; its existing report is kept)" printf '%s\\tskipped\\n' "$sample" >> "$TOOLS_DIR/samples_\${STAMP}.tsv" return 0 fi # v16: optionally assemble on the WSL drive and copy the finished phage to the output folder local work="" if is_true "$WORK_LOCAL"; then local_recover "$sample" work="$LOCAL_RUN_DIR/$sample" rm -rf -- "$work" if [[ -s "$OUT_ROOT/$sample/\${sample}.fasta" ]] && ! is_true "$OVERWRITE"; then echo "Copying the previous results of $sample to the WSL drive…" cp -r --preserve=timestamps -- "$OUT_ROOT/$sample" "$work" || warn "$sample: could not copy the previous results; they will be rebuilt." fi fi trap - ERR set +e ( set -e trap 'st=$?; echo "❌ $sample failed at line $LINENO (exit $st)" >&2; exit $st' ERR if [[ -n "$work" ]]; then OUT_ROOT="$LOCAL_RUN_DIR"; fi process_sample "$sample" "$@" ) status=$? if (( status != 0 )) && is_true "$RUN_COVERAGE"; then ( set +e; failed_report "$sample" "$( [[ -n "$work" ]] && echo "$LOCAL_RUN_DIR" || echo "$OUT_ROOT" )" "$@" ) || true fi if [[ -n "$work" ]]; then local_copy_back "$sample" "$work"; fi set -e trap "$MAIN_TRAP" ERR local d="$OUT_ROOT/$sample" local result="ok" { printf '%s\\t%s' "$sample" "$( (( status == 0 )) && echo ok || echo failed )"; printf '\\t%s' "$@"; printf '\\n'; } \\ >> "$TOOLS_DIR/samples_\${STAMP}.tsv" if (( status == 0 )); then OK_COUNT=$((OK_COUNT + 1)) else result="FAILED" FAIL_COUNT=$((FAIL_COUNT + 1)) fi local ph ph_cds ph_hyp po po_hyp ph="$(gbk_counts "$d/\${sample}.pharokka.gbk")" po="$(gbk_counts "$d/\${sample}.phold.gbk")" ph_cds="\${ph%%$'\\t'*}" ph_hyp="\${ph##*$'\\t'}" po_hyp="\${po##*$'\\t'}" local end_txt end_txt="$(kv "$d/qc/genome_ends.txt" end_detected)" if [[ "$end_txt" == "yes" ]]; then end_txt="$(kv "$d/qc/genome_ends.txt" end_position) ($(kv "$d/qc/genome_ends.txt" end_type_guess))"; fi printf '%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\n' \\ "$sample" "$result" \\ "$(kv "$d/qc/longest_contig.tsv" n_contigs)" \\ "$(kv "$d/qc/longest_contig.tsv" longest_length)" \\ "$(kv "$d/qc/longest_contig.tsv" flye_circular)" \\ "$(kv "$d/qc/longest_contig.tsv" flye_coverage)" \\ "$(kv "$d/qc/coverage_stats.txt" median_depth)" \\ "$(kv "$d/qc/coverage_stats.txt" low_agreement_positions)" \\ "$(kv "$d/qc/\${sample}.flye_vs_medaka.totals.txt" changes)" \\ "$(kv "$d/qc/\${sample}.flye_vs_medaka.totals.txt" homopolymer)" \\ "$ph_cds" "$ph_hyp" "$po_hyp" \\ "$d/\${sample}.fasta" "$*" \\ "$ASSEMBLY_MODE" \\ "$(kv "$d/qc/sample_flags.txt" phage_sized_sequences)" \\ "$end_txt" \\ "$(kv "$d/qc/rotation.txt" rotated_by)" \\ "$(kv "$d/qc/annotation_qc.txt" coding_density_pct)" \\ "$(kv "$d/qc/sample_flags.txt" flags)" \\ "$(kv "$d/qc/host_removal.txt" percent_reads_phage)" \\ "$(kv "$d/qc/assembly_method.txt" method)" \\ "$(kv "$d/qc/coverage_dips.tsv" regions)" \\ "$(awk -F'\\t' 'NR == 2 { print $1 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(awk -F'\\t' 'NR == 2 { print $4 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(awk -F'\\t' 'NR == 2 { print $5 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(awk -F'\\t' 'NR == 2 { print $6 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(awk -F'\\t' 'NR == 2 { print $8 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(kv "$d/qc/reference_start.txt" overhang_check)" \\ "$(awk -F'\\t' 'NR == 2 { print $10 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(awk -F'\\t' 'NR == 2 { print $15 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(awk -F'\\t' 'NR == 2 { print $17 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" \\ "$(awk -F'\\t' 'NR == 2 { print $11 }' "$d/qc/closest_phagesdb.tsv" 2>/dev/null)" >> "$SUMMARY_FILE" } # ============================================================== # Discover samples and choose names # ============================================================== find_fastqs() { # find_fastqs DIR [maxdepth] local depth_args=() skip_args=() if [[ -n "\${2:-}" ]]; then depth_args=(-maxdepth "$2"); fi # Dorado puts reads without a barcode in "unclassified"; they are not a phage sample. if is_true "$SKIP_UNCLASSIFIED"; then skip_args=(-not -iname '*unclassified*' -not -ipath "$1/*unclassified*/*"); fi find "$1" "\${depth_args[@]}" -path "$OUT_ROOT" -prune -o -type f \\ \$ -iname '*.fastq' -o -iname '*.fastq.gz' -o -iname '*.fq' -o -iname '*.fq.gz' \$ "\${skip_args[@]}" -print0 } SHEET_MATCHES="" declare -A ALIAS_FILES=() declare -A ALIAS_BARCODE=() UNIT_KEYS=() SAMPLE_NAMES=() if [[ -n "$SAMPLE_SHEET" ]]; then # The Nanopore Workflow app's Dorado sample sheet decides names and barcodes. [[ -f "$SAMPLE_SHEET" ]] || fail "Dorado sample sheet not found: $SAMPLE_SHEET" FILE_LIST="$TOOLS_DIR/fastqs_\${STAMP}.nul" if [[ "$INPUT_MODE" == "single" ]]; then printf '%s\\0' "$INPUT_PATH" > "$FILE_LIST" else find_fastqs "$INPUT_PATH" "\${INPUT_DEPTH:-}" | sort -z > "$FILE_LIST" fi SHEET_MATCHES="$TOOLS_DIR/sheet_matches_\${STAMP}.tsv" python3 "$TOOLS" sheetmatch "$SAMPLE_SHEET" "$INPUT_PATH" "$FILE_LIST" "$SHEET_MATCHES" declare -A ALIAS_BARCODE=() UNMATCHED=() while IFS=$'\\t' read -r f alias bc; do [[ "$f" == \\#* ]] && continue if [[ -z "$alias" ]]; then UNMATCHED+=("$f"); continue; fi if [[ -z "\${ALIAS_FILES["$alias"]:-}" ]]; then ALIAS_FILES["$alias"]="$f" ALIAS_BARCODE["$alias"]="$bc" else ALIAS_FILES["$alias"]+=$'\\n'"$f" fi done < "$SHEET_MATCHES" if (( \${#UNMATCHED[@]} > 0 )); then warn "\${#UNMATCHED[@]} FASTQ file(s) are not in the sample sheet and were skipped (e.g. unclassified reads):" printf ' %s\\n' "\${UNMATCHED[@]:0:10}" >&2 fi mapfile -t SORTED_ALIASES < <(for a in "\${!ALIAS_FILES[@]}"; do printf '%s\\t%s\\n' "\${ALIAS_BARCODE[$a]}" "$a"; done | sort -n | cut -f2) for a in "\${SORTED_ALIASES[@]}"; do UNIT_KEYS+=("barcode\${ALIAS_BARCODE[$a]} ($(printf '%s' "\${ALIAS_FILES[$a]}" | grep -c '') file(s))") SAMPLE_NAMES+=("$(sanitize_sample "$a")") ALIAS_FILES["$(sanitize_sample "$a")"]="\${ALIAS_FILES[$a]}" ALIAS_BARCODE["$(sanitize_sample "$a")"]="\${ALIAS_BARCODE[$a]}" done (( \${#SAMPLE_NAMES[@]} > 0 )) || fail "No FASTQ files matched the sample sheet. Check the input folder and the sheet." else UNIT_KEYS=() # FASTQ file (file/single mode) or folder (group mode) UNIT_PRIMARY=() # name from the containing folder UNIT_FALLBACK=() # name from the relative path + file name if [[ "$INPUT_MODE" == "single" ]]; then UNIT_KEYS+=("$INPUT_PATH") UNIT_PRIMARY+=("$(sanitize_sample "$(basename -- "$(dirname -- "$INPUT_PATH")")")") UNIT_FALLBACK+=("$(sanitize_sample "$(strip_fastq_ext "$(basename -- "$INPUT_PATH")")")") elif [[ "$FOLDER_BEHAVIOR" == "file" ]]; then while IFS= read -r -d '' f; do rel="\${f#"$INPUT_PATH"/}" UNIT_KEYS+=("$f") if [[ "$rel" != */* ]]; then # FASTQ sits directly in the selected folder: use its file name. UNIT_PRIMARY+=("$(sanitize_sample "$(strip_fastq_ext "$rel")")") elif [[ "$NAMING_MODE" == "immediate" ]]; then UNIT_PRIMARY+=("$(sanitize_sample "$(basename -- "$(dirname -- "$f")")")") else UNIT_PRIMARY+=("$(sanitize_sample "\${rel%%/*}")") fi UNIT_FALLBACK+=("$(sanitize_sample "$(strip_fastq_ext "$rel")")") done < <(find_fastqs "$INPUT_PATH" "\${INPUT_DEPTH:-}" | sort -z) else declare -A SEEN_DIRS=() while IFS= read -r -d '' f; do SEEN_DIRS["$(dirname -- "$f")"]=1 done < <(find_fastqs "$INPUT_PATH" "\${INPUT_DEPTH:-}") if (( \${#SEEN_DIRS[@]} > 0 )); then mapfile -t dirs < <(printf '%s\\n' "\${!SEEN_DIRS[@]}" | sort) for d in "\${dirs[@]}"; do UNIT_KEYS+=("$d") if [[ "$d" == "$INPUT_PATH" ]]; then UNIT_PRIMARY+=("$(sanitize_sample "$(basename -- "$d")")") UNIT_FALLBACK+=("$(sanitize_sample "$(basename -- "$d")")") else rel="\${d#"$INPUT_PATH"/}" if [[ "$NAMING_MODE" == "immediate" ]]; then UNIT_PRIMARY+=("$(sanitize_sample "$(basename -- "$d")")") else UNIT_PRIMARY+=("$(sanitize_sample "\${rel%%/*}")") fi UNIT_FALLBACK+=("$(sanitize_sample "$rel")") fi done fi fi (( \${#UNIT_KEYS[@]} > 0 )) || fail "No FASTQ files found under $INPUT_PATH" declare -A NAME_COUNT=() for i in "\${!UNIT_KEYS[@]}"; do n="\${UNIT_PRIMARY[$i]}" NAME_COUNT["$n"]=$(( \${NAME_COUNT["$n"]:-0} + 1 )) done declare -A NAME_USED=() SAMPLE_NAMES=() for i in "\${!UNIT_KEYS[@]}"; do if [[ "$NAMING_MODE" != "path" && "\${NAME_COUNT["\${UNIT_PRIMARY[$i]}"]}" -eq 1 ]]; then n="\${UNIT_PRIMARY[$i]}" else n="\${UNIT_FALLBACK[$i]}" if [[ "$NAMING_MODE" != "path" ]]; then warn "Folder name '\${UNIT_PRIMARY[$i]}' is shared by several FASTQ sets; using '$n' instead." fi fi base="$n" k=2 while [[ -n "\${NAME_USED["$n"]:-}" ]]; do n="\${base}_$k"; k=$((k + 1)); done NAME_USED["$n"]=1 SAMPLE_NAMES+=("$n") done fi echo echo "Samples to process (\${#SAMPLE_NAMES[@]}):" for i in "\${!SAMPLE_NAMES[@]}"; do printf ' %-28s ← %s\\n' "\${SAMPLE_NAMES[$i]}" "\${UNIT_KEYS[$i]}" done # v26: reports only — say so in the log, and stop if none of the phages has an assembly to rebuild reports for if [[ "\${REPORTS_ONLY:-false}" == "true" ]]; then ro_have=0 for n in "\${SAMPLE_NAMES[@]}"; do [[ -s "$OUT_ROOT/$n/$n.fasta" ]] && ro_have=$((ro_have + 1)); done echo echo "📄 REPORTS ONLY: $ro_have of \${#SAMPLE_NAMES[@]} phage(s) have an assembly in $OUT_ROOT; only their reports are rebuilt." if (( ro_have == 0 )); then fail "'Reports only' is on, but none of the \${#SAMPLE_NAMES[@]} phages has an assembly in $OUT_ROOT, so there is nothing to do. Untick 'Reports only' (section 1 of the generator) to assemble them." fi fi # ============================================================== # Run # ============================================================== # unit_files INDEX: the FASTQ file(s) of one sample, into the array UNIT_FILES unit_files() { local i="$1" UNIT_FILES=() if [[ -n "$SAMPLE_SHEET" ]]; then mapfile -t UNIT_FILES <<< "\${ALIAS_FILES["\${SAMPLE_NAMES[$i]}"]}" elif [[ "$INPUT_MODE" != "single" && "$FOLDER_BEHAVIOR" == "group" ]]; then mapfile -d '' -t UNIT_FILES < <(find_fastqs "\${UNIT_KEYS[$i]}" 1 | sort -z) else UNIT_FILES=("\${UNIT_KEYS[$i]}") fi } # Which samples get host-read removal: shown before anything runs. if [[ "$HOST_REMOVAL" == "plaque" ]] && is_true "$HOST_ALL"; then echo echo "🧹 Host-read removal will run on all \${#SAMPLE_NAMES[@]} samples (reads matching the host genome are removed;" echo " if fewer than \${HOST_MIN_KEEP_PCT}% of a sample's bases would be left, that sample keeps all its reads)." elif [[ "$HOST_REMOVAL" == "plaque" ]]; then HR_LIST=() for i in "\${!SAMPLE_NAMES[@]}"; do unit_files "$i" if is_plaque_sample "\${SAMPLE_NAMES[$i]}" "\${UNIT_FILES[@]}"; then HR_LIST+=("\${SAMPLE_NAMES[$i]}"); fi done echo echo "🧹 Host-read removal will run on \${#HR_LIST[@]} of \${#SAMPLE_NAMES[@]} samples (picked-plaque list):" if (( \${#HR_LIST[@]} )); then printf ' %s\\n' "\${HR_LIST[@]}"; else echo " none: no sample matched the list"; fi if (( \${#HR_LIST[@]} * 2 > \${#SAMPLE_NAMES[@]} )); then warn "More than half of the samples are treated as picked plaques. If that's not right, stop now (Ctrl+C) and fix the list in the generator." fi fi run_unit() { unit_files "$1" run_sample "\${SAMPLE_NAMES[$1]}" "\${UNIT_FILES[@]}" } # v20: check only — what a real run would do with each phage, then stop if [[ "$CHECK_ONLY" == "true" ]]; then echo echo "==============================================================" echo "CHECK ONLY — nothing is assembled" echo "==============================================================" printf ' %-28s %5s %9s %s\\n' "phage" "files" "Mb reads" "a real run would…" n_new=0; n_reuse=0; n_redo=0; n_over=0; n_old=0 for i in "\${!SAMPLE_NAMES[@]}"; do n="\${SAMPLE_NAMES[$i]}" unit_files "$i" mb="$(du -cb -- "\${UNIT_FILES[@]}" 2>/dev/null | tail -n 1 | awk '{ printf "%.1f", $1 / 1e6 }')" d="$OUT_ROOT/$n" if [[ "\${REPORTS_ONLY:-false}" == "true" ]]; then if [[ -s "$d/$n.fasta" ]]; then act="rebuild the report (assembly kept as it is)"; n_reuse=$((n_reuse + 1)) else act="skip (no assembly; its existing report is kept)"; n_new=$((n_new + 1)); fi elif [[ ! -s "$d/$n.fasta" ]]; then act="assemble (new)"; n_new=$((n_new + 1)) elif [[ "$OVERWRITE" == "true" ]]; then act="DELETE the existing results and reassemble (Overwrite is on)"; n_over=$((n_over + 1)) elif [[ ! -s "$d/qc/assembly_signature.txt" ]]; then act="keep the existing assembly (made before v17)"; n_old=$((n_old + 1)) elif [[ "$(cat "$d/qc/assembly_signature.txt")" != "$(assembly_signature "\${UNIT_FILES[@]}")" ]]; then act="reassemble: settings or reads changed (old results kept in previous_run/)"; n_redo=$((n_redo + 1)) else act="reuse the existing assembly (same settings and reads)"; n_reuse=$((n_reuse + 1)); fi printf ' %-28s %5s %9s %s\\n' "$n" "\${#UNIT_FILES[@]}" "$mb" "$act" done echo if [[ "\${REPORTS_ONLY:-false}" == "true" ]]; then echo "Phages: \${#SAMPLE_NAMES[@]} — reports only: $n_reuse report(s) rebuilt, $n_new skipped (no assembly)" else echo "Phages: \${#SAMPLE_NAMES[@]} — $n_new new, $n_redo reassembled, $n_reuse reused, $n_old kept (older version), $n_over deleted and redone" fi echo "Settings: $ASSEMBLY_MODE mode · read budget \${AC_MAX_MB} Mb · genome size \${GENOME_SIZE:-auto} (smallest estimate \${AC_MIN_GSIZE}) · first Flye \${FLYE_FIRST_MB} Mb" if [[ "$HOST_REMOVAL" == "off" ]]; then echo "Host reads: off" elif [[ "\${HOST_ALL:-false}" == "true" ]]; then echo "Host reads: removed from all samples" else echo "Host reads: removed from \${PLAQUE_SAMPLES:-no samples (none listed)}"; fi echo "After assembly: Pharokka $RUN_PHAROKKA · Phold $RUN_PHOLD · genome checks $CP_RUN · PhageTracker export $EXPORT_PHAGETRACKER · upload folder $MAKE_UPLOAD" echo "Output: $OUT_ROOT" echo echo "To run for real: the same command without --check." exit 0 fi # v16: work on the WSL drive (much faster than /mnt/c for the many small files assemblers write) WORK_LOCAL="\${WORK_LOCAL:-false}" LOCAL_RUN_DIR="" if is_true "$WORK_LOCAL"; then if [[ "$OUT_ROOT" != /mnt/* ]]; then echo "ℹ️ The output folder is already on the WSL drive; working there directly." WORK_LOCAL=false else LOCAL_WORK_DIR="\${LOCAL_WORK_DIR:-~/phage_pipeline_work}" LOCAL_WORK_DIR="\${LOCAL_WORK_DIR/#\\~/$HOME}" LOCAL_RUN_DIR="$LOCAL_WORK_DIR/$(basename "$OUT_ROOT")_\${STAMP}" if mkdir -p "$LOCAL_RUN_DIR" 2>/dev/null; then free_gb="$(df -Pk "$LOCAL_RUN_DIR" | awk 'NR == 2 { print int($4 / 1048576) }')" echo "Working on the WSL drive: $LOCAL_RUN_DIR (\${free_gb} GB free); each phage is copied to $OUT_ROOT when it finishes." echo "If the run is stopped, unfinished phages stay there and can be deleted." if (( free_gb < 5 * PARALLEL_SAMPLES )); then warn "Only \${free_gb} GB free on the WSL drive for $PARALLEL_SAMPLES phage(s) at once; a deep phage in accuracy mode can need several GB." fi else warn "Could not create $LOCAL_RUN_DIR; working in $OUT_ROOT instead." WORK_LOCAL=false fi fi fi if (( PARALLEL_SAMPLES <= 1 )); then for i in "\${!SAMPLE_NAMES[@]}"; do run_unit "$i" done else SLOG_DIR="$TOOLS_DIR/sample_logs_\${STAMP}" mkdir -p "$SLOG_DIR" echo echo "Running \${#SAMPLE_NAMES[@]} samples, $PARALLEL_SAMPLES at a time. Each sample's output is shown when it finishes" echo "(and saved as /.pipeline.log)." n_started=0 JOB_PIDS=() for i in "\${!SAMPLE_NAMES[@]}"; do # wait for a free slot (polling: a plain "wait" would also wait for the log's tee process) while (( $(jobs -rp | wc -l) >= PARALLEL_SAMPLES )); do sleep 2; done n_started=$((n_started + 1)) name="\${SAMPLE_NAMES[$i]}" echo "[$(date +%H:%M:%S)] ▶ $n_started/\${#SAMPLE_NAMES[@]} started: $name" ( slog="$SLOG_DIR/\${name}.log" run_unit "$i" > "$slog" 2>&1 || true res="$(awk -F'\\t' -v s="$name" '$1 == s { r = $2 } END { print r }' "$TOOLS_DIR/samples_\${STAMP}.tsv" 2>/dev/null)" if [[ -d "$OUT_ROOT/$name" && "$res" != "skipped" ]]; then cp -- "$slog" "$OUT_ROOT/$name/\${name}.pipeline.log" 2>/dev/null || true; fi { if command -v flock >/dev/null 2>&1; then flock 9; fi echo echo "==============================================================" echo "[$(date +%H:%M:%S)] $( [[ "$res" == "ok" ]] && echo "✅" || { [[ "$res" == "skipped" ]] && echo "⏭ " || echo "❌"; } ) $name finished (\${res:-unknown}) — its output:" cat -- "$slog" } 9>> "$TOOLS_DIR/print.lock" ) & JOB_PIDS+=("$!") done for pid in "\${JOB_PIDS[@]}"; do wait "$pid" 2>/dev/null || true; done echo echo "[$(date +%H:%M:%S)] All samples finished." fi if [[ -n "$LOCAL_RUN_DIR" ]]; then rmdir -- "$LOCAL_RUN_DIR" 2>/dev/null \\ || warn "Some phages were not copied back from the WSL drive; they are in $LOCAL_RUN_DIR." fi # counts and summary order (samples finish in any order when run in parallel) OK_COUNT=$(awk -F'\\t' '$2 == "ok"' "$TOOLS_DIR/samples_\${STAMP}.tsv" 2>/dev/null | wc -l) FAIL_COUNT=$(awk -F'\\t' '$2 == "failed"' "$TOOLS_DIR/samples_\${STAMP}.tsv" 2>/dev/null | wc -l) if (( PARALLEL_SAMPLES > 1 )); then { head -n 1 "$SUMMARY_FILE"; tail -n +2 "$SUMMARY_FILE" | sort -f -t $'\\t' -k1,1; } > "$SUMMARY_FILE.tmp" && mv -- "$SUMMARY_FILE.tmp" "$SUMMARY_FILE" fi # v17: compare this run's assemblies (all contigs): the same phage under two names, and small shared elements if [[ -n "$MM2_ENV" ]]; then XLIST="$TOOLS_DIR/assemblies_\${STAMP}.tsv" : > "$XLIST" for n in "\${SAMPLE_NAMES[@]}"; do [[ -s "$OUT_ROOT/$n/$n.fasta" ]] && printf '%s\\t%s\\n' "$n" "$OUT_ROOT/$n/$n.fasta" >> "$XLIST" done if (( $(wc -l < "$XLIST") > 1 )); then use_env "$MM2_ENV" python3 "$TOOLS" crosscheck "$XLIST" "$OUT_ROOT/shared_sequences_\${STAMP}.tsv" "$FLYE_THREADS" || true use_env "$CORE_ENV" fi fi # time spent per step if [[ -d "$TOOLS_DIR/timings_\${STAMP}" ]]; then use_env "$CORE_ENV" python3 "$TOOLS" timings "$TOOLS_DIR/timings_\${STAMP}" "$OUT_ROOT/pipeline_timings_\${STAMP}.tsv" || true fi # ============================================================== # NanoComp: one report comparing every sample in this run # ============================================================== if is_true "$RUN_NANOCOMP"; then NC_FILES=() NC_NAMES=() for n in "\${SAMPLE_NAMES[@]}"; do f="$OUT_ROOT/$n/input/\${n}.unique.fastq.gz" if [[ -s "$f" ]]; then NC_FILES+=("$f"); NC_NAMES+=("$n"); fi done if (( \${#NC_FILES[@]} >= 2 )); then NC_DIR="$OUT_ROOT/nanocomp_\${STAMP}" echo echo "📊 NanoComp: comparing \${#NC_FILES[@]} samples (deduplicated reads)" use_env "$CORE_ENV" if run_limited "$NANOCOMP_TIME_LIMIT" NanoComp --fastq "\${NC_FILES[@]}" --names "\${NC_NAMES[@]}" --outdir "$NC_DIR" \\ --threads "$NANO_THREADS" "\${NANOCOMP_EXTRA[@]}" > "$OUT_ROOT/nanocomp_\${STAMP}.log" 2>&1; then echo "NanoComp report: $NC_DIR/NanoComp-report.html" echo "NanoStats for the Nanopore Workflow app's Dorado tab: $NC_DIR/NanoStats.txt" else warn "NanoComp failed (see $OUT_ROOT/nanocomp_\${STAMP}.log)." fi else echo "ℹ️ NanoComp needs at least two samples with reads; skipped." fi fi # ============================================================== # Genome checks: closest phages (PhagesDB + GenBank), host DNA %, CheckV, similarity # ============================================================== CLOSEST_TSV="" if is_true "$CP_RUN"; then echo echo "🔎 Genome checks (closest phages, host DNA %, CheckV, similarity) for the phages in $OUT_ROOT" if [[ -z "\${PHAGESDB_FASTA:-}" || ! -s "$PHAGESDB_FASTA" ]]; then warn "Genome checks need the PhagesDB FASTA (section 5); skipped." else use_env "$CORE_ENV" CP_DIR="$OUT_ROOT/closest_phages" cat > "$TOOLS_DIR/closest_phages.py" <<'CLOSEST_PHAGES_PY' ${closestPy()} CLOSEST_PHAGES_PY CP_ARGS=(--input "$OUT_ROOT" --out "$CP_DIR" --phagesdb "$PHAGESDB_FASTA" --threads "$NANO_THREADS" --ncbi "$CP_NCBI_MODE" --ncbi-hosts "$CP_NCBI_HOSTS") [[ -n "\${PHAGESDB_TABLE:-}" ]] && CP_ARGS+=(--table "$PHAGESDB_TABLE") for hg in "\${HOST_GENOME_1:-}" "\${HOST_GENOME_2:-}" "\${HOST_GENOME_3:-}"; do [[ -n "$hg" ]] && CP_ARGS+=(--host-genome "$hg"); done is_true "$CP_SIMILARITY" || CP_ARGS+=(--no-similarity) if is_true "$CP_CHECKV"; then CV_DB="\${CP_CHECKV_DB/#\\~/$HOME}" CV_BIN="" if [[ -x "$CONDA_BASE/envs/$CP_CHECKV_ENV/bin/checkv" ]]; then CV_BIN="$CONDA_BASE/envs/$CP_CHECKV_ENV/bin" elif is_true "$CP_CHECKV_INSTALL"; then echo "Creating the conda environment \\"$CP_CHECKV_ENV\\" with CheckV (once; a few minutes)…" if command -v mamba >/dev/null 2>&1; then CV_PM=mamba; else CV_PM=conda; fi if "$CV_PM" create -y -n "$CP_CHECKV_ENV" -c conda-forge -c bioconda checkv >/dev/null 2>&1 \\ && [[ -x "$CONDA_BASE/envs/$CP_CHECKV_ENV/bin/checkv" ]]; then CV_BIN="$CONDA_BASE/envs/$CP_CHECKV_ENV/bin" fi fi if [[ -n "$CV_BIN" ]]; then CP_ARGS+=(--checkv-db "$CV_DB" --checkv-bin "$CV_BIN") is_true "$CP_CHECKV_DOWNLOAD" && CP_ARGS+=(--checkv-download) else warn "CheckV is not installed (conda environment \\"$CP_CHECKV_ENV\\"); genome checks run without it." fi fi if python3 "$TOOLS_DIR/closest_phages.py" "\${CP_ARGS[@]}" > "$OUT_ROOT/closest_phages_\${STAMP}.log" 2>&1; then CLOSEST_TSV="$CP_DIR/closest_phages.tsv" python3 "$TOOLS" cpinject "$CLOSEST_TSV" "$OUT_ROOT" || warn "Could not add the genome-check results to the coverage reports." grep -E '^ [^ ]' "$OUT_ROOT/closest_phages_\${STAMP}.log" | head -n 60 || true echo "Genome checks report: $CP_DIR/closest_phages.html" else tail -n 15 "$OUT_ROOT/closest_phages_\${STAMP}.log" >&2 || true warn "Genome checks failed (see closest_phages_\${STAMP}.log); the assemblies are not affected." fi fi fi if is_true "$EXPORT_PHAGETRACKER"; then META="$TOOLS_DIR/meta_\${STAMP}.tsv" PHAGE_MAP_FILE="" if [[ -n "$PT_PHAGE_MAP" ]]; then PHAGE_MAP_FILE="$TOOLS_DIR/phage_map_\${STAMP}.tsv" printf '%s\\n' "$PT_PHAGE_MAP" > "$PHAGE_MAP_FILE" fi { printf 'generated_at\\t%s\\n' "$(date -Iseconds)" printf 'generator_version\\t%s\\n' "$GENERATOR_VERSION" printf 'phage_map\\t%s\\n' "$PHAGE_MAP_FILE" printf 'closest_tsv\\t%s\\n' "$CLOSEST_TSV" printf 'sample_set_identifier\\t%s\\n' "$PT_SAMPLE_SET" printf 'sequencing_run_id\\t%s\\n' "$PT_SEQ_RUN" printf 'sharepoint_base_url\\t%s\\n' "$PT_SHAREPOINT_URL" if [[ -n "$SAMPLE_SHEET" ]]; then printf 'sample_sheet\\t%s\\n' "$SAMPLE_SHEET" printf 'sample_sheet_matches\\t%s\\n' "$SHEET_MATCHES" fi if command -v wslpath >/dev/null 2>&1; then printf 'output_root_windows\\t%s\\n' "$(wslpath -w "$OUT_ROOT" 2>/dev/null || true)" fi for k in "\${!TOOL_VERSION[@]}"; do printf 'tool.%s\\t%s\\n' "$k" "\${TOOL_VERSION[$k]}"; done for k in MIN_LENGTH MIN_MEAN_Q TARGET_BASES KEEP_PERCENT FLYE_READ_TYPE GENOME_SIZE FLYE_ITERATIONS \\ MIN_OVERLAP ASM_COVERAGE FLYE_META FLYE_SCAFFOLD MEDAKA_READS MEDAKA_DEPTH MEDAKA_MODEL \\ MEDAKA_BACTERIA PHAROKKA_GENE_CALLER PHAROKKA_FAST PHOLD_COMPUTE AGREE_PCT LOWCOV_X \\ ASSEMBLY_MODE AC_ASSEMBLERS AC_COUNT AC_GENOME_SIZE AC_MAX_DEPTH AC_MAX_MB AC_MIN_GSIZE AC_READ_TYPE ROTATE FIND_ENDS \\ HOST_REMOVAL HOST_METHOD HOST_PROPHAGE HOST_NAME_1 HOST_GENOME_1 HOST_PROPHAGE_1 \\ HOST_NAME_2 HOST_GENOME_2 HOST_PROPHAGE_2 HOST_NAME_3 HOST_GENOME_3 HOST_PROPHAGE_3 \\ PLAQUE_SAMPLES HOST_MIN_ALIGNED_PCT HOST_CONTIG_FRACTION HOST_MIN_KEEP_PCT PARALLEL_SAMPLES WORK_LOCAL \\ AC_CROSSCHECK LIBRARY_KIT AC_FLYE_VOTE AC_FLYE_WEIGHT AC_FLYE_TIERS FLYE_FIRST_MB PHAGESDB_FASTA PHAGESDB_TABLE AC_TIMEOUT_MIN PHOLD_TIMEOUT_MIN; do printf 'setting.%s\\t%s\\n' "$k" "\${!k}" done } > "$META" use_env "$CORE_ENV" python3 "$TOOLS" bundle "$OUT_ROOT" "$TOOLS_DIR/samples_\${STAMP}.tsv" "$META" \\ "$OUT_ROOT/phagetracker_import_\${STAMP}.json" \\ || warn "Could not write the PhageTracker import file." fi # ============================================================== # SharePoint upload folder: only the files worth keeping, same layout # (/) so the PhageTracker SharePoint links work. # Files are hard-linked when possible (no extra disk space), else copied. # ============================================================== link_or_copy() { # link_or_copy SRC DST [[ -f "$1" ]] || return 0 mkdir -p "$(dirname "$2")" rm -f -- "$2" ln -- "$1" "$2" 2>/dev/null || cp -p -- "$1" "$2" } if is_true "$MAKE_UPLOAD"; then UP="$OUT_ROOT/SharePoint_Upload" mkdir -p "$UP" echo echo "📦 Building the SharePoint upload folder: $UP" up_count=0 for n in "\${SAMPLE_NAMES[@]}"; do d="$OUT_ROOT/$n" if [[ ! -s "$d/\${n}.fasta" ]]; then # v19: a phage that could not be assembled: its report (why it failed, its reads), qc text files and log if [[ -s "$d/\${n}.coverage.html" ]]; then rm -rf -- "$UP/$n" link_or_copy "$d/\${n}.coverage.html" "$UP/$n/\${n}.coverage.html" link_or_copy "$d/\${n}.pipeline.log" "$UP/$n/\${n}.pipeline.log" for f in "$d"/qc/*.txt "$d"/qc/*.tsv; do [[ -f "$f" ]] && link_or_copy "$f" "$UP/$n/qc/$(basename "$f")" done echo " $n: not assembled; its report (why, and what is in its reads) is included" up_count=$((up_count + 1)) else echo " skipped $n (no polished assembly)" fi continue fi rm -rf -- "$UP/$n" for f in "\${n}.fasta" "\${n}.longest_contig.fasta" "\${n}.flye.fasta" "\${n}.autocycler.fasta" \\ "\${n}.pharokka.gbk" "\${n}.phold.gbk" "\${n}.coverage.html" "input/\${n}.deduplication.txt"; do link_or_copy "$d/$f" "$UP/$n/$f" done if [[ -d "$d/nanoplot" ]]; then for f in "$d"/nanoplot/*; do [[ -f "$f" ]] && link_or_copy "$f" "$UP/$n/nanoplot/$(basename "$f")" done fi if [[ -d "$d/qc" ]]; then for f in "$d"/qc/*.txt "$d"/qc/*.tsv "$d"/qc/*.tsv.gz "$d"/qc/*.json; do [[ -f "$f" ]] && link_or_copy "$f" "$UP/$n/qc/$(basename "$f")" done if is_true "$UPLOAD_BAM"; then link_or_copy "$d/qc/\${n}.reads_vs_polished.bam" "$UP/$n/qc/\${n}.reads_vs_polished.bam" link_or_copy "$d/qc/\${n}.reads_vs_polished.bam.bai" "$UP/$n/qc/\${n}.reads_vs_polished.bam.bai" fi fi if is_true "$UPLOAD_READS"; then link_or_copy "$d/input/\${n}.unique.fastq.gz" "$UP/$n/input/\${n}.unique.fastq.gz" fi up_count=$((up_count + 1)) done for f in "$SUMMARY_FILE" "$OUT_ROOT/phagetracker_import_\${STAMP}.json"; do link_or_copy "$f" "$UP/$(basename "$f")" done if [[ -d "$OUT_ROOT/nanocomp_\${STAMP}" ]]; then for f in "$OUT_ROOT/nanocomp_\${STAMP}"/*; do [[ -f "$f" ]] && link_or_copy "$f" "$UP/nanocomp_\${STAMP}/$(basename "$f")" done fi { echo "SharePoint upload folder — made by the Nanopore phage pipeline on $(date)" echo echo "Upload everything in this folder (not the folder itself) into the SharePoint folder whose link" echo "was entered in the pipeline generator, so the PhageTracker links (//) work." echo echo "Per sample: .fasta (polished) .longest_contig.fasta (annotated genome)" echo " .flye.fasta or .autocycler.fasta (unpolished draft) .pharokka.gbk .phold.gbk" echo " .coverage.html nanoplot/ qc/ (read stats, coverage, genome ends, rotation, warnings)" echo "Not assembled: .coverage.html (why it failed, its reads, host %, PhagesDB matches), qc/, the sample's log" is_true "$UPLOAD_BAM" && echo " qc/.reads_vs_polished.bam (+ .bai) for IGV" is_true "$UPLOAD_READS" && echo " input/.unique.fastq.gz (deduplicated reads)" echo "Run level: pipeline_summary_*.tsv phagetracker_import_*.json pipeline_*.log nanocomp_*/" echo echo "Left out (can be regenerated, or duplicates of the Dorado Phage_FASTQ reads): input/ and filtered/" echo "reads, flye/ autocycler/ medaka/ working folders, pharokka/ and phold/ working folders, .phage_pipeline/." } > "$UP/README.txt" echo " $up_count sample folder(s) ready. Upload the contents of SharePoint_Upload to SharePoint." fi echo echo "==============================================================" echo "Summary: $OK_COUNT succeeded, $FAIL_COUNT failed" list_where() { awk -F'\\t' "NR > 1 && ($1) { printf \\"%s \\", \\$1 }" "$SUMMARY_FILE"; } x="$(list_where '$23 == "flye_fallback"')"; [[ -n "$x" ]] && echo "Flye used because Autocycler failed: $x" x="$(list_where '$2 == "ok" && ($25 == "" || $27 + 0 < 50)')"; [[ -n "$PDB_INDEX" && -n "$x" ]] && echo "ℹ️ No PhagesDB phage covers half of the genome (possible new cluster/singleton, or not a phage): $x" x="$(list_where '$19 == "reference"')"; [[ -n "$x" ]] && echo "Genome start matched to the closest PhagesDB phage: $x" x="$(list_where '$30 ~ /^not found/')"; [[ -n "$x" ]] && echo "⚠️ The relative's 3' overhang was not found at the genome start (check the start / genome end): $x" x="$(list_where '$23 == "flye_chosen"')"; [[ -n "$x" ]] && echo "Flye kept after the cross-check (more complete): $x" x="$(list_where '$2 == "ok" && $7 != "NA" && $7 + 0 < 15')"; [[ -n "$x" ]] && echo "⚠️ Median read depth below 15× (genome probably incomplete; resequence): $x" x="$(list_where '$2 == "ok" && $4 != "NA" && $4 + 0 < 10000')"; [[ -n "$x" ]] && echo "⚠️ Longest contig under 10 kb (check it is a phage, e.g. BLAST): $x" x="$(list_where '$24 != "" && $24 != "NA" && $24 + 0 > 0')"; [[ -n "$x" ]] && echo "ℹ️ Regions with under 25% of the median read depth (see qc/coverage_dips.tsv; PCR/modification bias or a joined element): $x" echo "==============================================================" if command -v column >/dev/null 2>&1; then cut -f1-13,18-20 "$SUMMARY_FILE" | column -t -s $'\\t' else cut -f1-13,18-20 "$SUMMARY_FILE" fi if [[ "$ASSEMBLY_MODE" == "accuracy" ]] || grep -q '⚠\\|possible\\|doubled\\|mixed' <(cut -f21 "$SUMMARY_FILE" | tail -n +2); then echo echo "Warnings per sample (column 'flags' in the summary table):" awk -F'\\t' 'NR > 1 && $21 != "" && $21 != "none" && $21 != "NA" { print " " $1 ": " $21 }' "$SUMMARY_FILE" fi echo echo "Summary table: $SUMMARY_FILE" echo "Full log: $LOG_FILE" if is_true "$MAKE_UPLOAD"; then echo "Upload folder: $OUT_ROOT/SharePoint_Upload" sleep 1 cp -p -- "$LOG_FILE" "$OUT_ROOT/SharePoint_Upload/$(basename "$LOG_FILE")" 2>/dev/null || true fi if is_true "$OPEN_EXPLORER" && command -v wslpath >/dev/null 2>&1; then explorer.exe "$(wslpath -w "$OUT_ROOT")" >/dev/null 2>&1 || true fi if (( FAIL_COUNT > 0 )); then exit 1 fi`; const shQuote = (value) => "'" + String(value).replace(/'/g, "'\"'\"'") + "'"; const boolWord = (value) => (value ? "true" : "false"); const num = (id, fallback) => { const v = Number($(id).value); return Number.isFinite(v) ? v : fallback; }; const int = (id, min, fallback) => Math.max(min, Math.floor(num(id, fallback))); function toWslPath(value) { let p = String(value ?? "").trim(); if (!p) return ""; if ((p.startsWith('"') && p.endsWith('"')) || (p.startsWith("'") && p.endsWith("'"))) { p = p.slice(1, -1).trim(); } if (/^file:\/\//i.test(p)) { try { const u = new URL(p); p = decodeURIComponent(u.pathname); if (/^\/[A-Za-z]:\//.test(p)) p = p.slice(1); } catch (err) { return p; } } p = p.replace(/^\\\\\?\\/, ""); const driveMatch = p.match(/^([A-Za-z]):[\\/]?(.*)$/); if (driveMatch) { const drive = driveMatch[1].toLowerCase(); const rest = driveMatch[2].replace(/\\/g, "/").replace(/^\/+/, "").replace(/\/+$/, ""); return rest ? `/mnt/${drive}/${rest}` : `/mnt/${drive}`; } if (/^\/mnt\/[A-Za-z](\/|$)/.test(p)) { p = p.replace(/^\/mnt\/([A-Za-z])/, (_, d) => `/mnt/${d.toLowerCase()}`); } return p.length > 1 ? p.replace(/\/+$/, "") : p; } // "12, barcode15 = Malina-LP1, Hitchcock-LP2" -> ["bc:12", "bc:15", "nm:malinalp1", "nm:hitchcocklp2"] function plaqueTokens(text) { const out = []; for (const item of String(text || "").split(/[,;\n]+/)) { for (const part of item.split("=")) { const p = part.trim(); if (!p) continue; // a barcode range such as "16-28", "16–28" or "barcode16-barcode28" const r = p.match(/^(?:barcode|bc|nb)?\s*0*(\d{1,3})\s*(?:-|–|—|to|\.\.)\s*(?:barcode|bc|nb)?\s*0*(\d{1,3})$/i); if (r) { const lo = Math.min(Number(r[1]), Number(r[2])), hi = Math.max(Number(r[1]), Number(r[2])); for (let b = lo; b <= hi && b - lo < 400; b++) if (!out.includes(`bc:${b}`)) out.push(`bc:${b}`); continue; } const m = p.match(/^(?:barcode|bc|nb)?\s*0*(\d{1,3})$/i); const tok = m ? `bc:${Number(m[1])}` : `nm:${p.toLowerCase().replace(/[^a-z0-9]/g, "")}`; if (tok !== "nm:" && !out.includes(tok)) out.push(tok); } } return out; } // ["bc:3","bc:4","bc:5","nm:malinalp1"] -> "barcodes 3–5; names: malinalp1" function describePlaqueTokens(tokens) { const bcs = tokens.filter((x) => x.startsWith("bc:")).map((x) => Number(x.slice(3))).sort((a, b) => a - b); const runs = []; for (const b of bcs) { const last = runs[runs.length - 1]; if (last && b === last[1] + 1) last[1] = b; else runs.push([b, b]); } const parts = []; if (bcs.length) parts.push(`${bcs.length} barcode${bcs.length === 1 ? "" : "s"} (${runs.map(([a, b]) => (a === b ? a : `${a}–${b}`)).join(", ")})`); const names = tokens.filter((x) => x.startsWith("nm:")).map((x) => x.slice(3)); if (names.length) parts.push(`name${names.length === 1 ? "" : "s"}: ${names.join(", ")}`); return parts.join("; "); } // Host genomes for host-read removal: HOST_NAME_i / HOST_GENOME_i / HOST_PROPHAGE_i (i = 1..3), // plus HOST_PROPHAGE = "no" only when every listed host has no prophage. function hostGenomeSettings() { const ref = $("hostMethod").value === "ref" || $("hostRemoval").value === "off"; const out = {}; let anyUnsure = false; let n = 0; for (let i = 1; i <= 3; i++) { const path = ref ? toWslPath($(`hostGenome${i}`).value) : ""; const pro = path ? $(`hostProphage${i}`).value : ""; out[`HOST_NAME_${i}`] = path ? ($(`hostName${i}`).value.trim() || `host${i}`) : ""; out[`HOST_GENOME_${i}`] = path; out[`HOST_PROPHAGE_${i}`] = pro; if (path) n++; if (path && pro !== "no") anyUnsure = true; } out.HOST_GENOME_COUNT = n; out.HOST_PROPHAGE = ref && n ? (anyUnsure ? "unsure" : "no") : ""; return out; } function readSettings() { const minQ = Math.max(0, num("minQ", 0)); const targetMb = Math.max(0, num("targetMb", 0)); const targetBases = Math.round(targetMb * 1e6); const keepPercent = Math.min(100, Math.max(0.1, num("keepPercent", 100))); const inputMode = $("inputMode").value; return { INPUT_MODE: inputMode, FOLDER_BEHAVIOR: $("folderBehavior").value, INPUT_DEPTH: inputMode === "single" ? "" : $("inputDepth").value, NAMING_MODE: $("namingMode").value, INPUT_PATH: toWslPath(inputMode === "single" ? $("singleFastq").value : $("parentFolder").value), SAMPLE_SHEET: toWslPath($("sampleSheet").value), OUT_ROOT: toWslPath($("outputFolder").value), RENAME_HEADERS: boolWord($("renameHeaders").checked), OVERWRITE: boolWord($("overwrite").checked && !$("reportsOnly").checked), REPORTS_ONLY: boolWord($("reportsOnly").checked), SKIP_UNCLASSIFIED: boolWord($("skipUnclassified").checked), CONDA_BASE_INPUT: $("condaBase").value.trim(), CORE_ENV: $("coreEnv").value.trim(), MEDAKA_ENV: $("medakaEnv").value.trim(), PHAROKKA_ENV: $("pharokkaEnv").value.trim(), PHOLD_ENV: $("pholdEnv").value.trim(), RUN_NANOPLOT: boolWord($("runNanoplot").checked), RUN_NANOCOMP: boolWord($("runNanocomp").checked), NANO_THREADS: int("nanoThreads", 1, 4), MIN_LENGTH: int("minLength", 0, 0), MIN_MEAN_Q: minQ > 0 ? (100 * (1 - Math.pow(10, -minQ / 10))).toFixed(4) : "", TARGET_BASES: $("useTarget").checked ? targetBases : 0, KEEP_PERCENT: $("useKeep").checked ? String(keepPercent) : "", HOST_REMOVAL: $("hostRemoval").value, HOST_METHOD: $("hostRemoval").value === "off" ? "ref" : $("hostMethod").value, ...hostGenomeSettings(), PLAQUE_SAMPLES: plaqueTokens($("plaqueSamples").value).join(" "), HOST_MIN_ALIGNED_PCT: Math.min(100, Math.max(10, num("hostMinAligned", 50))), HOST_CONTIG_FRACTION: Math.min(1, Math.max(0.01, num("hostContigFrac", 0.25))), HOST_MIN_KEEP_PCT: Math.min(100, Math.max(0, num("hostMinKeep", 20))), PARALLEL_SAMPLES: Math.min(8, int("parallelSamples", 1, 3)), WORK_LOCAL: boolWord($("workLocal").checked), LOCAL_WORK_DIR: $("localWorkDir").value.trim() || "~/phage_pipeline_work", AC_CROSSCHECK: boolWord($("acCrosscheck").checked), LIBRARY_KIT: $("libKit").value, FLYE_READ_TYPE: $("flyeReadType").value, GENOME_SIZE: $("genomeSize").value.trim(), FLYE_ITERATIONS: int("flyeIterations", 0, 0), FLYE_THREADS: int("flyeThreads", 1, 1), MIN_OVERLAP: int("minOverlap", 0, 0), ASM_COVERAGE: int("asmCoverage", 0, 0), FLYE_META: boolWord($("flyeMeta").checked), FLYE_SCAFFOLD: boolWord($("flyeScaffold").checked), ASSEMBLY_MODE: $("assemblyMode").value, AUTOCYCLER_ENV: $("autocyclerEnv").value.trim(), AC_ASSEMBLERS: $("acAssemblers").value.trim().toLowerCase().replace(/[,;]+/g, " ").replace(/\s+/g, " "), AC_COUNT: Math.min(8, int("acCount", 2, 4)), AC_TIMEOUT_MIN: int("acTimeout", 0, 30), AC_GENOME_SIZE: $("acGenomeSize").value.trim().toLowerCase() || "flye", AC_FLYE_VOTE: boolWord($("acFlyeVote").checked), AC_FLYE_WEIGHT: Math.min(6, Math.max(1, int("acFlyeWeight", 1, 1))), AC_FLYE_TIERS: $("acFlyeTiers").value.trim().replace(/[,;x×]+/gi, " ").replace(/\s+/g, " "), FLYE_FIRST_MB: Math.min(100, Math.max(1, int("flyeFirstMb", 1, 8))), AC_MAX_DEPTH: int("acMaxDepth", 0, 400), AC_MAX_MB: Math.max(0, num("acMaxMb", 30)), AC_MIN_GSIZE: int("acMinGsize", 0, 50000), AC_READ_TYPE: $("acReadType").value, AC_KEEP_ASSEMBLIES: boolWord($("acKeep").checked), ROTATE: $("rotate").value, PHAGESDB_FASTA: toWslPath($("phagesdbFasta").value), PHAGESDB_TABLE: toWslPath($("phagesdbTable").value), NCBI_BLAST_PROXY: $("ncbiBlastProxy").value.trim(), PHAGESDB_BLAST_URL: $("phagesdbBlastUrl").value.trim(), PHAGESDB_SEQ_FIELD: $("phagesdbSeqField").value.trim() || "SEQUENCE", PHAGESDB_DATALIB: $("phagesdbDatalib").value.trim(), REF_MIN_ALIGNED_PCT: 50, DNAAPLER_ENV: $("dnaaplerEnv").value.trim(), FIND_ENDS: boolWord($("findEnds").checked && $("libKit").value === "ligation"), MEDAKA_READS: $("medakaReads").value, MEDAKA_DEPTH: int("medakaDepth", 1, 300), MEDAKA_THREADS: int("medakaThreads", 1, 1), MEDAKA_MODEL: $("medakaBacteria").checked ? "" : $("medakaModel").value.trim(), MEDAKA_BACTERIA: boolWord($("medakaBacteria").checked), COMPARE: boolWord($("compare").checked), RUN_COVERAGE: boolWord($("runCoverage").checked), COVERAGE_READS: $("coverageReads").value, AGREE_PCT: Math.min(100, Math.max(50, num("agreePct", 80))), LOWCOV_X: Math.max(1, num("lowcovX", 20)), RUN_PHAROKKA: boolWord($("runPharokka").checked), PHAROKKA_DB_INPUT: $("pharokkaDb").value.trim() || "~/pharokka_databases", PHAROKKA_GENE_CALLER: $("pharokkaGene").value, PHAROKKA_FAST: boolWord($("pharokkaFast").checked), PHAROKKA_THREADS: int("pharokkaThreads", 1, 8), RUN_PHOLD: boolWord($("runPharokka").checked && $("runPhold").checked), PHOLD_DB_INPUT: $("pholdDb").value.trim() || "~/phold_databases", PHOLD_COMPUTE: $("pholdCompute").value, PHOLD_THREADS: int("pholdThreads", 1, 8), PHOLD_BATCH: int("pholdBatch", 1, 1), PHOLD_TIMEOUT_MIN: int("pholdTimeout", 0, 30), EXPORT_PHAGETRACKER: boolWord($("exportPt").checked), PT_SAMPLE_SET: $("ptSampleSet").value.trim(), PT_SEQ_RUN: $("ptSeqRun").value.trim(), PT_SHAREPOINT_URL: $("ptSharepoint").value.trim(), INSTALL_DBS: boolWord($("installDbs").checked), MAKE_UPLOAD: boolWord($("makeUpload").checked), UPLOAD_BAM: boolWord($("makeUpload").checked && $("uploadBam").checked), UPLOAD_READS: boolWord($("makeUpload").checked && $("uploadReads").checked), OPEN_EXPLORER: boolWord($("openExplorer").checked), PT_PHAGE_MAP: phageMapTextV15(), CP_RUN: boolWord($("cpRun").checked), CP_NCBI_MODE: $("cpNcbi").value, CP_NCBI_HOSTS: [...new Set([...[...wrap.querySelectorAll(".pp-genus:checked")].map((x) => x.value), ...$("cpGeneraOther").value.trim().split(/[\s,;]+/).filter(Boolean)])].join(" "), CP_CHECKV: boolWord($("cpRun").checked && $("cpCheckv").checked), CP_CHECKV_ENV: $("cpCheckvEnv").value.trim() || "checkv", CP_CHECKV_DB: $("cpCheckvDb").value.trim() || "~/databases/checkv", CP_CHECKV_INSTALL: boolWord($("cpCheckvInstall").checked), CP_CHECKV_DOWNLOAD: boolWord($("cpCheckvDownload").checked), CP_SIMILARITY: boolWord($("cpSimilarity").checked), GENERATOR_VERSION: "nanopore-phage-pipeline v26" }; } function makeScript(s) { const config = Object.entries(s) .map(([k, v]) => `${k}=${typeof v === "number" ? v : shQuote(v)}`) .join("\n"); return [ "#!/usr/bin/env bash", "", "# Nanopore bacteriophage pipeline (generated in Observable)", s.ASSEMBLY_MODE === "accuracy" ? "# Accuracy mode: FASTQ -> deduplicate -> Filtlong -> Autocycler consensus -> Medaka -> genome ends / rotation" : "# FASTQ -> deduplicate -> Filtlong -> Flye -> Medaka -> Flye-vs-Medaka report", "# -> longest contig -> Pharokka -> Phold", "# Paste into a WSL terminal or save as a .sh file and run with: bash phage_pipeline.sh", "", "(", "set -Eeuo pipefail", "", "# ------------------------- settings -------------------------", config, "# -------------------------------------------------------------", "", PIPELINE_BODY, ")", "", "PIPELINE_STATUS=$?", 'if [[ "$PIPELINE_STATUS" -ne 0 ]]; then', " echo", ' echo "Pipeline finished with exit code $PIPELINE_STATUS (see the log in the output folder)."', ' echo "Your WSL shell remains open."', "fi", "" ].join("\n"); } function updateVisibility() { const folderMode = $("inputMode").value === "folder"; wrap.querySelectorAll(".folder-only").forEach((el) => (el.style.display = folderMode ? "" : "none")); wrap.querySelectorAll(".single-only").forEach((el) => (el.style.display = folderMode ? "none" : "")); $("medakaDepthField").classList.toggle("phage-dim", $("medakaReads").value !== "capped"); const acc = $("assemblyMode").value === "accuracy"; $("accFields").style.display = acc ? "" : "none"; { const a = $("advAcc"); if (a) a.style.display = acc ? "" : "none"; } // v16: the Flye settings also apply to the Flye runs in Accuracy mode, so they are no longer dimmed there $("medakaModel").disabled = $("medakaBacteria").checked; $("coverageFields").classList.toggle("phage-dim", $("runCoverage").checked === false); { const pf = $("phagesdbFasta").value.trim(); $("phagesdbFastaPreview").textContent = pf ? `WSL: ${toWslPath(pf)}` : "Not set: no closest-phage search"; } const kit = $("libKit").value; const transposase = kit !== "ligation"; $("findEnds").disabled = transposase; $("libKitNote").textContent = transposase ? "The transposase cuts DNA at random, so reads don't start at the genome's real ends: genome-end detection is off and circular genomes start at terL. " + (kit === "rapid_pcr" ? "PCR also copies away base modifications and can lower read depth over modified or GC-rich regions." : "") : "Reads keep the molecule ends, so read start/stop pile-ups can show cos/pac ends."; const hostOn = $("hostRemoval").value !== "off"; $("plaqueField").style.display = $("hostRemoval").value === "plaque" ? "" : "none"; // v20: shown only when used if ($("hostRemoval").value === "all" && $("hostMethod").value !== "ref") $("hostMethod").value = "ref"; $("hostParamFields").style.display = hostOn ? "" : "none"; $("hostMethodField").style.display = hostOn ? "" : "none"; const refMode = $("hostMethod").value === "ref"; $("hostRefFields").style.display = refMode || !hostOn ? "" : "none"; { const t = $("hostRefsTitle"); if (t) t.textContent = hostOn ? "Host genomes (reads matching them are removed)" : "Host genomes — not used to remove reads now, but to recognise host DNA joined to contigs and for the host-DNA % in the genome checks"; } for (let i = 1; i <= 3; i++) { const hg = $(`hostGenome${i}`).value.trim(); $(`hostGenomePreview${i}`).textContent = hg ? `WSL: ${toWslPath(hg)}` : (i === 1 ? "No path entered" : ""); $(`hostProphage${i}`).style.visibility = hg ? "visible" : "hidden"; // v22: keeps its table column } if (!$("plaqueStatus").dataset.loaded) { const toks = plaqueTokens($("plaqueSamples").value); $("plaqueStatus").textContent = !hostOn ? "Off: every sample uses all reads." : (toks.length ? `Host reads will be removed for ${describePlaqueTokens(toks)}. All other samples use all reads.` : "No picked-plaque samples listed, so no reads are removed."); } $("ptFields").classList.toggle("phage-dim", $("exportPt").checked === false); $("cpFields").classList.toggle("phage-dim", $("cpRun").checked === false); $("cpGenusCount").textContent = `(${wrap.querySelectorAll(".pp-genus:checked").length} ticked)`; $("uploadFields").classList.toggle("phage-dim", $("makeUpload").checked === false); const pharokka = $("runPharokka").checked; $("pharokkaFields").classList.toggle("phage-dim", !pharokka); $("runPhold").disabled = !pharokka; $("pholdFields").classList.toggle("phage-dim", !pharokka || !$("runPhold").checked); } function updatePathPreviews() { for (const [field, preview] of [ ["parentFolder", "parentFolderPreview"], ["singleFastq", "singleFastqPreview"], ["outputFolder", "outputFolderPreview"] ]) { const v = $(field).value.trim(); $(preview).textContent = v ? `WSL: ${toWslPath(v)}` : "No path entered"; } const sheet = $("sampleSheet").value.trim(); $("sampleSheetPreview").textContent = sheet ? `WSL: ${toWslPath(sheet)}` : "No sample sheet: samples are named from folders"; $("namingMode").disabled = Boolean(sheet); $("folderBehavior").disabled = Boolean(sheet); } function updateTree(s) { const name = s.NAMING_MODE === "path" ? "PhageA__barcode07" : "PhageA"; const lines = [ `${s.OUT_ROOT || ""}/`, `├── ${name}/`, `│ ├── ${name}.fasta ← polished Medaka assembly`, s.ASSEMBLY_MODE === "accuracy" ? `│ ├── ${name}.autocycler.fasta ← unpolished Autocycler consensus` : `│ ├── ${name}.flye.fasta ← unpolished Flye assembly`, `│ ├── ${name}.longest_contig.fasta ← header >${name}${s.ASSEMBLY_MODE === "accuracy" && s.ROTATE !== "none" ? " (rotated if circular)" : ""}` ]; if (s.RUN_PHAROKKA === "true") lines.push(`│ ├── ${name}.pharokka.gbk`); if (s.RUN_PHOLD === "true") lines.push(`│ ├── ${name}.phold.gbk`); if (s.RUN_COVERAGE === "true") lines.push(`│ ├── ${name}.coverage.html ← coverage + consensus plot`); if (s.COMPARE === "true") lines.push(`│ ├── qc/${name}.flye_vs_medaka.summary.tsv / .changes.tsv`); if (s.RUN_NANOPLOT === "true") lines.push(`│ ├── nanoplot/${name}_NanoPlot-report.html, ${name}_NanoStats.txt`); if (s.FIND_ENDS === "true") lines.push(`│ ├── qc/genome_ends.txt ← read start/stop pile-ups (cos/pac ends)`); if (s.ASSEMBLY_MODE === "accuracy" && s.ROTATE !== "none") lines.push(`│ ├── qc/rotation.txt ← how the genome start was chosen`); if (s.RUN_PHAROKKA === "true") lines.push(`│ ├── qc/annotation_qc.txt ← coding density / split-gene check`); lines.push(`│ ├── qc/sample_flags.txt ← mixed sample / doubled genome warnings`); lines.push(`│ ├── qc/input_check.tsv ← v17: does every input read set (barcode/run) fit this genome`); lines.push(`│ ├── qc/failure.txt ← v18: not assembled: where it stopped and the likely cause`); lines.push(`│ ├── qc/assembly_signature.txt ← v17: settings + reads used (a change → automatic reassembly)`); lines.push(`│ ├── previous_run/ ← v17: the earlier genome/report when settings changed`); if (s.HOST_REMOVAL === "plaque" && s.PLAQUE_SAMPLES || s.HOST_REMOVAL === "all") { lines.push(`│ ├── qc/host_removal.txt ← % of reads that were phage (host reads removed)`); lines.push(`│ ├── input/${name}.phage_reads.fastq.gz ← reads used for assembly (host reads removed)`); } lines.push(s.ASSEMBLY_MODE === "accuracy" ? `│ └── input/ filtered/ autocycler/ medaka/ pharokka/ phold/ *.log` : `│ └── input/ filtered/ flye/ medaka/ pharokka/ phold/ *.log`); if (s.RUN_NANOCOMP === "true") lines.push(`├── nanocomp_/NanoComp-report.html, NanoStats.txt`); if (s.MAKE_UPLOAD === "true") lines.push(`├── SharePoint_Upload/ ← upload its contents: ${name}/ (results only), summary, JSON, log`); lines.push(`├── pipeline_summary_.tsv ← one row per sample`); if (s.EXPORT_PHAGETRACKER === "true") lines.push(`├── phagetracker_import_.json ← import in Tracker Sync (4. Import results)`); if (s.CP_RUN === "true") lines.push(`├── closest_phages/closest_phages.html ← closest phages (status, ICTV, duplicates), CheckV, host DNA %, similarity`); lines.push(`└── pipeline_.log`); $("treePreview").textContent = lines.join("\n"); } function validateInputs(s) { const errors = []; const warnings = []; const parent = $("parentFolder").value.trim(); const single = $("singleFastq").value.trim(); const output = $("outputFolder").value.trim(); const activeInput = s.INPUT_MODE === "single" ? single : parent; if (s.INPUT_MODE === "single" && !single) errors.push("Enter a FASTQ file path."); if (s.INPUT_MODE === "folder" && !parent) errors.push("Enter a parent folder path."); if (!output) errors.push("Enter an output folder path."); const envRe = /^(\/[^'\s]+|[A-Za-z0-9._-]+)$/; for (const [key, label] of [ ["CORE_ENV", "Filtlong + Flye"], ["MEDAKA_ENV", "Medaka"], ...(s.ASSEMBLY_MODE === "accuracy" ? [["AUTOCYCLER_ENV", "Autocycler"]] : []), ...(s.ASSEMBLY_MODE === "accuracy" && s.ROTATE !== "none" && s.DNAAPLER_ENV ? [["DNAAPLER_ENV", "dnaapler"]] : []), ...(s.RUN_PHAROKKA === "true" ? [["PHAROKKA_ENV", "Pharokka"]] : []), ...(s.RUN_PHOLD === "true" ? [["PHOLD_ENV", "Phold"]] : []) ]) { if (!s[key]) errors.push(`Enter the ${label} Conda environment.`); else if (!envRe.test(s[key])) errors.push(`The ${label} environment name contains unsupported characters.`); } if (s.INPUT_MODE === "folder" && /\.(fastq|fq)(\.gz)?$/i.test(parent)) { errors.push("The parent-folder path looks like a FASTQ file. Choose 'One FASTQ file' or remove the file name."); } if (s.INPUT_MODE === "single" && single && !/\.(fastq|fq)(\.gz)?$/i.test(single)) { warnings.push("The single-file path does not end in .fastq, .fastq.gz, .fq, or .fq.gz."); } if (/^\\\\[^\\?]/.test(activeInput)) errors.push("UNC network paths cannot be converted automatically."); if (/^\\\\[^\\?]/.test(output)) errors.push("The output path appears to be a UNC network path."); if (s.INPUT_PATH && s.OUT_ROOT && s.INPUT_MODE === "folder" && s.INPUT_PATH === s.OUT_ROOT) { errors.push("Use a different folder for output than for input."); } if (s.SAMPLE_SHEET && /\.csv$/i.test(s.SAMPLE_SHEET) === false) { warnings.push("The sample sheet path does not end in .csv."); } if (s.SAMPLE_SHEET === "" && s.EXPORT_PHAGETRACKER === "true") { warnings.push("Adding the app's Dorado sample sheet makes matching samples to barcodes exact."); } if (s.ASSEMBLY_MODE !== "accuracy" && s.ASM_COVERAGE > 0 && !s.GENOME_SIZE) errors.push("Flye --asm-coverage requires an estimated genome size."); if (s.ASSEMBLY_MODE === "accuracy") { const known = ["flye", "raven", "miniasm", "canu", "redbean", "metamdbg", "myloasm", "lja", "plassembler", "hifiasm", "necat", "nextdenovo"]; const asms = s.AC_ASSEMBLERS.split(" ").filter(Boolean); const bad = asms.filter((a) => !known.includes(a)); if (bad.length) errors.push(`Unknown Autocycler assembler(s): ${bad.join(", ")}`); if (new Set(asms).size < 2) errors.push("Accuracy mode needs at least two different assemblers."); if (s.AC_GENOME_SIZE !== "auto" && s.AC_GENOME_SIZE !== "flye" && !/^\d+(\.\d+)?[kKmM]?$/.test(s.AC_GENOME_SIZE)) errors.push("Autocycler genome size must be 'flye', 'auto' or a size like 50000 or 50k."); if (s.ROTATE === "reference" && !s.PHAGESDB_FASTA) errors.push("Genome start 'Match the closest PhagesDB phage' needs the PhagesDB genomes FASTA (section 4)."); if (s.PHAGESDB_FASTA && !/\.(fa|fasta|fna|fas)(\.gz)?$/i.test(s.PHAGESDB_FASTA)) warnings.push("The PhagesDB FASTA path does not end in .fasta, .fa or .fna (optionally .gz); check the file name."); if (s.AC_FLYE_VOTE === "true" && !/^\d+( \d+)*$/.test(s.AC_FLYE_TIERS)) errors.push("Read depths for the Flye votes: whole numbers separated by spaces, e.g. 200 100 50."); if (asms.includes("canu")) warnings.push("Canu adds a few minutes per phage (4 subsets). If it isn't installed, miniasm is used instead. Install: mamba install -n " + (s.AUTOCYCLER_ENV || "autocycler") + " -c conda-forge -c bioconda canu"); if (s.AC_MAX_DEPTH > 0 && s.AC_MAX_DEPTH < 100) warnings.push("Autocycler works best with at least ~100x depth before subsetting (25x+ per subset)."); if (s.ROTATE !== "none" && !s.DNAAPLER_ENV) warnings.push("No dnaapler environment: circular genomes without clear read ends will be left as assembled."); warnings.push(`Accuracy mode runs ${new Set(asms).size} assemblers × ${s.AC_COUNT} subsets per sample; expect several times the Standard run time.`); } if (num("minQ", 0) >= 20) warnings.push("Q20 is stringent; confirm that enough phage reads remain after filtering."); if (s.MEDAKA_THREADS > 4) warnings.push("Medaka generally does not benefit much from more than ~4 threads."); if (s.MEDAKA_READS === "all") { warnings.push("Medaka will use every deduplicated read; for very deep phage runs consider 'capped at a target depth'."); } if (s.HOST_REMOVAL === "all" && s.HOST_METHOD !== "ref") errors.push("Host-read removal for all samples uses the host genome method: choose 'Host genome' and enter the host genome(s)."); if ((s.HOST_REMOVAL === "plaque" && s.PLAQUE_SAMPLES || s.HOST_REMOVAL === "all") && s.HOST_METHOD === "ref") { if (!s.HOST_GENOME_COUNT) errors.push("Host-read removal by host genome: enter at least one host genome FASTA path (section 3)."); const names = []; for (let i = 1; i <= 3; i++) { const g = s[`HOST_GENOME_${i}`]; if (!g) continue; if (!/\.(fa|fasta|fna|fas)(\.gz)?$/i.test(g)) warnings.push(`Host ${i}: the genome path does not end in .fasta, .fa or .fna (optionally .gz).`); if (!s[`HOST_PROPHAGE_${i}`]) errors.push(`Host ${i} (${s[`HOST_NAME_${i}`]}): say whether this host carries a prophage (section 3).`); if (/\t/.test(s[`HOST_NAME_${i}`])) errors.push(`Host ${i}: the name cannot contain a tab.`); names.push(g); } if (new Set(names).size < names.length) warnings.push("The same host genome file is listed twice."); } if (s.HOST_REMOVAL === "plaque" && /(^| )bc:/.test(s.PLAQUE_SAMPLES)) { warnings.push("The picked-plaque list uses barcode numbers. They match that barcode in THIS run's FASTQ names; check the list belongs to this sample set (names are safer)."); } if (s.PARALLEL_SAMPLES > 4) warnings.push(`${s.PARALLEL_SAMPLES} phages at once may run out of memory (Pharokka and Medaka need a few GB each).`); if (s.PARALLEL_SAMPLES > s.FLYE_THREADS) warnings.push("More phages at once than threads: each phage will get a single thread."); if (s.HOST_REMOVAL === "plaque" && s.PLAQUE_SAMPLES && s.HOST_MIN_ALIGNED_PCT < 30) { warnings.push("A phage-read minimum aligned length below 30% may keep chimeric host reads."); } if (s.EXPORT_PHAGETRACKER === "true" && s.PT_SAMPLE_SET === "") { warnings.push("Enter the PhageTracker sample set identifier so the import can match barcodes to phages."); } if (s.PT_SHAREPOINT_URL && /^https?:\/\//i.test(s.PT_SHAREPOINT_URL) === false) { errors.push("The SharePoint link must start with https://"); } if (s.TARGET_BASES === 0 && s.ASSEMBLY_MODE !== "accuracy") { warnings.push("No --target_bases cap: very deep phage data can make Flye slow or fragment the assembly."); } return { errors, warnings }; } function validateGeneratedScript(script) { const errors = []; if (!script.startsWith("#!/usr/bin/env bash\n")) errors.push("Generated script does not start with a Bash shebang."); if (script.split("\n").length < 100) errors.push("Generated script appears to be missing line breaks."); if (/^(INPUT_PATH|OUT_ROOT)='[A-Za-z]:[\\/]/m.test(script)) errors.push("A Windows path was not converted to WSL format."); if (script.includes("@@PHAGE_TOOLS_PY@@")) errors.push("Embedded Python helper is missing."); return errors; } function update() { updateVisibility(); syncRadios(); updatePathPreviews(); const s = readSettings(); updateTree(s); { // Downloads folder of the Windows user, from the input/output paths (/mnt/c/Users//...) const m = String(s.OUT_ROOT || s.INPUT_PATH || "").match(/^(\/mnt\/[a-z]\/Users\/[^/]+)\//); const dl = m ? `${m[1]}/Downloads` : "/mnt/c/Users//Downloads"; $("runCmd").textContent = `f="$(ls -t "${dl}"/phage_pipeline*.sh | head -n 1)"; echo "Running: $f"; bash "$f"`; } $("qConversion").textContent = s.MIN_MEAN_Q ? `Q${num("minQ", 0)} → Filtlong --min_mean_q ${s.MIN_MEAN_Q}` : "Disabled"; // v20: pre-run check — the run in plain words, with warnings for combinations that caused trouble before { const yes = (v) => v === "true" || v === true; const size = (v) => { const m = String(v || "").trim().toLowerCase().match(/^([\d.]+)\s*([kmg]?)b?$/); return m ? parseFloat(m[1]) * ({ k: 1e3, m: 1e6, g: 1e9 }[m[2]] || 1) : 0; }; const gs = Math.max(size(s.GENOME_SIZE), Number(s.AC_MIN_GSIZE) || 0); const items = [], warns = []; items.push(`Input: ${s.INPUT_PATH || "—"} — ` + (s.INPUT_MODE === "single" ? "one FASTQ file" : (s.INPUT_DEPTH === "1" ? "this folder only" : "including subfolders") + (s.SAMPLE_SHEET ? ", named by the Dorado sample sheet" : s.FOLDER_BEHAVIOR === "group" ? ", FASTQs combined per folder" : ", each FASTQ file a separate sample"))); items.push(`Output: ${s.OUT_ROOT || "—"}`); items.push(`Assembly: ${s.ASSEMBLY_MODE === "accuracy" ? "Accuracy (Autocycler + Flye cross-check)" : "Standard (Flye)"} · read budget ${s.AC_MAX_MB} Mb · ` + `genome size ${s.GENOME_SIZE || "—"} (smallest estimate ${s.AC_MIN_GSIZE}) · first Flye run ${s.FLYE_FIRST_MB} Mb`); const hr = s.HOST_REMOVAL === "off" ? "off (all reads used)" : s.HOST_REMOVAL === "all" ? "removed from all samples" : `removed only from: ${s.PLAQUE_SAMPLES || "(no samples listed)"}`; items.push(`Host reads: ${hr}`); items.push(`After assembly: Pharokka ${yes(s.RUN_PHAROKKA) ? "on" : "off"} · Phold ${yes(s.RUN_PHOLD) ? "on" : "off"} · genome checks ${yes(s.CP_RUN) ? "on" : "off"} · ` + `PhageTracker export ${yes(s.EXPORT_PHAGETRACKER) ? `on (sample set ${s.PT_SAMPLE_SET || "—"}${s.PT_SEQ_RUN ? ", run " + s.PT_SEQ_RUN : ""})` : "off"} · ` + `SharePoint upload folder ${yes(s.MAKE_UPLOAD) ? "on" : "off"}`); if (yes(s.REPORTS_ONLY)) warns.push("Reports only is on: nothing is assembled. Phages that already have an assembly in the output folder get new reports; all others are skipped. Untick it (section 1) for a new folder or new phages."); if (yes(s.OVERWRITE)) warns.push("Overwrite is on: every phage that already has results in the output folder will be deleted and reassembled. Turn it off to keep them (changed settings or reads are reassembled automatically)."); if (gs >= 100000) warns.push(`Genome size is set for large phages (${s.GENOME_SIZE} / ${s.AC_MIN_GSIZE}). Fine for ~90–150 kb phages; a batch of ~40–70 kb phages would run slower and Flye may fail — use the ~40–70 kb profile for those.`); if (s.HOST_REMOVAL === "plaque" && !s.PLAQUE_SAMPLES) warns.push("Host removal is set to “only these samples” but no samples are listed, so no host reads will be removed."); if (yes(s.EXPORT_PHAGETRACKER) && !s.PT_SAMPLE_SET) warns.push("PhageTracker export is on but the sample set identifier is empty."); if (s.PHAGESDB_DATALIB) warns.push(`The PhagesDB BLAST database is fixed to ${s.PHAGESDB_DATALIB}; leave it blank to use the newest automatically.`); $("precheck").innerHTML = "
Before you run
" + warns.map((w) => `
⚠️ ${w}
`).join("") + items.map((x) => `
${x}
`).join("") + "
" + '
Not sure what will happen to each phage? Use Copy check command: it lists them and what a run would do, without assembling.
'; // v20: the download is named after the run, e.g. phage_pipeline_SampleSet-3.sh (the run command finds it) const lab = String(s.PT_SAMPLE_SET || String(s.OUT_ROOT || "").split("/").filter(Boolean).pop() || "").replace(/[^A-Za-z0-9._-]+/g, "-").replace(/^-+|-+$/g, "").slice(0, 60); $("downloadBtn").setAttribute("download", lab ? `phage_pipeline_${lab}.sh` : "phage_pipeline.sh"); $("downloadBtn").textContent = lab ? `Download .sh (phage_pipeline_${lab}.sh)` : "Download .sh"; } const { errors: inputErrors, warnings } = validateInputs(s); const script = inputErrors.length === 0 ? makeScript(s) : ""; const errors = [...inputErrors, ...(script ? validateGeneratedScript(script) : [])]; const warningBox = $("warning"); warningBox.classList.toggle("phage-error", errors.length > 0); if (errors.length || warnings.length) { warningBox.style.display = "block"; warningBox.textContent = [...errors.map((x) => `❌ ${x}`), ...warnings.map((x) => `⚠️ ${x}`)].join("\n"); } else { warningBox.style.display = "none"; warningBox.textContent = ""; } $("code").textContent = errors.length ? "Complete/correct the required fields above to generate the WSL script." : script; $("codeSummary").textContent = errors.length ? "Show the generated script (not ready: complete the required fields above)" : `Show the generated script (${script.split("\n").length.toLocaleString()} lines; Copy / Download above use it)`; if (downloadUrl) { URL.revokeObjectURL(downloadUrl); downloadUrl = null; } const ok = errors.length === 0; if (ok) { downloadUrl = URL.createObjectURL(new Blob([script], { type: "text/x-shellscript" })); $("downloadBtn").href = downloadUrl; } else { $("downloadBtn").removeAttribute("href"); } $("downloadBtn").style.pointerEvents = ok ? "auto" : "none"; $("downloadBtn").style.opacity = ok ? "1" : "0.5"; $("copyBtn").disabled = !ok; $("copyRunBtn").disabled = !ok; $("copyCheckBtn").disabled = !ok; $("copyStatus").textContent = ""; wrap.value = ok ? script : ""; (wrap.__hooks || []).forEach((f) => { try { f(s, ok); } catch (e) { console.error(e); } }); } wrap.querySelectorAll("input, select").forEach((el) => { el.addEventListener("input", update); el.addEventListener("change", update); }); // v21: radio buttons drive the (hidden) input-mode, depth and folder-behaviour settings const scopeGroup = $("inputScope"); const syncRadios = () => { const mode = $("inputMode").value, depth = $("inputDepth").value; const want = mode === "single" ? "single" : depth === "1" ? "top" : "all"; for (const r of wrap.querySelectorAll('input[name="inputScope"]')) r.checked = r.value === want; for (const r of wrap.querySelectorAll('input[name="folderBeh"]')) r.checked = r.value === $("folderBehavior").value; $("folderBehaviorRadios").classList.toggle("phage-disabled", $("folderBehavior").disabled); // keep the scope radios directly under the visible path box const host = (mode === "single" ? $("singleFastq") : $("parentFolder")).closest(".phage-field"); const after = mode === "single" ? $("singleFastq") : $("parentFolder"); if (scopeGroup.parentElement !== host || scopeGroup.previousElementSibling !== after) after.insertAdjacentElement("afterend", scopeGroup); }; for (const r of wrap.querySelectorAll('input[name="inputScope"]')) { r.addEventListener("change", (ev) => { ev.stopPropagation(); $("inputMode").value = r.value === "single" ? "single" : "folder"; $("inputDepth").value = r.value === "top" ? "1" : ""; $("inputMode").dispatchEvent(new Event("change", { bubbles: true })); $("inputDepth").dispatchEvent(new Event("input", { bubbles: true })); }); } for (const r of wrap.querySelectorAll('input[name="folderBeh"]')) { r.addEventListener("change", (ev) => { ev.stopPropagation(); $("folderBehavior").value = r.value; $("folderBehavior").dispatchEvent(new Event("change", { bubbles: true })); }); } // v18: genome size group fills the three genome-size settings $("gsizeGroup").addEventListener("change", () => { const g = { small: ["50k", 50000, 10], large: ["135k", 130000, 20] }[$("gsizeGroup").value]; if (!g) return; $("genomeSize").value = g[0]; $("acMinGsize").value = g[1]; $("flyeFirstMb").value = g[2]; for (const id of ["genomeSize", "acMinGsize", "flyeFirstMb"]) $(id).dispatchEvent(new Event("input", { bubbles: true })); }); for (const id of ["plaqueSamples", "hostRemoval"]) { $(id).addEventListener("input", () => { delete $("plaqueStatus").dataset.loaded; updateVisibility(); }); $(id).addEventListener("change", () => { delete $("plaqueStatus").dataset.loaded; updateVisibility(); }); } $("loadPlaqueBtn").addEventListener("click", async () => { const status = $("plaqueStatus"); const id = $("ptSampleSet").value.trim(); status.dataset.loaded = "1"; if (!id) { status.textContent = "Enter the sample set identifier in section 9 first."; return; } status.textContent = "Loading…"; try { let db = window.__rudnerTrackerDb; if (!db) { const lib = await import("https://cdn.jsdelivr.net/npm/@supabase/supabase-js@2.45.4/+esm"); const createClient = lib && (lib.createClient || (lib.default && lib.default.createClient)); if (!createClient) throw new Error("the Supabase library could not be loaded"); db = window.__rudnerTrackerDb = createClient( "https://hbsqfdcdvluygqhlmsub.supabase.co", "sb_publishable_Bi5WDW6j4NkWTnPJUA-AiA_4KgpqM8g", { auth: { persistSession: true, storageKey: "rudner-tracker-sync" } }); } const { data: sess } = await db.auth.getSession(); if (!sess || !sess.session) throw new Error("not signed in. Sign in with an editor account in the Tracker Sync cell, then press the button again (or type the barcodes)"); const { data, error } = await db.rpc("get_sample_set_pipeline_map", { p_identifier: id }); if (error) throw error; const rows = (data && data.rows) || []; const plaque = rows.filter((r) => String(r.source_type || "").toLowerCase() === "plaque"); const others = rows.length - plaque.length; $("plaqueSamples").value = plaque.map((r) => r.phage).join(", "); status.textContent = plaque.length ? `${data.sample_set}: ${plaque.length} picked-plaque sample${plaque.length === 1 ? "" : "s"} (${plaque.map((r) => r.phage).join(", ")}); ${others} Lysate/DNA sample${others === 1 ? "" : "s"} keep all reads.` : `${data.sample_set}: no sample has DNA source "Plaque", so no host reads will be removed.`; update(); } catch (err) { status.textContent = "Could not load: " + ((err && err.message) || err); } }); async function trackerDbV15() { let db = window.__rudnerTrackerDb; if (!db) { const lib = await import("https://cdn.jsdelivr.net/npm/@supabase/supabase-js@2.45.4/+esm"); const createClient = lib && (lib.createClient || (lib.default && lib.default.createClient)); if (!createClient) throw new Error("the Supabase library could not be loaded"); db = window.__rudnerTrackerDb = createClient( "https://hbsqfdcdvluygqhlmsub.supabase.co", "sb_publishable_Bi5WDW6j4NkWTnPJUA-AiA_4KgpqM8g", { auth: { persistSession: true, storageKey: "rudner-tracker-sync" } }); } const { data: sess } = await db.auth.getSession(); if (!sess || !sess.session) throw new Error("not signed in. Sign in with an editor account in the Tracker Sync cell, then press the button again"); return db; } $("loadPhageIdsBtn").addEventListener("click", async () => { const status = $("phageIdsStatus"); const id = $("ptSampleSet").value.trim(); if (!id) { status.textContent = "Enter the sample set identifier first."; return; } status.textContent = "Loading…"; try { const db = await trackerDbV15(); const { data, error } = await db.rpc("get_sample_set_pipeline_map", { p_identifier: id }); if (error) throw error; const rows = (data && data.rows) || []; phageMapV15 = { key: id.toLowerCase().replace(/[^a-z0-9]+/g, ""), set: data.sample_set, rows }; const linked = rows.filter((r) => r.phage_id).length; status.textContent = `${data.sample_set}: ${linked} phage(s) linked by ID` + (rows.length - linked ? `, ${rows.length - linked} barcode(s) without a phage` : "") + (rows.some((r) => r.phage_id) ? "" : " — this sample set has no linked phages yet"); if (!$("ptSeqRun").value.trim() && data.sequencing_runs && data.sequencing_runs.length === 1) $("ptSeqRun").value = data.sequencing_runs[0]; update(); } catch (err) { status.textContent = "Could not load: " + ((err && err.message) || err); } }); $("ptSampleSet").addEventListener("input", () => { const st = $("phageIdsStatus"); if (phageMapV15 && phageMapV15.key !== $("ptSampleSet").value.trim().toLowerCase().replace(/[^a-z0-9]+/g, "")) { st.textContent = "Sample set changed — load the phage IDs again."; } }); $("copyCheckBtn").addEventListener("click", async () => { try { await navigator.clipboard.writeText($("runCmd").textContent.replace(/bash "\$f"$/, 'bash "$f" --check')); $("copyStatus").textContent = "Check command copied: download the .sh, paste into WSL. It lists each phage and what a run would do, then stops."; } catch (err) { $("copyStatus").textContent = "Copy failed; add --check at the end of the run command under the buttons."; } }); $("copyRunBtn").addEventListener("click", async () => { try { await navigator.clipboard.writeText($("runCmd").textContent); $("copyStatus").textContent = "Run command copied. Download the .sh first, then paste the command into WSL."; } catch (err) { $("copyStatus").textContent = "Copy failed; select the command under the buttons manually."; } }); $("copyBtn").addEventListener("click", async () => { try { await navigator.clipboard.writeText($("code").textContent); $("copyStatus").textContent = "Copied."; } catch (err) { $("copyStatus").textContent = "Copy failed; select the script manually."; } }); // ---- v336: remembered form values + Lab settings --------------------------------------------------- { const FORM_KEY = "rudnerForm.nanopore-pipeline.v1", LAB_KEY = "rudnerLabSettings.v1"; const LAB_MAP = {"condaBase": "condaBase", "coreEnv": "coreEnv", "medakaEnv": "medakaEnv", "parentFolder": "readLibrary", "phagesdbFasta": "phagesdbFasta", "phagesdbTable": "phagesdbTable", "pharokkaDb": "pharokkaDb", "pholdDb": "pholdDb", "hostName1": "hostName1", "hostGenome1": "hostGenome1", "hostName2": "hostName2", "hostGenome2": "hostGenome2", "hostName3": "hostName3", "hostGenome3": "hostGenome3", "cpCheckvEnv": "checkvEnv", "cpCheckvDb": "checkvDb"}; const GENUS_SEL = ".pp-genus"; const lsGet = k => { try { return JSON.parse(localStorage.getItem(k) || "null"); } catch (_) { return null; } }; const lsSet = (k, v) => { try { localStorage.setItem(k, JSON.stringify(v)); return true; } catch (_) { return false; } }; const fields = () => [...wrap.querySelectorAll("input[id], select[id], textarea[id]")].filter(el => el.type !== "file" && el.type !== "button"); const valOf = el => (el.type === "checkbox" ? el.checked : el.value); const setVal = (el, v) => { if (el.type === "checkbox") el.checked = !!v; else el.value = v; }; const genera = () => (GENUS_SEL ? [...wrap.querySelectorAll(GENUS_SEL + ":checked")].map(x => x.value).sort().join(" ") : ""); const setGenera = list => { const s = new Set(String(list).split(" ")); wrap.querySelectorAll(GENUS_SEL).forEach(x => { x.checked = s.has(x.value); }); }; const defaults = Object.fromEntries(fields().map(el => [el.id, valOf(el)])); const defGenera = genera(); function applyLab(force) { const lab = lsGet(LAB_KEY) || {}, saved = lsGet(FORM_KEY) || {}; let n = 0; for (const [id, key] of Object.entries(LAB_MAP)) { const el = wrap.querySelector("#" + id); if (!el || lab[key] == null || lab[key] === "") continue; if (force || !(id in saved)) { setVal(el, lab[key]); n++; } } return n; } function restore() { const saved = lsGet(FORM_KEY) || {}; for (const el of fields()) if (el.id in saved) setVal(el, saved[el.id]); if (GENUS_SEL && "__genera" in saved) setGenera(saved.__genera); } let timer = null; function save() { clearTimeout(timer); timer = setTimeout(() => { const lab = lsGet(LAB_KEY) || {}, out = {}; for (const el of fields()) { const v = valOf(el), labv = LAB_MAP[el.id] ? lab[LAB_MAP[el.id]] : undefined; if (v !== defaults[el.id] && !(labv != null && labv !== "" && v === labv)) out[el.id] = v; } if (GENUS_SEL && genera() !== defGenera) out.__genera = genera(); lsSet(FORM_KEY, out); }, 300); } const bar = document.createElement("div"); bar.style.cssText = "display:flex;gap:10px;align-items:center;flex-wrap:wrap;font-size:12.5px;color:#5f6b72;margin:0 0 6px"; bar.innerHTML = `Your changes here are remembered in this browser; empty fields come from the Lab settings cell. `; wrap.prepend(bar); const st = bar.querySelector("[data-mem-status]"); bar.querySelector("[data-mem-lab]").addEventListener("click", () => { const n = applyLab(true); st.textContent = n ? `${n} field(s) set from Lab settings.` : "No Lab settings saved yet (add the Lab settings cell to this page)."; save(); update(); }); bar.querySelector("[data-mem-forget]").addEventListener("click", () => { lsSet(FORM_KEY, {}); for (const el of fields()) setVal(el, defaults[el.id]); if (GENUS_SEL) setGenera(defGenera); applyLab(true); st.textContent = "Back to the defaults and Lab settings."; update(); }); applyLab(false); restore(); wrap.addEventListener("input", save); wrap.addEventListener("change", save); window.addEventListener("rudner-lab-settings", () => { applyLab(false); update(); }); } // ================================================================ v26: run panel and layout (function v26Layout() { // --- wide path fields for (const id of ["parentFolder", "singleFastq", "outputFolder", "sampleSheet", "namingMode", "phagesdbFasta", "phagesdbTable"]) { const f = $(id) && $(id).closest(".phage-field"); if (f && f.parentElement && f.parentElement.classList.contains("phage-grid")) f.classList.add("pp-wide"); } // --- host table: WSL previews only on hover / focus for (const i of [1, 2, 3]) { const p = $(`hostGenomePreview${i}`); if (p) p.classList.add("hp"); } // --- layout: form on the left, run panel on the right (stacked below on narrow pages, with a sticky mini bar) wrap.style.maxWidth = "1340px"; const pre = $("precheck"), warnBox = $("warning"), actions = $("downloadBtn").closest(".phage-actions"); const runNote = actions.nextElementSibling; const layout = document.createElement("div"); layout.className = "pp-layout"; const main = document.createElement("div"); main.className = "pp-main"; const aside = document.createElement("div"); aside.className = "pp-aside"; const firstBox = wrap.querySelector(".phage-profiles") || wrap.querySelector(".phage-legend") || wrap.querySelector(".phage-box"); wrap.insertBefore(layout, firstBox); while (layout.nextSibling) main.appendChild(layout.nextSibling); layout.append(main, aside); const changes = document.createElement("div"); changes.className = "pp-changes"; changes.id = "ppChanges"; aside.append(changes, pre, warnBox, actions, runNote); // actions: one main Download button; the copy buttons in a menu const dl = $("downloadBtn"); dl.classList.add("pp-dl"); const row = document.createElement("div"); row.className = "pp-row"; const copyBtns = ["copyRunBtn", "copyCheckBtn", "copyBtn"].map((id) => $(id)), copyStatus = $("copyStatus"); copyBtns[0].textContent = "Copy run command"; copyBtns[1].textContent = "Copy check command (what a run would do)"; copyBtns[2].textContent = "Copy the whole script"; actions.prepend(dl); row.append(...copyBtns, copyStatus); actions.appendChild(row); // narrow pages: sticky mini bar with the essentials const mini = document.createElement("div"); mini.className = "pp-mini"; mini.innerHTML = '' + ''; layout.insertAdjacentElement("afterend", mini); mini.addEventListener("click", (ev) => { const b = ev.target.closest("button"); if (!b) return; if (b.dataset.a === "dl") dl.click(); else changes.scrollIntoView({ behavior: "smooth", block: "start" }); }); // hide the mini bar while the run panel itself is on screen try { new IntersectionObserver((en) => { mini.classList.toggle("hide", en.some((x) => x.isIntersecting) && !layout.classList.contains("wide")); }).observe(actions); } catch (e) {} const fit = () => layout.classList.toggle("wide", wrap.getBoundingClientRect().width >= 1120); try { new ResizeObserver(fit).observe(wrap); } catch (e) { window.addEventListener("resize", fit); } fit(); // --- what changed since the last download for this output folder const KEY = "phagePipelineLastDownload.v1"; const SIG = ["ASSEMBLY_MODE", "MIN_LENGTH", "MIN_MEAN_Q", "TARGET_BASES", "KEEP_PERCENT", "HOST_REMOVAL", "HOST_METHOD", "PLAQUE_SAMPLES", "FLYE_READ_TYPE", "GENOME_SIZE", "FLYE_ITERATIONS", "MIN_OVERLAP", "ASM_COVERAGE", "FLYE_META", "FLYE_SCAFFOLD", "AC_ASSEMBLERS", "AC_COUNT", "AC_GENOME_SIZE", "AC_FLYE_VOTE", "AC_FLYE_WEIGHT", "AC_FLYE_TIERS", "FLYE_FIRST_MB", "AC_MAX_DEPTH", "AC_MAX_MB", "AC_MIN_GSIZE", "AC_READ_TYPE", "AC_CROSSCHECK", "MEDAKA_READS", "MEDAKA_DEPTH", "MEDAKA_MODEL", "MEDAKA_BACTERIA"]; const READS = ["INPUT_PATH", "INPUT_MODE", "INPUT_DEPTH", "FOLDER_BEHAVIOR", "NAMING_MODE", "SAMPLE_SHEET", "SKIP_UNCLASSIFIED"]; const IGNORE = ["GENERATOR_VERSION", "OUT_ROOT", "PT_PHAGE_MAP", "REPORTS_ONLY"]; const LBL = { ASSEMBLY_MODE: "Assembly mode", MIN_LENGTH: "Minimum read length", MIN_MEAN_Q: "Minimum mean read quality", TARGET_BASES: "Filtlong target bases", KEEP_PERCENT: "Filtlong keep %", HOST_REMOVAL: "Host-read removal", HOST_METHOD: "Host-removal method", PLAQUE_SAMPLES: "Picked-plaque samples", GENOME_SIZE: "Genome size", AC_MAX_MB: "Reads for Autocycler (Mb)", AC_MIN_GSIZE: "Smallest genome-size estimate", FLYE_FIRST_MB: "First Flye run (Mb)", AC_ASSEMBLERS: "Autocycler assemblers", AC_COUNT: "Autocycler read subsets", MEDAKA_READS: "Reads for Medaka", MEDAKA_DEPTH: "Medaka read depth", MEDAKA_MODEL: "Medaka model", INPUT_PATH: "Input folder", OVERWRITE: "Overwrite", REPORTS_ONLY: "Reports only", RUN_PHAROKKA: "Pharokka", RUN_PHOLD: "Phold", CP_RUN: "Genome checks", EXPORT_PHAGETRACKER: "PhageTracker export", MAKE_UPLOAD: "SharePoint upload folder", PARALLEL_SAMPLES: "Phages at once", PT_SAMPLE_SET: "PhageTracker sample set", HOST_GENOME_1: "Host genome 1", HOST_GENOME_2: "Host genome 2", HOST_GENOME_3: "Host genome 3" }; const load = () => { try { return JSON.parse(localStorage.getItem(KEY) || "{}") || {}; } catch (e) { return {}; } }; const short = (v) => { v = String(v == null ? "" : v); return v === "" ? "(empty)" : (v.length > 46 ? "…" + v.slice(-44) : v); }; const esc = (t) => String(t).replace(/&/g, "&").replace(/ { const prev = load()[st.OUT_ROOT || "(none)"]; let html = "", red = 0, n = 0, cls = ""; const ro = st.REPORTS_ONLY === "true"; if (!st.OUT_ROOT) html = "
Changes
Enter the output folder.
"; else if (!prev) { html = "
Changes since your last download
No download recorded for " + esc(short(st.OUT_ROOT)) + " in this browser " + "(a new output folder, or the script was made elsewhere)." + (ro ? "" : " If the folder already has results, use Copy check command first to see what will happen to each phage.") + "
"; } else { const ch = []; for (const k of Object.keys(st)) { if (IGNORE.includes(k)) continue; if (String(prev.s[k] == null ? "" : prev.s[k]) === String(st[k])) continue; const kind = SIG.includes(k) ? "sig" : READS.includes(k) ? "reads" : "other"; ch.push([kind, k, prev.s[k], st[k]]); } ch.sort((a, b) => ["sig", "reads", "other"].indexOf(a[0]) - ["sig", "reads", "other"].indexOf(b[0])); n = ch.length; red = ro ? 0 : ch.filter((c) => c[0] !== "other").length; const when = new Date(prev.t); html = "
Changes since your last download
for this output folder, " + when.toLocaleDateString() + " " + when.toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" }) + "
"; if (!n) html += "
✅ Same settings: existing assemblies will be reused.
"; else { if (red) html += "
⚠️ " + red + " change(s) make the pipeline reassemble every phage in this folder" + " (the earlier results are kept in each phage's previous_run/). Undo them, or tick Reports only to keep the assemblies.
"; else if (ro && ch.some((c) => c[0] !== "other")) html += "
Reports only is on: the assembly-setting changes below have no effect.
"; html += "
" + ch.slice(0, 14).map((c) => "
" + esc(LBL[c[1]] || c[1]) + ": " + esc(short(c[2])) + " → " + esc(short(c[3])) + (c[0] === "sig" && !ro ? " (reassembles)" : c[0] === "reads" && !ro ? " (reads change: reassembles the phages concerned)" : "") + "
").join("") + (ch.length > 14 ? "
… and " + (ch.length - 14) + " more
" : "") + "
"; } } if (ro) html = (prev ? "
📄 Reports only: assemblies are kept; coverage reports and genome checks are rebuilt.
" : "
⚠️ Reports only is on, and no earlier download is recorded for this output folder. If it is a new folder, nothing will be assembled: untick Reports only (section 1).
") + html; changes.innerHTML = html; changes.classList.toggle("bad", red > 0); last = { red, n, prev: !!prev }; const mtxt = mini.querySelector(".pp-mini-txt"); mtxt.innerHTML = (ro ? "📄 Reports only · " : (st.ASSEMBLY_MODE === "accuracy" ? "Accuracy" : "Standard") + " mode · ") + "→ " + esc(String(st.OUT_ROOT || "—").split("/").filter(Boolean).pop() || "—") + "" + (red ? " · ⚠️ " + red + " change(s) will reassemble" : (prev && !n ? " · same settings as last download" : "")); $("overwrite").disabled = ro; $("overwrite").closest("label").style.opacity = ro ? ".45" : ""; }); // --- one-line summaries on closed sections (wrap.__hooks = wrap.__hooks || []).push((st) => { const H = wrap.__hints || {}, on = (v) => v === "true", last2 = (p) => String(p || "—").split("/").filter(Boolean).pop() || "—"; const sum = { "1.": `${last2(st.INPUT_PATH)} → ${last2(st.OUT_ROOT)}` + (st.REPORTS_ONLY === "true" ? " · reports only" : "") + (on(st.OVERWRITE) ? " · OVERWRITE on" : ""), "1b.": st.SAMPLE_SHEET ? "names from the sample sheet " + last2(st.SAMPLE_SHEET) : "names from " + ({ top: "the sub-folder of the parent folder", immediate: "the folder of each FASTQ", path: "path + file name" }[st.NAMING_MODE] || st.NAMING_MODE), "2.": [st.CORE_ENV, st.MEDAKA_ENV, st.PHAROKKA_ENV, st.PHOLD_ENV].filter(Boolean).join(", "), "3.": `reads ≥ ${st.MIN_LENGTH} bp, Q${$("minQ").value || 0}` + (st.ASSEMBLY_MODE !== "accuracy" && st.TARGET_BASES ? ` · target ${Math.round(st.TARGET_BASES / 1e6)} Mb` : "") + ` · host reads ${st.HOST_REMOVAL === "off" ? "kept" : st.HOST_REMOVAL === "all" ? "removed (all samples)" : "removed (listed samples)"}`, "4.": `${st.ASSEMBLY_MODE === "accuracy" ? "Accuracy (Autocycler)" : "Standard (Flye)"} · ${st.ASSEMBLY_MODE === "accuracy" ? st.AC_MAX_MB + " Mb reads · " : ""}genome ${st.GENOME_SIZE || "auto"} · ${st.PARALLEL_SAMPLES} at once`, "5.": `reads: ${st.MEDAKA_READS}${st.MEDAKA_READS === "capped" ? " " + st.MEDAKA_DEPTH + "×" : ""} · model ${st.MEDAKA_MODEL || "auto"}`, "6.": on(st.COMPARE) ? "on" : "off", "7.": on(st.RUN_COVERAGE) ? `on · ${st.COVERAGE_READS || "default"} reads · flag < ${st.AGREE_PCT}% agreement, < ${st.LOWCOV_X}×` : "off", "8.": `Pharokka ${on(st.RUN_PHAROKKA) ? "on" : "off"} · Phold ${on(st.RUN_PHOLD) ? "on (" + st.PHOLD_COMPUTE + ")" : "off"}`, "9.": on(st.EXPORT_PHAGETRACKER) ? `on · sample set ${st.PT_SAMPLE_SET || "—"}` : "off", "9b.": on(st.CP_RUN) ? `on · GenBank ${st.CP_NCBI_MODE === "none" ? "off" : st.CP_NCBI_MODE} · CheckV ${on(st.CP_CHECKV) ? "on" : "off"}` : "off", "10.": `SharePoint folder ${on(st.MAKE_UPLOAD) ? "on" : "off"}`, }; for (const [k, el] of Object.entries(H)) if (sum[k] != null) el.textContent = "— " + sum[k]; }); })(); update(); return wrap; }4d:[] 48:{"id":"a4583f8762dde68e","slug":"nanopore-phage-pipeline","title":"Nanopore Phage Pipeline","thumbnail":"c38d66de329282262ab97cceecbac2ce5db442158b9102db0c9550d4462d3c08","access_level":"public","create_time":"$D2026-11-03T15:10:17.350Z","update_time":"$D2026-11-03T16:51:52.040Z","view_time":null,"first_public_time":"$D2026-11-03T15:18:01.665Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"d1dfc5f860fb8699","login":"phage","name":"WabushOnce","avatar_url":"https://avatars-next.observableusercontent.com/9303f26b98fd8f7e04b66a6ac3218bfd2ca7e146f15ba24ca32ffac15f802763","tier":"starter_2024","bio":"","followed_by_you":false,"follows_you":false},"version":19,"publish_version":19,"role":null,"authored":false,"authors":[{"type":"individual","id":"3035ce62f81ebcd9","login":"phage","name":"WabushOnce","avatar_url":"https://avatars-next.observableusercontent.com/9303f26b98fd8f7e04b66a6ac3218bfd2ca7e146f15ba24ca32ffac15f802763","tier":null,"bio":"","followed_by_you":false,"follows_you":false}],"members":[{"type":"team","id":"d1dfc5f860fb8699","login":"phage","name":"WabushOnce","avatar_url":"https://avatars-next.observableusercontent.com/9303f26b98fd8f7e04b66a6ac3218bfd2ca7e146f15ba24ca32ffac15f802763","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Nanopore Phage Pipeline","cells":[{"id":1,"value":"# Nanopore Phage Pipeline","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":3,"value":"WSL script generator to assemble and annotate phage genomes and provide a sequencing report","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":2,"value":"$4c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W4d","stdlib":"2"}} 24:null