NGS Pipeline Atlas

One Nextflow workflow (main.nf) takes a samplesheet of paired-end FASTQ files and a reference genome, and processes every sample through the same stages. The run cohort is ENA study PRJNA234495 — Escherichia coli K-12 whole-genome resequencing, Illumina HiSeq 2000 paired-end — aligned against the MG1655 complete genome, NC_000913.3. A sample that fails a gate has its BAM removed inside the gate step and never reaches variant calling; the reason is written to its gate log.

Every sample travels the same path. The only exit is the gate: pass all three thresholds and the sample is called; fail one and it stops there, with the reason logged.

Per-stage timing — median task time, run 1 trace

FETCH's median reflects the local mirror the reproducibility runs ingested from (see “Fetching from the archive” below), not the throttled archive itself.

A sample passes only if all three hold — thresholds fixed before the cohort run, at standard bacterial WGS values: mean coverage ≥ 10.0× (samtools coverage), reads mapped ≥ 80% (samtools flagstat), Q30 bases ≥ 70% (fastp, post-filtering).

Passed vs stopped

The six stops — mean coverage against the 10.0× gate

Coverage was the deciding metric. All six stopped samples had ~98% of reads mapped and Q30 of 96% — they failed on depth alone, five genuine samples landing between 8.82× and 9.99×, just under the line.

The gate step's own log line for the deliberate bad input, verbatim — identical in both runs:

[GATE] BADINPUT01 STOPPED reason: mean_coverage=0.01x < gate 10.00x (mean_coverage=0.01x pct_mapped=97.42 q30_pct=96.00)

BADINPUT01 is the first 400 read pairs of a cohort sample (~0.02× coverage), included in the samplesheet to prove the coverage gate stops under-covered samples. No BADINPUT01 VCF exists in either run's output.

Calling (bcftools mpileup + call) ran on the 27 passed samples only. Across them the cohort holds 130,614 variant records at 81,810 unique sites, with per-sample counts from 3,259 to 7,689. The full per-sample table — coverage, mapped percentage, Q30 and variants called for all 33 samples — is in the cohort reports under Downloads.

The pipeline was executed twice end to end from the same samplesheet, in fully separate work and output directories — run 2 in an isolated launch directory, so no task result could be reused from run 1's cache. Inputs are content-addressed (FETCH refuses any payload whose MD5 differs from the samplesheet) and the environment is locked to exact tool builds (environment-explicit.txt).

• Cohort reports: byte-identical — MD5 822aad36… for both runs, with the same attempted / passed / stopped counts, stop reasons, 130,614 records and 81,810 unique sites.
• Per-sample variant tables (vcf/*.calls.tsv): 27 of 27 byte-identical between run 1 and run 2.
• Sanitize statistics and gate decisions: identical in both runs, for all 33 samples, including all six stops.

Given the MD5-pinned inputs and the locked environment, two independent executions produce identical variant calls and identical reports.

The legacy files in this study are corrupt in two ways, and the corruption is in the archive, not the transfer: the files' MD5 checksums match the archive's published values. A structural pre-screen of 50 candidate runs admitted 32 and rejected 18 — every rejection a mate-2 record-structure desync. A full-content validation pass then found in-line byte garbage (non-IUPAC bytes inside sequence lines, concentrated in mate 2) even inside admitted files, so that second corruption class is handled at read level by the pipeline's own sanitize stage (bin/sanitize_pairs.py): it parses tolerantly, validates every record — IUPAC sequence alphabet, printable quality bytes, matched lengths — drops corrupt pairs, writes canonical 4-line FASTQ, and counts everything it drops, per sample.

104,684 corrupt pairs dropped (0.39%) out of 27,175,960 processed — identical totals in both runs, and 0 unparseable fragments. BADINPUT01: 400 pairs in, 390 kept.

The archive rate-limits sustained fetching hard: HTTP 403 pages under load, and transfers that occasionally truncate or stall. FETCH is built for exactly that — downloads retry with backoff until the payload is byte-identical to the samplesheet's MD5, and a payload that differs is refused, never silently accepted. Because the throttling was hard enough to stall the cohort's 66-file download set, the files were additionally staged in a local mirror with resumable, MD5-verified downloads; the reproducibility runs ingest from the mirror, while the canonical samples.tsv keeps the original archive URLs and FETCH performs the same MD5 verification either way.

The project's own results documents, byte-identical to the files in the project:

• RESULTS.md — the full results document behind every figure on this page
• cohort_report_run1.txt — run 1's cohort report (130,614 records, 81,810 unique sites)
• cohort_report_run2.txt — run 2's cohort report, byte-identical to run 1's (MD5 822aad36…)

• One study and one reference: PRJNA234495 E. coli K-12 against MG1655 NC_000913.3. Nothing here is a claim about other organisms, other references, or human data.
• The gates are fixed, standard thresholds — six of 33 samples stopping is the gates doing their job, and stopped samples contribute no variant calls.
• The reproducibility runs ingested from a local mirror of the archive files — identical bytes, MD5-pinned, with the same verification FETCH applies to live downloads.
• Timing medians are from run 1's trace on one machine; they describe this pipeline on this hardware, not a benchmark against other pipelines.

Every figure on this page is taken from the project's RESULTS.md and the two runs' cohort reports, and is cross-checked against both at build time.

The standalone proof page for this product: Evaluations — the protocol, the versioned results, and every result file as a download.