Skip to content

MIINT Documentation

MIINT provides a range of bioinformatic capabilities. The materials included here are coupled where possible with detailed explanation and examples.

Passing relations by name

Many MIINT functions take a table or view by name, as a quoted string literal rather than a relation argument — align_minimap2('reads', subject_table := 'refs'). Those names are resolved against the catalog, so a CREATE TEMP TABLE or CREATE TEMP VIEW works anywhere an ordinary table does. This matters most under a read-only catalog, where a temporary relation may be the only one you can create.

Three exceptions:

  • massql() and the per-sample (sample_id := ...) paths of woltka_ogu, sylph_profile and detect_chimera_uchime cannot read a temporary relation. Materialize the source as an ordinary table first.
  • A relation created inside an explicit BEGIN that has not been committed yet is not visible to any of these functions, temporary or not.
  • A CTE (WITH name AS (...)) cannot be passed by name: it exists only inside its query, not in the catalog. Passing one is an error naming the CTE, even when a table of the same name exists, because reading that table instead would silently give the wrong answer. Create a view or table under a different name and pass that instead, e.g. CREATE TEMP VIEW name_view AS ... (an ordinary view or table for the functions in the first bullet). Reusing the CTE's name would make a view that reads itself whenever the CTE's query reads the same-named table. Inside a CTE's own body the name still refers to the table, as in plain SQL, so that is not an error. Real CTE support is tracked in #286.

A subquery cannot be used in place of the literal — DuckDB rejects it outright with Binder Error: Table function cannot contain subqueries. To choose the relation at runtime, use SET VARIABLE plus getvariable():

SET VARIABLE reads = 'my_temp_reads';
SELECT * FROM align_minimap2(getvariable('reads'), subject_table := 'refs');

Arrow relations

A relation registered from Arrow works by name like any other, because DuckDB's registration creates a temporary view. This includes a RecordBatchReader streamed from an external source such as Arrow Flight, so a streaming query set can be handed to a function directly rather than spilled to Parquet first:

con.register('reads', flight_reader)          # pyarrow Table or RecordBatchReader
con.execute("SELECT * FROM align_minimap2('reads', subject_table := 'refs')")

Two constraints come with it:

  • A RecordBatchReader is consumed once. A second reference to the same registered name returns zero rows, silently — that is Arrow's semantics, not something MIINT can detect. align_minimap2 (both modes) and align_minimap2_sharded read their relation exactly once and are safe; align_bowtie2_sharded re-reads per shard and will raise an error rather than return a partial result. Other relation-taking functions have not been audited for this yet — notably the per-sample (sample_id := ...) paths, which read the relation once per sample — so prefer a materialized table there. See docs/internals/reading-tables-views.md for the audit status.
  • Do not register a reader that is itself backed by a DuckDB query on the same connection. In DuckDB 1.5.4 con.execute(...).arrow() returns exactly that — a lazy RecordBatchReader, not a materialized table — and passing it to a MIINT function deadlocks the process with no error and no timeout. The cause is a lock-ordering hazard inside DuckDB's own Arrow scan, and nothing in MIINT can intercept it. Materialize first: con.execute(...).arrow().read_all(), or CREATE TEMP TABLE ... AS SELECT.

Reading & writing

  • Reading files - File formats with read support in MIINT (FASTA/FASTQ, SAM/BAM, PAF, GFA, SFF, BIOM, mzML/mzXML, GFF, jplace, Newick), and myloasm assembly output.
  • Writing files - File formats with write support in MIINT.

Public data repositories

  • EBI/ENA - Interaction with EBI/ENA including submission and fetching of data.
  • NCBI - Interaction with NCBI including fetching of data and the NCBI taxonomy (tree, name classes, lineages, retired/deleted-taxid handling).

Alignment

Amplicon & community analysis

  • Quality control - Adapter / polyG / polyX / quality trimming and per-read filtering.
  • Denoising - Denoise sequencing data into sub-OTU variants.
  • Chimera checking - Reference-based and de novo chimera detection.
  • Sequence clustering - Cluster sequences into OTUs at an identity threshold.
  • Sequence search - Search sequences against a reference database.
  • Taxonomic classification - Classify sequences by minimizer/k-mer content (RYpe).
  • Profiling & feature tables - Abundance profiling (sylph) and OGU feature tables (Woltka).
  • Absolute quantification - Turn compositional read counts into absolute quantities: cell counts from synDNA spike-ins (Zaramela et al. 2022), and copies of each ORF's ssRNA.
  • Source tracking - SourceTracker: estimate what fraction of each sink community came from each source environment, with per-feature assignments and leave-one-out validation.

Phylogeny & diversity

  • Phylogeny estimation - Phylogenetic estimation, placement of query sequences onto a backbone and index building (krepp), tree manipulation (shear, resolve multifurcations/placements), and comparative methods (independent contrasts, ancestral state reconstruction: Brownian-motion, parsimony, and Mk maximum likelihood).
  • Alpha and beta diversity - Methods to compute and analyze alpha and beta diversity, including rarefaction, non-phylogenetic community distances, ordination (PCoA) and Procrustes alignment, sample clustering, and the two-sample Kolmogorov-Smirnov test.
  • Community simulation - Generate synthetic gradient / clustered OTU tables with known ground truth for benchmarking resemblance and ordination methods.
  • scikit-bio methods - Statistics computed by scikit-bio itself, through skb-arrow, such as the Mantel test's permutation p-value and ANCOM-BC differential abundance.
  • Multi-omics integration - MMvec: learn which metabolites co-occur with which microbes from paired count tables, without the spurious associations correlation produces on compositional data.
  • Sample classification & regression - Random forests over a feature table (sc, a reimplementation of QIIME 2's q2-sample-classifier): fit, predict, estimate generalization by cross-validation, and explain individual predictions with SHAP (SHapley Additive exPlanations) feature attributions, computed exactly with path-dependent TreeSHAP.

Mass spectrometry

  • MassQL - Operating on mass spectrometry data through MassQL.

Other

  • Utilities - Sequence helpers, versions/diagnostics, and optional-tool installation.