Metadata-Version: 2.4
Name: biomatch-nf
Version: 0.2.0
Summary: Thin CLI wrapper for the biomatch-nf Nextflow pipeline (BioMatch sample identification: k-mer panel generation, counting and deepKin evaluation)
Project-URL: Homepage, https://github.com/VonPoo/biomatch-nf
Project-URL: Source, https://github.com/VonPoo/biomatch-nf
Author: VonPoo
License: MIT
Keywords: bioinformatics,biomatch,genomics,k-mer,nextflow,sample-identification
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# biomatch-nf

A [Nextflow](https://www.nextflow.io/) pipeline for **BioMatch** — a data-driven framework for comprehensive sample identification.

It reproduces the full BioMatch CLI workflow as a scalable, reproducible pipeline:

- **Panel generation** — build a k-mer reference panel from a reference genome + population variants
- **K-mer counting** — count panel k-mers across sequencing samples (SE/PE)
- **Evaluation** — sample identity evaluation via `ntsmEval` and deepKin (R)

## Features

- All 5 BioMatch CLI modes supported (`genpanel`, `genpanel_count_eval`, `count_eval`, `deepkin_eval`, `eval_only`)
- Native Nextflow parallelism: one process per sample for counting, one per species for panel generation
- CPU-focused resource allocation (BioMatch is CPU-intensive; memory stays conservative)
- **Start-up dependency self-check** — verifies every external tool/R/Python package before running (`bin/check_deps.sh`)
- Dual environment support: **Conda** (default) and **Docker / Singularity**, plus plain **standard** (system PATH)
- Compatible with Nextflow `>= 22.10` (DSL2 only)

---

## Software requirements

### 1. Nextflow runtime

| Software | Version | Notes |
|---|---|---|
| [Nextflow](https://www.nextflow.io/) | `>= 22.10` | DSL2 |
| Java | 17+ (11+ for old Nextflow) | Nextflow runtime; not used by the pipeline itself |

### 2. Pipeline dependencies

Nextflow only *schedules* processes — every analysis tool is an external command resolved from `PATH`.
A plain `standard` run therefore requires all of the following to be installed **before** use:

| Dependency | Used by | Missing → affects |
|---|---|---|
| `biomatch` package (`ntsmSiteGen`, `ntsmCount`, `ntsmEval` + `ntsm-scripts/`) | panel generation / counting / ntsmEval | `genpanel`, `count_eval`, `genpanel_count_eval`, `eval_only` |
| `bwa` | `ntsmSiteGen` internal makefile (`bwa index/aln/samse`) | `genpanel`, `genpanel_count_eval` |
| `plink2` | VCF filtering / conversion | `genpanel*`, `deepkin_eval` |
| `plink` (1.9) | deepKin PLINK1 analysis | `deepkin_eval` |
| `bcftools`, `bgzip`, `tabix` | VCF processing in deepKin prep | `deepkin_eval` |
| R + packages `deepKin`, `data.table`, `tidyverse`, `caret`, `e1071`, `pls` | deepKin evaluation | `deepkin_eval` |
| Python3 + `pandas` | panel/analysis helper scripts | `genpanel*`, `deepkin_eval` |
| `make`, `gzip`, `bzip2` | `ntsmSiteGen` makefile / FASTA decompression | `genpanel*` |

> `filter_counts.py` and `select_panel.py` ship inside this repo's `bin/` and are added to `PATH` by Nextflow automatically — no install needed.

### 3. r-deepkin — conda-only

`r-deepkin` (the deepKin R package) has **no CRAN build and is only published as a conda package**
([`vonpoo::r-deepkin`](https://anaconda.org/channels/VonPoo/packages/r-deepkin)). Two options:

- **Conda** (recommended): `conda install -c conda-forge -c bioconda -c vonpoo r-deepkin`
- **No conda**: install the R packages from CRAN and `deepKin` from source:
  ```r
  install.packages(c("data.table","tidyverse","caret","e1071","pls"))
  install.packages("remotes"); remotes::install_github("qixininin/deepKin")
  ```
- References:
  - deepKin paper: <https://doi.org/10.1016/j.crmeth.2025.101053>
  - deepKin GitHub: <https://github.com/qixininin/deepKin>
  - r-deepkin conda package: <https://anaconda.org/channels/VonPoo/packages/r-deepkin>

### 4. Install at a glance

```bash
# --- Option A: conda (recommended, includes r-deepkin) ---
conda install -c conda-forge -c bioconda -c vonpoo \
    biomatch r-deepkin bwa plink plink2 bcftools tabix \
    r-data.table r-tidyverse r-caret r-e1071 r-pls pandas

# --- Option B: no conda ---
sudo apt install bwa plink plink2 bcftools tabix make gzip bzip2 r-base python3 python3-pandas
pip install biomatch          # provides ntsmSiteGen/ntsmCount/ntsmEval + ntsm-scripts/
# + the R packages shown above (incl. deepKin from GitHub)
```

#### Installing `ntsm` (alternative to the `biomatch`-bundled tools)

`ntsm` is the k-mer site toolkit behind panel generation. It can be installed with conda or compiled
from source ([GitHub](https://github.com/JustinChu/ntsm),
[paper](https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giae024/7687245)):

```bash
# conda
conda install bioconda::ntsm

# or from source
git clone https://github.com/JustinChu/ntsm
cd ntsm && ./autogen.sh && ./configure && make
./configure --prefix=/PATH && make install
```

#### Two known installation gotchas (checked by `bin/check_deps.sh`)

1. **`ntsm-scripts/` must sit next to `ntsmSiteGen`** — `ntsmSiteGen` locates its `makefile`
   via `dirname $BASH_SOURCE[0]`, so the executable and the `ntsm-scripts/` resource folder
   must live in the same directory. Conda/pip installs guarantee this; manual copies must not split them.
   The pipeline additionally re-creates a local `bin/` with both symlinked together, so it works even
   if only a symlink to `ntsmSiteGen` is on `PATH`.
2. **`bwa` is mandatory for panel generation** — `ntsmSiteGen` calls `bwa index/aln/samse`
   internally. It is easy to miss because it is not a Python dependency of the `biomatch` package.

### 5. Dependency self-check on startup

Every run starts with `bin/check_deps.sh` (disable with `--check_deps false`), which verifies all
commands, R packages, Python packages and the `ntsm-scripts/` resource:

```bash
nextflow run VonPoo/biomatch-nf -profile conda --mode count_eval ...   # checks first, then runs
```

Missing dependencies are listed with install instructions and the run stops (`errorStrategy 'terminate'`).
To **auto-install** missing dependencies (requires `conda`, incl. `r-deepkin`):

```bash
nextflow run VonPoo/biomatch-nf -profile conda --deps_auto_install true --mode count_eval ...
```

### 6. pip wrapper (optional — fastest install)

A thin PyPI wrapper ships the pipeline *inside* the wheel, so no repository clone is needed
(useful on networks with restricted GitHub access):

```bash
pip install biomatch-nf

biomatch-nf run -profile conda --deps_auto_install true --mode count_eval ...   # first run bootstraps Nextflow
```

CLI commands:

| Command | Description |
|---|---|
| `biomatch-nf run [args...]` | Run the pipeline; every argument is passed through to `nextflow run` |
| `biomatch-nf check [--auto-install]` | Dependency self-check only (embedded `bin/check_deps.sh`) |
| `biomatch-nf init` | Check Java, bootstrap Nextflow, verify dependencies |
| `biomatch-nf version` | Show wrapper + embedded pipeline versions |
| `biomatch-nf update` | `pip install -U biomatch-nf` |

Notes:

- **Java 17+** is required; the wrapper warns if it is missing. Install e.g.
  `apt install openjdk-17-jre` or `conda install -c conda-forge openjdk=17`.
- **Nextflow** is auto-downloaded to `~/.biomatch-nf/nextflow` on first run when it is not on
  `PATH`; reuse an existing install via `BIOMATCH_NF_NEXTFLOW=/path/to/nextflow` or
  `biomatch-nf run --nextflow-home /path/to/nextflow ...`.
- **Relative paths** are resolved from the calling directory, exactly like a direct
  `nextflow run` (run from your project directory).
- Point to a development copy of the pipeline with `biomatch-nf run --pipeline /path/to/repo ...`.
- The wheel is built from the pipeline git tag via `scripts/build_wheel.sh` — no pipeline
  source changes are needed to repackage.

---

## Quick start

```bash
nextflow run VonPoo/biomatch-nf -profile conda -mode count_eval \
    --panel_name <panel.fa> \
    --count <reads_dir> \
    --count_db <count_out_dir> \
    --eval_result <eval_out_dir>
```

### Run modes

| Mode | Description | Required params |
|---|---|---|
| `genpanel` | Generate a reference k-mer panel | `--ref --pop_vcf --panel_name` (+`--species`/`--chr_set`) |
| `genpanel_count_eval` | Generate panel, count reads, ntsmEval | `--ref --pop_vcf --panel_name --count --count_db --eval_result` |
| `count_eval` | Count with existing panel + ntsmEval | `--panel_name --count --count_db --eval_result` |
| `deepkin_eval` | deepKin evaluation on VCF/PLINK | `--match --eval_result` (+`--species`/`--chr_set`, `--keep-base`) |
| `eval_only` | ntsmEval on existing counts | `--count_db --eval_result` |

## Parameters

| Param | Type | Default | Description |
|---|---|---|---|
| `--mode` | String | `count_eval` | Run mode (see table above) |
| `--ref` | Path | — | Reference genome FASTA (`.fa/.fna`, optional `.gz/.bz2`) |
| `--pop_vcf` | Path | — | Population VCF (or PLINK `.bed`/`.pgen` prefix) |
| `--panel_name` | String | — | Panel name (e.g. `rice_ref_v1`, or a built-in name below) or path to an existing panel `.fa` |
| `--list_panels` | Boolean | `false` | Print available built-in k-mer reference panels and exit |
| `--species` | String | — | Species name for autosome lookup (see `assets/species_chroms.tsv`) |
| `--chr_set` | Integer | — | Override autosome chromosome count |
| `--target_seqs` | Integer | 150000 | Panel target sequence count |
| `--count` | Path | — | Sample reads directory or single FASTQ/FASTA file |
| `--count_db` | Path | — | Output directory for `.counts.txt` files |
| `--threads` | Integer | 10 | Counting parallelism (samples in parallel) |
| `--match` | Path | — | deepKin eval input: VCF or PLINK prefix |
| `--keep_base` | String | — | Base-set retention (e.g. `ATC`) |
| `--eval_result` | Path | — | Output directory for evaluation results |
| `--panels_dir` | Path | `panels` | Where generated panels are published |
| `--check_deps` | Boolean | `true` | Run the dependency self-check before analysis |
| `--deps_auto_install` | Boolean | `false` | Auto-install missing deps via conda (`check_deps.sh --auto-install`) |
| `--conda_env` | Path/Name | — | Reuse an existing conda environment with `-profile conda` (default: auto-create `biomatch`) |
| `--install_panel_dir` | Path | — | Also copy a generated panel into this directory (“insert” into a panel library) |
| `--help` | Boolean | `false` | Print usage |

## Profiles

| Profile | Description |
|---|---|
| `standard` | Default local executor (no container/conda) — tools must be on `PATH` |
| `conda` | Run inside the `biomatch` conda env (auto-created); reuse an existing env via `--conda_env` |
| `docker` | Run inside a Biocontainers image |
| `singularity` | Run inside a Singularity/Apptainer image (HPC) |

Example with Singularity on an HPC cluster:

```bash
nextflow run VonPoo/biomatch-nf -profile singularity -mode genpanel \
    --ref genome.fa --pop_vcf pop.vcf --panel_name rice_ref_v1 --species rice
```

## Panel library

- Generated panels are **published to `panels/`** (`--panels_dir`), which acts as the built-in
  panel library: afterwards `--panel_name panels/<name>.fa` reuses them in `count_eval`.
- To also *insert* the panel elsewhere (e.g. the `biomatch` package's internal
  `kmer_ref_panels/` directory, or a shared/central library), pass
  `--install_panel_dir <dir>` to `genpanel` / `genpanel_count_eval`:
  ```bash
  nextflow run VonPoo/biomatch-nf -profile conda --mode genpanel \
      --ref genome.fa --pop_vcf pop.vcf --panel_name rice_ref_v1 --species rice \
      --install_panel_dir /path/to/kmer_ref_panels
  ```
  The panel `<panel_name>.fa` is copied there and the panel remains usable by name across runs.

### Built-in reference panels

Pre-built k-mer panels ship with the pipeline — no manual download or setup needed:

- **Repository runs** (`nextflow run VonPoo/biomatch-nf`): panels are bundled in
  `assets/kmer_ref_panels/`.
- **Conda users** (incl. the pip wrapper): panels are already installed by the conda
  `biomatch` package at `$CONDA_PREFIX/share/biomatch/kmer_ref_panels/`.

Pass a bare name (no path) and the pipeline resolves it automatically:

```bash
nextflow run VonPoo/biomatch-nf -profile conda --mode count_eval \
    --panel_name Cattle_ref_ARS-UCD1.2 --count reads/ --count_db out_counts/ --eval_result out_eval/

# list all available built-in panels
nextflow run VonPoo/biomatch-nf -profile conda --list_panels
```

Available panels (27 species):

```
Arabian_camel_ref_CamDro2           Cat_ref_Felis_catus_9.0
Cattle_ref_ARS-UCD1.2               Chicken_ref_bGalGal1.mat.broiler.GRCg7b
Chimpanzee_ref_Pan_tro_3.0          Chukar_partridge_ref_Chukar.1.0
Darwin_finches_ref_GeoFor_1.0       Dog_ref_Dog10K_Boxer_Tasha
Donkey_ref_ASM303372v1              Dugong_ref_ASM4018294v1
Fox_ref_VulVul2.2                   Goat_ref_ARS1
Honey_bee_ref_Amel_HAv3.1           Horse_ref_EquCab3.0
Lion_ref_PanLeo1.0                  Macaque_ref_Mmul_10
Mallard_ref_ASM874695v1             Mouse_ref_GRCm39
Norway_rat_ref_Rnor_6.0             Pig_ref_Sscrofa11.1
pig11                               Rabbit_ref_OryCun2.0
Sheep_ref_ARS-UI_Ramb_v2.0          Swan_goose_ref_GooseV1.0
Turkey_ref_Turkey_5.1               Water_buffalo_ref_NDDB_SH_1
Zebra_finch_ref_Taeniopygia_guttata-3.2.4
```

### Adding a new panel

To add a new built-in panel to this repository:

1. Drop the panel FASTA into `assets/kmer_ref_panels/<Species>/<panel_name>.fa`
   (e.g. `assets/kmer_ref_panels/Rice/rice_ref_v1.fa`).
2. Commit and push:
   ```bash
   git add assets/kmer_ref_panels/<Species>/
   git commit -m "Add <Species> k-mer reference panel"
   git push origin master
   ```
3. It is now usable as `--panel_name <panel_name>` by anyone running the pipeline
   from GitHub, and is listed by `--list_panels`.

> Note: panels are **not** bundled inside the PyPI wheel (too large for the 100MB
> PyPI file limit). `pip install biomatch-nf` users get panels from the conda
> `biomatch` package — keep the conda package panels in sync for pip users, or add
> the panel to both places.

## Output

Results are published to the directories you specify (`--count_db`, `--eval_result`, `--panels_dir`), consistent with the BioMatch CLI semantics:

- `*.counts.txt` — per-sample k-mer counts
- `<panel>_BioMatch.csv` — ntsmEval identity results
- `<panel>_filtered_match.tsv` — rows with `same=1`
- `<panel>.fa` — generated k-mer reference panel

## How it works

```
genpanel:
  CHECK_DEPS -> PREPARE_REF -> FILTER_VCF (plink2 MAF 0.45-0.55) -> EXPORT_VCF
  -> RENAME_REF (ref_map_new.py) -> NTSM_SITEGEN (bwa + ntsm-scripts) -> SELECT_PANEL
  -> [INSTALL_PANEL]

count_eval:
  CHECK_DEPS -> NTSM_COUNT (ntsmCount, one process per sample) -> COLLECT_COUNTS -> NTSM_EVAL

deepkin_eval:
  CHECK_DEPS -> DEEPKIN_EVAL (bgzip/plink2 prep -> params.RData -> VCF/PLINK_Geno_nonGeno.R)
```

## Acknowledgements

- [BioMatch](https://github.com/VonPoo/BioMatch) — VonPoo, Zhejiang University
- [ntsm](https://github.com/JustinChu/ntsm) — k-mer site counting tools
  ([paper](https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giae024/7687245), `conda install bioconda::ntsm`)
- [deepKin](https://github.com/qixininin/deepKin) — relatedness inference used by deepKin evaluation
  ([paper](https://doi.org/10.1016/j.crmeth.2025.101053), conda package: [`vonpoo::r-deepkin`](https://anaconda.org/channels/VonPoo/packages/r-deepkin))

## License

MIT
