Metadata-Version: 2.1
Name: gffkit
Version: 0.7.0
Summary: GFF annotation integration and analysis toolkit
Author: caijunhao
License: MIT
Project-URL: Homepage, https://github.com/qunjie-zhang/gffkit
Project-URL: Repository, https://github.com/qunjie-zhang/gffkit
Project-URL: Issues, https://github.com/qunjie-zhang/gffkit/issues
Keywords: GFF3,GTF,genome annotation,bioinformatics,UTR,gene annotation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: rna
Requires-Dist: pysam >=0.22 ; extra == 'rna'

# gffkit

`gffkit` is a lightweight toolkit for GFF/GTF annotation integration and analysis.
Version 0.7.0 provides six standalone utilities:

1. `detect-bridge`: detect suspicious merged-gene artifacts caused by bridge transcripts.
2. `complement`: complement/merge annotations, with optional region-swap mode.
3. `add-utr`: reconstruct `five_prime_UTR` and `three_prime_UTR` features from exon/CDS coordinates.
4. `rename-sort`: rename gene/transcript/child IDs with a prefix and sort the final GFF3.
5. `get_longest_transcript`: retain one longest transcript isoform per gene.
6. `stat`: calculate basic gene-annotation statistics for one GFF or a directory of GFF files.

The `integrate` workflow continues to use only the original four integration
utilities. `get_longest_transcript` and `stat` are independent commands and are
not automatically run by `integrate`.

## Installation

```bash
pip install gffkit
```


## Quick start

### Full integration pipeline

```bash
gffkit integrate \
  --annotation-a EviAnn.gff3 \
  --annotation-b ANNEVO.gff3 \
  --outdir gffkit_out \
  --prefix sample \
  -t 8
```

Outputs:

- `gffkit_out/sample.suspicious.tsv`
- `gffkit_out/sample.merged.gff3`
- `gffkit_out/sample.final.withUTR.gff3.pre_rename.gff3`
- `gffkit_out/sample.final.withUTR.gff3`
- `gffkit_out/sample.final.withUTR.gff3.id_map.tsv`

In `integrate`, `--prefix sample` is also used for final ID renaming. Final gene IDs are written like `sample_C01g00001`, transcript IDs like `sample_C01g00001.t1`, and child IDs like `sample_C01g00001.t1.exon1`.

### Step-by-step usage

```bash
# 1. Detect suspicious merged genes in Annotation A
gffkit detect-bridge -i EviAnn.gff3 -o suspicious.tsv -t 8

# 2. Use A as the global reference, but switch to B in suspicious regions
gffkit complement \
  --ref EviAnn.gff3 \
  --add ANNEVO.gff3 \
  --swap_region_tsv suspicious.tsv \
  --swap_region_flank 100 \
  --output merged.gff3 \
  -t 8

# 3. Add UTR features
gffkit add-utr -i merged.gff3 -o final.annotation.withUTR.pre_rename.gff3

# 4. Rename IDs, drop unplaced seqids, and sort the final GFF3
gffkit rename-sort \
  -i final.annotation.withUTR.pre_rename.gff3 \
  -o final.annotation.withUTR.gff3 \
  --prefix sample
```

### Adaptive bridge threshold scan

The default detector keeps the fixed `--min-gap 10000` behavior. To scan an
annotation-specific threshold range, use:

```bash
gffkit detect-bridge \
  -i EviAnn.gff3 \
  -o suspicious.auto.tsv \
  --threshold-mode auto
```

Auto mode deduplicates introns by chromosome, strand, and coordinates, then
uses Q95 through `max(Q99.5, --min-gap)` as its search range. It evaluates only
thresholds where a transcript's bridge classification can change. Candidate
rows include the threshold ranges that support the call, the selected
representative threshold, and a threshold stability label. The adjacent
`suspicious.auto.tsv.meta.json` records the background size, quantile method,
and search range. With fewer than 1,000 unique introns, auto mode reports that
it fell back to the fixed threshold; this fallback is not considered stable.

A bridge must overlap actual exon blocks from at least two core transcript
clusters. Overlap with a cluster's broad genomic span alone is insufficient.
Missing exon/CDS structure is excluded from bridge evidence. `integrate
--threshold-mode auto` uses only `stable_candidate` rows for region swapping;
the standalone detector still reports sensitive candidates for review.
Threshold stability is a screening measure, not a calibrated probability.

### Optional RNA-seq evidence for merged and split genes

The local development source supports `--rna-bam` on `detect-bridge` and
`integrate`. This addition is not a published release. BAM support uses the
optional `pysam` dependency; a local package installation with this extra is
`pip install '.[rna]'` from this package directory. Existing commands without
BAM inputs do not require pysam.

For execution directly from this checkout, use `PYTHONPATH=src python -m gffkit`
in place of `gffkit` in the examples below. This avoids accidentally running an
older installed version.

```bash
gffkit detect-bridge \
  -i annotation.gff3 \
  -o suspicious.tsv \
  --rna-bam sample1.sorted.bam \
  --rna-bam sample2.sorted.bam \
  --rna-library firststrand
```

Each BAM must be aligned to the same genome assembly as the annotation, be
coordinate-sorted (`HD:SO=coordinate`), and have a readable BAI or CSI index.
Chromosome names must match. The program reads BAMs without sorting, indexing,
or modifying them. BAMs should represent distinct samples; do not supply both
a pooled BAM and its constituent BAMs. All inputs share the specified library
type. The program cannot verify that identically named chromosomes come from
the same assembly, so this remains an input requirement.

By default, the original `suspicious.tsv` is preserved in format and selection
logic, and an additional `suspicious.tsv.rna.tsv` reports both event types:

| Event / status | Interpretation |
| --- | --- |
| `merged_gene` / `possible_overmerge` | Both exonic flanks have enough templates in the same sample(s), but no cross-boundary connection was observed. The legacy merged-gene candidate merits review. |
| `merged_gene` / `connection_supported` | Reliable RNA alignments connect the proposed parts; this argues against automatically splitting that model. |
| `split_genes` / `possible_oversplit` | Alignments connect adjacent annotated genes; review whether they are fragments of one gene or a readthrough transcript. |
| `ambiguous` | Some connection evidence exists, but it does not meet the read/sample threshold. |
| `insufficient_evidence` | Coverage, contig availability, or strand information is insufficient for a decision. No connection between adjacent genes does not establish that their boundaries are correct. |
| `unresolved_structure` | A legacy bridge does not connect actual core exons, so its span overlap is not usable boundary evidence. |

The report contains `event`, JSON `gene_ids`, chromosome, strand, the two exon
boundary coordinates, gap length, status, reason, supporting/coexpressed sample
counts, JSON `parameters` recording the options used, and JSON `sample_evidence`.
Each sample entry includes its BAM path,
left/right template counts, junction and contiguous-connection counts, unknown
strand connection counts, and observed splice boundaries. Coordinates are
**1-based and inclusive**, in genomic left-to-right order even on the minus
strand. They are evidence boundaries, not sequence-editing cut instructions.
Unresolvable boundaries without usable core exons have `.` coordinates.
Use `--rna-report PATH` to choose the report filename.

To explicitly use RNA evidence to filter the original merged-gene candidates:

```bash
gffkit integrate \
  --annotation-a EviAnn.gff3 \
  --annotation-b ANNEVO.gff3 \
  --outdir gffkit_rna_out \
  --prefix sample \
  --rna-bam sample.sorted.bam \
  --rna-library firststrand \
  --rna-filter-merged
```

`--rna-filter-merged` retains only genes with a `possible_overmerge` boundary
in the original suspicious TSV. A supported or weakly supported alternative
connection anywhere among that gene's assessed boundaries protects it from
this gene-wide filter. The filtered TSV is the one consumed by the existing
region-swap workflow. The separate RNA report retains all assessed boundaries;
`possible_oversplit` events are **review candidates only** and are not sent to
region swapping or automatically joined into new gene models. Without
`--rna-filter-merged`, `integrate` still uses its original candidate selection;
supplying BAMs alone adds a report.

| Option | Default | Meaning |
| --- | --- | --- |
| `--rna-library` | `unstranded` | `firststrand`: read 1/single-end is antisense; `secondstrand`: read 1/single-end is sense. Read 2 is opposite to read 1. Unstranded mode uses `XS:A:+/-` when available; otherwise evidence has unknown strand. |
| `--rna-min-mapq` | `20` | Minimum mapping quality. MAPQ 255 is accepted only with `NH:i:1`, accommodating STAR. Known multimappers (`NH != 1`) are excluded. |
| `--rna-min-anchor` | `10` | Minimum aligned anchor on each side of a connection. |
| `--rna-min-junction-reads` | `3` | Supporting templates per sample for a connection call; also applies to contiguous connections. |
| `--rna-min-flank-reads` | `10` | Templates required on each flank in the same sample before interpreting missing connections. |
| `--rna-min-samples` | `1` | Number of samples meeting the per-sample support criteria. |
| `--rna-flank-window` | `100` | Exonic window size near each assessed boundary. |
| `--rna-max-split-gap` | `100000` | Maximum genomic gap between neighboring genes for split review. |

Counts use `(read group, query name)` within each BAM, so paired mates are not
double-counted as independent templates. Unmapped, secondary, supplementary,
duplicate-marked, QC-failed, low-MAPQ, and known multimapping records are
excluded. The filter does not itself mark PCR duplicates. A CIGAR `N` with
sufficient aligned anchors supports splicing; `D` does not. A single contiguous
aligned block spanning both flanks can support a split within an exon. Paired
mates landing on separate genes alone do not establish a connection. Missing
contigs are unavailable evidence, not zero expression. Evidence is measured
per sample before combining calls, avoiding artificial coexpression from
different tissues. Short-read evidence cannot generally prove a full-length
multi-junction transcript, exclude readthrough, or validate CDS/ORF structure.

Scope: merged-gene review currently starts from the original multi-transcript
bridge candidates; adding BAMs does not recover single-transcript fusions or
short-gap merges missed by that detector. Split review considers neighboring,
non-overlapping genes on the same chromosome and strand, using their terminal
exons (CDS if exon is absent and fallback is enabled). Overlapping genes,
non-neighboring fragments, and boundaries outside these windows need additional
structural analysis. Unknown-strand connections may include antisense signal.
These are transparent screening heuristics with tunable thresholds, not
calibrated probabilities or definitive gene-boundary corrections.

RNA analysis fetches indexed exonic windows sequentially, opening each BAM once;
`-t` still controls the existing structural detection and complement workers.
Memory for template tracking scales with local coverage. In BAM mode the
candidate TSV and RNA report must be new paths; existing files are refused.

Implementation references: [pysam API](https://pysam.readthedocs.io/en/stable/api.html)
and [SAM format specification](https://samtools.github.io/hts-specs/SAMv1.pdf).

### Merge three or more annotations

Use repeated `--add` arguments. Files are merged in the order provided.

```bash
gffkit complement \
  --ref EviAnn.gff3 \
  --add ANNEVO.gff3 \
  --add Helixer.gff3 \
  --add PASA.gff3 \
  --output merged.multi.gff3 \
  -t 8
```

### Extract the longest transcript

For each gene, `get_longest_transcript` selects the isoform with the longest
concatenated CDS. If none of a gene's isoforms has CDS features, it uses the
longest concatenated exon length. Ties retain the first isoform encountered in
the input file.

Process one GFF/GFF3 file:

```bash
gffkit get_longest_transcript \
  -i annotation.gff3 \
  -o longest_gff
```

Process all `.gff`/`.gff3` files directly inside a directory:

```bash
gffkit get_longest_transcript \
  -f input_gff_directory \
  -o longest_gff
```

Compressed `.gff.gz` and `.gff3.gz` inputs are also supported by this command.
Output files are named `*.longest.gff` or `*.longest.gff3`. Existing output
files are refused unless `--force` is supplied.

### Calculate annotation statistics

Process one GFF/GFF3 file and write the default `gffstat.txt`:

```bash
gffkit stat -i annotation.gff3
```

Process all `.gff`/`.gff3` files directly inside a directory and select an
output path:

```bash
gffkit stat \
  -f input_gff_directory \
  -o annotation.gffstat.txt
```

The tab-separated output contains species/file name, gene count, average gene
span, average transcript span, average CDS length per gene, average exon count
per gene, average exon length, and average UTR length per gene. Gene and
transcript lengths use their GFF genomic spans (`end - start + 1`); they are not
spliced mature-transcript lengths.

## Command overview

```bash
gffkit --help
gffkit detect-bridge --help
gffkit complement --help
gffkit add-utr --help
gffkit rename-sort --help
gffkit get_longest_transcript --help
gffkit stat --help
gffkit integrate --help
```

## Parallel Processing

Use `-t/--threads` to select the number of worker processes. The option name is
kept for command-line compatibility, but version 0.5.0 and later uses processes for
CPU-bound work so that Python's GIL does not restrict execution to one core.

- `detect-bridge` analyzes genes with a process pool and batches tasks to reduce process communication overhead.
- `complement` parses the reference and supplementary files in parallel, then merges them in the original command-line order.
- `complement` uses a dynamic chromosome interval index to compare each supplementary gene only with nearby overlapping reference genes.
- `integrate` passes the worker count to the detect and complement steps. `add-utr` and `rename-sort` remain single-process.

Example:

```bash
gffkit integrate --annotation-a EviAnn.gff3 --annotation-b ANNEVO.gff3 -t 16
```

During `detect-bridge`, multiple worker processes should be visible in `top` or
`htop`. CPU usage naturally drops during the single-process UTR and rename/sort
steps. More workers also increase memory use; start with `-t 4` or `-t 8` for
large annotations.

## Annotation integration strategy

- Annotation A, for example EviAnn/RNA-seq-supported GFF, is used as the global primary reference.
- Annotation B, for example ANNEVO/deep-learning GFF, is used as the local primary reference only in suspicious merged-gene regions.
- UTR features are reconstructed after merging using an exon-minus-CDS strategy.
- Version 0.4.0 and later run `rename-sort` as the final `integrate` step. The final GFF3 keeps chromosome-mounted records, removes unplaced/scaffold/contig records, sorts features, rewrites `ID`/`Parent`, and writes an ID map next to the output.
- Version 0.5.0 replaces CPU-bound threads with worker processes and adds a dynamic interval index for faster annotation overlap searches.
- Version 0.6.0 adds the independent `get_longest_transcript` and `stat` commands. Neither command is included in `integrate`.
- When multiple tools annotate the same gene locus, the GFF source column is combined with `|`, for example `EviAnn|ANNEVO`.

## Rename and Sort

Run this step independently when you already have a merged GFF3:

```bash
gffkit rename-sort \
  -i merged.withUTR.gff3 \
  -o sample.renamed.sorted.gff3 \
  --prefix sample \
  --digits 5 \
  --keep-old-ids
```

This writes `sample.renamed.sorted.gff3` and `sample.renamed.sorted.gff3.id_map.tsv`.

## Version 0.7.0 changes

- Added `detect-bridge --threshold-mode auto`, which deduplicates introns, derives a per-annotation search range, and scans only thresholds that can change bridge classification.
- Added candidate threshold ranges, selected threshold, stability classification, and a JSON metadata sidecar to auto-mode output; sparse backgrounds are marked as fixed-threshold fallback.
- Changed bridge confirmation to require overlap with actual core exon/CDS blocks instead of broad cluster spans; transcripts without usable exon/CDS structure cannot support a bridge call.
- Made `integrate --threshold-mode auto` use only stable candidates for region replacement and aligned RNA boundary reconstruction to the selected threshold.
- Preserved fixed-threshold mode as the default and retained its original TSV columns.
- Updated the package version to 0.7.0.

## Version 0.6.0 changes

- Added `gffkit get_longest_transcript` using the longest-CDS/longest-exon selection logic from `get_longest_transcript.py`.
- Added mutually exclusive `-i/--input` single-file and `-f/--folder` directory input modes for longest-transcript extraction.
- Added `gffkit stat` using the GFF statistics implementation from `gffstat.py`.
- Added mutually exclusive `-i/--input` single-file and `-f/--folder` directory input modes for annotation statistics.
- Kept both new commands independent from the four-step `gffkit integrate` workflow.
- Updated the package version from 0.5.0 to 0.6.0.
- Updated package author and credits to `caijunhao`.

## Maintainer notes

When command-line options or behavior changes, update this `README.md` in the versioned package directory before building and uploading to PyPI.

## License

MIT License.

## Credits

Author: caijunhao
