Metadata-Version: 2.5
Name: ipaapi
Version: 1.6.0
Summary: Command-line client for QIAGEN Ingenuity Pathway Analysis: map arbitrary column layouts, submit datasets in bulk, and resume cleanly when the analysis allowance runs out.
Author-email: Ken Jones <Ken.Jones@bioinformaticsolutions.com>
License: MIT License
        
        Copyright (c) 2026 Ken Jones
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: bioinformatics,ingenuity,ipa,pathway,qiagen
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Requires-Dist: pandas>=1.5
Requires-Dist: requests-oauthlib>=1.3
Requires-Dist: requests>=2.28
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: progress
Requires-Dist: tqdm>=4.64; extra == 'progress'
Description-Content-Type: text/markdown

# ipaapi

A Python package and command-line tool for QIAGEN Ingenuity Pathway Analysis
(IPA). It uploads datasets into an IPA project using an explicit column
mapping, submits them for analysis, and tracks the results — one file or
several hundred.

Free software (MIT). Built on QIAGEN's `python-api-demo` example code. This is
**not an official QIAGEN product** and is not endorsed by QIAGEN.

```bash
ipaapi submit ~/data --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy --pattern _DEG
```

---

## Contents

- [Why this exists](#why-this-exists)
- [Installation](#installation)
- [Quick start](#quick-start)
- [How the mapping works](#how-the-mapping-works)
- [Command-line reference](#command-line-reference)
- [Recipes](#recipes)
- [Working with IPA](#working-with-ipa) — the undocumented parts
- [Authentication](#authentication)
- [Python API](#python-api)
- [Troubleshooting](#troubleshooting)
- [How a submission is encoded](#how-a-submission-is-encoded)
- [Development](#development)
- [Contributing](#contributing)
- [Licence](#licence)

---

## Why this exists

QIAGEN's demo script works, but it assumes a rigid file layout: the gene ID in
column 0, then `n_observations × n_measurements` value columns in strict
repeating order, with every observation carrying the same measurement types in
the same positions. Analysis output rarely looks like that.

This package replaces the assumption with a declaration. You name the
identifier column and describe each observation as a set of
`(column, measurement type)` pairs. Columns can be in any order, named
anything, with columns you don't need in between.

It also fixes a number of things the demo got wrong or left out — see
[Differences from the demo](#differences-from-the-demo).

---

## Installation

```bash
python3 -m pip install ipaapi
```

Name the interpreter rather than calling `pip` or `pip3`. A bare `pip` does not
exist on macOS or on most Linux distributions, and `pip3` installs into
whichever Python comes first on `PATH`, which need not be the one that runs
`ipaapi`.

For development, an editable checkout:

```bash
git clone <this-repo> ipaapi && cd ipaapi
python3 -m pip install -e .
```

Or build and install a wheel:

```bash
python3 -m pip wheel . --no-deps -w dist
python3 -m pip install dist/ipaapi-*.whl
```

Requires Python 3.9 or newer, `requests`, `requests-oauthlib` and `pandas`.

`ipaapi --version` reports the version, the install location, and whether you
are running an editable checkout rather than a built wheel:

```bash
$ ipaapi --version
ipaapi 1.6.0
installed at /usr/lib/python3.11/site-packages/ipaapi
python 3.11.5 (/usr/bin/python3)
```

---

## Quick start

Say your file has a comment line, then a header, then data:

```
# generated by pipeline v3
Gene,Common_name,Control_mean,Treatment_mean,Fold_change,P-value,Q-value
ENSG00000229807,XIST,4.21,2.88,-1.33,0.001,0.02
```

Column positions are counted from the header row and are **0-based**:

```
0 Gene   1 Common_name   2 Control_mean   3 Treatment_mean   4 Fold_change   5 P-value   6 Q-value
```

Check the mapping without contacting IPA:

```bash
ipaapi validate results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1
```

```
results: 2,338 rows
gene id: 'Common_name' (hugo)
observations: 1
  results:
    'Fold_change' -> Log Ratio

     Common_name  Fold_change
0           XIST        -1.33
...
1 file valid. Nothing was uploaded.
```

When that looks right, submit:

```bash
ipaapi submit results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy
```

```
submitted results: 43595871

Submitted 1 analysis.
Analyses are running in IPA. Check on them with:
  ipaapi status 43595871
Recorded in ~/.local/state/ipaapi/submissions.tsv -- see 'ipaapi history'.
```

---

## How the mapping works

Three ideas, which mirror how IPA thinks about a dataset.

**Measurement** — one value column: which column, what kind of number it holds,
and an optional cutoff.

**Observation** — a named sample or contrast, and the measurement columns
belonging to it. One analysis is created per observation.

**ColumnMapping** — the identifier column, its type, and the observations.

```python
ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("drug A vs ctrl", [
            Measurement("A_log2fc", MeasurementType.LOG_RATIO),
            Measurement("A_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
        ]),
        Observation("drug B vs ctrl", [
            # declared in a different order on purpose -- this is fine
            Measurement("B_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
            Measurement("B_log2fc", MeasurementType.LOG_RATIO),
        ]),
    ],
)
```

One constraint comes from IPA rather than from this package. The wire format
declares `expvaltype`, `expvaltype2`, … and `cutoff`, `cutoff2`, … once for the
whole submission, then supplies per-observation column names against those
slots. So every observation must contribute exactly one column per measurement
type, and a given type carries one cutoff throughout. Both are checked before
anything is uploaded, with an error that explains why.

Within those limits, order and naming are free. Observations declared in
different column orders are normalised automatically.

Everything is validated against the data before upload: columns exist, none is
claimed twice, and values fall in the range IPA expects for their type. That
last check matters more than it looks — see
[measurement types](#measurement-types).

---

## Command-line reference

```
ipaapi validate   check a mapping against file(s) without uploading
ipaapi submit     upload into a project and start analyses
ipaapi status     check the state of existing analyses
ipaapi report     print IPA Interpret links (needs a commercial add-on)
ipaapi history    list analyses submitted through this tool
ipaapi login      authenticate and cache a token without submitting
```

### Mapping arguments

Used by `validate` and `submit`.

| Flag | Form | Meaning |
| --- | --- | --- |
| `PATH` | positional | a data file, or a directory to search |
| `--ID` | `COLUMN:TYPE` | 0-based identifier column and its IPA gene ID type. May be given twice — see [two identifier columns](#two-identifier-columns) |
| `--FC` | `COLUMN:TYPE[:CUTOFF]` | 0-based value column, [measurement type](#measurement-types), optional cutoff |
| `--skip-rows` | `N` | discard N lines above the header row |
| `--sep` | `CHAR` | field delimiter (sniffed from the header line by default) |
| `--pattern` | `TEXT` | when PATH is a directory: substring or glob selecting files |
| `--recursive` | flag | search subdirectories too |
| `--observation` | `NAME` | observation name in IPA (default: the filename). Single file only. Shortened to 60 characters if needed — see [long observation names](#long-observation-names) |
| `--pvalue` | `COLUMN[:CUTOFF]` | 0-based p-value column, optional cutoff. Values must lie in [0, 1] |
| `--fdr` | `COLUMN[:CUTOFF]` | 0-based FDR column, optional cutoff. **IPA reads this as a percentage** — see [measurement types](#measurement-types) |
| `--strip` | `TEXT` | remove TEXT from observation names before shortening. Repeatable |
| `--no-range-check` | flag | skip the value-range validation |
| `--list-id-types` | flag | print all 33 gene ID types and exit |

### `submit`

| Flag | Default | Meaning |
| --- | --- | --- |
| `--project` | *required* | destination IPA project. **Created if it doesn't exist**, so a typo silently makes a new one |
| `--reference-set` | `omit` | `ipkb`, `dataset`, or `omit`. See [the reference set](#the-reference-set) |
| `--wait` | off | poll until analyses finish and print report links |
| `--interval` / `--timeout` | 30s / 3600s | polling, only with `--wait` |
| `--dry-run` | off | validate and stop before login |
| `--analysis-name` / `--dataset-name` | filename | single file only |
| `--log-file` | `~/.local/state/ipaapi/submissions.tsv` | submission log |

### Authentication arguments

Used by every command that contacts IPA.

| Flag | Meaning |
| --- | --- |
| `--no-cache` | ignore any cached token |
| `--token-file` | token cache path (default `~/.cache/ipaapi/token.json`) |
| `--application-name` | `applicationname` IPA scopes the session to (default `PythonAPI`) |
| `--browser` | browser to launch for login, e.g. `firefox` |

### `login`

| Flag | Meaning |
| --- | --- |
| `--force` | sign in again even if a valid token is cached |
| `--forget` | delete the cached token and exit |
| `--no-browser` | print the authorization URL instead of opening a browser |
| `--timeout` | seconds to wait for authorization (default 300) |

### `history`

| Flag | Meaning |
| --- | --- |
| `--project` / `--since` / `--limit` | filters |
| `--status` | look up each analysis's current state (requires login) |
| `--log-file` | read a different log |

### Environment variables

| Variable | Purpose |
| --- | --- |
| `IPAAPI_TOKEN_FILE` | token cache location — set this if `$HOME` isn't writable |
| `IPAAPI_LOG_FILE` | submission log location |
| `IPAAPI_LOGIN_TIMEOUT` | seconds a login waits for authorization, for the logins other commands start on their own |

---

## Recipes

### Many files, one analysis each

```bash
ipaapi submit ~/data --pattern _DEG --ID 1:hugo --FC 4:logratio \
    --skip-rows 1 --reference-set ipkb --project Study1
```

`--pattern` takes plain text or a glob. Text with no `*`, `?` or `[` matches as
a substring, so `--pattern SampleA` finds `SampleA_DEG.txt` and
`SampleA_raw.tsv`. With no `--pattern`, `*.txt`, `*.tsv` and `*.csv` are
searched. Hidden files are skipped and results are sorted, so run order is
predictable.

Every matched file must fit the same `--ID` and `--FC` positions.

### Files are filed as they're processed

When `PATH` is a directory, each file moves as its outcome becomes known:

| Outcome | Destination |
| --- | --- |
| IPA accepted it | `submitted/` |
| The file is at fault | `failed/`, with a `.error.txt` note beside it |
| Allowance exhausted, or IPA declined | left in place for the next run |

```
submitted SampleA_DEG: 43595001
submitted SampleB_DEG: 43595002

Allowance exhausted while submitting SampleC_DEG:
REJECTED: the analysis allowance appears to be exhausted.
IPA said: 'Unable to run analysis: Analysis limit exceeded'

2 file(s) moved to submitted/
2 file(s) left in place for the next run
Re-run the same command later; the files left in place are exactly the ones
still to do.
```

The source directory shrinks to the work outstanding, and re-running the
identical command resumes. `submitted/` and `failed/` are excluded from
discovery, so a run cannot re-ingest its own output.

Nothing is moved when the command itself is at fault. A bad `--ID` type, or a
mapping that fails every file, leaves the directory untouched — that is a
mistake to fix rather than data to quarantine. Single-file submits are never
moved.

### Draining a backlog against a daily allowance

Because a stopped run resumes cleanly, this is safe to leave unattended:

```cron
0 6 * * * cd ~/data && ipaapi submit ./ --pattern _DEG --ID 1:hugo \
    --FC 4:logratio --skip-rows 1 --reference-set ipkb --project Study1 \
    >> ~/ipaapi-cron.log 2>&1
```

It submits until the allowance runs out, files what succeeded, and leaves the
rest. Check the log after the first few runs: a cron job whose refresh token
has expired fails into that file rather than prompting anyone.

### Finding analysis IDs later

IPA's API cannot list the analyses on an account, so the package keeps its own
log. Every submission appends a timestamped row.

```bash
ipaapi history
ipaapi history --project Study1 --since 2026-08-01
ipaapi history --status
```

```
2026-08-05T08:35:53-06:00  43595039  Study1  SampleA_DEG
2026-08-05T08:35:53-06:00  43595041  Study1  SampleB_DEG

2 submission(s).
```

It is plain TSV — grep it, or open it in a spreadsheet. It only covers
submissions made through this tool; anything submitted from the IPA client will
not appear.

### Comment lines above the header

```
# generated by pipeline v3, 2026-08-05
EnsemblID	log2FC	pval
```

`--skip-rows 1` discards the preamble. Column numbers count from the header, so
they do not change when you add it.

Skipping also fixes delimiter detection. The delimiter is sniffed from the
header line, and a comment line is a bad thing to sniff — the one above has
commas but no tabs, so without `--skip-rows` the file would be read as CSV and
collapse into nonsense. Rather than let that through, a header that looks like
a comment is rejected with a message pointing at this flag.

### Two identifier columns

`--ID` may be given twice. The first is the primary; the second fills rows
where the primary is blank (`.`, `NA`, empty and similar are all treated as
missing).

```bash
ipaapi submit data.csv --ID 0:ensembl --ID 1:hugo --FC 4:logratio --project S1
```

> **Read this before relying on it.** IPA accepts one `geneidtype` per
> submission. Rows filled from the second column are still uploaded under the
> primary's type, so they may fail to map. The fill count is always reported:
>
> ```
> Warning: 344 of 2,338 rows took their identifier from the fallback column
> 'Common_name' (hugo). IPA is told a single gene ID type for the submission --
> 'ensembl' -- so those rows are uploaded under that declaration and may not map.
> ```
>
> If a large fraction is being filled, using the fallback column as the only
> identifier is usually better than mixing.

---

## Working with IPA

Most of this is either undocumented or documented somewhere hard to find. It is
recorded here because getting it wrong is expensive — analyses consume a
metered allowance.

### Gene ID types

`--ID COLUMN:TYPE` takes any value from IPA's `geneidtype` list (Integration
Module §3.1). `ipaapi validate --list-id-types` prints all 33.

Common ones: `ensembl`, `hugo`, `entrezgene`, `refseq`, `swissprot`,
`affymetrix`, `illumina`, `agilent`.

Two things are not guessable:

- **Human gene symbols are `hugo`.** Not `genesymbol`, not `hgnc`, and not the
  desktop client's own label `Gene Symbol`. All three are rejected outright.
- **Species rides on the identifier type.** There is no species parameter:
  `hugo` is human, `mousesymeg` mouse, `ratsymeg` rat. Accession numbers name
  their own species, so `ensembl` covers every organism on its own.

Assign `hugo` to a column of mouse symbols and IPA will not care. It will match
very few IDs and score the analysis on what is left, and nothing in the output
says the species was wrong.

A type outside the documented list produces a warning with a near-match
suggestion but is still sent, since IPA is the authority and the list will age.
An unrecognised value fails before anything is uploaded, and IPA names it.

### Duplicate dataset names

**IPA refuses to create a dataset whose name already exists in a project, and
reports it as "The page you are looking for is currently unavailable."** The
wording says outage; the cause is a name collision. This is the most misleading
response the API produces.

Verified: the same 2 KB request, byte for byte, succeeded and then failed twice
in a row. With unique dataset names, three consecutive submissions all
succeeded.

Dataset names come from filenames, so re-running a batch retries names an
earlier run already created. The run then dies on its first file and looks like
a total outage.

The package prevents this using the submission log. Before uploading, it checks
whether that dataset name already went to that project:

```
skipping SampleA_DEG: already submitted to 'Study1' on 2026-08-05T08:35:53-06:00
as analysis 43595039. IPA would reject a second dataset of the same name.
Use --force to submit it again anyway.
```

Skipped files are filed under `submitted/`, because they are. `--force`
overrides, though IPA will normally reject it.

The guard only knows about submissions made through this tool with the same log
file. If you hit the collision anyway — a colleague's upload, or the IPA client
— the fix is a different `--project`, a different `--dataset-name`, or deleting
the existing dataset in IPA.

### Long observation names

**IPA rejects a long observation name, and reports it as "The page you are
looking for is currently unavailable."** That is the same page as a duplicate
dataset name, from an unrelated cause.

The observation name defaults to the filename, so descriptive pipeline output
names run past the limit without anyone choosing a long name. A long *dataset*
name is fine; only the observation is affected.

Established by A/B on one file, holding project, reference set and data
constant:

| dataset name | observation name | result |
| --- | --- | --- |
| short (25) | **long (82)** | rejected as an "outage" |
| **long (82)** | short (25) | accepted — reached IPA's allowance check |

65 characters is known good and 82 known bad; the exact limit is undocumented.
Since 1.2.0 the observation name is brought under 60 characters automatically.

What gets removed is chosen by what it means, not by where it sits. Two parts
of a differential expression filename carry information: the contrast
(`Treated_vs_Control`) and the cell type the comparison was computed from
(`Proximal_tubule_epithelium`). Everything the pipeline appends about how it
ran — `_naive_cell_t_significant_p0.05_rna`,
`_pseudobulk_t_significant_p0.05_rna` — is disposable, so that is what goes
first:

```
Treated_vs_Control_Proximal_tubule_epithelium_naive_cell_t_significant_p0.05_rna
 -> Treated_vs_Control_Proximal_tubule_epithelium

Treated_vs_Control_Glandular_epithelium_pseudobulk_t_significant_p0.05_rna
 -> Treated_vs_Control_Glandular_epithelium
```

A filename that marks its own significance is cut on that structure rather than
by guesswork. Single-cell pipelines commonly name their output
`<contrast>_<celltype>_<method>_<test>_significant_<threshold>_<assay>`, and
the `significant` literal is a reliable anchor:

```
Treated_vs_Control_Proximal_tubule_epithelium | naive_cell | t | significant_p0.05_rna
Treated_vs_Control_Glandular_epithelium       | pseudobulk | t | significant_p0.05_rna
```

Working backwards from the anchor: drop it and everything after, drop the
statistical test immediately before it, then drop the aggregation method. The
test needs no list of names — it is the token before the anchor, so `wilcox`
works as well as `t` without anyone maintaining a vocabulary. Methods are
matched as whole phrases in that one position only, because a cell type of
`Naive_T_cell` shares both words with the `naive_cell` method, and loose
matching cut it to `Naive_T`.

Removal is graded, least damaging first, and stops as soon as the names fit:

1. anything named with `--strip`
2. the pipeline tail, cut at its `significant` anchor as above
3. trailing pipeline metadata, for files from anything else — tokens like
   `significant`, `deseq2`, `filtered`, and cutoffs in any of the shapes
   `p0.05`, `fdr0.01`, `padj0.05`, `0.05`, read right to left and stopping at
   the first token that isn't recognisable as a setting
4. the suffix every file in the batch happens to share
5. the prefix every file shares — this costs the contrast, so it is late
6. a two-ended cut with `..` marking the gap, on a token boundary

Steps 2 to 4 always run together rather than stopping the moment the names
merely fit, since half a removed suffix reads worse than either whole.
Comparison is whole tokens at a time, so `Mature` is never treated as a prefix
of `Immature`, and `cell_type` is never left as `cell_t`. Names already within
the limit are returned untouched.

Any step is abandoned if it would make two names identical, or leave one under
four characters or without a letter. Reducing files to `1` and `2` keeps them
distinct and makes the analysis unreadable. Distinctness is the property being
protected: observation names are what IPA lists side by side in a comparison
analysis, so two files reduced to the same label would be worse than a long
name.

Everything removed is printed:

```
note: shortened 2 observation names. IPA rejects a long observation name and
reports it as an outage, so this is not optional.
      Datasets and analyses keep the full filename; only the observation label
      inside the analysis is shorter.
      Treated_vs_Control_Glandular_epithelium_pseudobulk_t_significant_p0.05_rna
   -> Treated_vs_Control_Glandular_epithelium
```

If your pipeline's suffix is not recognised, name it explicitly. `--strip` is
repeatable, applied before anything else, and ignored as a whole if it would
leave the names unusable:

```bash
ipaapi submit ~/data --ID 1:mousesymeg --FC 4:foldchange --skip-rows 1 \
    --project Study --strip _significant_p0.05_rna
```

Steps 4 and 5 depend on the other files in the run, so **the same file
submitted in a different batch can get a different observation label**. Steps 1
to 3 do not — they read only the name in front of them, which is why a
single-file submit shortens as well as a batch does. The dataset and analysis
always keep the full filename, so `ipaapi history` and the IPA project view are
unaffected.

`--observation` sets the name explicitly for a single file, and is shortened
the same way if it needs to be.

### The FDR percentage trap

**IPA reads `falsediscovery` as a percentage in [0, 100].** Statistical
software emits q-values as fractions in [0, 1]. Both are inside the accepted
range, so nothing is rejected and nothing is discarded. A q-value of `0.05` is
taken as `0.05%`, a threshold a hundred times stricter than intended, and a
cutoff applied in IPA silently keeps far less than you meant.

This is the one measurement type where the range check cannot help, because the
wrong scale is a legitimate value. So `ipaapi` warns on the shape of the
distribution instead — if every value in an `--fdr` column is at or below 1:

```
Warning: column 'Q_value' is declared 'falsediscovery' and every one of its
2,338 values is <= 1. IPA reads this type as a PERCENTAGE in [0, 100], so 0.05
means 0.05%, not 5%. If these are ordinary q-values, multiply the column by 100
before submitting -- IPA accepts them either way and cannot tell the difference,
so nothing will be rejected.
```

Multiply the column by 100, or pass the cutoff on IPA's scale and accept that
the stored values are hundredths.

### The reference set

The background enrichment is scored against — the denominator of the Fisher's
exact test behind every p-value.

| Value | Background |
| --- | --- |
| `ipkb` | Ingenuity Knowledge Base (Genes Only, or + Endogenous Chemicals if chemicals are present) |
| `dataset` | the genes you uploaded |
| `omit` (default) | IPA chooses |

Which to use depends on what you uploaded:

- A **complete measured transcriptome** with a cutoff: `dataset` is the better
  science, because the background is what your assay could actually detect,
  which controls for detection bias.
- A **pre-filtered hit list**: `dataset` makes the background nearly identical
  to the foreground. Use `ipkb`.

§4.1.3.1 states that with the parameter omitted IPA picks by size — `ipkb`
below 2000 identifiers, `dataset` at 2000 or more. In practice this has not
been observed to hold: files of 1,804–6,245 rows all came back as
`Ingenuity Knowledge Base (Genes Only)`. Since the behaviour is unpredictable,
set it explicitly for anything you intend to compare against itself.

Verify after the fact — the setting is recorded in every IPA export:

```bash
grep -h "^Reference set" *_IPA_output.txt | sort | uniq -c
```

Array platforms can also be named as reference sets, paired with a
`referencesettype`. Not exposed here; see §4.1.3.

### Measurement types

| Value | Meaning | Valid range |
| --- | --- | --- |
| `ratio` | Ratio | `[0, +∞)` |
| `foldchange` | Fold Change | `(-∞, -1]` and `[1, +∞)` |
| `logratio` | Log Ratio | `(-∞, +∞)` |
| `pvalue` | p-value | `[0, 1]` |
| `falsediscovery` | FDR / q-value | `[0, 100]` |
| `intensity` | Intensity | `[0, +∞)` |
| `other` | Other (normalised around zero) | `(-∞, +∞)` |
| `gain_loss` | Variant Gain/Loss | `-2, -1, 0, 1, 2` |
| `classification` | Variant ACMG Classification | `-2, -1, 0, 1, 2` |

> **Out-of-range values are silently discarded by IPA.** §3.1: "analysis will
> still proceed without errors or warning diagnostics" — offending entries are
> simply dropped. This is why the range check exists, and why it refuses rather
> than warns. Declaring log2 fold changes as `foldchange` would quietly discard
> every gene between −1 and 1, which in a typical scRNA-seq table is most of
> them.

The package helps in both directions:

- Values declared `foldchange` that cluster inside (−1, 1) suggest `logratio`.
- A column declared `logratio` with no values in (−1, 1) warns that it looks
  like signed fold change, since a real log ratio is centred on zero.

A column called `Fold_change` may hold either. Check the data, not the name.

### What the API cannot do

- **List your projects.** `--project` creates one if the name doesn't exist, so
  a typo silently makes a new project rather than erroring.
- **List your analyses.** Every endpoint needs an ID you already hold, which is
  why the submission log exists.
- **Tell you your remaining allowance.** You discover the limit by hitting it.

### Errors IPA actually returns

IPA answers a rejected submission with an HTML error page, not plain text, and
the reason is at the end, after support boilerplate. This package strips the
boilerplate and the page footer, and classifies what is left:

| IPA's message | Class | What the tool does |
| --- | --- | --- |
| `Unknown GeneId Type (X)` | `MalformedRequestError` | stops; names the flag; moves nothing |
| `Unable to run analysis: Analysis limit exceeded` | `QuotaExceededError` | stops; leaves remaining files for the next run |
| `Unable to run analysis: …` (other) | `AnalysisRefusedError` | as above — reached the analysis logic, so not a parameter fault |
| `currently unavailable` / `technical difficulties`, or 502/503/504 | `ServiceUnavailableError` | IPA is down; stops, moves nothing, says the command is fine |
| HTTP 401 with a token error | `TokenRefusedError` | renews the token and retries once — see [authentication](#authentication) |
| anything else | `SubmissionError` | files that one under `failed/` |

Quota matching is deliberately broad (`ipaapi.client.QUOTA_PATTERNS` plus HTTP
429). A false positive only leaves a file for the next run, while a false
negative would quarantine a retryable submission. The raw response is always
printed, so a misclassification is visible.

Note that IPA's status codes are not consistent between endpoints. An exhausted
allowance arrives as 403, a missing add-on as 500, a rejected submission as 200
with an HTML page. The message text carries more meaning than the code does.

### Interpret links

`ipaapi report <id>` fetches the IPA Interpret URL for a finished analysis and
prints it. `--open` also opens it in a browser. It returns a link and nothing
else — no data is downloaded. It checks status first, so an unfinished analysis
says so rather than surfacing a bare HTTP 500.

**This requires a separate commercial add-on to IPA, and most licences do not
include it.** Confirmed against a licence that does not carry it: the endpoint
answers HTTP 500 and there is nothing to fix. `ipaapi report` says so and
exits, rather than presenting it as a fault. The analyses themselves are
unaffected — they ran, they are in your project, and they open normally in IPA.

Everything else in this package works on an ordinary IPA licence. If you are
not sure whether yours carries the add-on, run `ipaapi report` on any finished
analysis: the answer takes a second and costs nothing.
`examples/probe_interpret.py` prints the raw response if you want to see it.

---

## Authentication

Browser-based OAuth 2.0 with PKCE. Your password never reaches this package.

1. A short-lived HTTP server binds `127.0.0.1:8000`.
2. Your browser opens QIAGEN's authorization page and you log in there.
3. QIAGEN redirects back to `localhost:8000` with a one-time code. The `state`
   parameter is verified, then the code plus the PKCE verifier is exchanged for
   a token.
4. The token is used as `Authorization: Bearer …` and the server shuts down.

Whichever account you log in as owns the datasets and projects.

The client ID is the public one any IPA user may use. It is not a secret.

### Signing in on its own

```bash
ipaapi login
```

This authenticates and caches a token without submitting anything, which is the
only way to check that credentials work without spending analysis allowance
finding out. It reports where the token went, when it expires, and whether a
refresh token was issued:

```
Signed in.
  token cache: /home/you/.cache/ipaapi/token.json
  expires in 11h 58m (at 2026-09-11 06:12:44)
  a refresh token was issued, so the next command should not need the browser
```

`--force` signs in again even when a valid token is cached. `--forget` clears
the cache. `--no-browser` prints the URL rather than opening one. `--timeout`
raises the 300-second wait, which is worth doing when a forwarded browser is
slow to appear over a VPN. For the logins other commands start on their own,
set `IPAAPI_LOGIN_TIMEOUT` instead, since `submit --timeout` is already the
analysis completion clock.

Worth running before a long batch, so a token cannot expire mid-run.

There is deliberately no `--client-id` or `--host`. The cache is keyed on
those, so a login under a different key would be invisible to every other
command — a login that appears to work and changes nothing.

### Token caching and refresh

Tokens are cached at `~/.cache/ipaapi/token.json`, owner-only (0600). Access
tokens are short-lived, but a refresh token comes with them and is spent
automatically: an expired cache is renewed over HTTP with no browser and no
prompt. A browser login is only needed when the refresh token itself is
rejected.

Since 1.5.0 that renewal also happens mid-run. If IPA refuses the token while a
command is working, the client refreshes it and replays the request once. If
the refresh is refused, it says so and runs the full login rather than
reporting a submission failure:

```
Your refresh token was refused ((invalid_grant)). Running `ipaapi login` now.
```

Deleting the cache is effectively logging out. `--no-cache` forces a fresh
login. Be aware the token is plaintext on disk — anyone who can read your home
directory can use it until it expires.

### Headless servers

Because refresh is automatic, a token copied from a machine with a browser
keeps renewing itself for as long as the refresh token stays valid:

```bash
# once, on a machine with a browser
ipaapi login

scp ~/.cache/ipaapi/token.json server:~/.cache/ipaapi/token.json
ssh server chmod 600 ~/.cache/ipaapi/token.json
```

The cache is keyed on the account, the application name and the IPA host, not
on the machine that made it, so it works anywhere.

**If `$HOME` isn't writable**, the cache cannot be saved and every run needs a
fresh login, which is crippling on a headless box. Point it somewhere writable:

```bash
export IPAAPI_TOKEN_FILE=$HOME/ipaapi-token.json
export IPAAPI_LOG_FILE=$HOME/ipaapi-submissions.tsv
```

Both failures are reported loudly rather than swallowed, because a cache that
never writes looks exactly like a token that expires instantly.

When an interactive login is genuinely needed, X forwarding is the cleanest
route. The server-side browser renders locally and `localhost:8000` resolves
server-side where the callback listens, so no port forwarding is required:

```bash
ssh -X you@server        # ssh -Y from macOS, with XQuartz running
```

Failing that, forward the callback port and use your own browser:

```bash
ssh -L 8000:localhost:8000 you@server
ipaapi login --no-browser
```

The error message distinguishes `DISPLAY` unset from no browser found.

> The redirect URI is pinned to `http://localhost:8000` by the OAuth client
> registration, so the port is not configurable in practice.

### Using a token obtained elsewhere

```python
import os
from ipaapi import Credentials, IPAClient

client = IPAClient(Credentials.from_token(os.environ["IPA_TOKEN"]))
```

---

## Python API

```python
from ipaapi import (
    ColumnMapping, Dataset, IPAClient, Measurement, MeasurementType,
    Observation, ReferenceSet, TokenCache,
)

mapping = ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("treated vs control", [
            Measurement("Fold_change", MeasurementType.LOG_RATIO),
        ]),
    ],
)

dataset = Dataset.from_file("results.csv", mapping, skip_rows=1)
print(dataset.describe())          # confirm before uploading

client = IPAClient.login(cache=TokenCache())
ids = client.submit(dataset, project="MyStudy", reference_set=ReferenceSet.IPKB)

for analysis_id, status in client.wait_for(ids).items():
    if status.succeeded:
        print(client.report_url(analysis_id))
```

Key objects:

| Object | Purpose |
| --- | --- |
| `ColumnMapping`, `Observation`, `Measurement` | describe the file |
| `Dataset.from_file` / `.from_frame` | load and validate |
| `IPAClient.login()` | OAuth, with caching and refresh |
| `.submit()` `.status()` `.wait_for()` `.results()` `.report_url()` | the API |
| `GENE_ID_TYPES` | all 33 identifier types and what they mean |
| `ipaapi.history` | the submission log |
| `ipaapi.errors` | everything derives from `IPAError` |

### Results

```python
results = client.results(analysis_id)
print(results.canonical_pathways.head())
cp, ur, df = results                  # unpacks like the demo's ipa_results()
```

> **These endpoints are not part of the documented API.** They came from
> QIAGEN's demo code, and they live under `/pa/ipa/analysisResults/` rather than
> the documented `/pa/api/v2/` surface that submission and status use. They are
> refused on at least one real licence, with an HTTP 401 reading "You have
> exceeded the lifetime limit for this operation." Treat result retrieval as
> unavailable unless you have confirmed otherwise for your own account.
> Submission, status, history and Interpret links are unaffected.

---

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `REJECTED: IPA does not recognise the gene ID type 'X'` | not in IPA's vocabulary | `--list-id-types`; human symbols are `hugo` |
| `declared 'foldchange' but holds N out-of-range value(s)` | log2 values declared as linear fold change | `--FC N:logratio` |
| `Could not find a header row … looks like a comment` | preamble above the header | `--skip-rows N` |
| `--FC refers to column N, but the file has only M column(s)` | 1-based counting, or wrong `--skip-rows` | positions are 0-based, from the header |
| `Every row is missing an identifier` | wrong column, or no header | check with `head -1 file \| tr '\t' '\n' \| nl -v0` |
| `the analysis allowance appears to be exhausted` | daily or period limit | re-run later; files left in place resume |
| `IPA appears to be down or having trouble` | IPA outage — **or a duplicate dataset name, or a long observation name**, all reported identically | `ipaapi --version` (1.2.0+ handles the name length); check `ipaapi history --project X` for the dataset name; otherwise wait |
| Batch dies on the first file, names come from long filenames | observation name too long | upgrade to 1.2.0+, or pass a short `--observation` |
| Batch dies on the *first* file after an earlier run | dataset names already exist in the project | expected — 1.1.0 skips them automatically; before that, use a new `--project` |
| `Cannot listen on 127.0.0.1:8000` | stale login process, or another user mid-login | the message names the PID and prints the `kill` command |
| `the request timed out before IPA answered` | over-long observation name, or a slow link or VPN | upgrade to 1.2.0+; the file may have been created anyway |
| `Timed out after 300s waiting for authorization` | forwarded browser too slow to appear | `ipaapi login --timeout 900`, or `IPAAPI_LOGIN_TIMEOUT` |
| FDR cutoff keeps far fewer genes than expected | q-values given as fractions, read as percentages | multiply the `--fdr` column by 100 |
| Login prompt on every run | token cache not writable | `export IPAAPI_TOKEN_FILE=...`; check for a root-owned cache |
| `Could not open a browser automatically` | headless | `ssh -X`, or copy a token across |
| `report` returns HTTP 500 on a succeeded analysis | the Interpret add-on is not on your licence | nothing to fix; open the analysis in IPA |
| `client.results()` returns HTTP 401 | undocumented endpoint, refused for this account | see [Results](#results); the analysis itself is fine |
| Analyses have z-scores but no p-values | reference set equals the gene list | `--reference-set ipkb` |
| Half of all pathways significant | list too large for the background | apply a cutoff, or upload unfiltered data with a cutoff |

Useful first move for any column problem:

```bash
head -1 yourfile.csv | tr ',\t' '\n' | nl -v0
```

---

## How a submission is encoded

Worth knowing when debugging. `--ID 1:hugo` becomes three separate things:

| From `--ID` | Wire parameter | Sent |
| --- | --- | --- |
| the type | `geneidtype=hugo` | once |
| the column, resolved from position to header name | `genecolname=Common_name` | once |
| that column's values | `geneid=XIST`, `geneid=UTY`, … | once per row |

The column number never leaves your machine.

The whole dataset travels in one `application/x-www-form-urlencoded` POST to
`/pa/api/v2/multiobsanalysis`, which both creates the dataset in the project
and starts one analysis per observation. Parameter naming is positional and
irregular — for measurement slot *k* and observation *i*, both zero-based:

| Parameter | Meaning |
| --- | --- |
| `expvaltype`, `expvaltypeK+1` | measurement type for slot *k* (global) |
| `cutoff`, `cutoffK+1` | cutoff for slot *k* (global, optional) |
| `obsI+1name` | observation name |
| `expvalname`, `expvalK+1name` | column label, first observation |
| `obsI+1expvalname`, `obsI+1expvalK+1name` | column label, later observations |
| `geneid` | one per data row |
| `expvalue`, `expvalK+1` | one per slot per observation, per row |

Per-row value parameters carry no observation prefix. They cycle through the
slots of observation 1, then observation 2, and so on. Order is load-bearing.

The body is properly percent-encoded. The demo concatenated it by hand, so any
value containing a space, `&`, `=`, `+` or `%` corrupted the request —
including the `Group Max Intensity` column in the demo's own sample dataset.

---

## Development

```
src/ipaapi/
  __init__.py    public API and the version (single source of truth)
  models.py      MeasurementType, AnalysisStatus, ReferenceSet, GENE_ID_TYPES
  mapping.py     Measurement, Observation, ColumnMapping
  dataset.py     Dataset, load_table
  _payload.py    multiobsanalysis body construction
  auth.py        OAuth 2.0 + PKCE, Credentials, TokenCache, refresh
  client.py      IPAClient, error classification
  history.py     the submission log
  triage.py      submitted/ and failed/ filing
  cli.py         the ipaapi console script
  errors.py      exception hierarchy
tests/           offline; no network required
examples/        runnable scripts and diagnostics
```

```bash
python3 -m pip install -e ".[dev]"
pytest
```

The suite is fully offline. It covers mapping validation, the exact parameter
layout of the submission body, encoding of hostile characters, error
classification, triage behaviour, and token cache and refresh logic.

**Versioning.** The version lives only in `src/ipaapi/__init__.py`;
`pyproject.toml` reads it at build time. Bump it there and nowhere else, and
add a `CHANGELOG.md` entry. `ipaapi --version` reports the install path too,
which is what actually answers "am I running the wheel I think I am".

### Differences from the demo

- Column mapping by name in any order, validated before upload.
- Request bodies are percent-encoded.
- OAuth: no CPU-spinning wait loop, `state` is verified, logins time out, the
  callback server is shut down, error redirects are handled, tokens are cached
  and refreshed.
- Submissions are never retried automatically, because a retried POST could
  create a duplicate analysis. GETs retry with backoff.
- Typed exceptions; access tokens excluded from `repr()`.
- No `install_dependencies()` shelling out to `pip3`.

---

## Contributing

Issues and pull requests are welcome. The most useful contributions are
**corrections to the [Working with IPA](#working-with-ipa) section**. Much of
it was established by trial against a live account, and several points have had
to be corrected more than once. If IPA behaves differently for you, that is
worth reporting even without a code change.

```bash
python3 -m pip install -e ".[dev]"
pytest
```

Tests are fully offline; none of them contact IPA.

---

## Status

**1.6** — stable and in production use against live IPA. The command line and
the Python API are settled; breaking changes from here mean a major version
bump. See `CHANGELOG.md`.

Known open questions, none of which affect submission:

- Result retrieval (`client.results()`) calls undocumented endpoints and is
  refused on at least one licence, with an HTTP 401 naming a lifetime limit.
  See [Results](#results).
- The documented reference-set size rule does not match observed behaviour; set
  `--reference-set` explicitly.
- Quota classification has been confirmed against one IPA wording only.

---

## Licence

MIT — see [LICENSE](LICENSE). Free to use, modify and redistribute.

Not affiliated with, endorsed by, or supported by QIAGEN. IPA is QIAGEN's
product; this is an independent client for its public API, built on the
`python-api-demo` example code QIAGEN publishes. For questions about the API
itself, QIAGEN's contact is `AdvancedGenomicsSupport@qiagen.com` — please don't
send them bug reports about this package.
