Metadata-Version: 2.5
Name: ipaapi
Version: 1.2.0
Summary: Command-line client for QIAGEN Ingenuity Pathway Analysis: map arbitrary column layouts, submit datasets in bulk, and resume cleanly when the analysis allowance runs out.
Author-email: Ken Jones <Ken.Jones@bioinformaticsolutions.com>
License: MIT License
        
        Copyright (c) 2026 Ken Jones
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: bioinformatics,ingenuity,ipa,pathway,qiagen
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Requires-Dist: pandas>=1.5
Requires-Dist: requests-oauthlib>=1.3
Requires-Dist: requests>=2.28
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: progress
Requires-Dist: tqdm>=4.64; extra == 'progress'
Description-Content-Type: text/markdown

# ipaapi

A Python package and command-line tool for QIAGEN Ingenuity Pathway Analysis
(IPA). Upload datasets into an IPA project using an explicit column mapping,
submit them for analysis, and track the results — one file or several hundred.

Free software (MIT). Built on QIAGEN's `python-api-demo` example code — **not
an official QIAGEN product**, and not endorsed by QIAGEN.

```bash
ipaapi submit ~/data --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy --pattern _DEG
```

---

## Contents

- [Why this exists](#why-this-exists)
- [Installation](#installation)
- [Quick start](#quick-start)
- [How the mapping works](#how-the-mapping-works)
- [Command-line reference](#command-line-reference)
- [Recipes](#recipes)
- [Working with IPA](#working-with-ipa) — the undocumented parts
- [Authentication](#authentication)
- [Python API](#python-api)
- [Troubleshooting](#troubleshooting)
- [How a submission is encoded](#how-a-submission-is-encoded)
- [Development](#development)
- [Contributing](#contributing)
- [Licence](#licence)

---

## Why this exists

QIAGEN's demo script works, but assumes a rigid file layout: the gene ID in
column 0, then `n_observations × n_measurements` value columns in strict
repeating order, every observation carrying the same measurement types in the
same positions. Real analysis output rarely looks like that.

This package replaces that assumption with a declaration. You name the
identifier column and describe each observation as a set of
`(column, measurement type)` pairs. Columns may be in any order, named
anything, and interleaved with columns the analysis should ignore.

It also fixes a number of things the demo got wrong or left out — see
[Differences from the demo](#differences-from-the-demo).

---

## Installation

```bash
git clone <this-repo> ipaapi && cd ipaapi
pip install -e .
```

Or build and install a wheel:

```bash
python3 -m pip wheel . --no-deps -w dist
python3 -m pip install dist/ipaapi-*.whl
```

Requires Python 3.9+, `requests`, `requests-oauthlib`, `pandas`.

Confirm what you're running — this reports the version, the install location,
and whether it's an editable checkout rather than a built wheel:

```bash
$ ipaapi --version
ipaapi 1.0.0
installed at /usr/lib/python3.11/site-packages/ipaapi
python 3.11.5 (/usr/bin/python3)
```

---

## Quick start

Say your file looks like this — a comment line, then a header, then data:

```
# generated by pipeline v3
Gene,Common_name,Control_mean,Treatment_mean,Fold_change,P-value,Q-value
ENSG00000229807,XIST,4.21,2.88,-1.33,0.001,0.02
```

Column positions are **0-based** and counted from the *header* row:

```
0 Gene   1 Common_name   2 Control_mean   3 Treatment_mean   4 Fold_change   5 P-value   6 Q-value
```

Check the mapping without contacting IPA:

```bash
ipaapi validate results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1
```

```
results: 2,338 rows
gene id: 'Common_name' (hugo)
observations: 1
  results:
    'Fold_change' -> Log Ratio

     Common_name  Fold_change
0           XIST        -1.33
...
1 file valid. Nothing was uploaded.
```

When that looks right, submit:

```bash
ipaapi submit results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy
```

```
submitted results: 43595871

Submitted 1 analysis.
Analyses are running in IPA. Check on them with:
  ipaapi status 43595871
  ipaapi report 43595871
Recorded in ~/.local/state/ipaapi/submissions.tsv -- see 'ipaapi history'.
```

---

## How the mapping works

Three ideas, and they mirror how IPA thinks about a dataset.

**Measurement** — one value column: which column, what kind of number it holds,
and an optional cutoff.

**Observation** — a named sample or contrast, and the measurement columns
belonging to it. One analysis is created per observation.

**ColumnMapping** — the identifier column, its type, and the observations.

```python
ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("drug A vs ctrl", [
            Measurement("A_log2fc", MeasurementType.LOG_RATIO),
            Measurement("A_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
        ]),
        Observation("drug B vs ctrl", [
            # declared in a different order on purpose -- this is fine
            Measurement("B_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
            Measurement("B_log2fc", MeasurementType.LOG_RATIO),
        ]),
    ],
)
```

**One constraint is imposed by IPA, not by this package.** The wire format
declares `expvaltype`, `expvaltype2`, … and `cutoff`, `cutoff2`, … *once for the
whole submission*, then supplies per-observation column names against those
slots. So every observation must contribute exactly one column per measurement
type, and a given type carries one cutoff throughout. Both are checked before
anything is uploaded, with an error that explains why.

Within those limits, order and naming are free — observations declared in
different column orders are normalised automatically.

Everything is validated against the actual data before upload: columns exist,
none is claimed twice, and values fall in the range IPA expects for their type.
That last check matters more than it looks — see
[measurement types](#measurement-types).

---

## Command-line reference

```
ipaapi validate   check a mapping against file(s) without uploading
ipaapi submit     upload into a project and start analyses
ipaapi status     check the state of existing analyses
ipaapi report     print IPA Interpret links
ipaapi history    list analyses submitted through this tool
```

### Mapping arguments

Used by `validate` and `submit`.

| Flag | Form | Meaning |
| --- | --- | --- |
| `PATH` | positional | a data file, or a directory to search |
| `--ID` | `COLUMN:TYPE` | 0-based identifier column and its IPA gene ID type. May be given twice — see [two identifier columns](#two-identifier-columns) |
| `--FC` | `COLUMN:TYPE[:CUTOFF]` | 0-based value column, [measurement type](#measurement-types), optional cutoff |
| `--skip-rows` | `N` | discard N lines above the header row |
| `--sep` | `CHAR` | field delimiter (sniffed from the header line by default) |
| `--pattern` | `TEXT` | when PATH is a directory: substring or glob selecting files |
| `--recursive` | flag | search subdirectories too |
| `--observation` | `NAME` | observation name in IPA (default: the filename). Single file only. Shortened to 60 characters if needed — see [long observation names](#long-observation-names) |
| `--strip` | `TEXT` | remove TEXT from observation names before shortening. Repeatable |
| `--no-range-check` | flag | skip the value-range validation |
| `--list-id-types` | flag | print all 33 gene ID types and exit |

### `submit`

| Flag | Default | Meaning |
| --- | --- | --- |
| `--project` | *required* | destination IPA project. **Created if it doesn't exist**, so a typo silently makes a new one |
| `--reference-set` | `omit` | `ipkb`, `dataset`, or `omit`. See [the reference set](#the-reference-set) |
| `--wait` | off | poll until analyses finish and print report links |
| `--interval` / `--timeout` | 30s / 3600s | polling, only with `--wait` |
| `--dry-run` | off | validate and stop before login |
| `--analysis-name` / `--dataset-name` | filename | single file only |
| `--log-file` | `~/.local/state/ipaapi/submissions.tsv` | submission log |

### Authentication arguments

Used by every command that contacts IPA.

| Flag | Meaning |
| --- | --- |
| `--no-cache` | ignore any cached token |
| `--token-file` | token cache path (default `~/.cache/ipaapi/token.json`) |
| `--application-name` | `applicationname` IPA scopes the session to (default `PythonAPI`) |
| `--browser` | browser to launch for login, e.g. `firefox` |

### `history`

| Flag | Meaning |
| --- | --- |
| `--project` / `--since` / `--limit` | filters |
| `--status` | look up each analysis's current state (requires login) |
| `--log-file` | read a different log |

### Environment variables

| Variable | Purpose |
| --- | --- |
| `IPAAPI_TOKEN_FILE` | token cache location — set this if `$HOME` isn't writable |
| `IPAAPI_LOG_FILE` | submission log location |

---

## Recipes

### Many files, one analysis each

```bash
ipaapi submit ~/data --pattern _DEG --ID 1:hugo --FC 4:logratio \
    --skip-rows 1 --reference-set ipkb --project Study1
```

`--pattern` takes plain text or a glob. Text with no `*`, `?` or `[` matches as
a **substring**, so `--pattern SampleA` finds `SampleA_DEG.txt` and
`SampleA_raw.tsv`. With no `--pattern`, `*.txt`/`*.tsv`/`*.csv` are searched.
Hidden files are skipped and results sorted, so run order is predictable.

Every matched file must fit the same `--ID`/`--FC` positions.

### Files are filed as they're processed

When `PATH` is a directory, each file moves as its outcome becomes known:

| Outcome | Destination |
| --- | --- |
| IPA accepted it | `submitted/` |
| The file is at fault | `failed/`, with a `.error.txt` note beside it |
| Allowance exhausted, or IPA declined | left in place for the next run |

```
submitted SampleA_DEG: 43595001
submitted SampleB_DEG: 43595002

Allowance exhausted while submitting SampleC_DEG:
REJECTED: the analysis allowance appears to be exhausted.
IPA said: 'Unable to run analysis: Analysis limit exceeded'

2 file(s) moved to submitted/
2 file(s) left in place for the next run
Re-run the same command later; the files left in place are exactly the ones
still to do.
```

The source directory shrinks to exactly the work outstanding, and re-running the
identical command resumes. `submitted/` and `failed/` are excluded from
discovery, so a run can't re-ingest its own output.

Nothing is moved when the *command* is at fault — a bad `--ID` type or a mapping
that fails every file leaves the directory untouched, because that's a mistake
to fix rather than data to quarantine. Single-file submits are never moved.

### Draining a backlog against a daily allowance

Because a stopped run resumes cleanly, this is safe to leave unattended:

```cron
0 6 * * * cd ~/data && ipaapi submit ./ --pattern _DEG --ID 1:hugo \
    --FC 4:logratio --skip-rows 1 --reference-set ipkb --project Study1 \
    >> ~/ipaapi-cron.log 2>&1
```

It submits until the allowance runs out, files what succeeded, leaves the rest.
Check the log after the first few runs — a cron job whose *refresh* token has
expired fails into that file rather than prompting anyone.

### Finding analysis IDs later

IPA's API cannot list the analyses on an account, so the package keeps its own
log — every submission appends a timestamped row.

```bash
ipaapi history
ipaapi history --project Study1 --since 2026-08-01
ipaapi history --status
```

```
2026-08-05T08:35:53-06:00  43595039  Study1  SampleA_DEG
2026-08-05T08:35:53-06:00  43595041  Study1  SampleB_DEG

2 submission(s). Report links: ipaapi report 43595039 43595041
```

Plain TSV — grep it, open it in a spreadsheet. It only covers submissions made
through this tool; anything submitted from the IPA client won't appear.

### Comment lines above the header

```
# generated by pipeline v3, 2026-08-05
EnsemblID	log2FC	pval
```

`--skip-rows 1` discards the preamble. Column numbers count from the header, so
they don't change when you add it.

Skipping also fixes delimiter detection: the delimiter is sniffed from the
header line, and a comment line is a bad thing to sniff — the one above has
commas but no tabs, so without `--skip-rows` the file would be read as CSV and
collapse into nonsense. Rather than let that through, a header that looks like a
comment is rejected with a message pointing at this flag.

### Two identifier columns

`--ID` may be given twice. The first is the primary; the second fills rows where
the primary is blank (`.`, `NA`, empty, and similar are all treated as missing).

```bash
ipaapi submit data.csv --ID 0:ensembl --ID 1:hugo --FC 4:logratio --project S1
```

> **Read this before relying on it.** IPA accepts one `geneidtype` per
> submission. Rows filled from the second column are still uploaded under the
> *primary's* type, so they may fail to map. The fill count is always reported:
>
> ```
> Warning: 344 of 2,338 rows took their identifier from the fallback column
> 'Common_name' (hugo). IPA is told a single gene ID type for the submission --
> 'ensembl' -- so those rows are uploaded under that declaration and may not map.
> ```
>
> If a large fraction is being filled, using the fallback column as the *only*
> identifier is usually better than mixing.

---

## Working with IPA

Most of this is either undocumented or documented somewhere hard to find. It's
recorded here because getting it wrong is expensive — analyses consume a
metered allowance.

### Gene ID types

`--ID COLUMN:TYPE` takes any value from IPA's `geneidtype` list (Integration
Module §3.1). `ipaapi submit --list-id-types` prints all 33.

Common ones: `ensembl`, `hugo`, `entrezgene`, `refseq`, `swissprot`,
`affymetrix`, `illumina`, `agilent`.

Two things are not guessable:

- **Human gene symbols are `hugo`.** Not `genesymbol`, not `hgnc`, and not the
  desktop client's own label `Gene Symbol` — all three are rejected outright.
- **Species rides on the identifier type.** There is no species parameter:
  `hugo` human, `mousesymeg` mouse, `ratsymeg` rat.

A type outside the documented list produces a warning with a near-match
suggestion but is still sent, since IPA is the authority and the list will age.
An unrecognised value fails before anything is uploaded, and IPA names it.

### Duplicate dataset names

**IPA refuses to create a dataset whose name already exists in a project — and
reports it as "The page you are looking for is currently unavailable."** The
wording says outage; the cause is a name collision. This is the single most
misleading response the API produces, and it cost a full day to identify.

Verified: the same 2 KB request, byte for byte, succeeded and then failed twice
in a row. With unique dataset names, three consecutive submissions all
succeeded.

Since dataset names come from filenames, re-running a batch retries names an
earlier run already created — so the run dies on its *first* file and looks
like a total outage.

The package prevents this using the submission log. Before uploading, it checks
whether that dataset name already went to that project:

```
skipping SampleA_DEG: already submitted to 'Study1' on 2026-08-05T08:35:53-06:00
as analysis 43595039. IPA would reject a second dataset of the same name.
Use --force to submit it again anyway.
```

Skipped files are filed under `submitted/`, because they are. `--force`
overrides, though IPA will normally reject it.

The guard only knows about submissions made through this tool with the same log
file. If you hit the collision anyway — a colleague's upload, or the IPA client
— the fix is a different `--project`, a different `--dataset-name`, or deleting
the existing dataset in IPA.

### Long observation names

**IPA rejects a long observation name, and reports it as "The page you are
looking for is currently unavailable."** The same misleading page as a
duplicate dataset name, from an entirely unrelated cause — which is what made
this one expensive to find: the obvious explanation had already been used up.

The observation name defaults to the filename, so descriptive pipeline output
names run past the limit without anyone choosing a long name. A long *dataset*
name is fine; only the observation is affected.

Established by A/B on one file, holding project, reference set and data
constant:

| dataset name | observation name | result |
| --- | --- | --- |
| short (25) | **long (82)** | rejected as an "outage" |
| **long (82)** | short (25) | accepted — reached IPA's allowance check |

65 characters is known good and 82 known bad; the exact limit is undocumented.
Since 1.2.0 the observation name is brought under 60 characters automatically.

**What gets removed is chosen by what it means, not by where it sits.** Two
parts of a Paralome filename carry information: the contrast (`Estrus_vs_2dpp`)
and the cell type the comparison was computed from
(`Immature_cortical_ovarian_stroma`). Everything the pipeline appends about how
it ran — `_naive_cell_t_significant_p0.05_rna`, `_pseudobulk_t_significant_p0.05_rna` —
is disposable. So that is what goes first:

```
Estrus_vs_2dpp_Immature_cortical_ovarian_stroma_naive_cell_t_significant_p0.05_rna
 -> Estrus_vs_2dpp_Immature_cortical_ovarian_stroma

Estrus_vs_2dpp_Glandular_epithelium_pseudobulk_t_significant_p0.05_rna
 -> Estrus_vs_2dpp_Glandular_epithelium
```

**Paralome output is cut on its own structure, not by guesswork.** Paralome
names its files
`<contrast>_<celltype>_<method>_<test>_significant_<threshold>_<assay>`, and
the `significant` literal is a reliable anchor:

```
Estrus_vs_2dpp_Immature_cortical_ovarian_stroma | naive_cell | t | significant_p0.05_rna
Estrus_vs_2dpp_Glandular_epithelium             | pseudobulk | t | significant_p0.05_rna
```

Working backwards from the anchor: drop it and everything after, drop the
statistical test immediately before it, then drop the aggregation method. The
test needs no list of names — it is simply the token before the anchor, so
`wilcox` works as well as `t` without anyone maintaining a vocabulary. Methods
are matched as **whole phrases in that one position only**, because a cell type
of `Naive_T_cell` shares both words with the `naive_cell` method and loose
matching cut it to `Naive_T`.

Removal is graded, least damaging first, and stops as soon as the names fit:

1. anything named with `--strip`
2. the Paralome tail, cut at its anchor as above
3. trailing pipeline metadata, for files from anything else — tokens like
   `significant`, `deseq2`, `filtered`, and cutoffs in any of the shapes
   `p0.05`, `fdr0.01`, `padj0.05`, `0.05`, read right to left and stopping at
   the first token that isn't recognisable as a setting
4. the suffix every file in the batch happens to share
5. the prefix every file shares — this costs the contrast, so it is late
6. a two-ended cut with `..` marking the gap, on a token boundary

Steps 2 to 4 always run together rather than stopping the moment the names
merely fit, since half a removed suffix reads worse than either whole. Comparison
is **whole tokens at a time**, so `Mature` is never treated as a prefix of
`Immature` and `cell_type` is never left as `cell_t`. Names already within the
limit are returned untouched.

Any step is abandoned if it would make two names identical, or leave one under
four characters or without a letter — reducing files to `1` and `2` keeps them
distinct and makes the analysis unreadable. Distinctness is the property being
protected: observation names are what IPA lists side by side in a comparison
analysis, so two files reduced to the same label would be worse than a long name.

Everything removed is printed:

```
note: shortened 2 observation names. IPA rejects a long observation name and
reports it as an outage, so this is not optional.
      Datasets and analyses keep the full filename; only the observation label
      inside the analysis is shorter.
      Estrus_vs_2dpp_Glandular_epithelium_pseudobulk_t_significant_p0.05_rna
   -> Estrus_vs_2dpp_Glandular_epithelium
```

If your pipeline's suffix is not recognised, name it explicitly. `--strip` is
repeatable, applied before anything else, and ignored as a whole if it would
leave the names unusable:

```bash
ipaapi submit ~/data --ID 1:mousesymeg --FC 4:foldchange --skip-rows 1 \
    --project Study --strip _significant_p0.05_rna
```

Because steps 4 and 5 depend on the other files in the run, **the same file
submitted in a different batch can get a different observation label**. Steps 1 to 3
do not — they read only the name in front of them, which is why a
single-file submit shortens as well as a batch does. The dataset and analysis
always keep the full filename, so `ipaapi history` and the IPA project view are
unaffected.

`--observation` sets the name explicitly for a single file and is shortened the
same way if it needs to be.

### The reference set

The background enrichment is scored against — the denominator of the Fisher's
exact test behind every p-value.

| Value | Background |
| --- | --- |
| `ipkb` | Ingenuity Knowledge Base (Genes Only, or + Endogenous Chemicals if chemicals are present) |
| `dataset` | the genes you uploaded |
| `omit` (default) | IPA chooses |

Which to use depends on **what you uploaded**:

- Uploading a **complete measured transcriptome** with a cutoff? `dataset` is
  the better science — the background is what your assay could actually detect,
  which controls for detection bias.
- Uploading a **pre-filtered hit list**? `dataset` makes the background nearly
  identical to the foreground. Use `ipkb`.

§4.1.3.1 states that with the parameter omitted IPA picks by size — `ipkb` below
2000 identifiers, `dataset` at 2000 or more. **In practice this has not been
observed to hold**: files of 1,804–6,245 rows all came back as
`Ingenuity Knowledge Base (Genes Only)`. Since the behaviour is unpredictable,
set it explicitly for anything you intend to compare against itself.

Verify after the fact — the setting is recorded in every IPA export:

```bash
grep -h "^Reference set" *_IPA_output.txt | sort | uniq -c
```

Array platforms can also be named as reference sets, paired with a
`referencesettype`. Not exposed here; see §4.1.3.

### Measurement types

| Value | Meaning | Valid range |
| --- | --- | --- |
| `ratio` | Ratio | `[0, +∞)` |
| `foldchange` | Fold Change | `(-∞, -1]` and `[1, +∞)` |
| `logratio` | Log Ratio | `(-∞, +∞)` |
| `pvalue` | p-value | `[0, 1]` |
| `falsediscovery` | FDR / q-value | `[0, 100]` |
| `intensity` | Intensity | `[0, +∞)` |
| `other` | Other (normalised around zero) | `(-∞, +∞)` |
| `gain_loss` | Variant Gain/Loss | `-2, -1, 0, 1, 2` |
| `classification` | Variant ACMG Classification | `-2, -1, 0, 1, 2` |

> **Out-of-range values are silently discarded by IPA.** §3.1: "analysis will
> still proceed without errors or warning diagnostics" — offending entries are
> simply dropped. This is why the range check exists and why it refuses rather
> than warns. Declaring log2 fold changes as `foldchange`, for instance, would
> quietly discard every gene between −1 and 1, which in a typical scRNA-seq
> table is most of them.

The package helps in both directions:

- Values declared `foldchange` that cluster inside (−1, 1) → suggests `logratio`.
- A column declared `logratio` with *no* values in (−1, 1) → warns that it looks
  like signed fold change, since a real log ratio is centred on zero.

A column called `Fold_change` may hold either. Check the data, not the name.

### What the API cannot do

- **List your projects.** `--project` creates one if the name doesn't exist, so
  a typo silently makes a new project rather than erroring.
- **List your analyses.** Every endpoint needs an ID you already hold — hence
  the local submission log.
- **Tell you your remaining allowance.** You discover the limit by hitting it.

### Errors IPA actually returns

IPA answers a rejected submission with an **HTML error page**, not plain text.
The reason is at the *end*, after support boilerplate. This package strips the
boilerplate and the page footer, and classifies what's left:

| IPA's message | Class | What the tool does |
| --- | --- | --- |
| `Unknown GeneId Type (X)` | `MalformedRequestError` | stops; names the flag; moves nothing |
| `Unable to run analysis: Analysis limit exceeded` | `QuotaExceededError` | stops; leaves remaining files for the next run |
| `Unable to run analysis: …` (other) | `AnalysisRefusedError` | as above — reached the analysis logic, so not a parameter fault |
| `currently unavailable` / `technical difficulties`, or 502/503/504 | `ServiceUnavailableError` | IPA is down; stops, moves nothing, says the command is fine |
| anything else | `SubmissionError` | files that one under `failed/` |

Quota matching is deliberately broad (`ipaapi.client.QUOTA_PATTERNS` plus HTTP
429): a false positive only leaves a file for the next run, while a false
negative would quarantine a retryable submission. The raw response is always
printed, so a misclassification is visible.

### Interpret links

`ipaapi report <id>` fetches the IPA Interpret URL for a finished analysis. It
checks status first, so an unfinished analysis says so rather than surfacing a
bare HTTP 500.

**These have been observed to return HTTP 500 even for succeeded analyses.**
The cause is unconfirmed — possibly the commercial add-on licence, possibly a
stale endpoint path inherited from the demo. `examples/probe_interpret.py`
prints the raw response for diagnosis. Analyses open fine in IPA itself.

---

## Authentication

Browser-based OAuth 2.0 with PKCE. Your password never reaches this package.

1. A short-lived HTTP server binds `127.0.0.1:8000`.
2. Your browser opens QIAGEN's authorization page; you log in there.
3. QIAGEN redirects back to `localhost:8000` with a one-time code. The `state`
   parameter is verified, then the code plus the PKCE verifier is exchanged for
   a token.
4. The token is used as `Authorization: Bearer …` and the server shuts down.

Whichever account you log in as owns the datasets and projects.

The client ID is the public one any IPA user may use — it is not a secret.

### Token caching and refresh

Tokens are cached at `~/.cache/ipaapi/token.json`, owner-only (0600). **Access
tokens are short-lived**, but a refresh token comes with them and is spent
automatically: an expired cache is renewed over HTTP with no browser and no
prompt. A browser login is only needed when the refresh token itself is
rejected.

Deleting the cache is effectively logging out. `--no-cache` forces a fresh
login. Be aware the token is plaintext on disk — anyone who can read your home
directory can use it until it expires.

### Headless servers

Because refresh is automatic, a token copied from a machine with a browser keeps
renewing itself indefinitely:

```bash
# once, on a machine with a browser
ipaapi submit ... # or any command that logs in

scp ~/.cache/ipaapi/token.json server:~/.cache/ipaapi/token.json
ssh server chmod 600 ~/.cache/ipaapi/token.json
```

**If `$HOME` isn't writable**, the cache can't be saved and every run needs a
fresh login — crippling on a headless box. Point it somewhere writable:

```bash
export IPAAPI_TOKEN_FILE=$HOME/ipaapi-token.json
export IPAAPI_LOG_FILE=$HOME/ipaapi-submissions.tsv
```

Both failures are reported loudly rather than swallowed, because a cache that
never writes looks exactly like a token that expires instantly.

When an interactive login is genuinely needed, X forwarding is the cleanest
route — the server-side browser renders locally *and* `localhost:8000` resolves
server-side where the callback listens, so no port forwarding is required:

```bash
ssh -X you@server        # ssh -Y from macOS, with XQuartz running
```

Failing that, forward the callback port and use your own browser:

```bash
ssh -L 8000:localhost:8000 you@server
```

The error message distinguishes `DISPLAY` unset from no browser found.

> The redirect URI is pinned to `http://localhost:8000` by the OAuth client
> registration, so the port is not configurable in practice.

### Using a token obtained elsewhere

```python
import os
from ipaapi import Credentials, IPAClient

client = IPAClient(Credentials.from_token(os.environ["IPA_TOKEN"]))
```

---

## Python API

```python
from ipaapi import (
    ColumnMapping, Dataset, IPAClient, Measurement, MeasurementType,
    Observation, ReferenceSet, TokenCache,
)

mapping = ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("HIV vs NEG", [
            Measurement("Fold_change", MeasurementType.LOG_RATIO),
        ]),
    ],
)

dataset = Dataset.from_file("results.csv", mapping, skip_rows=1)
print(dataset.describe())          # confirm before uploading

client = IPAClient.login(cache=TokenCache())
ids = client.submit(dataset, project="MyStudy", reference_set=ReferenceSet.IPKB)

for analysis_id, status in client.wait_for(ids).items():
    if status.succeeded:
        print(client.report_url(analysis_id))
```

Key objects:

| Object | Purpose |
| --- | --- |
| `ColumnMapping`, `Observation`, `Measurement` | describe the file |
| `Dataset.from_file` / `.from_frame` | load and validate |
| `IPAClient.login()` | OAuth, with caching and refresh |
| `.submit()` `.status()` `.wait_for()` `.results()` `.report_url()` | the API |
| `GENE_ID_TYPES` | all 33 identifier types and what they mean |
| `ipaapi.history` | the submission log |
| `ipaapi.errors` | everything derives from `IPAError` |

### Results

```python
results = client.results(analysis_id)
print(results.canonical_pathways.head())
cp, ur, df = results                  # unpacks like the demo's ipa_results()
```

> Programmatic result retrieval is a **commercial IPA add-on**. Without it these
> calls raise `ResultsUnavailableError`. Submission, status polling and report
> links are unaffected.

---

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `REJECTED: IPA does not recognise the gene ID type 'X'` | not in IPA's vocabulary | `--list-id-types`; human symbols are `hugo` |
| `declared 'foldchange' but holds N out-of-range value(s)` | log2 values declared as linear fold change | `--FC N:logratio` |
| `Could not find a header row … looks like a comment` | preamble above the header | `--skip-rows N` |
| `--FC refers to column N, but the file has only M column(s)` | 1-based counting, or wrong `--skip-rows` | positions are 0-based, from the header |
| `Every row is missing an identifier` | wrong column, or no header | check with `head -1 file \| tr '\t' '\n' \| nl -v0` |
| `the analysis allowance appears to be exhausted` | daily/period limit | re-run later; files left in place resume |
| `IPA appears to be down or having trouble` | IPA outage — **or a duplicate dataset name, or a long observation name**, all reported identically | `ipaapi --version` (1.2.0+ handles the name length); check `ipaapi history --project X` for the dataset name; otherwise wait |
| Batch dies on the first file, names come from long filenames | observation name too long | upgrade to 1.2.0+, or pass a short `--observation` |
| Batch dies on the *first* file after an earlier run | dataset names already exist in the project | expected — 1.1.0 skips them automatically; before that, use a new `--project` |
| `Cannot listen on 127.0.0.1:8000` | stale login process, or another user mid-login | `ss -ltnp 'sport = :8000'`, then kill it if it's yours |
| Login prompt on every run | token cache not writable | `export IPAAPI_TOKEN_FILE=...`; check for a root-owned cache |
| `Could not open a browser automatically` | headless | `ssh -X`, or copy a token across |
| `report` returns HTTP 500 on a succeeded analysis | unconfirmed; possibly add-on licence | open the analysis in IPA; see `examples/probe_interpret.py` |
| Analyses have z-scores but no p-values | reference set equals the gene list | `--reference-set ipkb` |
| Half of all pathways significant | list too large for the background | apply a cutoff, or upload unfiltered data with a cutoff |

Useful first move for any column problem:

```bash
head -1 yourfile.csv | tr ',\t' '\n' | nl -v0
```

---

## How a submission is encoded

Worth knowing when debugging. `--ID 1:hugo` becomes three separate things:

| From `--ID` | Wire parameter | Sent |
| --- | --- | --- |
| the type | `geneidtype=hugo` | once |
| the column, resolved from position to header name | `genecolname=Common_name` | once |
| that column's values | `geneid=XIST`, `geneid=UTY`, … | once per row |

The column *number* never leaves your machine.

The whole dataset travels in one `application/x-www-form-urlencoded` POST to
`/pa/api/v2/multiobsanalysis`, which both creates the dataset in the project and
starts one analysis per observation. Parameter naming is positional and
irregular — for measurement slot *k* and observation *i*, both zero-based:

| Parameter | Meaning |
| --- | --- |
| `expvaltype`, `expvaltypeK+1` | measurement type for slot *k* (global) |
| `cutoff`, `cutoffK+1` | cutoff for slot *k* (global, optional) |
| `obsI+1name` | observation name |
| `expvalname`, `expvalK+1name` | column label, first observation |
| `obsI+1expvalname`, `obsI+1expvalK+1name` | column label, later observations |
| `geneid` | one per data row |
| `expvalue`, `expvalK+1` | one per slot per observation, per row |

Per-row value parameters carry no observation prefix — they cycle through the
slots of observation 1, then observation 2, and so on. Order is load-bearing.

The body is properly percent-encoded. The demo concatenated it by hand, so any
value containing a space, `&`, `=`, `+` or `%` corrupted the request — including
the `Group Max Intensity` column in the demo's own sample dataset.

---

## Development

```
src/ipaapi/
  __init__.py    public API and the version (single source of truth)
  models.py      MeasurementType, AnalysisStatus, ReferenceSet, GENE_ID_TYPES
  mapping.py     Measurement, Observation, ColumnMapping
  dataset.py     Dataset, load_table
  _payload.py    multiobsanalysis body construction
  auth.py        OAuth 2.0 + PKCE, Credentials, TokenCache, refresh
  client.py      IPAClient, error classification
  history.py     the submission log
  triage.py      submitted/ and failed/ filing
  cli.py         the ipaapi console script
  errors.py      exception hierarchy
tests/           offline; no network required
examples/        runnable scripts and diagnostics
```

```bash
pip install -e ".[dev]"
pytest
```

The suite is fully offline — mapping validation, the exact parameter layout of
the submission body, encoding of hostile characters, error classification,
triage behaviour, token cache and refresh logic.

**Versioning.** The version lives only in `src/ipaapi/__init__.py`;
`pyproject.toml` reads it at build time. Bump it there and nowhere else, and add
a `CHANGELOG.md` entry. `ipaapi --version` reports the install path too, which
is what actually answers "am I running the wheel I think I am".

### Differences from the demo

- Column mapping by name in any order, validated before upload.
- Request bodies are percent-encoded.
- OAuth: no CPU-spinning wait loop, `state` is verified, logins time out, the
  callback server is shut down, error redirects are handled, tokens are cached
  and refreshed.
- Submissions are never retried automatically — a retried POST could create a
  duplicate analysis. GETs retry with backoff.
- Typed exceptions; access tokens excluded from `repr()`.
- No `install_dependencies()` shelling out to `pip3`.

---

## Contributing

Issues and pull requests are welcome. The most useful contributions are
**corrections to the [Working with IPA](#working-with-ipa) section** — much of
it was established by trial against a live account, and a few points have
already had to be corrected more than once. If IPA behaves differently for you,
that is worth reporting even without a code change.

```bash
pip install -e ".[dev]"
pytest
```

Tests are fully offline; none of them contact IPA.

---

## Status

**1.0** — stable and in production use against live IPA. The command line and
the Python API are settled; breaking changes from here mean a major version
bump. See `CHANGELOG.md`.

Known open questions, none of which affect submission:

- Interpret links (`ipaapi report`) have returned HTTP 500 for analyses that
  succeeded. Cause unconfirmed; possibly the commercial add-on licence.
- Programmatic result retrieval (`client.results()`) requires that same add-on
  and is largely untested here.
- The documented reference-set size rule does not match observed behaviour;
  set `--reference-set` explicitly.

---

## Licence

MIT — see [LICENSE](LICENSE). Free to use, modify and redistribute.

Not affiliated with, endorsed by, or supported by QIAGEN. IPA is QIAGEN's
product; this is an independent client for its public API, built on the
`python-api-demo` example code QIAGEN publishes. For questions about the API
itself, QIAGEN's contact is `AdvancedGenomicsSupport@qiagen.com` — please don't
send them bug reports about this package.
