Metadata-Version: 2.5
Name: carwatch
Version: 1.0.1
Summary: Processing of CARWatch sampling logs and saliva data.
Project-URL: Homepage, https://github.com/carwatch-tools/carwatch-python
Project-URL: Documentation, https://github.com/carwatch-tools/carwatch-python/tree/main/docs
Project-URL: Repository, https://github.com/carwatch-tools/carwatch-python
Project-URL: Issues, https://github.com/carwatch-tools/carwatch-python/issues
Project-URL: Changelog, https://github.com/carwatch-tools/carwatch-python/blob/main/CHANGELOG.md
Author-email: Robert Richer <robert.richer@fau.de>
License-Expression: MIT
License-File: LICENSE
Keywords: ambulatory assessment,cortisol,saliva,sampling compliance
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Requires-Python: <4.0,>=3.10
Requires-Dist: ipympl>=0.10.0
Requires-Dist: matplotlib<4,>=3.10
Requires-Dist: numpy<3,>=1.26
Requires-Dist: pandas<3,>=2
Requires-Dist: pingouin<1,>=0.5
Requires-Dist: scipy<2,>=1
Requires-Dist: seaborn<1,>=0.13
Provides-Extra: interactive
Requires-Dist: ipydatagrid<2,>=1.4; extra == 'interactive'
Requires-Dist: ipywidgets<9,>=8; extra == 'interactive'
Description-Content-Type: text/markdown

# `carwatch` for Python - Tools for processing CARWatch and saliva data

<img src="docs/_static/brand/logo.svg" align="right" width="120" alt="CARWatch logo">

[![PyPI](https://img.shields.io/pypi/v/carwatch)](https://pypi.org/project/carwatch/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![Documentation Status](https://readthedocs.org/projects/carwatch-python/badge/?version=latest)](https://carwatch-python.readthedocs.io/en/latest/?badge=latest)
[![Test and Lint](https://github.com/carwatch-tools/carwatch-python/actions/workflows/test-and-lint.yml/badge.svg)](https://github.com/carwatch-tools/carwatch-python/actions/workflows/test-and-lint.yml)
[![codecov](https://codecov.io/gh/carwatch-tools/carwatch-python/branch/main/graph/badge.svg?token=IK0QBHQKCO)](https://codecov.io/gh/carwatch-tools/carwatch-python)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![PyPI downloads](https://img.shields.io/pypi/dm/carwatch)](https://pypi.org/project/carwatch/)
![GitHub commit activity](https://img.shields.io/github/commit-activity/m/carwatch-tools/carwatch-python)

This Python package supports the processing of sampling logs recorded by
_CARWatch_ and their integration with laboratory biomarkers. It is designed
for ambulatory sampling studies in which researchers need auditable sampling
times, protocol deviations, manual diary fallbacks, and biomarker features.

The package reads the app log exports, reconstructs each study day, compares
planned and recorded sample times, lets you review unclear records, and
combines the result with laboratory data, such as cortisol. The original app
exports are never changed. The package creates a separate table of questions
and decisions when information is missing or inconsistent. This makes it
possible to rerun the same analysis later and understand how each decision was
made.

The features include:

- Import CARWatch logs from CSV files, ZIP archives, and participant folders
- Reconstruct expected study days, sampling positions, and timing compliance
- Create and reload a review table for conversion issues
- Review conversion anomalies in a structured, editable two-pass issue report
- Patch missing timestamps from manual measurement diary
- Read and write complete Study Results CSV files
- Merge laboratory saliva measurements by physical sample ID or scheduled
  sample position
- Correct documented tube swaps
- Compute cortisol response features and static quality-control plots

The [examples directory](https://github.com/carwatch-tools/carwatch-python/tree/main/examples)
contains end-to-end notebooks and focused gallery workflows. The
[user guides](https://carwatch-python.readthedocs.io/en/latest/guides/index.html)
explain individual parts of the workflow in more depth.

## Installation

CARWatch requires Python 3.10 or newer. Install it from PyPI in an existing
Python environment:

```bash
pip install carwatch
```

With [uv](https://docs.astral.sh/uv/):

```bash
uv add carwatch
```

The
[beginner setup tutorial](https://carwatch-python.readthedocs.io/en/latest/guides/python_setup.html)
explains how to install uv, create an isolated CARWatch analysis environment,
install the package, and run Jupyter notebooks on macOS, Linux, and Windows.

Install the optional Jupyter decision editor and interactive plots with:

```bash
uv add "carwatch[interactive]"
```

Install the current development version directly from GitHub with:

```bash
uv add "carwatch @ git+https://github.com/carwatch-tools/carwatch-python.git"
```

The `main` branch can contain unreleased changes. Development setup is
documented under [For developers](#for-developers).

## Typical workflow

### Workflow at a glance

CARWatch keeps data import, researcher decisions, and analysis as separate
steps:

```text
CARWatch app exports
        |
        v
Import logs and check which files were used
        |
        v
First conversion: reconstruct the study and create a review report
        |
        v
Review the reported issues and record your decisions
        |
        v
Final conversion: create complete Study Results
        |
        +----> inspect timing and compliance
        +----> save and reload the processed study
        +----> merge laboratory data and compute saliva features (e.g., AUC, max increase, slope)
```

The two conversion passes are intentional. The first pass shows how CARWatch
understands the app logs and identifies records that need review. The second
pass applies the submitted decisions to the original logs. The app exports
remain unchanged throughout.

### Basic concepts

| Term | Meaning |
| --- | --- |
| **Raw logs** (`raw_logs`) | The events exported by the CARWatch app, such as saved study settings, awakening events, and barcode scans. These are the original input data. |
| **File-import log** (`source_audit`) | A table showing which CSV or ZIP files were used or skipped and why. |
| **Registration** | A set of study settings saved in the app for a participant: study name, number of days, sample IDs in their planned order, and sampling times. Saving a changed setup creates another registration. |
| **Protocol** | The intended order of registrations, study days, and samples across the whole study. CARWatch normally reconstructs it from the registrations found in the app logs. |
| **Protocol manifest** (`protocol_manifest`) | An optional Python list in which you state the intended protocol order explicitly. Most studies do not need one. It is useful when the available app logs do not establish one unambiguous order. |
| **Conversion report** | The review table produced by the first conversion. It describes missing or inconsistent information and contains suggested decisions. Suggestions are not applied automatically. |
| **Manual diary** | A table of awakening and sampling times that were written down outside the app (e.g., on paper). This can be used as fallback information when the app logs are incomplete. It is used only when a decision explicitly sets the source of an awakening or sampling time to "manual diary" because the app logs are missing or inconsistent. The manual diary is not used automatically. |
| **Scheduled sample** | The tube expected at a particular position according to the registration. |
| **Recorded sample** | The tube actually scanned in the app. It may differ from the scheduled tube, for example after a tube swap. |
| **Sample position** | The first, second, third, and so on sampling position defined by the order saved in the registration. CARWatch does not derive this from the spelling of a tube ID or file name. |
| **Study Results** (`study_results`) | The final processed study data. They contain study days, planned and recorded sampling times, compliance, information about where each time came from, and later any merged laboratory values. |

### Detailed workflow

#### 1. Load raw CARWatch logs

For one folder per participant, provide the participant IDs explicitly. Folders
can contain nested CSV exports and ZIP archives.

```python
from pathlib import Path

import carwatch as cw

participant_folders = {
    "vp01": Path("data/carwatch/vp01"),
    "vp02": Path("data/carwatch/vp02"),
}

raw_logs, source_audit = cw.io.load_raw_logs_from_participant_folders(
    participant_folders,
    create_report=True,
)
```

`source_audit` records which files CARWatch used or skipped and why. When the
exact sources are already known:

```python
raw_logs = cw.io.load_raw_logs(
    ["data/carwatch/vp01.csv", "data/carwatch/vp02.zip"]
)
```

#### 2. Create a first review report

The first conversion creates provisional results and a list of records that
need review. It also calculates planned sample times and timing compliance.

```python
initial_results, conversion_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        errors="warn",
        create_report=True,
    )
)

print(conversion_report["summary"])
conversion_report["issues"].to_csv("conversion_issues.csv")
```

The suggested decisions are recommendations. Nothing is corrected until an
edited report is submitted in the final conversion.

##### If CARWatch cannot determine the protocol order

Inspect the reconstructed protocol with:

```python
cw.logs.summarize_protocol(raw_logs)
cw.logs.extract_registration_schedule_from_raw_logs(raw_logs)
```

Most studies do not need a `protocol_manifest`. Create one only when the logs
cannot establish the intended order reliably, for example because participants
completed registrations in conflicting orders, one registration is absent, or
only incomplete participant records are available.

```python
protocol_manifest = [
    {
        "study_name": "Control",
        "study_days": 2,
        "saliva_ids": ["control-1", "control-2", "control-3", "control-4"],
        "saliva_times": [0, 30, 15, 15],
        "saliva_absolute_times": [],
    },
    {
        "study_name": "Challenge",
        "study_days": 2,
        "saliva_ids": [
            "challenge-1",
            "challenge-2",
            "challenge-3",
            "challenge-4",
        ],
        "saliva_times": [0, 30, 15, 15],
        "saliva_absolute_times": [],
    },
]

initial_results, conversion_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        protocol_manifest=protocol_manifest,
        errors="warn",
        create_report=True,
    )
)
```

The list order defines registration order. `study_days` defines the number of
days in each registration. `saliva_ids` defines planned tube order, while
`saliva_times` contains relative timing intervals in minutes. Fixed clock times
belong in `saliva_absolute_times`. Pass the same manifest to the first and
final conversion and to the interactive editor.

#### 3. Review and apply decisions

Resolve the reported issues in a spreadsheet or in the optional Jupyter
editor. Both routes produce the `decisions` table used by the final conversion.

Load the manual diary when report decisions use it as a fallback. Omit this
line and the `manual_diary` arguments below when the study has no accepted
manual-diary decisions.

```python
manual_diary = cw.io.load_manual_diary("manual_diary.csv")
```

**Option A: edit the report in a spreadsheet.** Open the CSV, choose a decision
for every issue, and save it without changing identifying columns. Then reload
it:

```python
decisions = cw.logs.load_conversion_issue_report("conversion_issues.csv")
```

**Option B: resolve issues interactively.** This requires the `interactive`
extra.

![CARWatch conversion issue editor with issue table on the left and decision controls on the right](docs/images/conversion_issue_editor.png)

```python
from IPython.display import display

editor = cw.logs.interactive_conversion_issue_report(
    raw_logs,
    conversion_report["issues"],
    manual_diary=manual_diary,
)
display(editor.widget)

# After completing and refreshing the issue queue:
decisions = editor.decisions
decisions.to_csv("conversion_issues.csv")
```

Select an issue, choose a valid decision, and apply it. Use **Refresh remaining
issues** to rerun conversion with decisions made so far. Accepted upstream
decisions remain in the history while the visible queue is replaced by issues
that still need attention. If a diary-backed decision cannot be applied, that
issue is reset to **Leave unresolved** and remains visible while successfully
applied decisions are hidden.

The editor offers only decisions that are valid for the selected issue:

| Decision | Effect | Decision value |
| --- | --- | --- |
| `accept` | Apply the suggested action shown in `proposed_action`. Read its description before accepting because the effect depends on the issue. | Not required. |
| `keep` | Mark the issue as reviewed without applying the suggested correction. The current reconstruction, including any missing value, is retained. | Not required. |
| `drop_sample` | For an issue tied to a sample, clear that sample from the participant's Study Results while retaining the planned sample position. | Not required. |
| `drop_day` | For an issue tied to a study day, clear the complete participant-day from Study Results. | Not required. |
| `drop_participant` | Remove the participant associated with the issue from Study Results. | Not required. |
| `override_expected_sample` | Assign a scan whose expected sample cannot be resolved to another sample in the active registration. This option appears only for the corresponding issue type. | The exact registered sample ID to use. |
| `change` | Apply an alternative, issue-specific correction instead of the proposed action. This option appears only when the issue supports it. | Required; the allowed value depends on the issue, as shown below. |

When `change` is available, `user_decision_value` supports these values:

| Issue | Allowed value for `change` |
| --- | --- |
| Multiple collection dates | `use_earliest_collection_date`, `use_latest_collection_date`, or a complete JSON mapping such as `{"2026-01-05":"D1","2026-01-06":"D2"}`. |
| Possible re-registration | A JSON target such as `{"registration":2}` or `{"registration":"Challenge"}`. |
| Sampling times are not increasing | `sort_samples_by_time`. |
| Missing awakening time | `use_manual_diary_awakening_time`, or an explicit local timestamp such as `2026-01-05 07:10`. |
| Missing scheduled sample | `use_manual_diary_sampling_time` or `use_default`. The latter reconstructs the time from the registered schedule or supplied `sampling_schedule`. |

Run the final conversion against the original logs:

```python
study_results, final_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        errors="raise",
        create_report=True,
        issue_decisions=decisions,
        manual_diary=manual_diary,
    )
)
```

Omit `manual_diary` when no accepted decision uses it. If the first pass used a
`protocol_manifest` or `sampling_schedule`, pass the same object to the editor
and final conversion. Strict conversion stops if a submitted decision is
invalid or an issue remains unresolved.

#### 4. Inspect and save complete Study Results

`study_results` contains the complete information needed for later merging,
analysis, and plotting. Helper functions provide focused study-day and sample
tables for inspection.

```python
study_days = cw.logs.extract_day_summary_from_summary(study_results)
sample_events = cw.logs.extract_sample_events_from_summary(study_results)
cw.io.save_study_results(study_results, "study_results.csv")

analysis_results = cw.compliance.drop_non_compliant_samples(study_results)
cw.compliance.summarize_compliance(study_results)
cw.compliance.find_non_compliant_samples(study_results)
```

The study-day table includes collection dates, awakening information,
registration context, and day compliance. The sample table includes planned
and recorded times, sample positions, tube IDs, timing deviations, and sample
compliance. Filtering preserves the complete wide Study Results structure.

#### 5. Restore results or import a Study Manager export

```python
study_results = cw.io.load_study_results("study_results.csv")
display_results = cw.io.load_study_results("study_results.csv", simple=True)
study_results = cw.io.load_study_manager_export("study_manager_export.csv")
```

Use `simple=True` only for a compact table to inspect or share. Keep the
default complete form for merging, analysis, compliance checks, and plotting.

#### 6. Load laboratory saliva data

For matching by physical tube ID, use a long CSV with `participant`, `sample`,
and one biomarker column:

```text
participant,sample,cortisol
vp01,tube-a,8.2
vp01,tube-b,12.6
```

```python
saliva = cw.io.load_saliva("cortisol.csv", saliva_type="cortisol")
```

For position-based laboratory data, create an index containing participant,
canonical day, and sample position. Additional day-level labels such as
`condition` can be retained as named index levels.

```python
import pandas as pd

saliva_by_position = (
    pd.read_csv("cortisol_by_position.csv")
    .set_index(["participant", "day", "sample_position", "condition"])
)
```

#### 7. Merge sampling and laboratory data

Recorded physical IDs correct documented swaps by default. Set
`correct_swaps=False` to match scheduled IDs instead.

```python
merged_results = cw.merge.merge_saliva(
    study_results,
    saliva,
    match_on="sample",
    correct_swaps=True,
)

merged_results = cw.merge.merge_saliva(
    study_results,
    saliva_by_position,
    match_on="position",
)
```

The result remains complete Study Results. It records whether a laboratory
value was found, which tube was used, and whether a documented swap was
corrected.

#### 8. Compute features and quality-control plots

The CARWatch feature adapter groups by participant and day, orders samples by
`sample_position`, and uses actual sampling times.

```python
cortisol_features = cw.saliva.compute_features_from_carwatch(
    merged_results,
    saliva_type="cortisol",
)
```

The sampling timeline compares protocol targets, app-updated targets, and
recorded collection times for one participant-day. Arrows show signed timing
deviation; color indicates compliance and marker shape identifies the source
of the recorded time.

```python
fig, ax = cw.plotting.plot_sampling_timeline(
    study_results,
    participant="VP_01",
    day="D1",
)
```

![Sampling timeline showing planned and recorded collection times](docs/images/sampling_timeline.png)

The compliance overview summarizes compliant, non-compliant, and unassessed
samples at each sampling position.

```python
fig, ax = cw.plotting.plot_compliance_overview(study_results)
```

![Proportion of compliant, non-compliant, and unassessed samples by sampling position](docs/images/compliance_overview.png)

The deviation plot shows how early or late samples were collected at each
sampling position. Individual points remain visible behind the boxplots.

```python
fig, ax = cw.plotting.plot_timing_deviation(study_results)
```

![Distribution of sampling-time deviations by sample position](docs/images/timing_deviation.png)

The saliva curve retains individual participant-day trajectories and adds the
mean response with its confidence interval.

```python
fig, ax = cw.plotting.plot_saliva_curve(
    merged_results,
    value="cortisol",
    group_by="condition",
)
```

![Individual and mean cortisol response curves over time since awakening](docs/images/saliva_curve.png)

The figures are generated from deterministic synthetic data with:

```bash
uv run python tools/generate_readme_figures.py
```

The Python and R figure generators use the same study configuration, 40
participants, anomaly ratios, random seed, bootstrap settings, figure sizes,
and resolution.

For generic long-format saliva data, use `compute_features()`, `auc()`,
`max_value()`, `initial_value()`, `max_increase()`, or `slope()`.

### Optional interactive review

In Jupyter, launch a participant/day selector around the same static timeline:

```python
from IPython.display import display

timeline_widget = cw.plotting.interactive_sampling_timeline(study_results)
display(timeline_widget)
```

### Synthetic data

Generate deterministic local example data without changing real study files:

```python
study_root = cw.example_data.generate_synthetic_study_data(
    "carwatch-example",
    n_participants=4,
    random_state=42,
    non_compliant_sample_ratio=0.10,
    missing_awakening_time_ratio=0.01,
    missing_sampling_time_ratio=0.02,
    create_cortisol_data=True,
)
```

The generated directory contains raw logs, `manual_diary.csv`, a
ready-to-submit `issue_decisions.csv`, and position-indexed `cortisol.csv` when
requested.

## For developers

Install [uv](https://docs.astral.sh/uv/getting-started/installation/), clone the
repository, and synchronize the project environment:

```bash
git clone https://github.com/carwatch-tools/carwatch-python.git
cd carwatch-python
uv sync
```

Run the relevant checks after changing the package or documentation:

```bash
uv run poe format
uv run poe ci_check
uv run poe test
uv run poe docs_clean
```

Use `uv run poe docs_preview` to inspect the built documentation. Dependencies
are managed through `pyproject.toml`. Package invariants and task-specific
guidance are documented in
[`AGENTS.md`](https://github.com/carwatch-tools/carwatch-python/blob/main/AGENTS.md),
the
[repository-local skills](https://github.com/carwatch-tools/carwatch-python/tree/main/skills),
and the
[agent-assisted workflow guide](https://carwatch-python.readthedocs.io/en/latest/guides/agent_assisted_workflows.html).

## Contributing

Bug reports, feature requests, and reproducible examples belong in the
[GitHub issue tracker](https://github.com/carwatch-tools/carwatch-python/issues).
Changes should include tests and documentation for the affected research
workflow.

## Citation

Report the package version used in an analysis. For research using CARWatch,
cite:

> Richer, R., Abel, L., Küderle, A., Eskofier, B. M., & Rohleder, N. (2023).
> CARWatch — A smartphone application for improving the accuracy of cortisol
> awakening response sampling. *Psychoneuroendocrinology, 151*, 106073.
> https://doi.org/10.1016/j.psyneuen.2023.106073

The installed package version is available as:

```python
import carwatch

print(carwatch.__version__)
```

## License

CARWatch for Python is available under the [MIT License](LICENSE).
