Metadata-Version: 2.5
Name: pyautostat
Version: 0.5.0
Summary: Automated statistical analysis for research data: normality, hypothesis testing, correlation, outliers, and actionable insights.
Project-URL: Homepage, https://github.com/majikoushik/pyautostat
Project-URL: Issues, https://github.com/majikoushik/pyautostat/issues
Project-URL: Changelog, https://github.com/majikoushik/pyautostat/blob/main/CHANGELOG.md
Author-email: Koushik Chandra Maji <koushiknsec34@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: data-analysis,eda,hypothesis-testing,research,statistics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: numpy<3,>=1.23.5
Requires-Dist: pandas!=2.1.0,>=1.4
Requires-Dist: scipy!=1.9.2,>=1.8
Requires-Dist: statsmodels<0.16,>=0.15
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: openpyxl>=3; extra == 'dev'
Requires-Dist: plotly>=5; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: examples
Requires-Dist: openpyxl>=3; extra == 'examples'
Provides-Extra: report
Requires-Dist: plotly>=5; extra == 'report'
Description-Content-Type: text/markdown

# PyAutoStat

[![CI](https://github.com/majikoushik/pyautostat/actions/workflows/ci.yml/badge.svg)](https://github.com/majikoushik/pyautostat/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://github.com/majikoushik/PyAutoStat/blob/main/pyproject.toml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/majikoushik/PyAutoStat/blob/main/LICENSE)

PyAutoStat is a statistical research assistant for pandas DataFrames. It provides deterministic,
statistically bounded workflows for profiling data, recording a research question, recommending
and executing supported methods, interpreting results, and producing reproducible reports.

The package is in **alpha**. Researchers remain responsible for study-design facts, measurement
meaning, sampling assumptions, and the scientific use of results. PyAutoStat leaves unknown design
facts unresolved rather than guessing them.

## Installation

PyAutoStat requires Python 3.10 or newer:

```bash
python -m pip install pyautostat
```

Interactive Plotly reports are optional:

```bash
python -m pip install "pyautostat[report]"
```

## Quick start: profile a DataFrame

Profiling needs only a DataFrame. It does not require a research question.

```python
import pandas as pd

from pyautostat import ResearchAssistant

df = pd.DataFrame(
    {
        "class_section": ["A", "A", "B", "B", "A", "B"],
        "hours_studied": [2, 3, 4, 5, 6, 7],
        "exam_score": [55, 61, 67, 74, 81, 88],
    }
)

assistant = ResearchAssistant(df)
profile = assistant.profile()

print(profile["overview"])
print(profile["data_quality"]["issues"])
print(profile["resource_info"])
print(assistant.summarize())  # Plain-text view of the same profile
print(assistant.summarize(mode="story"))  # Opt-in connected dataset narrative
```

The profile includes descriptive summaries (including P5, P25, P50, P75, P95 and a safeguarded
coefficient of variation), missingness, duplicates, distributions, histograms,
outlier review cues, correlation records, type and role suggestions, and structured resource
advisories. Profiling never samples, truncates, imputes, or modifies the source DataFrame.
Story mode reads those same recorded values, prioritizes at most three profile-driven first steps,
and does not infer domain explanations. The established `summarize()` output remains the default.

An individual numerical column can be narrated from its existing descriptive record without
rerunning statistics:

```python
from pyautostat import column_story

print(column_story("exam_score", profile["descriptive"]["exam_score"]))
```

Categorical exploration is similarly direct. Results remain JSON-safe and include narration as a
separate field over the structured counts and percentages:

```python
frequency = assistant.frequency_table("class_section")
print(frequency["levels"])
print(frequency["narrative"])

cross_tab = assistant.cross_tab(
    "class_section",
    "exam_score",
    data_dictionary={"exam_score": {"type": "ordinal", "ordinal_order": [55, 61, 67, 74, 81, 88]}},
)
print(cross_tab["counts"])
print(cross_tab["row_percent"])
```

Frequency percentages use non-missing observations; missing counts are separate. Cross-tabs use
complete cases for the selected pair and expose counts, row percentages, column percentages, and
total percentages separately. Identifiers and continuous measurements require an explicit,
scientifically justified categorical declaration before these APIs will accept them.

## Guided analysis

Supply the scientific target and design facts that cannot be inferred safely:

```python
scores = pd.DataFrame(
    {
        "teaching_method": ["standard"] * 6 + ["new"] * 6,
        "exam_score": [62, 65, 68, 70, 72, 75, 67, 71, 74, 78, 80, 84],
    }
)

assistant = ResearchAssistant(scores)
workflow = assistant.run(
    objective="compare_groups",
    outcome="exam_score",
    predictor="teaching_method",
    estimand="mean",
    design="independent",
    variable_types={"exam_score": "continuous"},
)

print(workflow.status.value)  # completed
print(workflow.analysis.method_id)  # welch_t (stable machine identifier)
print(workflow.analysis.method_label)  # Welch independent-samples t-test
print(workflow.recommendation.rationale_text)  # Why this test, alternatives, verification
print(workflow.explain())  # Qualified, printable result overview
print(workflow.interpretation.findings_plain)
print(workflow.audit.status)
html = workflow.report.to_html()  # Includes a deterministic executive summary
```

`run()` connects question validation, one method recommendation, one statistical execution,
deterministic interpretation, reporting, consistency audit, and reproducibility metadata. It does
not write files or replay the analysis automatically. Recommendation explanations and report
summaries use only the stored specification, decision trace, diagnostics, results, effect labels,
warnings, and limitations; they do not use generative AI.

### Linear regression

Simple and multiple ordinary least-squares regression use a declared continuous outcome and one
or more ordered predictors. The target is the conditional mean, and independent observational
units must be declared explicitly:

```python
regression = ResearchAssistant(df).run(
    objective="regression",
    outcome="exam_score",
    predictors=["hours_studied", "attendance"],
    estimand="conditional_mean",
    design="independent",
    variable_types={
        "exam_score": "continuous",
        "hours_studied": "continuous",
        "attendance": "continuous",
    },
    covariance_type="HC3",  # or "classical"
)
print(regression.explain())
```

Numerical predictors remain in their original units. Boolean, nominal, and ordinal predictors use
explicit treatment coding; ordinal levels are categorical and are never assumed equally spaced.
Set references with `reference_levels={"study_method": "standard"}`. Otherwise the declared
category order, pandas categorical order, or first observed complete-case level is recorded as the
reference. One complete-case sample is formed across the outcome and every predictor, an
intercept is included, and rank-deficient designs are blocked without dropping terms.

Results include coefficient estimates, standard errors, t tests, coefficient intervals, model F,
R-squared, adjusted R-squared, residual error measures, and continuous-predictor standardized
betas. VIF, Breusch-Pagan, Jarque-Bera, influence heuristics, and the condition number are advisory
records; they never select a covariance estimator, remove observations, or change the model.
HC3 is an explicit alternative to classical covariance and changes uncertainty estimates, not OLS
point coefficients. Coefficients are conditional associations, not causal effects; R-squared is
in-sample fit, not out-of-sample predictive accuracy. See
[`examples/linear_regression.py`](examples/linear_regression.py).

### Binary outcomes and extended association

Binary logistic regression uses the existing regression objective with
`estimand="event_probability"` and an explicit modeled event:

```python
logistic = ResearchAssistant(df).run(
    objective="regression",
    outcome="churned",
    predictors=["age", "monthly_spend", "membership"],
    estimand="event_probability",
    design="independent",
    event_level="yes",
    variable_types={
        "churned": "nominal",
        "age": "continuous",
        "monthly_spend": "continuous",
        "membership": "nominal",
    },
)
```

The result records likelihood fit, McFadden pseudo-R-squared, convergence, VIF and condition
diagnostics, and coefficient-level log-odds and odds ratios with intervals. Odds ratios are not
constant probability changes, the fit is not out-of-sample predictive validation, and the model
does not establish causation. Non-Boolean categories require `event_level`; Boolean outcomes use
`True`, while analytically declared binary `{0, 1}` outcomes use `1` when no level is supplied.

Paired binary proportions use exact two-sided McNemar inference with explicit `unit_id`,
`condition_order`, and `event_level`. Point-biserial correlation uses an explicit positive level.
Kendall tau-b is selected with `estimand="monotonic", association_measure="kendall"`; Spearman
remains the default monotonic method. Partial Pearson uses `estimand="partial_linear"` and an
explicit nonempty `controls=[...]` list. Its linear adjustment describes a conditional
association; it does not show that confounding was removed. All four effect intervals resample
whole observation pairs, units, or complete rows as required. See
[`examples/binary_and_extended_association.py`](examples/binary_and_extended_association.py).

### Survey and scale reliability

Declare the scored items that form a proposed scale; PyAutoStat never discovers a scale or
reverse-scores an item automatically:

```python
reliability = ResearchAssistant(df).reliability(
    items=["q1", "q2", "q3", "q4"],
)
print(reliability.explain())
```

The complete-case workflow reports Cronbach's alpha, a deterministic respondent-row bootstrap
interval, corrected item-total correlations, alpha if each item is deleted, item distributions,
missingness, and the complete inter-item correlation matrix. Optional reverse scoring is explicit,
for example `reverse_scoring={"q4": (1, 5)}`, and operates on an internal copy. Alpha is an
internal-consistency estimate, not a hypothesis test, universal pass/fail grade, proof of
unidimensionality, or evidence of construct validity. See [`examples/scale_reliability.py`](examples/scale_reliability.py).

### Rater reliability and agreement (ICC)

Assess inter-rater or test-retest reliability across targets and raters with the Intraclass Correlation Coefficient (ICC):

```python
icc_workflow = ResearchAssistant(df).intraclass_correlation(
    target="subject_id",
    rater="rater_id",
    value="score",
    model="two_way_random",
    definition="absolute_agreement",
    unit="single",
)
print(icc_workflow.explain())
```

Supports all 6 canonical Shrout & Fleiss (1979) and McGraw & Wong (1996) variants: ICC(1,1), ICC(1,k), ICC(2,1), ICC(2,k), ICC(3,1), ICC(3,k) as well as McGraw-Wong C(2) and A(3) aliases. The workflow decomposes variance into ANOVA mean squares ($BMS, JMS, EMS, WMS$), provides method-of-moments variance components, analytical F-inversion confidence intervals (exact or Satterthwaite effective df approximation), hypothesis testing against zero, and complete-target filtering without clamping negative sample estimates. See the [ICC Guide](docs/ICC_GUIDE.md).

### Repeated-measures analysis (3+ conditions)

For 3 or more repeated conditions or measurement occasions on the same units, declare `design="repeated"`, `unit_id`, and `condition_order`:

```python
repeated = ResearchAssistant(df).run(
    objective="compare_groups",
    outcome="score",
    predictor="visit",
    estimand="mean",  # or "distribution" for Friedman
    design="repeated",
    unit_id="participant_id",
    condition_order=("baseline", "week4", "week8"),
    variable_types={"score": "continuous", "visit": "ordinal"},
)
print(repeated.explain())
```

For continuous mean targets, PyAutoStat executes one-way repeated-measures ANOVA, evaluates sphericity via Mauchly's test, computes Greenhouse-Geisser epsilon and corrected degrees of freedom / p-values when sphericity is violated, reports partial eta-squared, and computes complete pairwise paired t-tests with Holm multiplicity control. For repeated rank/distribution targets, it executes the Friedman test with Kendall's W concordance and complete pairwise Wilcoxon signed-rank tests with Holm adjustment. Complete-case panels across all declared conditions are audited and reported; two-condition paired designs continue to route through the paired t-test or paired Wilcoxon. See [`examples/repeated_measures_analysis.py`](examples/repeated_measures_analysis.py).

### Two-way factorial ANOVA (independent groups)

For two categorical factors on a continuous outcome under independent sampling, declare `factor_a` and `factor_b`:

```python
factorial = ResearchAssistant(df).run(
    objective="compare_groups",
    outcome="exam_score",
    factor_a="teaching_method",
    factor_b="class_size",
    estimand="mean",
    design="independent",
    sum_of_squares="type2",  # or "type3"
)
print(factorial.explain())
```

PyAutoStat fits the full factorial model ($Y = \mu + A + B + A \times B + \epsilon$), evaluates Type II or Type III sums of squares, reports partial eta-squared with exact noncentral-$F$ inversion confidence intervals, and computes planned follow-up contrasts (simple main effects, marginal comparisons, and interaction contrasts) with Holm multiplicity adjustment.

## When information is missing

Omit an essential design fact and the workflow returns a structured request without running a
test:

```python
pending = assistant.run(
    objective="compare_groups",
    outcome="exam_score",
    predictor="teaching_method",
    estimand="mean",
    variable_types={"exam_score": "continuous"},
)

print(pending.status)  # needs_input
print(pending.explain())
print(pending.missing_information[0])

revised = assistant.update_question(
    pending.draft,
    design="independent",
)
workflow = assistant.run(draft=revised)
```

PyAutoStat does not silently default independence, pairing, clustering, unit identifiers,
estimands, or causal meaning. `needs_input`, `data_limited`, `unsupported`, `partial`, and `failed`
remain distinct from a completed workflow.

## Supported capabilities

- Dataset profiling with structured quality and resource metadata.
- Welch independent-means analysis and explicit Student t-test.
- Explicit two-condition paired t-test using a declared unit-ID column.
- One-sample t inference against an explicit reference mean, including the raw signed mean
  difference, analytical interval, and one-sample Cohen's d when defined.
- Explicit paired Wilcoxon signed-rank inference for a declared paired rank/distribution target.
- Guided Welch one-way ANOVA with complete Games-Howell comparisons for independent multi-group
  mean questions.
- Mann-Whitney U and guided Kruskal-Wallis with complete Dunn-Holm comparisons.
- Explicit standard one-way ANOVA with complete Tukey-Kramer comparisons when equal-variance
  assumptions are independently justified.
- Pearson linear and Spearman monotonic numerical association inference.
- Explicit Kendall tau-b inference with tie-aware concordance and paired-row bootstrap uncertainty.
- Point-biserial correlation for explicitly oriented binary/continuous association.
- Partial Pearson correlation after linear adjustment for declared numerical controls.
- Pearson chi-square categorical association plus Fisher's exact fallback for sparse 2x2 tables.
- Simple and multiple OLS conditional-mean regression with continuous, Boolean, nominal, and
  categorical ordinal predictors; explicit classical or HC3 covariance and diagnostics.
- Binary logistic regression for a declared event-probability target, with odds-ratio inference,
  convergence and separation safeguards, likelihood fit, and design diagnostics.
- Exact two-sided McNemar inference for two-condition paired binary outcomes identified by unit ID.
- Researcher-declared multi-item scale reliability with Cronbach's alpha, deterministic bootstrap
  uncertainty, item diagnostics, missingness accounting, and optional explicit reverse scoring.
- One-way repeated-measures ANOVA for 3+ repeated conditions on the same units, including
  Mauchly's sphericity test, Greenhouse-Geisser correction, partial eta-squared, and complete
  Holm-adjusted pairwise paired t-tests.
- Friedman rank-sum test for 3+ repeated conditions, including Kendall's W concordance and
  complete Holm-adjusted pairwise Wilcoxon signed-rank tests.
- Two-way factorial ANOVA for two independent categorical factors, evaluating full models (main
  effects and interaction) with Type II and Type III sums of squares, partial eta-squared effect
  sizes with exact noncentral-F inversion confidence intervals, unweighted estimated marginal
  means, residual diagnostics, and planned follow-ups with Holm multiplicity adjustment.
- Effect estimates, supported confidence intervals, sample accounting, assumptions, and warnings.
- Deterministic interpretation and canonical research reports.
- Static HTML, Markdown, JSON, CSV tables, safe LaTeX, and optional interactive Plotly HTML.
- Decision ledger, report/export audit, reproducibility metadata, and explicit supplied-data replay.
- Declared sensitivity scenarios and researcher-defined practical-significance thresholds.
- Statistical analysis plans, plan-adherence comparison, and prospective independent or paired
  mean power and precision planning.
- Reporting-completeness checks and UI-independent session snapshots.

See the [capabilities and support matrix](docs/CAPABILITIES.md) for exact supported and unsupported
method families.

## Advanced workflows

### Explicit paired analysis

Pairing uses an identifier, never row order:

```python
paired = pd.DataFrame(
    {
        "participant_id": [1, 1, 2, 2, 3, 3, 4, 4],
        "condition": ["before", "after"] * 4,
        "exam_score": [62, 68, 70, 74, 65, 71, 73, 79],
    }
)

paired_workflow = ResearchAssistant(paired).run(
    objective="compare_groups",
    outcome="exam_score",
    predictor="condition",
    estimand="mean",
    design="paired",
    unit_id="participant_id",
    condition_order=("after", "before"),
    variable_types={"exam_score": "continuous"},
)
```

Duplicate unit/condition rows are blocked rather than averaged. Incomplete pairs are counted and
excluded transparently. See [advanced planning and presentation](docs/ADVANCED_PLANNING_AND_PRESENTATION.md).

For concise synthetic examples of one-sample t, paired Wilcoxon, inferential Spearman, and sparse
2x2 Fisher workflows, run `python examples/basic_inference.py`. Wilcoxon is a signed-rank analysis,
not universally a median test; Spearman targets monotonic rank association, not “Pearson for
non-normal data”; and Fisher support does not extend beyond 2x2 tables.

### Analysis and study planning

```python
from pyautostat import StudyPlanner

draft = assistant.prepare_question(
    objective="compare_groups",
    outcome="exam_score",
    predictor="teaching_method",
    estimand="mean",
    design="independent",
    variable_types={"exam_score": "continuous"},
)
plan = assistant.analysis_plan(draft, report_style="apa")

prospective = StudyPlanner().independent_mean_power(
    target_difference=5,
    sd_group1=10,
    sd_group2=12,
    target_power=0.80,
)
```

Planning inputs are researcher assumptions and are never derived automatically from an observed
result. Observed post-hoc power is not implemented.

### Sensitivity and practical significance

```python
from pyautostat import MeaningfulEffectThreshold, SensitivitySpecification

scenario = SensitivitySpecification(
    name="pooled variance",
    specification=workflow.analysis.specification,
    method_id="student_t",
    assumptions=("Equal population variances",),
)
sensitivity = assistant.sensitivity_analysis(
    workflow.analysis,
    scenarios=[scenario],
)

threshold = MeaningfulEffectThreshold(
    quantity="mean_difference",
    minimum_magnitude=5,
    direction="two_sided",
    unit="points",
)
practical = assistant.practical_significance(
    workflow.analysis,
    threshold=threshold,
)

print(sensitivity.compare())
print(practical.verdict)
```

Every sensitivity attempt remains visible, and different estimands or paired contrasts are not
presented as ordinary same-estimand robustness checks. Practical and statistical significance are
reported separately. `compare()` describes hypothesis-decision consistency only among completed
same-estimand comparisons and does not establish robustness. Formal equivalence and
noninferiority inference are not implemented. See
[sensitivity and practical significance](docs/ROBUSTNESS_AND_PRACTICAL_SIGNIFICANCE.md).

### Reports, audit, and replay

```python
report = assistant.report(
    workflow.analysis,
    sensitivity=sensitivity,
    practical_significance=practical,
)

html = report.to_html(style="apa")
markdown = report.to_markdown(style="apa")
latex = report.to_latex(style="apa")
audit = assistant.audit(report)

record = assistant.reproducibility_record(
    workflow.analysis,
    sensitivity=sensitivity,
    practical_significance=practical,
)
```

Construction and in-memory rendering write no files. Explicit `save_*` methods require a path and
protect existing files unless overwrite is requested. Replay requires the caller to supply data.
Auditing checks internal consistency, not scientific truth. See
[provenance and replay](docs/PROVENANCE_AND_REPLAY.md) and the
[research report schema](docs/RESEARCH_REPORT_SCHEMA.md).

## Resource awareness

`profile()["resource_info"]` reports row and column counts, the deep pandas memory estimate,
analytical numeric-column count, correlation-matrix dimensions, a local performance category, and
structured warnings. The default advisory policy is:

- `large` at 256 MiB;
- `very_large` at 1 GiB;
- a correlation-width advisory at 100 analytical numeric columns.

These are performance heuristics rather than scientific or universal hardware limits. Full
profiling still runs exactly as requested. If deep memory estimation fails, profiling continues
with an `unknown` resource level and an advisory warning.

## Statistical safeguards and limits

- Normality-test nonrejection is not proof of normality.
- Statistical significance is not effect magnitude, practical importance, or causation.
- Automatic selection requires the estimand and design; it does not choose a favorable p-value.
- Source values are not automatically recoded, imputed, removed, sampled, or truncated.
- Outlier flags are review cues and never automatic deletion rules.
- Two-way repeated-measures ANOVA, mixed ANOVA, clustered models, mixed-effects models,
  logistic and other generalized regression families, interactions,
  survival analysis, causal inference, sparse exact-table alternatives beyond Fisher's 2x2 test,
  broad post-hoc procedures, and formal equivalence/noninferiority tests are unsupported.
- Current templates are publication-oriented aids, not journal or regulatory certification.

Read [scientific limitations](docs/SCIENTIFIC_LIMITATIONS.md) before applying results in research
or decisions.

## Detailed APIs and legacy interfaces

The [API reference](API_REFERENCE.md) documents `ResearchAssistant`, configuration and result
records, planning, reports, audit, replay, and the legacy `StatisticalAnalyzer`, `InsightEngine`,
and `ReportGenerator` interfaces.

Legacy calls remain available:

```python
from pyautostat import InsightEngine, ReportGenerator, StatisticalAnalyzer

analysis = StatisticalAnalyzer(df).analyze_all()
insights = InsightEngine(analysis).generate_insights()
report = ReportGenerator(analysis, insights)
```

## Examples and project documents

- [Examples guide](examples/README.md)
- [Documentation index](docs/README.md)
- [API reference](API_REFERENCE.md)
- [Product vision](PRODUCT_VISION.md)
- [Future roadmap](ROADMAP.md)
- [Architecture](docs/ARCHITECTURE.md)
- [Statistical validation](docs/STATISTICAL_VALIDATION.md)

## Development

```bash
git clone https://github.com/majikoushik/PyAutoStat.git
cd PyAutoStat
python -m pip install -e ".[dev]"
python -m ruff check src tests
python -m ruff format --check src tests
python -m mypy src/pyautostat
python -m pytest -q --cov=pyautostat --cov-report=term-missing --cov-fail-under=90
python -m build
python -m twine check dist/*
```

See [AGENTS.md](AGENTS.md) for contribution guidance.

## License

MIT. See [LICENSE](LICENSE).
