Metadata-Version: 2.4
Name: systemgmmkit
Version: 0.5.14
Summary: Panel-data econometrics workflows for OLS, FE, RE, IV/2SLS, Difference GMM, System GMM, diagnostics, post-estimation, visualization, and ML-style validation in Python.
Author: Oluwajuwon Mayomi Akanbi
License: MIT
Project-URL: Homepage, https://github.com/Akanom/systemgmmkit
Project-URL: Repository, https://github.com/Akanom/systemgmmkit
Project-URL: Issues, https://github.com/Akanom/systemgmmkit/issues
Keywords: econometrics,panel-data,fixed-effects,random-effects,panel-iv,2sls,difference-gmm,system-gmm,dynamic-panel,arellano-bond,blundell-bond,windmeijer,stata,xtabond2,post-estimation,forecasting,panel-cross-validation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3,>=1.23
Requires-Dist: pandas<4,>=1.5
Requires-Dist: scipy<2,>=1.9
Provides-Extra: pydynpd
Requires-Dist: pydynpd<1,>=0.2.2; extra == "pydynpd"
Provides-Extra: fe
Requires-Dist: linearmodels<8,>=5; extra == "fe"
Provides-Extra: plots
Requires-Dist: matplotlib<4,>=3.8; extra == "plots"
Provides-Extra: reports
Requires-Dist: tabulate<1,>=0.9; extra == "reports"
Provides-Extra: outputhub
Requires-Dist: universal-output-hub<1,>=0.2.2; python_version >= "3.10" and extra == "outputhub"
Provides-Extra: dev
Requires-Dist: matplotlib<4,>=3.8; extra == "dev"
Requires-Dist: statsmodels<1,>=0.14; extra == "dev"
Requires-Dist: tomli<3,>=2; python_version < "3.11" and extra == "dev"
Requires-Dist: pytest<10,>=8; extra == "dev"
Requires-Dist: ruff<1,>=0.6; extra == "dev"
Requires-Dist: mypy<3,>=1.10; extra == "dev"
Requires-Dist: build<2,>=1.2; extra == "dev"
Requires-Dist: twine<7,>=5; extra == "dev"
Requires-Dist: universal-output-hub<1,>=0.2.2; python_version >= "3.10" and extra == "dev"
Provides-Extra: typing
Requires-Dist: mypy<3,>=1.20; extra == "typing"
Requires-Dist: pandas-stubs<3,>=2.0; extra == "typing"
Requires-Dist: scipy-stubs<2,>=1.9; extra == "typing"
Requires-Dist: linearmodels<8,>=5; extra == "typing"
Provides-Extra: all
Requires-Dist: pydynpd<1,>=0.2.2; extra == "all"
Requires-Dist: linearmodels<8,>=5; extra == "all"
Requires-Dist: matplotlib<4,>=3.8; extra == "all"
Requires-Dist: tabulate<1,>=0.9; extra == "all"
Requires-Dist: universal-output-hub<1,>=0.2.2; python_version >= "3.10" and extra == "all"
Dynamic: license-file

# systemgmmkit

[![PyPI version](https://img.shields.io/pypi/v/systemgmmkit.svg)](https://pypi.org/project/systemgmmkit/)
[![Python versions](https://img.shields.io/pypi/pyversions/systemgmmkit.svg)](https://pypi.org/project/systemgmmkit/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![CI](https://github.com/Akanom/systemgmmkit/actions/workflows/ci.yml/badge.svg)](https://github.com/Akanom/systemgmmkit/actions/workflows/ci.yml)
[![Publish](https://github.com/Akanom/systemgmmkit/actions/workflows/publish.yml/badge.svg)](https://github.com/Akanom/systemgmmkit/actions/workflows/publish.yml)

`systemgmmkit` is a Python package for applied panel-data econometrics and dynamic-panel GMM workflows.

It provides a unified workflow for:

* baseline linear models;
* pooled OLS, fixed-effects, and random-effects models;
* panel IV / 2SLS;
* Difference GMM and System GMM;
* easy dynamic-GMM wrapper APIs;
* explicit GMM instrument design;
* global, role-specific, and variable-specific GMM lag windows;
* dynamic-panel diagnostics;
* post-estimation analysis;
* diagnostic visualization;
* ML-style workflow utilities around fitted econometric models;
* reproducible reporting and export.

The package is designed for empirical researchers working in economics, finance, management, operations, productivity analysis, public policy, development economics, political economy, industrial organization, and other panel-data settings.

The objective is not only to estimate models. The objective is to make modelling choices clear enough for replication, review, publication, and applied decision-making.

## Public runnable example

Run the public [systemgmmkit Kaggle quickstart](https://www.kaggle.com/code/akanom/systemgmmkit)
to explore deterministic panel-data setup, pooled OLS, fixed and random effects,
panel-aware forecast validation, OutputHub reporting, and a compact Difference GMM
diagnostic workflow. The notebook is a reproducible cloud demonstration, not a
cross-software parity certificate or paper artifact.

## Universal Output Hub integration

Install the optional reporting bridge and add any fitted SystemGMMKit result to
an OutputHub report. SystemGMMKit's core supports Python 3.9, while Universal
Output Hub and this optional integration require Python 3.10 or newer:

```bash
python -m pip install "systemgmmkit[outputhub]"
```

```python
from universal_output_hub import OutputHub
from systemgmmkit import add_to_outputhub

hub = OutputHub("Panel model report")
add_to_outputhub(hub, pooled_result, name="Pooled OLS")
add_to_outputhub(
    hub,
    system_gmm_result,
    name="System GMM",
    include_diagnostics=True,
)
```

The adapter maps coefficients, standard errors, p-values, sample statistics,
backend and covariance metadata, and available GMM diagnostics. It does not
change estimates or convert predictive evidence into identification evidence.

For compact notebook display of linear combinations and Wald tests, use
`format_inference_frame(frame)`. It preserves the raw result frame, removes
redundant test-statistic fields, and displays very small p-values as bounds such
as `<0.0001` instead of rounding them to zero.

## Import surface

The package root exposes the dependency-free estimator, workflow, diagnostics,
reporting, and post-estimation API. This includes the easy-GMM interfaces:

```python
from systemgmmkit import DynamicGMMWorkflowResult, difference_gmm, system_gmm
```

Plot themes, plotting functions, SGM-Viz dashboards, and result-plot helpers
belong to the optional `systemgmmkit.postestimation` namespace:

```python
from systemgmmkit.postestimation import coefficient_plot, health_dashboard
```

Existing explicit imports of plotting names from `systemgmmkit` remain
supported through lazy compatibility aliases. Wildcard imports intentionally
include only the dependency-free root API and do not import optional plotting
names.

## Controlled performance mode

Native dynamic-panel GMM and native panel IV retain their validated preparation
paths by default. Native fixed effects use the compact within transformation on
both preparation paths. For large or repeatedly evaluated specifications,
opt-in preparation engines remove measured preparation overhead without changing
estimator algebra or output ordering:

```python
from systemgmmkit import run_native_dynamic_panel_gmm

result = run_native_dynamic_panel_gmm(
    spec,
    data,
    entity="firm_id",
    time="year",
    preparation_engine="accelerated",
)
```

The same selector is available on `run_fixed_effects()` and `run_panel_2sls()`.
For native fixed effects, it accelerates full-rank collinearity screening on the
compact transformed design; it does not restore an explicit dummy matrix:

```python
from systemgmmkit import run_fixed_effects

result = run_fixed_effects(
    spec,
    data,
    entity="firm_id",
    time="year",
    preparation_engine="accelerated",
)
```

Use `preparation_engine="reference"` for the permanent audit and rollback path.
See [controlled performance benchmarks](docs/PERFORMANCE.md) for the benchmark
contract, exact-parity requirements, measured scope, and reproduction commands.

---

# Why systemgmmkit?

Applied panel-data work often requires several disconnected tools:

* OLS and pooled OLS;
* fixed-effects and random-effects estimators;
* panel IV / 2SLS;
* Difference GMM and System GMM;
* overidentification diagnostics;
* serial-correlation diagnostics;
* post-estimation analysis;
* coefficient tables;
* diagnostic plots;
* model comparison;
* forecasting and backtesting;
* reproducible output pipelines.

`systemgmmkit` aims to bring these pieces into a consistent Python workflow.

The package is built around five principles.

## 1. Explicit model specification

Model assumptions should be visible in the code, not hidden inside estimation defaults.

Users should be able to see:

* the dependent variable;
* structural regressors;
* lagged dependent variables;
* endogenous variables;
* predetermined variables;
* exogenous variables;
* GMM-style instruments;
* IV-style instruments;
* lag-window design;
* collapsed instrument settings;
* backend metadata;
* diagnostic outputs.

## 2. Reproducible empirical workflows

Estimation, diagnostics, post-estimation, visualization, validation, forecasting, and reporting should be easy to rerun and inspect.

## 3. Transparent dynamic-panel diagnostics

Dynamic-panel GMM results should expose:

* sample size;
* group count;
* instrument count;
* instrument/group ratio;
* Hansen diagnostics;
* Sargan diagnostics;
* Arellano-Bond AR(1) diagnostics;
* Arellano-Bond AR(2) diagnostics;
* covariance type;
* Windmeijer correction status;
* backend metadata;
* instrument architecture.

## 4. Verification against reference implementations

Where practical, estimators are benchmarked against established Stata implementations, including `regress`, `xtreg`, `ivregress`, `xtabond2`, and `xtdpdgmm`.

## 5. Publication-oriented communication

Results should be interpretable not only as coefficient tables, but also through diagnostics, model-health dashboards, persistence plots, instrument-architecture displays, prediction outputs, validation tables, forecast outputs, and report-ready graphics.

---

# Current Development Focus

The current development line strengthens the dynamic-GMM user workflow.

The main improvements are:

* easy Difference GMM and System GMM wrappers;
* compact default easy-GMM specifications;
* clean structural lag handling through `L1_<dependent>` columns;
* no duplicated `L1.y` / `L1_y` lagged-dependent notation in easy workflows;
* explicit global GMM lag windows through `gmm_lags`;
* explicit role-specific GMM lag windows through `gmm_lags_by_role`;
* explicit variable-specific GMM lag windows through `gmm_lags_by_variable`;
* deterministic lag-window precedence;
* tests confirming exogenous variables remain IV-style by default;
* Difference GMM and System GMM validation using separate `gmm()` blocks;
* deterministic instrument-name and instrument-count validation;
* FD001 real-data validation for easy dynamic-GMM lag-window workflows;
* sanitized diagnostic p-values so impossible backend p-values are not reported as valid diagnostics.

The easy API remains a convenience layer. It does not introduce a new estimator. It builds on the validated lower-level specification and runner APIs.

Public roadmap, parity, and adoption discussions are part of the development
record. See [Open development](docs/OPEN_DEVELOPMENT.md) and the GitHub
Discussion templates for specification questions, validation reports, and use
cases.

The dedicated Python 3.12 all-extras job enforces combined, statement, branch,
and dynamic-panel routing coverage floors. See the
[coverage policy](docs/validation/COVERAGE_POLICY.md) for measured baselines,
local reproduction commands, and ratchet rules.

Kaggle and Google Colab notebooks may be used as reproducibility aids only when
they stay within the `systemgmmkit` package boundary. See
[Kaggle and Google Colab usage](docs/CLOUD_NOTEBOOKS.md) for credential, data,
and evidence rules.

---

# Quick User Path

Most users can start from the top-level package namespace:

```python
import systemgmmkit as sgk

workflow = sgk.system_gmm(
    data=df,
    entity="firm",
    time="year",
    dependent="growth",
    regressors=["investment", "leverage"],
    endogenous=["investment"],
    exogenous=["leverage"],
    return_workflow=True,
)

post = sgk.quick_postestimation(
    workflow.result,
    workflow.data,
    y="growth",
    lincoms={"total_effect": "investment + leverage"},
    wald_tests={"joint_zero": "investment = 0, leverage = 0"},
)

post.metrics
post.linear_combinations
post.wald_tests
post.to_markdown()
```

For automatic dynamic-GMM specification search:

```python
search = sgk.auto_dynamic_gmm(
    df,
    y="growth",
    entity="firm",
    time="year",
    regressors=["investment", "leverage"],
    endogenous=["investment"],
    exogenous=["leverage"],
)

search.best_spec
search.best_result
search.to_markdown()
```

These are wrapper APIs. They compose the accepted estimators, post-estimation
helpers, forecasting tools, and diagnostics without changing estimator internals.

---

# Installation

Stable release:

```bash
pip install systemgmmkit
```

Development version:

```bash
pip install git+https://github.com/Akanom/systemgmmkit.git
```

Local development installation:

```bash
pip install -e ".[dev,all]"
```

Core estimator type-checking uses a separate, typed dependency set and an
explicit progressive scope:

```bash
pip install -e ".[typing]"
python -m mypy
```

See [Progressive Type Checking](docs/TYPE_CHECKING.md) for the enforced files,
current debt baseline, and expansion plan.

For graphics support, ensure `matplotlib` is available:

```bash
pip install matplotlib
```

Check the installed version:

```python
import systemgmmkit

print(systemgmmkit.__version__)
```

---

# Current Feature Coverage

## Linear Models

* Ordinary Least Squares
* Robust OLS
* Pooled OLS
* Clustered OLS

## Panel Models

* One-way Fixed Effects
* Two-way Fixed Effects
* Random Effects
* Panel IV / 2SLS

## Dynamic Panel Models

* Difference GMM
* System GMM
* Easy Difference GMM wrapper through `difference_gmm()`
* Easy System GMM wrapper through `system_gmm()`
* One-step estimation
* Two-step estimation
* Windmeijer-corrected standard errors
* Collapsed instruments
* Restricted global GMM lag windows
* Role-specific GMM lag windows
* Variable-specific GMM lag windows
* Deterministic GMM lag-window precedence
* Hansen diagnostics
* Sargan diagnostics
* Arellano-Bond AR(1) diagnostics
* Arellano-Bond AR(2) diagnostics
* Instrument-name validation
* Instrument-count validation

## Post-Estimation

* Predictions
* Fitted values
* Residuals
* Variance-covariance extraction
* Confidence intervals
* Linear combinations
* Wald tests
* Marginal effects for linear estimators

## Graphics and Diagnostics

* Coefficient plots
* Marginal effects plots
* Prediction / margins plots
* Interaction plots
* Conditional effects plots
* Residual diagnostics
* Fixed-effects plots
* Panel trajectory plots
* Instrument count plots
* Hansen / AR diagnostic plots
* Counterfactual scenario plots
* 3D effect surfaces
* SGM-Viz model-health dashboards
* SGM-Viz persistence analytics
* SGM-Viz instrument architecture dashboards
* SGM-Viz publication panels
* HTML gallery export
* PNG / SVG / PDF-compatible figure export

## ML-Style Workflow

* Prediction from fitted result objects
* Fitted-value and residual extraction
* Regression metrics
* Panel-aware train/test splitting
* Expanding-window panel cross-validation
* Model comparison
* Recursive forecasting
* Forecast backtesting
* GMM specification-search scaffolding

## Reporting

* Markdown export
* CSV export
* LaTeX export
* HTML figure galleries
* Structured result objects
* Integration with `universal-output-hub`

---

# Quick Start

## Ordinary Least Squares

```python
from systemgmmkit import OLSSpec, run_ols

spec = OLSSpec(
    dependent="y",
    regressors=["x1", "x2"],
    controls=["z1", "z2"],
    covariance="robust",
)

result = run_ols(spec, df)

print(result.summary_frame())
```

Equivalent Stata idea:

```stata
regress y x1 x2 z1 z2, vce(robust)
```

---

## Pooled OLS

```python
from systemgmmkit import PooledOLSSpec, run_pooled_ols

spec = PooledOLSSpec(
    dependent="y",
    regressors=["x1", "x2"],
    controls=["z1"],
    covariance="clustered",
)

result = run_pooled_ols(
    spec,
    df,
    entity="firm_id",
    time="year",
)

print(result.summary_frame())
```

Equivalent Stata idea:

```stata
regress y x1 x2 z1, vce(cluster firm_id)
```

---

## Fixed Effects

```python
from systemgmmkit import build_fixed_effects_spec, run_fixed_effects

spec = build_fixed_effects_spec(
    dependent="y",
    regressors=["x1", "x2"],
    controls=["z1"],
)

result = run_fixed_effects(
    spec,
    df,
    entity="firm_id",
    time="year",
    preparation_engine="accelerated",
)

print(result.summary_frame())
```

Equivalent Stata idea:

```stata
xtset firm_id year
xtreg y x1 x2 z1, fe
```

---

## Random Effects

```python
from systemgmmkit import RandomEffectsSpec, run_random_effects

spec = RandomEffectsSpec(
    dependent="y",
    regressors=["x1", "x2", "z1"],
)

result = run_random_effects(
    spec,
    df,
    entity="firm_id",
    time="year",
)

print(result.summary_frame())
```

Equivalent Stata idea:

```stata
xtset firm_id year
xtreg y x1 x2 z1, re
```

---

## Panel IV / 2SLS

```python
from systemgmmkit import PanelIVSpec, run_panel_2sls

spec = PanelIVSpec(
    dependent="y",
    exog=["x1", "z1"],
    endogenous=["x2"],
    instruments=["z2"],
)

result = run_panel_2sls(
    spec,
    df,
    entity="firm_id",
    time="year",
    preparation_engine="accelerated",
)

print(result.summary_frame())
```

Equivalent Stata idea:

```stata
ivregress 2sls y x1 z1 (x2 = z2)
```

---

# Dynamic Panel GMM

`systemgmmkit` supports both Difference GMM and System GMM.

The recommended workflow is:

1. Define the structural model.
2. Create or request structural lagged variables.
3. Classify regressors as endogenous, predetermined, or exogenous.
4. Define the GMM instrument strategy.
5. Control the lag windows and collapsed-instrument design.
6. Run the estimator.
7. Inspect diagnostics before interpreting coefficients.
8. Export tables and diagnostic figures.

Users can run dynamic-panel GMM through either:

* the easy wrapper API: `difference_gmm()` and `system_gmm()`;
* the lower-level specification API: `build_difference_gmm_spec()`, `run_difference_gmm()`, `build_system_gmm_spec()`, and `run_system_gmm()`.

---

# Easy Dynamic GMM API

The easy API is designed for readable one-call workflows while preserving explicit modelling choices.

```python
from systemgmmkit import difference_gmm, system_gmm
```

The easy API handles:

* sorting panel data by entity and time;
* creating lagged dependent-variable columns such as `L1_y`;
* adding the lagged dependent variable to the structural equation;
* classifying the lagged dependent variable as endogenous, predetermined, exogenous, or excluded from GMM-style treatment;
* applying global, role-specific, and variable-specific GMM lag windows;
* keeping exogenous variables IV-style by default;
* returning either the fitted result or a full workflow object.

The easy API does not create arbitrary structural lags for other regressors. Users should create those manually.

---

## Easy System GMM

```python
from systemgmmkit import system_gmm

result = system_gmm(
    data=df,
    entity="firm_id",
    time="year",
    dependent="y",
    lagged_dependent=1,
    regressors=[
        "investment",
        "cashflow",
        "firm_size",
    ],
    endogenous=[
        "investment",
    ],
    predetermined=[
        "cashflow",
    ],
    exogenous=[
        "firm_size",
    ],
    gmm_lags=(2, 2),
    collapse=True,
    windmeijer=True,
)
```

By default, `lagged_dependent=1` creates a structural lag column named `L1_y`, adds it to the model equation, and classifies it as endogenous.

The easy API intentionally avoids duplicated lag notation. It should generate a command architecture like:

```text
y investment cashflow firm_size L1_y | gmm(L1_y, 2:2) ...
```

not:

```text
y L1.y investment cashflow firm_size L1_y | gmm(y, 2:2) gmm(L1_y, 2:2) ...
```

---

## Easy Difference GMM

```python
from systemgmmkit import difference_gmm

result = difference_gmm(
    data=df,
    entity="firm_id",
    time="year",
    dependent="y",
    lagged_dependent=1,
    regressors=[
        "investment",
        "cashflow",
        "firm_size",
    ],
    endogenous=[
        "investment",
    ],
    predetermined=[
        "cashflow",
    ],
    exogenous=[
        "firm_size",
    ],
    gmm_lags=(2, 2),
    collapse=True,
)
```

The easy Difference GMM wrapper follows the same structural-lag and variable-classification logic as the easy System GMM wrapper.

---

## Lagged dependent-variable role

The lagged dependent variable can be classified through `lagged_dependent_role`.

```python
result = system_gmm(
    data=df,
    entity="firm_id",
    time="year",
    dependent="y",
    lagged_dependent=1,
    lagged_dependent_role="predetermined",
    regressors=["investment", "firm_size"],
    endogenous=["investment"],
    exogenous=["firm_size"],
    gmm_lags=(2, 2),
)
```

Allowed values are:

* `"endogenous"`;
* `"predetermined"`;
* `"exogenous"`;
* `"none"`.

Unclassified regressors are treated as exogenous by default for usability, but researchers should classify variables explicitly in serious empirical work.

---

## Inspecting the generated workflow

For inspection, set `return_workflow=True`.

```python
from systemgmmkit import system_gmm

workflow = system_gmm(
    data=df,
    entity="firm_id",
    time="year",
    dependent="y",
    regressors=["investment", "firm_size"],
    endogenous=["investment"],
    exogenous=["firm_size"],
    gmm_lags=(2, 2),
    return_workflow=True,
)

result = workflow.result
spec = workflow.spec
model_data = workflow.data
```

The workflow object exposes:

* fitted result;
* generated specification;
* model dataframe after lag creation and missing-value handling;
* final regressors;
* endogenous variables;
* predetermined variables;
* exogenous variables;
* global GMM lag window;
* role-specific GMM lag windows;
* variable-specific GMM lag windows;
* collapse setting;
* time-effect setting;
* model type.

---

# Advanced Dynamic GMM API

Advanced users can use the lower-level API directly.

## Difference GMM

```python
from systemgmmkit import build_difference_gmm_spec, run_difference_gmm

spec = build_difference_gmm_spec(
    dependent="y",
    regressors=[
        "L1_y",
        "investment",
        "firm_size",
    ],
    endogenous=[
        "L1_y",
        "investment",
    ],
    exogenous=[
        "firm_size",
    ],
    gmm_lags=(2, 4),
    collapse=True,
)

result = run_difference_gmm(
    spec,
    data=df,
    entity="firm_id",
    time="year",
    backend="auto",
)

print(result)
```

Equivalent Stata idea:

```stata
xtabond2 y L.y investment firm_size, ///
    gmm(L.y investment, lag(2 4) collapse) ///
    iv(firm_size) ///
    robust
```

---

## System GMM

```python
from systemgmmkit import build_system_gmm_spec, run_system_gmm

spec = build_system_gmm_spec(
    dependent="y",
    regressors=[
        "L1_y",
        "investment",
        "firm_size",
    ],
    endogenous=[
        "L1_y",
        "investment",
    ],
    exogenous=[
        "firm_size",
    ],
    gmm_lags=(2, 4),
    collapse=True,
    windmeijer=True,
)

result = run_system_gmm(
    spec,
    data=df,
    entity="firm_id",
    time="year",
    backend="auto",
)

print(result)
```

Equivalent Stata idea:

```stata
xtabond2 y L.y investment firm_size, ///
    gmm(L.y investment, lag(2 4) collapse) ///
    iv(firm_size) ///
    twostep robust small
```

The lower-level API gives full control over the specification object and remains the reference API for validation and parity workflows.

---

# Variable Classification Guide

Correct variable classification is one of the most important modelling decisions in dynamic-panel estimation.

| Classification | Interpretation                                       | Typical instrument treatment                                                              | Examples                                                                     |
| -------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Endogenous     | May be correlated with current and past disturbances | GMM-style instruments using deeper lags                                                   | investment, aid, leverage, R&D, production decisions                         |
| Predetermined  | May react to past shocks but not current shocks      | GMM-style instruments, often allowing shorter lag windows than fully endogenous variables | cash flow, backlog, lagged policy variables, delayed implementation measures |
| Exogenous      | Assumed independent of the disturbance process       | IV-style instruments by default                                                           | firm size, year dummies, industry dummies, externally determined controls    |

Researchers should perform robustness checks using alternative classifications when the theoretical justification is uncertain.

---

# Structural Lags vs Instrument Lags

Dynamic GMM has two different uses of lags.

| Use                          | Meaning                                               | Example                         |
| ---------------------------- | ----------------------------------------------------- | ------------------------------- |
| Lagged variable in the model | The lag enters the structural equation as a regressor | `L1_investment` in `regressors` |
| Lagged value as instrument   | Past values are used internally as GMM instruments    | `gmm_lags=(2, 4)`               |

This distinction is central.

```python
regressors = ["L1_investment"]
```

means:

```text
Include lagged investment as an explanatory variable in the model equation.
```

while:

```python
gmm_lags = (2, 4)
```

means:

```text
Use lags 2 through 4 as GMM instruments.
```

Safe rule:

```text
Create lagged regressors yourself when they belong in the model equation.
Classify each lagged regressor according to its maintained exogeneity assumption.
Use GMM lag-window arguments only to control instrument construction.
```

---

# Creating Structural Lags

The lower-level API treats lagged regressors as ordinary columns supplied by the user. It does not automatically create structural `L1_` or `L2_` model variables.

```python
df = df.sort_values(["firm_id", "year"]).copy()

df["L1_y"] = df.groupby("firm_id")["y"].shift(1)
df["L1_investment"] = df.groupby("firm_id")["investment"].shift(1)
df["L2_investment"] = df.groupby("firm_id")["investment"].shift(2)

df = df.dropna(
    subset=[
        "L1_y",
        "L1_investment",
        "L2_investment",
    ]
)
```

The easy API can create lagged dependent-variable columns automatically through:

```python
lagged_dependent=1
```

or:

```python
lagged_dependent=2
```

This convenience applies to the dependent variable only. Other structural lags should still be created explicitly by the user.

---

# GMM Lag-Window Strategy

`systemgmmkit` supports three layers of GMM lag-window control.

## 1. Global GMM lag window

```python
gmm_lags = (2, 4)
```

This means GMM-style variables use lags 2 through 4 unless overridden by a more specific rule.

## 2. Role-specific GMM lag windows

```python
gmm_lags_by_role = {
    "endogenous": (2, 3),
    "predetermined": (1, 2),
}
```

This allows endogenous and predetermined variables to use different lag windows.

## 3. Variable-specific GMM lag windows

```python
gmm_lags_by_variable = {
    "L1_y": (2, 2),
    "cashflow": (1, 2),
}
```

This gives specific variables their own instrument lag window.

## Precedence rule

The precedence rule is deterministic:

```text
gmm_lags_by_variable > gmm_lags_by_role > gmm_lags
```

Variable-specific settings override role-specific settings. Role-specific settings override the global setting.

Example:

```python
from systemgmmkit import system_gmm

result = system_gmm(
    data=df,
    entity="firm_id",
    time="year",
    dependent="y",
    lagged_dependent=1,
    regressors=[
        "investment",
        "cashflow",
        "firm_size",
    ],
    endogenous=[
        "investment",
    ],
    predetermined=[
        "cashflow",
    ],
    exogenous=[
        "firm_size",
    ],
    gmm_lags=(2, 2),
    gmm_lags_by_role={
        "endogenous": (2, 3),
        "predetermined": (1, 2),
    },
    gmm_lags_by_variable={
        "L1_y": (2, 2),
        "cashflow": (1, 3),
    },
    collapse=True,
)
```

Under this design:

* `L1_y` uses `(2, 2)` because variable-specific settings win;
* `cashflow` uses `(1, 3)` because variable-specific settings win;
* other endogenous variables use `(2, 3)`;
* other predetermined variables use `(1, 2)`;
* any remaining GMM-style variables use the global `gmm_lags`.

---

# Exogenous Variables Remain IV-Style by Default

Exogenous variables are treated as IV-style instruments by default.

```python
exogenous = [
    "firm_size",
    "L1_firm_size",
]
```

The package does not force exogenous variables into GMM-style instrumentation.

If users need lagged exogenous variables in the model equation, they should create those structural lags manually:

```python
df["L1_firm_size"] = df.groupby("firm_id")["firm_size"].shift(1)
```

Then include them as regressors and classify them as exogenous if strict exogeneity is defensible:

```python
regressors = [
    "firm_size",
    "L1_firm_size",
]

exogenous = [
    "firm_size",
    "L1_firm_size",
]
```

If strict exogeneity is too strong, the lagged variable should be classified as predetermined instead.

---

# Instrument Control

Instrument count should be controlled to reduce overfitting and avoid weakening the Hansen test.

Recommended practice:

* keep the instrument count below the number of groups where possible;
* use collapsed instruments when appropriate;
* restrict global GMM lag windows;
* use role-specific lag windows where theoretically justified;
* use variable-specific lag windows where identification requires finer control;
* report the number of instruments;
* report the instrument/group ratio;
* report AR(1), AR(2), Hansen, and Sargan diagnostics;
* compare alternative lag-window choices as robustness checks.

---

# Dynamic GMM Diagnostics

For dynamic-panel GMM models, users should inspect diagnostics before interpreting coefficients.

Recommended diagnostics include:

* number of observations;
* number of groups;
* number of instruments;
* instrument/group ratio;
* AR(1) test;
* AR(2) test;
* Hansen test;
* Sargan test;
* covariance estimator;
* one-step or two-step setting;
* Windmeijer correction status;
* transformation;
* estimation backend;
* instrument architecture.

A statistically significant AR(1) test is expected in many differenced dynamic-panel models. The AR(2) test is usually more important for checking second-order serial correlation in differenced residuals.

The Hansen and Sargan tests should not be interpreted mechanically. Very high Hansen p-values can indicate instrument proliferation, while very low p-values may indicate invalid instruments or misspecification.

---

# Post-Estimation Utilities

```python
from systemgmmkit import (
    predict,
    fitted_values,
    residuals,
    vcov,
    estat_vce,
    confint,
    lincom,
    wald_test,
    marginal_effects,
    margins,
    predict_stata,
)
```

## Predictions

```python
pred = predict(result)
```

or:

```python
pred = result.predict()
```

## Fitted Values

```python
fit = fitted_values(result)
```

## Residuals

```python
resid = residuals(result)
```

## Variance-Covariance Matrix

```python
V = vcov(result)
V = estat_vce(result)
```

## Confidence Intervals

```python
ci = confint(result)
ci90 = confint(result, level=90)
```

## Linear Combinations

Stata-style syntax is supported:

```stata
lincom x1 + x2
lincom x1 + x2 = 0
```

Python:

```python
effect = lincom(result, "x1 + x2")
effect_against_null = lincom(result, "x1 + x2 = 0")
effect90 = lincom(result, "x1 + x2", level=90)

print(effect)
```

If the result exposes inference degrees of freedom, `lincom` reports t-style
p-values and confidence intervals. Otherwise it falls back to z-style inference.

## Wald Tests

Stata-style syntax is supported:

```stata
test x1 x2
test (x1 = 0) (x2 = 0)
test x1 + x2 = 0
```

Python:

```python
test_result = wald_test(result, "x1 = 0, x2 = 0")
same_test = wald_test(result, "test (x1 = 0) (x2 = 0)")
combo_test = wald_test(result, "x1 + x2 = 0")

print(test_result)
```

If the result exposes inference degrees of freedom, `wald_test` reports
Stata-like F tests. Otherwise it falls back to chi-squared Wald tests.

## Marginal Effects

```python
me = marginal_effects(result)
mfx = margins(result, dydx=["x1", "x2"], level=90)

print(me)
```

For linear estimators, marginal effects correspond to estimated slopes.

## Stata-Style Prediction

```stata
predict xbhat, xb
predict ehat, residuals
```

Python:

```python
xbhat = predict_stata(result, option="xb")
ehat = predict_stata(result, option="residuals")
```

---

# Standard Post-Estimation Graphics

## Coefficient plot

```python
from systemgmmkit.postestimation import coefficient_plot

coefficient_plot(
    result,
    style="sgm",
    preset="paper",
    save="outputs/coefficient_plot.png",
)
```

## Marginal effects plot

```python
from systemgmmkit.postestimation import marginal_effects_plot

marginal_effects_plot(
    effects_df,
    style="sgm",
    preset="paper",
    save="outputs/marginal_effects.png",
)
```

Expected input:

```python
effects_df = pd.DataFrame({
    "term": ["techshare", "polity", "fragility"],
    "effect": [0.18, 0.04, -0.09],
    "std_error": [0.04, 0.02, 0.03],
})
```

## Residual diagnostics

```python
from systemgmmkit.postestimation import (
    residuals_vs_fitted_plot,
    qq_residual_plot,
    residual_histogram,
)

residuals_vs_fitted_plot(result, save="outputs/residuals_vs_fitted.png")
qq_residual_plot(result.residuals, save="outputs/qq_residuals.png")
residual_histogram(result.residuals, save="outputs/residual_histogram.png")
```

---

# SGM-Viz Diagnostic Dashboards

SGM-Viz is the package's higher-level diagnostic visualization system.

It combines:

* econometric diagnostic discipline;
* publication-quality layout;
* dashboard-style readability;
* dynamic-panel-specific interpretation.

## Model health dashboard

```python
result.plot.health(
    save="outputs/model_health.png",
)
```

This figure summarizes:

* Hansen diagnostic;
* Sargan diagnostic;
* AR(1);
* AR(2);
* observations;
* groups;
* instruments;
* instrument/group ratio;
* collapse status.

## Dynamic persistence dashboard

```python
result.plot.persistence(
    phi=result.params["L1.y"],
    save="outputs/persistence.png",
)
```

This figure reports:

* persistence coefficient;
* shock decay path;
* half-life;
* long-run multiplier;
* persistence class.

## Instrument architecture dashboard

```python
from systemgmmkit.postestimation import InstrumentArchitecture

architecture = InstrumentArchitecture(
    estimator="System GMM",
    difference_equation=("L2.y", "L3.y"),
    level_equation=("D.y",),
    standard_instruments=("x", "w", "time effects"),
    lag_range=(2, 3),
    collapsed=True,
    transformation="FOD",
    total_instruments=result.instruments,
    groups=result.groups,
)

result.plot.instruments(
    architecture=architecture,
    save="outputs/instruments.png",
)
```

## One-command report export

```python
result.plot.export_all(
    "outputs/sgm_report",
    prefix="model",
    architecture=architecture,
    gallery_mode="dashboard",
)
```

Report modes:

```text
dashboard    = individual dashboards only
publication  = composed publication panel only
full         = all figures
```

---

# ML-Style Workflow Layer

`systemgmmkit.ml` adds machine-learning-style workflow utilities around already fitted econometric result objects.

This layer is intentionally additive:

```text
validated econometric estimator
        ↓
result object
        ↓
ML-style workflow utilities
```

It does not replace the econometric estimators and does not rewrite the validated estimation core.

## Public API

```python
from systemgmmkit.ml import (
    ResultAdapter,
    adapt_result,
    predict,
    fitted_values,
    residuals,
    regression_metrics,
    panel_train_test_split,
    PanelTimeSeriesSplit,
    cross_validate_panel,
    compare_models,
    quick_postestimation,
    quick_forecast,
    quick_ml,
    forecast,
    backtest_forecast,
    GMMGridSearch,
    DynamicGMMHybridSearch,
    auto_dynamic_gmm,
    dynamic_gmm_candidate_grid,
    GMMSearchResult,
)
```

## Prediction and residuals

```python
from systemgmmkit.ml import predict, fitted_values, residuals

pred = predict(result, df)
fit = fitted_values(result, df)
err = residuals(result, df, y="growth_rate")
```

## Regression metrics

```python
from systemgmmkit.ml import regression_metrics

scores = regression_metrics(
    y_true=df["growth_rate"],
    y_pred=pred,
)
```

Metrics include:

* MAE;
* MSE;
* RMSE;
* MAPE;
* SMAPE;
* R²;
* evaluated observation count.

## Panel-aware train/test split

```python
from systemgmmkit.ml import panel_train_test_split

train, test = panel_train_test_split(
    df,
    time="year",
    test_size=0.2,
)
```

The split is time-respecting and does not randomly split panel rows.

## Panel-aware cross-validation

```python
from systemgmmkit.ml import PanelTimeSeriesSplit, cross_validate_panel

cv = PanelTimeSeriesSplit(
    n_splits=5,
    min_train_periods=10,
    test_periods=1,
)

scores = cross_validate_panel(
    estimator=my_estimator,
    data=df,
    y="growth_rate",
    time="year",
    cv=cv,
)
```

## Model comparison

```python
from systemgmmkit.ml import compare_models

comparison = compare_models(
    models={
        "OLS": ols_result,
        "Fixed Effects": fe_result,
        "System GMM": sysgmm_result,
    },
    data=test_df,
    y="growth_rate",
)
```

The comparison table reports prediction metrics such as MAE, MSE, RMSE, MAPE, SMAPE, and R². Where available, scalar diagnostics are also included with a `diag_` prefix.

## Simple wrapper UI

```python
from systemgmmkit import lincom, wald_test
from systemgmmkit.ml import quick_postestimation, quick_forecast, quick_ml

post = quick_postestimation(
    result,
    df,
    y="growth_rate",
)

post.metrics
post.confidence_intervals
post.marginal_effects

total_effect = lincom(result, "investment + trade_open")
joint_test = wald_test(result, "investment = 0, trade_open = 0")

fc = quick_forecast(
    result,
    history=df,
    y="growth_rate",
    entity="country",
    time="year",
    horizon=4,
    future_exog=future_controls,
)

workflow = quick_ml(
    result,
    df,
    y="growth_rate",
    entity="country",
    time="year",
    horizon=4,
    future_exog=future_controls,
)
```

These helpers only compose the accepted post-estimation, forecasting, and ML
utilities. They do not re-estimate models or modify estimator internals.

## Recursive forecasting

```python
from systemgmmkit.ml import forecast

fc = forecast(
    result=sysgmm_result,
    history=df,
    y="growth_rate",
    entity="country",
    time="year",
    horizon=4,
    future_exog=future_controls,
)
```

For dynamic-panel models, lagged dependent-variable terms are detected from coefficient names such as:

* `L1.y`;
* `L2.y`;
* `L.y`;
* `y_lag1`;
* `lag1_y`;
* `L1_y`.

The function recursively updates lagged dependent variables using previous forecasts.

## Forecast backtesting

```python
from systemgmmkit.ml import backtest_forecast

scores = backtest_forecast(
    result_factory=my_estimator,
    data=df,
    y="growth_rate",
    entity="country",
    time="year",
    horizon=4,
    min_train_periods=10,
)
```

This supports expanding-window forecast validation for panel-data workflows.

## GMM specification-search scaffold

```python
from systemgmmkit.ml import GMMGridSearch

search = GMMGridSearch(
    build_spec=build_system_gmm_spec,
    run_model=run_system_gmm,
    param_grid=[
        {"gmm_lags": (2, 2), "collapse": True},
        {"gmm_lags": (2, 3), "collapse": True},
        {"gmm_lags": (2, 4), "collapse": True},
    ],
    y="growth_rate",
    entity="country",
    time="year",
    diagnostic_rules={
        "hansen_p": (">", 0.05),
        "ar2_p": (">", 0.05),
    },
)

search_result = search.fit(df)
```

## Dynamic GMM Hybrid Loop

```python
from systemgmmkit.ml import auto_dynamic_gmm

hybrid_result = auto_dynamic_gmm(
    df,
    y="growth_rate",
    entity="country",
    time="year",
    regressors=["investment", "trade_open"],
    endogenous=["investment"],
    exogenous=["trade_open"],
    test_size=2,
)

best_result = hybrid_result.best_result
best_spec = hybrid_result.best_spec
report = hybrid_result.to_markdown()
```

For advanced control, instantiate the search object directly:

```python
from systemgmmkit.ml import DynamicGMMHybridSearch

search = DynamicGMMHybridSearch(
    y="growth_rate",
    entity="country",
    time="year",
    regressors=["investment", "trade_open"],
    endogenous=["investment"],
    exogenous=["trade_open"],
    models=["system", "difference"],
    steps=["twostep", "onestep"],
    lag_windows=[(2, 2), (2, 3), (3, 4)],
    transformations=["fod", "fd"],
    test_size=2,
)

hybrid_result = search.fit(df)

best_result = hybrid_result.best_result
best_spec = hybrid_result.best_spec
report = hybrid_result.to_markdown()
```

The hybrid loop generates candidate Difference/System GMM specifications,
estimates them through the existing easy API, rejects diagnostically unsafe
models, ranks the surviving candidates, and produces a compact Markdown report.
Econometric validity comes before prediction quality: AR(2), Hansen,
convergence, and instrument-proliferation failures are not recommended.

The search layer repeatedly calls existing validated estimators. It does not implement a new GMM estimator.

## Smoke demonstration

A reviewer-facing smoke script is available:

```bash
python scripts/ml/run_ml_workflow_smoke.py --outdir artifacts/ml_workflow
```

The script writes reproducible workflow artifacts including:

* synthetic static and dynamic panel data;
* predictions and residuals;
* panel cross-validation scores;
* model comparison output;
* GMM grid-search output;
* recursive forecasts;
* forecast backtest metrics;
* a machine-readable summary file.

---

# Verification Philosophy

Verification is a core design principle of `systemgmmkit`.

Where practical, estimators are benchmarked against established Stata implementations, including:

* `regress`;
* `xtreg`;
* `ivregress`;
* `xtabond2`;
* `xtdpdgmm`.

Benchmark scripts, comparison workflows, and validation artifacts are maintained in the repository.

The goal is not merely to produce estimates. The goal is to provide transparent evidence that estimates match trusted reference implementations under maintained benchmark specifications.

---

# Current Validation Status

| Component                            | Status                   |
| ------------------------------------ | ------------------------ |
| OLS                                  | PASS_STATA_PARITY        |
| Robust OLS                           | PASS_STATA_PARITY        |
| Clustered OLS                        | PASS_STATA_PARITY        |
| Confidence intervals                 | PASS_STATA_PARITY        |
| `lincom`                             | PASS_STATA_PARITY        |
| Wald / F tests                       | PASS_STATA_PARITY        |
| Fixed Effects                        | PASS_STATA_COMPARISON    |
| Random Effects                       | PASS_STATA_COMPARISON    |
| Panel IV / 2SLS                      | PASS_STATA_COMPARISON    |
| Difference GMM                       | PASS_XTABOND2_PARITY     |
| System GMM structural coefficients   | PASS_XTABOND2_PARITY     |
| Windmeijer standard errors           | PASS_XTABOND2_PARITY     |
| Hansen diagnostics                   | PASS_XTABOND2_PARITY     |
| Sargan diagnostics                   | PASS_XTABOND2_PARITY     |
| AR(1) diagnostics                    | PASS_XTABOND2_PARITY     |
| AR(2) diagnostics                    | PASS_XTABOND2_PARITY     |
| SGM-Viz dashboards                   | PASS_TESTED_EXPORT       |
| Standard graphics gallery            | PASS_TESTED_EXPORT       |
| Result plot accessors                | PASS_TESTED_EXPORT       |
| ML workflow smoke script             | PASS_TESTED_WORKFLOW     |
| Easy dynamic-GMM wrappers            | PASS_TESTED_API          |
| Role-specific GMM lag windows        | PASS_TESTED_API          |
| Variable-specific GMM lag windows    | PASS_TESTED_API          |
| GMM instrument-name validation       | PASS_TESTED_VALIDATION   |
| GMM instrument-count validation      | PASS_TESTED_VALIDATION   |
| FD001 easy-GMM lag-window validation | PASS_REALDATA_VALIDATION |

Validation claims apply to the maintained benchmark specifications and validation workflows in the repository. The controlled `xtabond2` benchmark is used for strict certification. CMAPSS FD001 is an application scaffold, not part of the formal parity certificate.

For Hansen, Sargan, and AR rows, `PASS_XTABOND2_PARITY` means numerical
agreement with `xtabond2`; it does not establish instrument validity or endorse
the benchmark specification. Stata rejects both Hansen and Sargan tests at 5%
for the no-controls (`0.02356` / `0.00568`), three-way-controls (`0.02144` /
`0.0000369`), decomposition (`0.00640` / `0.0000128`), and variable-missing
(`0.01151` / `0.0007838`) fixtures. The baseline (`0.15998` / `0.08792`) and
unbalanced-panel (`0.23240` / `0.29722`) fixtures do not reject. The unified
certificate preserves the raw p-values and reject-at-0.05 flags.

Users should still inspect their own model diagnostics, instrument counts, sample construction, lag-window choices, and identification assumptions.

---

# System GMM xtabond2 Certification

The native System GMM implementation has benchmark-specific parity against Stata
`xtabond2` on six maintained, collapsed two-step specifications: baseline
controls, no controls, three-way controls, decomposition controls, an unbalanced
panel, and a variable-specific missing-data design.
`system_gmm_three_way_no_controls` is outside this certified set and remains
pending because no dedicated Stata comparison artifact has been committed.

Certified components include:

* the complete expected parameter set and coefficient estimates;
* Windmeijer-corrected two-step standard errors;
* observations, groups, instruments, and overidentification degrees of freedom;
* exact native/Stata estimation-sample keys for the unbalanced-panel and
  variable-missing designs;
* Hansen and Sargan statistics and p-values; and
* signed Arellano-Bond AR(1)/AR(2) statistics and p-values.

The authoritative gate reruns the current native engine in an isolated temporary
workspace and compares those fresh outputs directly with the committed Stata exports.
The sole specification/oracle/tolerance registry is
`artifacts/parity/xtabond2/system_gmm_certification_specs.json`; the sole
numerical decision artifact is
`artifacts/parity/xtabond2/diagnostic_parity_certificate.csv`. The gate verifies
the certificate against LF-normalized canonical SHA-256 digests of the registry,
comparator, generators, support files, fixtures, do-files, parameter exports,
diagnostic exports, and the sample exports where declared. Counts and degrees of
freedom must match exactly; coefficient absolute
differences must be at most `1e-6`; Windmeijer-SE tolerances are specification
specific; Hansen/Sargan absolute differences must be at most `1e-6`; signed AR
z-statistic differences must be at most `0.10`; and AR p-value differences must be
at most `0.03`. The current six-spec certificate passes every gate. The largest
observed signed-AR differences are approximately `0.02264` for z and `0.00638` for
p. See the [System GMM parity matrix](docs/parity/system_gmm_parity_matrix.md) and
the unified machine-readable certificate at
`artifacts/parity/xtabond2/diagnostic_parity_certificate.csv`.

Comparator identity for the current historical Stata exports is carried by a
path-free, machine-generated provenance attestation. Its local source log is not
committed because it contains machine-specific paths. The attestation explicitly
records that the tracked-output and installed-ado hashes were observed when the
attestation was generated. The completed log recorded all six run markers and
completion; the hash-bound Stata diagnostic exports embed Stata 17 and
`xtabond2` 3.7.2 / `e(version)=03.07.00` comparator metadata directly.

The exact builder, Stata-driver, attestation, certificate, report, and pytest
commands are documented in the
[System GMM parity matrix](docs/parity/system_gmm_parity_matrix.md#evidence-and-reproduction).

## Certified unbalanced and missing-data extension

The native engine now preserves variable-specific missing histories, selects
differenced residuals from explicit equation metadata, and matches AR residual
pairs on the exact panel-time grid. The `system_gmm_unbalanced_panel` and
`system_gmm_variable_missing` fixtures exercise those paths and are registered in
the authoritative six-spec certificate. Both pass complete-parameter,
Windmeijer-SE, exact count, exact estimation-sample-key, Hansen/Sargan, and signed
AR gates. This evidence remains specific to those two controlled designs.

For integral numeric panel-time labels, the native engine follows Stata's default
unit-period delta, including a period absent from every entity. It fails with
recode guidance before a unit-spaced grid would exceed 100,000 periods or five
million entity-period rows. Non-integral and datetime labels retain ordered-rank
semantics until the public specification exposes an explicit panel-time delta.

The extension fixture and native-output builders are:

```text
python scripts/parity/build_xtabond2_unbalanced_missing_extension.py
python scripts/parity/run_native_unbalanced_missing_extension.py
```

For certification, use the complete six-spec Stata driver, provenance attestation,
and central comparator sequence documented in the
[System GMM parity matrix](docs/parity/system_gmm_parity_matrix.md#evidence-and-reproduction).
The extension manifest is a reproducibility index; the central registry and unified
certificate remain the only specification and numerical decision authorities.

Stata `xtabond2` is the formal certification oracle. `pydynpd` is an optional
execution backend and auxiliary comparator, not part of the current formal native
System-GMM certificate. A `pydynpd` result is parity- or speed-comparable only after
the effective sample, equations, lag semantics, instruments, collapsing, estimation
steps, finite-sample corrections, Windmeijer treatment, and covariance scaling have
all been aligned.

The maintained benchmark uses a controlled dynamic-panel specification with:

* collapsed instruments;
* restricted global GMM lag windows;
* two-step robust estimation;
* Windmeijer correction;
* strict numerical comparison against `xtabond2`.

Under these maintained benchmarks, the native implementation reproduces the
certified estimation quantities and diagnostics within the declared numerical
tolerances. This is not a claim of universal Stata identity. The Stata-reported date
and time are informational metadata only; fixture, do-file, parameter-export, and
diagnostic-export hashes identify the exact evidence used by the gate.

---

# FD001 Easy-GMM Lag-Window Validation

The easy GMM lag-window workflow was validated on CMAPSS FD001 real-data panel specifications.

Panel structure:

```text
entity = unit
time   = cycle
```

Target model:

```text
risk ~ L1_risk + degradation_index + sensor_mean_z + pc2 + op_setting1 + op_setting2
```

Variable classification:

| Variable            | Classification |
| ------------------- | -------------- |
| `L1_risk`           | Endogenous     |
| `degradation_index` | Predetermined  |
| `sensor_mean_z`     | Predetermined  |
| `pc2`               | Predetermined  |
| `op_setting1`       | Exogenous      |
| `op_setting2`       | Exogenous      |

The validation confirms:

* no duplicated symbolic lag notation;
* no `L1.risk` / `L1_risk` duplication in easy commands;
* no duplicated `gmm(risk, ...)` plus `gmm(L1_risk, ...)` instrumentation;
* exact agreement between actual and expected compact instrument counts;
* correct Difference GMM and System GMM command construction;
* correct global, role-specific, and variable-specific lag-window precedence.

Observed validation results:

| Scenario                          |      Estimator | Actual instruments | Expected compact instruments | Status |
| --------------------------------- | -------------: | -----------------: | ---------------------------: | ------ |
| `global_compact_22`               | Difference GMM |                  6 |                            6 | PASS   |
| `global_compact_22`               |     System GMM |                 11 |                           11 | PASS   |
| `role_endog_23_predet_12`         | Difference GMM |                 10 |                           10 | PASS   |
| `role_endog_23_predet_12`         |     System GMM |                 15 |                           15 | PASS   |
| `variable_override_sensor_l1risk` | Difference GMM |                  9 |                            9 | PASS   |
| `variable_override_sensor_l1risk` |     System GMM |                 14 |                           14 | PASS   |

The FD001 validation is a real-data application check. The controlled `xtabond2` benchmark remains the strict certification benchmark.

---

# External CMAPSS FD001 Validation Scaffold

The repository includes a CMAPSS FD001 publication-style validation scaffold for
two System GMM specifications.

Two validation models were used.

Risk model:

```text
risk ~ L1.risk + degradation_index + sensor_mean_z + pc2 + op_setting1 + op_setting2
```

Degradation model:

```text
degradation_index ~ L1.degradation_index + sensor_mean_z + pc2 + pc3 + op_setting1 + op_setting2
```

This scaffold is not part of the current formal certificate: its comparator still
depends on an external prepared-data location and does not yet have committed,
portable numerical guards for coefficients, standard errors, and diagnostics. It
must therefore not be cited as certified cross-software parity. The six controlled
`xtabond2` fixtures above remain the strict System GMM certification benchmark.

---

# Verified OLS and Pooled OLS Benchmarks

The OLS and pooled OLS implementations are validated against established Python, R, and Stata reference implementations under aligned benchmark specifications.

Committed validation artifacts report:

| Model | Reference | Maximum coefficient difference | Maximum standard-error difference | Status |
|---|---|---:|---:|---|
| Pooled OLS | Python linearmodels | 8.88178e-16 | 6.93889e-18 | PASS_NUMERIC |
| Pooled OLS | R plm | 1.22125e-15 | 1.00614e-16 | PASS_NUMERIC |
| Pooled OLS | Stata | 2.15841e-08 | 4.01439e-10 | PASS_COEFFICIENTS |

These results indicate numerical agreement with established reference implementations under the maintained benchmark specifications. Stata comparisons are reported with the precision available from the exported Stata results, so the Stata comparison is conservatively reported as coefficient-level agreement rather than overstated as universal machine-precision parity.

The corresponding validation artifacts are stored under:

- artifacts/joss/tables/26_static_postestimation_comparison/
- artifacts/joss/tables/27_static_cross_software_comparison/
---

# Reporting and Export

Results can be exported to:

* Markdown;
* CSV;
* LaTeX;
* PNG;
* SVG;
* PDF-compatible figures;
* HTML galleries.

`systemgmmkit` is designed to integrate with:

* `universal-output-hub`;
* publication pipelines;
* reproducible research workflows;
* model-comparison tables;
* diagnostic figure workflows.

---

# Example Reporting Workflow with universal-output-hub

```python
from universal_output_hub import outreg

outreg(
    [result],
    model_names=["System GMM"],
    path="tables/system_gmm_results.md",
)
```

The reporting layer is intentionally separate from estimation. This allows users to estimate models in `systemgmmkit` and export publication-style tables through `universal-output-hub`.

---

# Practical Modelling Guidance

## Use OLS and panel estimators as baselines

Dynamic GMM should not usually be the first model estimated.

A defensible empirical workflow often starts with:

1. OLS or pooled OLS;
2. fixed effects;
3. random effects where appropriate;
4. IV / 2SLS where identification requires external instruments;
5. Difference GMM or System GMM for dynamic-panel settings.

This helps users understand how estimates change across assumptions.

## Fixed-effects runtime

The native fixed-effects backend uses a compact within transformation for one-way and
two-way fixed-effects slope estimation. This avoids constructing a full least-squares
dummy-variable matrix when many entity or time effects are present, while preserving
slope equivalence with the corresponding LSDV estimator on the maintained balanced and
unbalanced benchmark paths.

Native fixed-effects results report:

* backend: `native-within`;
* structural slopes and a descriptive intercept where fixed effects are included;
* robust or clustered covariance using the compact residualized design;
* fixed-effect dummies kept out of the public coefficient table.

The internal LSDV construction remains available as an audit reference in tests. For
workflows that need an upstream result object or additional `PanelOLS` features, use
`run_fixed_effects(..., prefer_backend="linearmodels")`.

## Control instrument proliferation

Instrument proliferation is one of the most common problems in applied dynamic GMM.

Recommended practice:

* use `collapse=True`;
* restrict `gmm_lags`;
* use role-specific and variable-specific lag windows where justified;
* report instrument count;
* compare instrument count with number of groups;
* check whether Hansen p-values are suspiciously high;
* run robustness checks with alternative lag windows;
* inspect the SGM-Viz instrument architecture dashboard.

## Do not overinterpret one specification

Dynamic GMM estimates are sensitive to:

* lag-window choices;
* variable classification;
* transformation choice;
* instrument count;
* sample construction;
* missing-value handling;
* persistence of the dependent variable;
* weak instruments.

Users should treat dynamic GMM as part of a specification family, not as a single automatic estimator.

---

# Recommended Dynamic GMM Reporting

For published research, report:

* dependent variable;
* sample period;
* number of observations;
* number of groups;
* estimator type;
* transformation;
* lagged dependent-variable treatment;
* endogenous variables;
* predetermined variables;
* exogenous variables;
* structural lags included in the model;
* GMM lag-window design;
* global lag windows;
* role-specific lag windows;
* variable-specific lag windows;
* collapse setting;
* number of instruments;
* instrument/group ratio;
* AR(1) diagnostic;
* AR(2) diagnostic;
* Hansen test;
* Sargan test;
* covariance estimator;
* Windmeijer correction status;
* estimation backend;
* package version;
* model-health dashboard or equivalent diagnostics.

---

# Roadmap

The next technical extensions should focus on robustness, reviewer-facing validation, and production usability.

Planned work includes:

* preflight feasibility checks for impossible lag windows on short panels;
* clearer validation errors when requested GMM lags exceed usable panel depth;
* stronger handling of unknown variables in `gmm_lags_by_variable`;
* explicit rejection or warning when users try to apply GMM lag windows to exogenous-only variables;
* additional Stata comparison scripts for role-specific and variable-specific lag-window specifications;
* deeper documentation examples for instrument architecture;
* more post-estimation coverage for nonlinear combinations and richer marginal-effects workflows;
* additional real-data examples;
* further integration with `universal-output-hub`.

Implemented and validated features should remain in the current-feature sections, not in the roadmap.

---

# Citation

If you use `systemgmmkit` in academic work, please cite:

```text
Akanbi, Oluwajuwon Mayomi.

systemgmmkit:
Panel-Data Econometrics and Dynamic-Panel GMM Workflows in Python.

https://github.com/Akanom/systemgmmkit
```

BibTeX:

```bibtex
@software{akanbi_systemgmmkit,
  author = {Akanbi, Oluwajuwon Mayomi},
  title = {systemgmmkit: Panel-Data Econometrics and Dynamic-Panel GMM Workflows in Python},
  year = {2026},
  url = {https://github.com/Akanom/systemgmmkit}
}
```

Replace or supplement this citation with DOI information once a Zenodo archive or software paper is available.

---

# License

MIT License.

See `LICENSE` for details.
