Metadata-Version: 2.4
Name: ghonn_models_pytorch
Version: 0.2.0
Summary: Gated Higher Order Neural Networks models. PyTorch.
Author-email: Ondrej Budik <obudik@jcu.cz>
Project-URL: Homepage, https://gmp.readthedocs.io/en/latest/
Project-URL: Repository, https://github.com/carnosi/ghonn_models_pytorch
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch~=2.13
Requires-Dist: numpy~=2.5
Requires-Dist: pandas~=3.0
Provides-Extra: docs
Requires-Dist: sphinx~=9.1; extra == "docs"
Requires-Dist: autodocsumm~=0.2.15; extra == "docs"
Requires-Dist: sphinx-book-theme~=1.2; extra == "docs"
Provides-Extra: dev
Requires-Dist: ruff~=0.15; extra == "dev"
Requires-Dist: mypy~=2.2; extra == "dev"
Requires-Dist: setuptools~=83.0; extra == "dev"
Requires-Dist: pylint~=4.0; extra == "dev"
Requires-Dist: build~=1.5; extra == "dev"
Requires-Dist: pytest~=9.1; extra == "dev"
Provides-Extra: examples
Requires-Dist: matplotlib~=3.11; extra == "examples"
Requires-Dist: jupyter~=1.1.1; extra == "examples"
Requires-Dist: scikit-learn~=1.9; extra == "examples"
Requires-Dist: tqdm~=4.48; extra == "examples"
Dynamic: license-file

<div align="center">

<img src="https://raw.githubusercontent.com/carnosi/ghonn_models_pytorch/main/docs/source/_static/logo.png" alt="logo" width="50%" />

**Python library with polynomial neural networks**

[![Project Status: Active](https://img.shields.io/badge/repo_status-active-brightgreen?style=for-the-badge)](https://www.repostatus.org/#active) [![Read the Docs](https://img.shields.io/readthedocs/gmp?style=for-the-badge&logo=readthedocs&logoColor=white)](https://gmp.readthedocs.io/en/latest/)

[![PyPI](https://img.shields.io/pypi/v/ghonn-models-pytorch?color=red&style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/ghonn-models-pytorch/) [![Python - Version](https://img.shields.io/badge/PYTHON-3.12+-red?style=for-the-badge&logo=python&logoColor=white)](https://pepy.tech/project/ghonn-models-pytorch) [![PyTorch - Version](https://img.shields.io/badge/PYTORCH-2.13+-red?style=for-the-badge&logo=pytorch)](https://pepy.tech/project/ghonn-models-pytorch)

[![License](https://img.shields.io/badge/License-MIT-<COLOR>?style=for-the-badge&color=blue)](https://github.com/carnosi/ghonn_models_pytorch/blob/main/LICENSE)

</div>

**GHONN Models Pytorch** brings advanced neural architectures to your PyTorch projects: Higher Order Neural Units (HONU), Higher Order Neural Networks (HONN), Gated Higher Order Neural Units (GHONU), Gated Higher Order Neural Networks (GHONN), and the Self-Attention Gated Higher Order Neural Network (SA-GHONN).

✨ **Polynomial neurons at the core:** These models excel at capturing complex, nonlinear relationships—especially when working with polynomial signals. Their adaptable design makes them a strong choice for a wide range of machine learning tasks.

🔗 **Gated variants for extra power:** The gated architectures use a dual HONU neuron setup—one as a dynamic gate, the other as the main predictor—enabling richer and more expressive modeling.

🧠 **Full forecasting network:** SA-GHONN normalizes the input (optionally with RevIN), maps each timestep through a HONU/GHONU embedding, adds positional encoding, and passes the sequence through a Transformer encoder with self-attention. Its forecast head is configurable: `flat` uses a linear projection of the full encoded sequence, `pool` mean-pools the sequence and applies an MLP, and `query` uses learned forecast queries with cross-attention over the self-attended sequence before a final linear projection. Thus, self-attention is always used, while cross-attention is specific to the `query` head.

🛠️ **Modular and flexible:** Build your own architectures with ease. Layers can be stacked directly or connected via linear mappings, giving you full control over your network’s structure.

👉 **Curious how it works in practice?** Check out the example notebooks and usage guides included in this repository.

## 📖 [Project Documentation](https://gmp.readthedocs.io/) 📖
Visit [Read The Docs Project Page](https://gmp.readthedocs.io/) or read the following README to know more about Gated Higher Order Neural Network Models Pytorch (GHONN for short) library.

## ✨ Features <a name="features"></a>

- **Polynomial neurons:** Capture complex, nonlinear relationships using higher-order neural units.
- **Gated architectures:** Leverage dual-neuron setups for richer modeling capacity.
- **SA-GHONN:** Use a complete self-attention forecasting network built around HONU/GHONU components.
- **Conv-GHONN:** Apply GHONU polynomial filters to causal temporal windows for sequence feature extraction.
- **Modular design:** Easily stack and combine layers for custom architectures.
- **Efficient computation:** Optimized for high-order polynomial calculations, even on CPUs.
- **Seamless PyTorch integration:** All components are standard PyTorch modules.
- **Supports regression & classification:** Flexible for a wide range of ML tasks.
- **Ready-to-use examples:** Example notebooks and guides included.

**Neuron Types** ⚡
- **HONU:** The fundamental building block for higher-order modeling. For example, a 2nd order HONU is defined as:

  ![HONU equation](https://latex.codecogs.com/png.image?\dpi{120}\bg_white\tilde{y}(k)=\sum_{i=0}^{n}\sum_{j=i}^{n}w_{i,j}x_ix_j=\mathbf{w}\cdot\mathrm{col}^{r=2}(\mathbf{x}))

  where:
  - $\tilde{y}(k)$ is the neuron output for input sample $k$
  - $w_{i,j}$ are the learnable weights
  - $x_i, x_j$ are input features
  - $\mathbf{w}$ is the weight vector
  - $\mathrm{col}^{r=2}(\mathbf{x})$ is the column vector of all 2nd order combinations of input features
  - $r$ is the polynomial order

  This structure ensures polynomial relationships between input datapoints and high computation performance.
- **gHONU:** Combines two HONUs—one as a predictor (typically linear activation), the other as a dynamic gate (e.g., `tanh`)—multiplying their outputs for enhanced ability to capture complex patterns.

**Network Layers** 🧩
- **HONN:** Single-layer networks of HONU neurons. Supports both raw outputs for stacking and linear heads for custom output dimensions.
- **gHONN:** Single-layer networks of gHONU neurons, with the same flexible output options as HONN.

**Full Models and Components** 🧠
- **SA-GHONN:** Self-attention Gated Higher-Order Neural Network forecaster. The pipeline is `RevIN` (optional) -> HONU/GHONU embedding (`GhonuBank` or temporal `ConvGhonn`) -> positional encoding -> Transformer encoder self-attention -> forecast head. The `flat` head uses a linear projection, the `pool` head uses mean pooling followed by an MLP, and the `query` head uses learned forecast queries to cross-attend to the self-attended sequence before a linear projection.
- **`HonuBank` and `GhonuBank`:** Grouped, vectorized banks of HONU and GHONU neurons.
- **`ConvGhonn` (Conv-GHONN):** Applies GHONU polynomial filters to causal temporal windows and can be used directly or as the SA-GHONN embedding when `temporal_lookback` is greater than zero.
- **`RevIN`:** Reversible instance normalization for non-stationary time series.

**Why Choose GHONN Models?** 🚀
- **Efficient high-order computation:** Optimized for fast polynomial calculations, even at high orders and on CPUs.
- **Flexible & modular:** Easily stack, combine, or adapt layers and neurons for custom architectures.
- **PyTorch-native:** All components are standard PyTorch modules for seamless integration.
- **Versatile:** Supports both regression and classification tasks.
- **Quick start:** Example notebooks and guides included to help you get going fast.

## 🧪 Examples & Usage <a name="examples"></a>

You can find helpful, step-by-step Jupyter notebooks in the [examples](./examples/) folder, which offer practical demonstrations and implementation suggestions.

You may also find the code snippets below useful as a starting point.

**HONU initialization**
```python
import ghonn_models_pytorch as gmp

kwargs = {
    "weight_divisor": 100,  # Divides weights to help with numerical stability
    "bias": True            # Whether to use a bias term in the model
}

# Create a Higher Order Neural Unit (HONU) with 3 inputs and degree 2
honu_neuron = gmp.HONU(
    in_features=3,          # Number of input features
    order=2,                # Degree of the polynomial
    activation="identity",  # Activation function
    **kwargs
)
```

**HONN initialization**
```python
import ghonn_models_pytorch as gmp

kwargs = {
    "weight_divisor": 100,
    "bias": True
}

# Create single HONU based layer - HONN with 4 neurons of different orders and activation functions.
honn_layer = gmp.HONN(
    in_features=3,                          # Number of input features
    out_features=2,                         # Number of output features
    layer_size=4,                           # Number of neurons in the layer
    polynomial_orders=(2, 3),               # Degree of the polynomials; cycles if shorter than layer_size
    activations=("identity", "sigmoid"),    # Activation functions for the neurons in the layer. If shorter work like a rolling buffer
    output_type="linear",                   # Output type of the layer. Can be "linear" or "sum" or "raw"
    **kwargs
)
```
**Neuron, Layer or Model training as usual**
```python
for i in range(0, data.size(0), batch_size):
    # Get the batch
    batch = data[i:i+batch_size]
    # Forward pass
    output = honn_layer(batch)
    # Compute loss
    loss = criterion(output, target)
    # Backward pass
    loss.backward()
    # Update weights
    optimizer.step()
```

## 💡 Tips & Tricks <a name="tips_n_tricks"></a>

**Default SA-GHONN initialization**
```python
import torch
import ghonn_models_pytorch as gmp

# Forecast 12 steps from a 24-step, 3-feature input sequence.
sa_ghonn = gmp.SAGHONN(
    input_length=24,
    in_features=3,
    out_features=12,
)

sequence = torch.randn(8, 24, 3)
forecast = sa_ghonn(sequence)
# forecast.shape == (8, 12, 1)
```

SA-GHONN uses a GHONU bank as its default sequence embedding. It then adds
positional encoding, processes the sequence with a Transformer encoder, and
applies the selected forecasting head. Set `temporal_lookback` to use a temporal
`ConvGhonn` embedding, or set `head_type="query"` to use learned forecast
queries with cross-attention. Use `predicted_feature_indices` and
`aux_all_features` when forecasting multiple input features.
* In the case of GHONU based units it is often benefitial to have different initial learning rate between the two neurons.
* more TBD

## 🛠️ Installation <a name="installation"></a>

**PyPI version:**
```bash
pip install ghonn-models-pytorch
```

**The latest version from GitHub:**
```bash
pip install git+https://github.com/carnosi/ghonn_models_pytorch
```

## Compatibility notice
The verion 0.2+ API uses vectorized PyTorch modules and does not guarantee backward compatibility with
earlier releases. `HONU` accepts inputs shaped `(..., in_features)` and returns
`(..., out_features)`. A `GHONU` with `gate_order=0` omits its gate and behaves as a HONU.
Gates can be frozen with standard PyTorch controls:

```python
ghonu.gate.requires_grad_(False)
```

## 📚 References <a name="references"></a>
This repository is inspired by the foundational research presented in the following papers. While the original studies utilized legacy implementations, this PyTorch-based version offers a more user-friendly and computationally efficient alternative, maintaining the same core objectives and functionality.

**HONU**:
```plaintext
[1] P. M. Benes and I. Bukovsky, “Railway Wheelset Active Control and Stability via Higher Order Neural Units,” IEEE/ASME Transactions on Mechatronics, vol. 28, no. 5, pp. 2964–2975, Oct. 2023, doi: 10.1109/TMECH.2023.3258909.

[2] I. Bukovsky, G. Dohnal, P. M. Benes, K. Ichiji, and N. Homma, “Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures With Gradient Learnings,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 8, pp. 5189–5192, Aug. 2023, doi: 10.1109/TNNLS.2021.3123533.

[3] I. Bukovsky, “Deterministic behavior of temperature field in turboprop engine via shallow neural networks,” Neural Comput & Applic, vol. 33, no. 19, pp. 13145–13161, Oct. 2021, doi: 10.1007/s00521-021-06013-7.

[4] P. M. Benes, I. Bukovsky, M. Vesely, J. Voracek, K. Ichiji, and N. Homma, “Framework for Discrete-Time Model Reference Adaptive Control of Weakly Nonlinear Systems with HONUs,” in Computational Intelligence, C. Sabourin, J. J. Merelo, K. Madani, and K. Warwick, Eds., Cham: Springer International Publishing, 2019, pp. 239–262. doi: 10.1007/978-3-030-16469-0_13.
```
**GHONU**:
```plaintext
[1] O. Budik, I. Bukovsky, and N. Homma, “Potentials of Gated Higher Order Neural Units for Signal Decomposition and Process Monitoring,” Procedia Computer Science, vol. 253, pp. 2278–2287, Jan. 2025, doi: 10.1016/j.procs.2025.01.288.
```

### Our other project

**[AISLEX](https://github.com/carnosi/AISLEX)**: A Python package for Approximate Individual Sample Learning Entropy (LE) anomaly detection. Easily integrate LE-based novelty detection into your neural network workflows, with both Python and JAX implementations.

## 📝 How To Cite <a name="how_to_cite"></a>
If `ghonn_models_pytorch` has been useful in your research or work, please consider citing our article:

```plaintext
Work in progress. Use GHONU (10.1016/j.procs.2025.01.288) for now please.
```

BibText:
```bibtex
Work in progress. Use GHONU (10.1016/j.procs.2025.01.288) for now please.
```
## 📄 License <a name="lisence"></a>

This project is licensed under the terms of the [MIT License](https://github.com/carnosi/ghonn_models_pytorch/blob/main/LICENSE).
