Metadata-Version: 2.4
Name: dthinking
Version: 0.1.0
Summary: A modular, production-grade Deep Reinforcement Learning framework built on PyTorch.
Author-email: Developer <developer@example.com>
License-Expression: MIT
Keywords: reinforcement-learning,deep-rl,dqn,ppo,sac,ddpg,a2c,pytorch,gymnasium
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0.0
Requires-Dist: numpy>=1.20.0
Requires-Dist: gymnasium>=0.28.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: matplotlib>=3.5.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Dynamic: license-file

# dthinking 🧠⚡

[![Python Versions](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://pypi.org/project/dthinking/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.0%2B-ee4c2c.svg)](https://pytorch.org/)
[![Gymnasium](https://img.shields.io/badge/Gymnasium-compatible-brightgreen.svg)](https://gymnasium.farama.org/)
[![Tests: 84 Passed](https://img.shields.io/badge/tests-84%20passed-success.svg)](tests/)

`dthinking` is a modular, production-grade Deep Reinforcement Learning framework built on PyTorch. It generalizes empirical state-of-the-art algorithms researched and benchmarked in `Sanke_RL` into a decoupled, extensible architecture capable of training on **Gymnasium benchmarks, custom game environments, multi-agent worlds, and offline datasets** with zero boilerplate.

---

## 📑 Table of Contents

1. [Architectural Overview & Taxonomy](#-architectural-overview--taxonomy)
2. [Categorized Algorithm Suite](#-categorized-algorithm-suite)
   - [1. Online Reinforcement Learning](#1-online-reinforcement-learning)
   - [2. Offline Reinforcement Learning](#2-offline-reinforcement-learning)
   - [3. Multi-Agent Reinforcement Learning (MARL)](#3-multi-agent-reinforcement-learning-marl)
   - [4. Interactive Imitation & Preference Learning](#4-interactive-imitation--preference-learning)
3. [Snake Game Environment Specifications](#-snake-game-environment-specifications)
4. [Installation & Setup](#-installation--setup)
5. [Complete Command-Line Interface (CLI) Guide](#-complete-command-line-interface-cli-guide)
   - [Inspecting Algorithms & Presets](#inspecting-algorithms--presets)
   - [Training Online Agents on Snake](#training-online-agents-on-snake)
   - [Training with Tuned Presets](#training-with-tuned-presets)
   - [Evaluating Saved Checkpoints](#evaluating-saved-checkpoints)
6. [Python API Quickstart & Code Examples](#-python-api-quickstart--code-examples)
   - [Online Training (DQN & PPO)](#1-online-training-dqn--ppo)
   - [Offline Training (CQL & BCQ)](#2-offline-training-cql--bcq)
   - [Multi-Agent Training (QMIX on Multi-Snake)](#3-multi-agent-training-qmix-on-multi-snake)
   - [Direct Preference Optimization (DPO)](#4-direct-preference-optimization-dpo)
7. [Directory Structure](#-directory-structure)
8. [Testing & Verification](#-testing--verification)
9. [License](#-license)

---

## 🏛️ Architectural Overview & Taxonomy

```mermaid
graph TD
    subgraph dthinking_Core ["dthinking Algorithmic Ecosystem"]
        Online["1. Online RL (dthinking.agents.online)"]
        Offline["2. Offline RL (dthinking.agents.offline)"]
        MARL["3. Multi-Agent RL (dthinking.agents.marl)"]
        Imitation["4. Imitation & Preference (dthinking.agents.imitation)"]
    end

    Online --> DQN_Fam["DQN / Double DQN (DDQN) / Dueling DQN / Dyna-Q"]
    Online --> PG_Fam["REINFORCE / A2C / GAE / PPO / TRPO / A3C"]
    Online --> Cont_Fam["DDPG / TD3 / SAC"]
    Online --> Search_Fam["MCTS (Monte Carlo Tree Search)"]

    Offline --> BCQ["BCQ (Batch-Constrained Q-Learning)"]
    Offline --> CQL["CQL (Conservative Q-Learning)"]
    Offline --> Off_Base["Offline DQN / Offline DDPG / Offline PPO"]
    Offline --> Off_Data["OfflineDataset & Batch Sampler"]

    MARL --> MADDPG["MADDPG (CTDE Centralized Critic)"]
    MARL --> QMIX["QMIX (Hypernetwork Monotonic Mixing)"]
    MARL --> VDN["VDN (Additive Value Factorization)"]
    MARL --> IQL["IQL (Independent Q-Learning)"]
    MARL --> MAPPO["MAPPO (Multi-Agent PPO)"]

    Imitation --> BC["Behavioral Cloning (BC)"]
    Imitation --> DAgger["DAgger (Interactive Dataset Aggregation)"]
    Imitation --> DPO["DPO (Direct Preference Optimization)"]
```

---

## 🧠 Categorized Algorithm Suite

### 1. Online Reinforcement Learning
Located in `dthinking.agents.online`.

#### Value-Based Algorithms
* **DQN (`DQNAgent`)**: Classical Deep Q-Network with experience replay, target networks, and $\epsilon$-greedy exploration scheduling.
* **Double DQN (`DoubleDQNAgent`)**: Decouples action selection from target evaluation to suppress greedy overestimation:
  $$y_t = r_t + \gamma Q\left(s_{t+1}, \arg\max_{a'} Q(s_{t+1}, a'; \theta); \theta^-\right)$$
* **Dueling DQN (`DuelingDQNAgent`)**: Decomposes state-action values into state value $V(s)$ and action advantage $A(s, a)$ with identifiability centering:
  $$Q(s, a) = V(s) + \left(A(s, a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s, a')\right)$$
* **Dyna-Q (`DynaQAgent`)**: Model-Based RL integrating model-free Q-learning with a learned transition dynamics neural network $T(s, a) \to (s', r, d)$ generating synthetic planning rollouts.

#### Policy Gradients & Actor-Critic
* **REINFORCE (`ReinforceAgent`)**: Monte Carlo policy gradient with state-value baseline subtraction for variance reduction and entropy bonus exploration.
* **A2C (`A2CAgent`)**: Synchronous Advantage Actor-Critic using temporal-difference value baselines.
* **GAE (`GAEAgent`)**: Actor-Critic with Generalized Advantage Estimation parameter $\lambda \in [0, 1]$ balancing bias and variance:
  $$\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^\infty (\gamma \lambda)^l \delta_{t+l}^V, \quad \delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t)$$
* **PPO (`PPOAgent`)**: Clipped surrogate objective policy optimization preventing destructive policy updates:
  $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \left[ \min\left(r_t(\theta)\hat{A}_t, \, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\right) \right]$$
* **TRPO (`TRPOAgent`)**: Trust Region Policy Optimization using Fisher-Vector Products (FVP), Conjugate Gradient, and Backtracking Line Search under strict KL constraint $\bar{D}_{KL}(\theta_{\text{old}}, \theta) \le \delta$.
* **A3C (`A3CAgent`)**: Asynchronous Advantage Actor-Critic architecture supporting multi-threaded workers with shared master network gradient synchronization.

#### Continuous Control & Planning
* **DDPG (`DDPGAgent`)**: Deep Deterministic Policy Gradient with continuous action target networks, Polyak averaging ($\tau$), and Ornstein-Uhlenbeck / Gaussian exploration noise.
* **TD3 (`TD3Agent`)**: Twin Delayed DDPG addressing value overestimation via Clipped Double-Q networks, Target Policy Smoothing Noise, and Delayed Actor Updates.
* **SAC (`SACAgent`)**: Soft Actor-Critic maximum entropy continuous control with twin Q-critics and automatic temperature ($\alpha$) tuning.
* **MCTS (`MCTSAgent`)**: Monte Carlo Tree Search with UCB1 exploration and lookahead rollout simulations.

---

### 2. Offline Reinforcement Learning
Located in `dthinking.agents.offline`.

Trains agents directly from static transition datasets without active environment interaction, overcoming out-of-distribution (OOD) extrapolation collapse.

* **BCQ (`BCQAgent` / `DiscreteBCQAgent`)**: Batch-Constrained Q-Learning. Trains a behavior cloning generative prior $\hat{\pi}_\beta(a|s)$ and restricts action selection strictly to supported data:
  $$a^* = \arg\max_{a: \hat{\pi}_\beta(a|s) > \tau} Q(s, a)$$
* **CQL (`CQLAgent` / `DiscreteCQLAgent`)**: Conservative Q-Learning. Directly penalizes unobserved actions via log-sum-exp while pushing up dataset action values, guaranteeing lower-bound value estimates:
  $$\min_Q \alpha \left(\log \sum_{a} \exp(Q(s, a)) - \mathbb{E}_{a \sim \mathcal{D}}[Q(s, a)]\right) + \frac{1}{2}\mathcal{L}_{\text{TD}}(Q)$$
* **Offline DQN (`OfflineDQNAgent`)**: Standard Q-learning optimized on static batch datasets.
* **Offline DDPG (`OfflineDDPGAgent`)**: Offline DDPG supporting continuous and discrete actions via Gumbel-Softmax relaxation.
* **Offline PPO (`OfflinePPOAgent`)**: Importance-weighted clipped surrogate updates over static trajectories.
* **OfflineDataset (`OfflineDataset`)**: PyTorch `Dataset` with high-throughput tensor sampling and file serialization (`.save()` / `.load()`).

---

### 3. Multi-Agent Reinforcement Learning (MARL)
Located in `dthinking.agents.marl`.

Addresses non-stationary transitions and team coordination in multi-agent environments using Centralized Training with Decentralized Execution (CTDE).

* **MADDPG (`MADDPGAgent`)**: Decentralized actors $\pi_i(s_i)$ combined with centralized critics $Q_i(s_1, \dots, s_N, a_1, \dots, a_N)$ conditioned on joint state-action vectors.
* **QMIX (`QMIXAgent`)**: Monotonic Value Factorization network. Uses state-conditioned Hypernetworks with absolute-value weights enforcing:
  $$\frac{\partial Q_{tot}}{\partial Q_i} \ge 0$$
* **VDN (`VDNAgent`)**: Value-Decomposition Network with additive linear factorization:
  $$Q_{tot}(\mathbf{s}, \mathbf{a}) = \sum_{i=1}^N Q_i(s_i, a_i)$$
* **IQL (`IQLAgent`)**: Independent Q-Learning with decentralized parameter sharing across agents.
* **MAPPO (`MAPPOAgent`)**: Multi-Agent PPO with decentralized policy actors and a centralized state-value critic $V(s_{\text{global}})$.
* **MultiSnakeEnv (`MultiSnakeEnv`)**: Multi-agent cooperative/competitive snake game supporting $N$ agents.

---

### 4. Interactive Imitation & Preference Learning
Located in `dthinking.agents.imitation`.

* **Behavioral Cloning (`BehavioralCloningAgent`)**: Supervised cross-entropy / MSE policy optimization on demonstration state-action pairs.
* **DAgger (`DAggerAgent`)**: Dataset Aggregation. Iteratively rolls out the learner's policy, queries an expert oracle for corrected labels, aggregates the dataset, and retrains to overcome distribution shift.
* **DPO (`DPOAgent`)**: Direct Preference Optimization for RL agents using the closed-form Bradley-Terry objective without requiring an explicit reward model:
  $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(s, a_w, a_l)} \left[ \log \sigma \left( \beta \left( \log \frac{\pi_\theta(a_w|s)}{\pi_{\text{ref}}(a_w|s)} - \log \frac{\pi_\theta(a_l|s)}{\pi_{\text{ref}}(a_l|s)} \right) \right) \right]$$

---

## 🐍 Snake Game Environment Specifications

### Single-Agent Snake (`SnakeEnv`)
* **State Vector (14 Dimensions)**:
  * `[0..3]`: Danger detection in 4 cardinal directions (Up, Down, Left, Right)
  * `[4..7]`: Current movement direction (One-hot encoded)
  * `[8..11]`: Food relative position flags (Up, Down, Left, Right)
  * `[12..13]`: Normalized food distance along x and y coordinates
* **Action Space**: `Discrete(4)` (0: UP, 1: DOWN, 2: LEFT, 3: RIGHT)
* **Reward Structure**:
  * `+50.0`: Eating food (snake body grows)
  * `+1.0`: Moving closer to food
  * `-1.0`: Moving farther from food
  * `-100.0`: Collision with wall or self (Game Over)
  * `-50.0`: Timeout / stagnation

### Multi-Agent Snake (`MultiSnakeEnv`)
* **Observation Space**: List of 14-dim sensory vectors for $N$ agents.
* **Collision Logic**: Wall collisions, body self-collisions, and inter-snake body/head collisions.
* **Cooperative Reward**: Individual food eating reward (`+50.0`) plus team cooperative bonus (`+20.0`) shared across all alive teammates.

---

## 📦 Installation & Setup

You can install `dthinking` via `pip` either as a built package (wheel/tarball) or from source:

### 1. Install via pip (Wheel / Distribution)
If installing the built wheel:
```bash
# Direct install from distribution wheel
pip install dist/dthinking-0.1.0-py3-none-any.whl

# Or if published to PyPI
pip install dthinking
```

### 2. Install from Source
```bash
# Clone the repository
git clone https://github.com/yourusername/dthinking.git
cd dthinking

# Install in standard mode
pip install .

# Or install in editable development mode with testing tools
pip install -e ".[dev]"
```

### 3. Build Distribution Packages from Source
To create fresh distribution archives (`.whl` and `.tar.gz` in `dist/`):
```bash
pip install build
python -m build
```

Dependencies: `torch >= 2.0.0`, `gymnasium >= 0.28.0`, `numpy >= 1.20.0`, `pyyaml >= 6.0`, `matplotlib >= 3.5.0`.

---

## 💻 Complete Command-Line Interface (CLI) Guide

Upon installation, the CLI tool `dthinking` is available in your shell:

### Inspecting Algorithms & Presets
```bash
# List all 26+ categorized algorithms across Online, Offline, MARL, and Imitation
dthinking list-algos

# List all pre-tuned benchmark presets
dthinking list-presets
```

### Training Online Agents on Snake
```bash
# Double DQN (DDQN) on Snake
dthinking train --env snake --algo ddqn --timesteps 50000

# Standard DQN on Snake
dthinking train --env snake --algo dqn --timesteps 50000 --lr 0.001

# Dueling DQN on Snake
dthinking train --env snake --algo dueling_dqn --timesteps 50000

# PPO on Snake
dthinking train --env snake --algo ppo --timesteps 50000 --lr 0.0003

# TRPO (Trust Region Policy Optimization) on Snake
dthinking train --env snake --algo trpo --timesteps 30000

# Dyna-Q (Model-Based RL) on Snake
dthinking train --env snake --algo dynaq --timesteps 30000

# A2C on Snake
dthinking train --env snake --algo a2c --timesteps 40000
```

### Training with Tuned Presets
```bash
# Train using the tuned Snake Double-DQN profile
dthinking train --preset snake_dqn --timesteps 50000

# Train using the tuned Snake PPO profile
dthinking train --preset snake_ppo --timesteps 50000

# Train using the tuned Snake Conservative Q-Learning profile
dthinking train --preset snake_cql --timesteps 30000

# Train using the tuned Snake QMIX profile
dthinking train --preset snake_qmix --timesteps 40000

# Train classical benchmarks
dthinking train --preset cartpole_dqn
dthinking train --preset lunarlander_ppo
dthinking train --preset pendulum_sac
```

### Evaluating Saved Checkpoints
```bash
# Evaluate a trained PPO model over 10 deterministic test episodes
dthinking evaluate --model ./checkpoints/final_model.pth --env snake --algo ppo --episodes 10

# Evaluate a trained DQN model
dthinking evaluate --model ./checkpoints/snake_dqn/final_model.pth --env snake --algo dqn --episodes 10
```

### Exporting Configurations to YAML
```bash
dthinking export-config --preset snake_dqn --output snake_dqn_config.yaml
```

---

## 🚀 Python API Quickstart & Code Examples

`dthinking` enables unified, high-level model training via `dthinking.train(...)` across all 4 reinforcement learning paradigms, alongside lower-level agent instantiation:

### 1. Online Training (DQN, PPO, SAC, TD3, MCTS, etc.)
Train on Gymnasium benchmarks or custom environments with full automatic logging and evaluation:
```python
import dthinking as dt

# Train PPO on CartPole
result = dt.train(
    env="CartPole-v1",
    algorithm="ppo",
    total_timesteps=20_000,
    learning_rate=3e-4,
    verbose=1,
)

# Evaluate trained agent
metrics = dt.evaluate(result.agent, env="CartPole-v1", num_episodes=10)
print(f"PPO Mean Reward: {metrics['mean_reward']:.2f} +/- {metrics['std_reward']:.2f}")

# Train Continuous Control (SAC) on Pendulum
sac_result = dt.train(
    env="Pendulum-v1",
    algorithm="sac",
    total_timesteps=10_000,
    learning_rate=1e-3,
    verbose=1,
)
```

### 2. Offline Training (CQL, BCQ, Offline DQN/DDPG/PPO)
Train directly from an offline dataset without live environment interaction:
```python
import dthinking as dt
from dthinking.agents.offline import collect_offline_dataset

# 1. Collect demonstration dataset (or load pre-existing dataset)
dataset = collect_offline_dataset("CartPole-v1", num_transitions=10_000)

# 2. Train Conservative Q-Learning (CQL) or BCQ on the dataset
result = dt.train(
    env="CartPole-v1",
    algorithm="cql",
    dataset=dataset,
    total_timesteps=2_000,
    batch_size=64,
    verbose=1,
)

# Evaluate offline-learned policy on live environment
metrics = dt.evaluate(result.agent, env="CartPole-v1", num_episodes=10)
print(f"CQL Policy Mean Reward: {metrics['mean_reward']:.2f}")
```

### 3. Multi-Agent Reinforcement Learning (QMIX, MADDPG, MAPPO, VDN, IQL)
Train coordinated teams in multi-agent environments with Centralized Training and Decentralized Execution (CTDE):
```python
import dthinking as dt
from dthinking.envs.multi_snake import MultiSnakeEnv

# Multi-Agent Snake with 2 cooperating/competing snakes
env = MultiSnakeEnv(num_agents=2, width=15, height=15)

# Train QMIX team policy
result = dt.train(
    env=env,
    algorithm="qmix",
    total_timesteps=5_000,
    learning_rate=5e-4,
    verbose=1,
)

# Evaluate team performance
metrics = dt.evaluate(result.agent, env=env, num_episodes=5)
print(f"QMIX Team Reward: {metrics['mean_reward']:.2f}")
```

### 4. Interactive Imitation & Preference Learning (BC, DAgger, DPO)
Overcome distribution shift with interactive expert oracles (DAgger) or optimize directly from preferences (DPO):
```python
import dthinking as dt

# DAgger with interactive expert policy query
class RuleBasedExpert:
    def act(self, state, deterministic=True):
        # Expert heuristic (e.g. angle-balancing for CartPole)
        angle = state[2]
        return 1 if angle > 0 else 0

result = dt.train(
    env="CartPole-v1",
    algorithm="dagger",
    expert_policy=RuleBasedExpert(),
    total_timesteps=2_000,
    learning_rate=1e-3,
    verbose=1,
)

# Direct Preference Optimization (DPO)
dpo_result = dt.train(
    env="CartPole-v1",
    algorithm="dpo",
    total_timesteps=1_000,
    learning_rate=1e-3,
    verbose=1,
)
```

### 5. Universal All-Algorithms Runner Script
Run the built-in script demonstrating end-to-end training across all 28 supported algorithm configurations:
```bash
python examples/train_all_algorithms.py
```

---

## 📂 Directory Structure

```
dthinking/
├── pyproject.toml                     # Packaging configuration & CLI entrypoints
├── setup.py                           # Fallback setuptools build script
├── requirements.txt                   # Production dependencies (torch, gymnasium, numpy, pyyaml, matplotlib)
├── requirements-dev.txt               # Development dependencies (pytest, black, flake8, mypy)
├── LICENSE                            # MIT License
├── README.md                          # Master documentation & architectural reference
├── BUILD_DTHINKING_PROMPT.md          # Architectural specifications & requirements
├── src/
│   └── dthinking/
│       ├── __init__.py                # Top-level exports and agent re-exports
│       ├── api.py                     # High-level API: train(), evaluate(), play(), load_model()
│       ├── cli.py                     # CLI entrypoint (dthinking train, evaluate, list-algos, etc.)
│       ├── envs/
│       │   ├── __init__.py            # Environment module exports
│       │   ├── adapter.py             # UniversalEnvAdapter (Gymnasium, custom classes, native strings)
│       │   ├── snake.py               # Gymnasium-compatible 14-dim SnakeEnv
│       │   ├── multi_snake.py         # MultiSnakeEnv (cooperative/competitive N-agent Snake)
│       │   ├── spaces.py              # Space inspection & dimension extractors
│       │   └── wrappers.py            # ObservationNormalizer, FrameStack, RewardScaler, TimeLimit
│       ├── models/
│       │   ├── __init__.py            # Model module exports
│       │   ├── base.py                # BaseModel with Xavier & Orthogonal initializations
│       │   ├── mlp.py                 # Configurable MLP with LayerNorm, BatchNorm, Dropout
│       │   ├── cnn.py                 # NatureCNN convolutional visual feature extractor
│       │   ├── actor_critic.py        # DiscreteActorCritic, ContinuousActorCritic, TwinQCritic
│       │   ├── dueling.py             # DuelingQNetwork (Value & Advantage streams)
│       │   ├── dynamics.py            # EnvDynamicsModel for Dyna-Q planning rollouts
│       │   └── marl.py                # QMIXMixer, VDNMixer, CentralizedCritic
│       ├── buffers/
│       │   ├── __init__.py            # Buffer module exports
│       │   ├── base.py                # BaseBuffer abstract protocol
│       │   ├── replay_buffer.py       # Circular vector replay buffer
│       │   ├── rollout_buffer.py      # RolloutBuffer with vectorized GAE computation
│       │   └── prioritized.py         # Prioritized Experience Replay (SumTree)
│       ├── agents/
│       │   ├── __init__.py            # Central agent imports and paradigm re-exports
│       │   ├── base.py                # BaseAgent abstract interface
│       │   ├── factory.py             # Central AGENT_REGISTRY & create_agent() dispatcher
│       │   ├── online/                # Online Reinforcement Learning
│       │   │   ├── __init__.py        # Online agent exports
│       │   │   ├── dqn.py             # DQNAgent, DoubleDQNAgent, DuelingDQNAgent
│       │   │   ├── dynaq.py           # DynaQAgent (Model-Based RL)
│       │   │   ├── reinforce.py       # ReinforceAgent (Policy Gradients)
│       │   │   ├── a2c.py             # A2CAgent (Advantage Actor-Critic)
│       │   │   ├── gae.py             # GAEAgent (Generalized Advantage Estimation)
│       │   │   ├── ppo.py             # PPOAgent (Proximal Policy Optimization)
│       │   │   ├── trpo.py            # TRPOAgent (Trust Region Policy Optimization)
│       │   │   ├── a3c.py             # A3CAgent (Asynchronous Actor-Critic)
│       │   │   ├── ddpg.py            # DDPGAgent (Deep Deterministic Policy Gradient)
│       │   │   ├── td3.py             # TD3Agent (Twin Delayed DDPG)
│       │   │   ├── sac.py             # SACAgent (Soft Actor-Critic)
│       │   │   └── mcts.py            # MCTSAgent (Monte Carlo Tree Search)
│       │   ├── offline/               # Offline Reinforcement Learning
│       │   │   ├── __init__.py        # Offline agent exports
│       │   │   ├── bcq.py             # BCQAgent, DiscreteBCQAgent (Batch-Constrained Q-Learning)
│       │   │   ├── cql.py             # CQLAgent, DiscreteCQLAgent (Conservative Q-Learning)
│       │   │   ├── offline_dqn.py     # OfflineDQNAgent
│       │   │   ├── offline_ddpg.py    # OfflineDDPGAgent (Continuous & Discrete Gumbel-Softmax)
│       │   │   ├── offline_ppo.py     # OfflinePPOAgent
│       │   │   └── dataset.py         # OfflineDataset & collect_offline_dataset helper
│       │   ├── marl/                  # Multi-Agent Reinforcement Learning (MARL)
│       │   │   ├── __init__.py        # MARL agent exports
│       │   │   ├── maddpg.py          # MADDPGAgent (Centralized Critic CTDE)
│       │   │   ├── qmix.py            # QMIXAgent & VDNAgent (Monotonic Value Factorization)
│       │   │   ├── iql.py             # IQLAgent (Independent Q-Learning with Parameter Sharing)
│       │   │   └── mappo.py           # MAPPOAgent (Multi-Agent PPO with Centralized Critic)
│       │   └── imitation/             # Imitation & Preference Learning
│       │       ├── __init__.py        # Imitation agent exports
│       │       ├── bc.py              # BehavioralCloningAgent / BCAgent
│       │       ├── dagger.py          # DAggerAgent (Dataset Aggregation with Oracle)
│       │       └── dpo.py             # DPOAgent (Direct Preference Optimization)
│       ├── trainer/
│       │   ├── __init__.py            # Trainer module exports
│       │   ├── engine.py              # Universal TrainingEngine loop coordinator
│       │   └── evaluator.py           # Deterministic policy evaluation engine
│       ├── callbacks/
│       │   ├── __init__.py            # Callback module exports
│       │   ├── base.py                # BaseCallback lifecycle hooks
│       │   ├── checkpoint.py          # Periodic & best model checkpointing
│       │   ├── evaluation.py          # Periodic evaluation callback
│       │   ├── progress.py            # Console FPS and reward progress logging
│       │   └── visualizer.py          # Matplotlib 4-panel training progress plots
│       ├── config/
│       │   ├── __init__.py            # Config module exports
│       │   ├── schema.py              # TrainConfig, AlgoConfig, NetworkConfig dataclasses
│       │   ├── builder.py             # train_config() fluent constructor
│       │   ├── presets.py             # Tuned presets (snake_dqn, snake_ppo, snake_cql, etc.)
│       │   └── validator.py           # Pre-flight environment-algorithm validation
│       └── utils/
│           ├── __init__.py            # Utility module exports
│           ├── device.py              # Auto hardware detection (cuda, mps, cpu)
│           ├── logger.py              # Formatted diagnostic logger
│           └── seeding.py             # Global deterministic RNG seeding
├── tests/                             # Comprehensive test suite (84 tests)
│   ├── __init__.py
│   ├── test_all_algorithms_training.py # End-to-end training tests for all 28 algorithm setups
│   ├── test_agents.py                 # Core agent test & factory registry coverage
│   ├── test_categorized_online.py     # Online RL unit tests (DDQN, Dueling, Dyna-Q, TRPO, etc.)
│   ├── test_categorized_offline.py    # Offline RL unit tests (BCQ, CQL, Offline DQN, Dataset)
│   ├── test_categorized_marl.py       # MARL unit tests (MADDPG, QMIX, VDN, MAPPO, MultiSnakeEnv)
│   ├── test_categorized_imitation.py  # Imitation & preference tests (BC, DAgger, DPO)
│   ├── test_buffers.py                # ReplayBuffer, RolloutBuffer, PER SumTree
│   ├── test_cli.py                    # CLI commands & argument parsing
│   ├── test_config.py                 # Serialization (YAML/JSON) & presets
│   ├── test_envs.py                   # Environment adapters, spaces, and wrappers
│   ├── test_integration.py            # End-to-end training runs
│   └── test_models.py                 # Neural networks forward passes & shapes
└── examples/                          # Executable training and demonstration scripts
    ├── train_all_algorithms.py        # Master script training ALL 28 algorithms end-to-end
    ├── train_snake_all_paradigms.py   # Multi-paradigm Snake showcase (Online, Offline, MARL, DPO)
    ├── train_snake_generalized.py     # Ported SnakeEnv training pipeline
    ├── play_snake_pygame.py           # Real-time visual Pygame human/AI demo
    ├── play_trained_agent.py          # Checkpoint playback script
    ├── train_cartpole_dqn.py          # CartPole benchmark
    ├── train_lunarlander_ppo.py       # LunarLander benchmark
    ├── train_pendulum_sac.py          # Pendulum continuous control
    └── train_custom_gridworld.py      # Custom Python class without Gym
```

---

## 🧪 Testing & Verification

Run the complete test suite:

```bash
pytest -v tests/
```

```
============================== 84 passed in 10.90s ==============================
```

All 84 unit, integration, and training tests verify:
- End-to-end model training for all 28 supported algorithm variants across Online, Offline, MARL, and Imitation/Preference learning.
- Categorized action selection and loss computations for all algorithms.
- Monotonicity constraints in QMIX and centralized critic loss in MADDPG.
- Out-of-distribution masking in BCQ and conservative penalties in CQL.
- Closed-form preference loss minimization in DPO.
- Prioritized experience replay, multi-agent buffer transitions, and Generalized Advantage Estimation (GAE).
- Checkpoint persistence (`.pth`) and round-trip model weight restoration.
- Pre-flight config validation and CLI command parsing.

---

## 📜 License

MIT License. See [LICENSE](LICENSE) for details.
