Metadata-Version: 2.1
Name: lm-monitor
Version: 0.1.1
Summary: Callable algorithm primitives for cyberspace security risk monitoring.
Author: LM Monitor contributors
License: MIT License
        
        Copyright (c) 2026 LM Monitor contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Keywords: cybersecurity,model-monitoring,knowledge-distillation,open-set
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch >=2.0
Provides-Extra: examples
Requires-Dist: transformers >=4.40 ; extra == 'examples'
Requires-Dist: torchvision >=0.15 ; extra == 'examples'
Requires-Dist: pillow >=9.0 ; extra == 'examples'

# LM Monitor

A PyTorch toolkit for extending model applicability across domains, tasks, and evolving data distributions. LM Monitor provides algorithm interfaces for knowledge integration, metric adaptation, continuous performance monitoring, and corrective workflows.

English | [简体中文](README.zh-CN.md) | [TestPyPI](https://test.pypi.org/project/lm-monitor/)

## Features

| Technical direction | Technique | API |
| --- | --- | --- |
| Cross-domain knowledge integration | Divergence-aligned knowledge distillation | `abkd_loss` |
| Cross-domain knowledge integration | Parameter-efficient tensor fine-tuning | `tucka_loss` |
| Metric adaptation | Adaptive AUC optimization across domains | `openworldauc_loss` |
| Adaptive metric optimization | Distribution-agnostic mixture-of-experts fine-tuning | `dirmixe_loss` |
| Adaptive metric optimization | Distribution-robust SAM optimization | `FocalSAM`, `focal_sam_step` |
| Continuous performance monitoring and correction | Response-consistency-based anomaly monitoring | `check_prompt_trigger`, `keyword_attack_score` |
| Continuous performance monitoring and correction | Confidence-guided anomalous response correction | `interneg_score` |

## Algorithm Pipeline

### Step 1: Cross-domain Knowledge Integration

Cross-domain knowledge integration consists of two techniques: divergence-aligned knowledge distillation and parameter-efficient tensor fine-tuning. First, `abkd_loss` uses an alpha-beta divergence to align the output distributions of the reference model and the model being adapted, placing new-domain learning and existing-knowledge retention in one training objective. TuckA adapters then restrict updates to compact parameter increments, while `tucka_loss` provides label supervision for the trainable increments.

When combined, `abkd_loss` supplies the output-distribution constraint, and the TuckA adapter with `tucka_loss` performs compact parameter updates. This step produces a model with integrated cross-domain knowledge and passes the model, domain labels, and evaluation results to Step 2.

### Step 2: Metric Adaptation and Adaptive Optimization

Step 2 covers adaptive AUC optimization across domains, distribution-agnostic mixture-of-experts fine-tuning, and distribution-robust SAM optimization. For ranking tasks, `openworldauc_loss` optimizes an AUC ranking objective from positive-negative score differences, providing adaptive AUC optimization across domains. For multi-expert classification, `dirmixe_loss` incorporates training-set class frequencies and jointly optimizes expert and aggregated predictions, providing distribution-agnostic mixture-of-experts fine-tuning. For classification parameter updates, `FocalSAM` and `focal_sam_step` apply parameter-neighborhood perturbations for distribution-robust SAM optimization.

Select these techniques according to the task structure and apply them to the model from Step 1. After fine-tuning, select the model on a fixed validation set and record domain metrics, performance grouped by input condition, and reference-period `interneg_score` values for vision-language tasks. These records form the operational baselines for Steps 3 and 4.

### Step 3: Response-consistency-based Anomaly Monitoring

Step 3 implements response-consistency-based anomaly monitoring. Use the model from Step 2 on incoming data and store sample IDs, domains, time windows, prompt text, model outputs, and task results in shared evaluation records. `check_prompt_trigger` determines whether an input matches a specified condition, and `keyword_attack_score` records the associated condition metadata.

The external evaluator groups responses and performance metrics for the same task by condition and compares them with the Step 2 baselines. This identifies domains and input subsets where response consistency has changed unexpectedly. The outputs are condition-tagged records and subsets for reassessment. Vision-language data proceeds to Step 4, while text-only conditioned data proceeds directly to Step 5.

### Step 4: Confidence-guided Anomalous Response Correction

Step 4 implements confidence-guided anomalous response correction. Apply `interneg_score` to image features linked to Step 3 records to measure relative matching against known and negative semantics. Use the same encoder, text features, and temperature for reference and current windows, and attach scores to Step 3 records using sample IDs.

Compare current scores with the reference scores and threshold established in Step 2 to select candidates for reassessment. The application combines scores, condition labels, and task results to route each candidate to continued response, human review, or an alternative response. This produces confidence-guided anomalous response correction. Candidate samples, window statistics, and handling results proceed to Step 5.

### Step 5: Component Reuse and Baseline Updates

Annotate and reassess samples collected in Steps 3 and 4, then map each result back to the corresponding technique in the feature overview:

| Reassessment finding | Corrective or update action | Corresponding technique and API |
| --- | --- | --- |
| Gaps in new-domain knowledge coverage | Add domain data and constrain further learning with a reference model | Divergence-aligned knowledge distillation, `abkd_loss` |
| A domain capability requires a localized update | Freeze base parameters and update compact parameter increments | Parameter-efficient tensor fine-tuning, `tucka_loss` |
| Cross-domain positive-negative ranking declines | Adjust ranking data, domain sampling, and sample weights | Adaptive AUC optimization across domains, `openworldauc_loss` |
| Class-distribution changes affect classification | Refresh training-set class statistics and fine-tune expert objectives | Distribution-agnostic mixture-of-experts fine-tuning, `dirmixe_loss` |
| Generalization after classification updates is insufficient | Repeat parameter-neighborhood optimization during additional training | Distribution-robust SAM optimization, `FocalSAM`, `focal_sam_step` |
| Response performance changes under a specific input condition | Retain condition labels and update grouped-performance baselines | Response-consistency-based anomaly monitoring, `check_prompt_trigger`, `keyword_attack_score` |
| Candidate samples require different response handling | Update response-handling rules using scores and reassessment results | Confidence-guided anomalous response correction, `interneg_score` |

After corrective training, compare performance on existing and new data using the same domain definitions and evaluation rules. Register the validated model and its baselines in Step 2, then continue monitoring in Steps 3 and 4. When no model update is required, retain the current model and update the relevant grouped records, score statistics, or response-handling results.

## Requirements and Installation

Python 3.9 or later and PyTorch 2.0 or later are required. The Conda environment uses Python 3.10, with Python dependencies installed from `requirements.txt`.

### Create the Conda Environment

`environment.yml`:

```yaml
name: lm-monitor
channels:
  - defaults
dependencies:
  - python=3.10
  - pip
```

Run these commands from the directory containing the configuration files:

```bash
conda env create -f environment.yml
conda activate lm-monitor
```

### Install Dependencies

`requirements.txt`:

```text
torch>=2.0
```

```bash
python -m pip install -r requirements.txt
```

### Install LM Monitor

Install version 0.1.0 from TestPyPI:

```bash
python -m pip install --index-url https://test.pypi.org/simple/ --no-deps lm-monitor==0.1.0
```

Or install from a local wheel:

```bash
python -m pip install ./lm_monitor-0.1.0-py3-none-any.whl
```

The package name is `lm-monitor`; the Python import name is `lm_monitor`.

## Quickstart

Store training data as a dictionary containing `features` and `labels`:

| Field | Type and shape | Description |
| --- | --- | --- |
| `features` | Floating-point Tensor `[N, D]` | Sample features extracted with a fixed encoder and preprocessing procedure |
| `labels` | Integer Tensor `[N]` | Contiguous class IDs `0 ... C-1` for multiclass tasks; `0/1` for binary tasks |

Save features with `torch.save({"features": features.cpu(), "labels": labels.cpu()}, path)`. Supply data files and models separately; they are not included in the wheel. Use a separate validation set for hyperparameter selection and performance evaluation.

### 1. ABKD: Divergence-Aligned Knowledge Distillation

ABKD implements divergence-aligned knowledge distillation in Step 1 and returns to additional training when Step 5 confirms gaps in new-domain knowledge coverage.

`abkd_loss` aligns teacher and student output distributions with a configurable α-β divergence and combines it with ground-truth supervision. `alpha` and `beta` control the divergence, `temperature` controls distribution smoothing, and `kd_ratio` controls the distillation weight.

Prepare `data/train_features.pt` and a teacher model at `models/teacher.pt`. The model file must contain the `state_dict` of a `Linear(D, 256) → ReLU → Linear(256, C)` network.

```python
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from lm_monitor import abkd_loss

# Use CUDA when available and keep the model and each batch on the same device.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load local features and labels into CPU memory before moving individual batches.
data = torch.load("data/train_features.pt", map_location="cpu", weights_only=True)
# features has shape [N, D] with float values; labels contains N integer class IDs.
features, labels = data["features"].float(), data["labels"].long()
num_features = features.shape[1]
num_classes = int(labels.max().item()) + 1
# Require contiguous class IDs starting at zero to match classifier output indices.
assert torch.equal(torch.unique(labels), torch.arange(num_classes))
# Shuffle observations and load batches of up to 64 samples.
loader = DataLoader(TensorDataset(features, labels), batch_size=64, shuffle=True)

# Teacher and student share input dimensions and class order; hidden width controls model size.
def make_classifier(hidden_dim):
    return nn.Sequential(
        nn.Linear(num_features, hidden_dim),
        nn.ReLU(),
        nn.Linear(hidden_dim, num_classes),
    ).to(device)

# Build the teacher and load its trained parameters using the matching architecture.
teacher = make_classifier(256)
teacher.load_state_dict(torch.load(
    "models/teacher.pt", map_location=device, weights_only=True
))
# Keep the teacher in evaluation mode and freeze it to provide target distributions only.
teacher.eval()
teacher.requires_grad_(False)

# Use a narrower hidden layer for the student and optimize only student parameters.
student = make_classifier(64)
student.train()
optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3)

total_loss = 0.0
for inputs, targets in loader:
    # Move features and labels to the model device before computing predictions and loss.
    inputs, targets = inputs.to(device), targets.to(device)
    # Run the teacher without an autograd graph to reduce memory use during distillation.
    with torch.no_grad():
        teacher_logits = teacher(inputs)

    # Clear gradients from the previous batch to prevent unintended accumulation.
    optimizer.zero_grad(set_to_none=True)
    # Combine output-distribution alignment with label supervision using matching class meanings.
    loss = abkd_loss(
        student(inputs), teacher_logits, targets,
        # alpha and beta configure divergence, temperature smooths distributions, and kd_ratio weights distillation.
        alpha=0.5, beta=0.5, temperature=2.0, kd_ratio=0.8,
    )
    # Compute gradients of the current loss with respect to trainable parameters.
    loss.backward()
    # Update the parameters managed by the optimizer using their gradients.
    optimizer.step()
    # Weight batch losses by sample count so a smaller final batch does not distort the mean.
    total_loss += loss.item() * inputs.size(0)

# Report the sample-weighted training loss for one epoch; evaluate metrics on held-out data.
print({"train_loss": total_loss / len(loader.dataset)})
```

`kd_ratio=0.8` assigns 80% of the objective to distillation and 20% to cross-entropy; `kd_ratio=1.0` uses distillation alone. Teacher and student outputs must share the same dimensions and class meanings. Token-level distillation also requires vocabulary, token, and position alignment; label `-100` can mask positions.

### 2. TuckA: Parameter-Efficient Tensor Fine-Tuning

TuckA implements parameter-efficient tensor fine-tuning in Step 1 and can be combined with the ABKD output-distribution constraint. Localized corrections in Step 5 reuse the same adapter structure.

Compact parameter fine-tuning freezes the base model and trains only adaptation increments. `tucka_loss` provides the supervised classification objective; the external model defines the tensor adapter structure and parameter sharing.

Prepare `data/train_features.pt` and a trained `nn.Linear(D, C)` classifier model at `models/base_head.pt`.

```python
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from lm_monitor import tucka_loss

# Use CUDA when available and keep the model and each batch on the same device.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load local features and labels into CPU memory before moving individual batches.
data = torch.load("data/train_features.pt", map_location="cpu", weights_only=True)
# features has shape [N, D] with float values; labels contains N integer class IDs.
features, labels = data["features"].float(), data["labels"].long()
num_features = features.shape[1]
num_classes = int(labels.max().item()) + 1
# Require contiguous class IDs starting at zero to match classifier output indices.
assert torch.equal(torch.unique(labels), torch.arange(num_classes))
# Shuffle observations and load batches of up to 64 samples.
loader = DataLoader(TensorDataset(features, labels), batch_size=64, shuffle=True)

# Freeze the base classifier and learn a domain update through an added low-rank branch.
class LowRankHead(nn.Module):
    def __init__(self, input_dim, output_dim, rank):
        super().__init__()
        self.base = nn.Linear(input_dim, output_dim)
        # Project inputs to rank dimensions, then map them to class scores through up.
        self.down = nn.Linear(input_dim, rank, bias=False)
        self.up = nn.Linear(rank, output_dim, bias=False)
        # Initialize the update output to zero so initial predictions match the base model.
        nn.init.zeros_(self.up.weight)
        # Disable gradients for base parameters so training focuses on the update branch.
        self.base.requires_grad_(False)

    def forward(self, x):
        # Add base predictions and the low-rank update to obtain adapted class scores.
        return self.base(x) + self.up(self.down(x))

# Keep rank within input and output dimensions; select its value on validation data.
rank = min(4, num_features, num_classes)
model = LowRankHead(num_features, num_classes, rank).to(device)
# Load only the base classifier parameters, leaving the low-rank branch separately initialized.
model.base.load_state_dict(torch.load(
    "models/base_head.pt", map_location=device, weights_only=True
))
# Pass only adapter parameters with requires_grad=True to the optimizer.
optimizer = torch.optim.AdamW(
    [p for p in model.parameters() if p.requires_grad], lr=1e-3
)

model.train()
for inputs, targets in loader:
    # Move features and labels to the model device before computing predictions and loss.
    inputs, targets = inputs.to(device), targets.to(device)
    # Clear gradients from the previous batch to prevent unintended accumulation.
    optimizer.zero_grad(set_to_none=True)
    # Supervise adapted predictions with class labels; only unfrozen parameters receive updates.
    loss = tucka_loss(model(inputs), targets)
    # Compute gradients of the current loss with respect to trainable parameters.
    loss.backward()
    # Update the parameters managed by the optimizer using their gradients.
    optimizer.step()

# Count trainable update parameters, excluding the frozen base parameters.
print({"trainable_parameters": sum(
    p.numel() for p in model.parameters() if p.requires_grad
)})
```

`tucka_loss` currently provides a cross-entropy objective. `LowRankHead` uses a low-rank matrix factorization; TuckA tensor factorization requires an external adapter implementation. The trainable parameter count depends on model dimensions, rank, and sharing structure.

### 3. OpenworldAUC: Adaptive AUC Optimization Across Domains

OpenworldAUC implements adaptive AUC optimization across domains in Step 2. When Step 5 confirms a decline in ranking performance, it receives reassessed samples and adjusted weights.

`openworldauc_loss` constructs a differentiable objective from the relative ranking of positive and negative samples and supports weights for both groups. It can be integrated into ranking-oriented training, with data organization and weighting controlling sample contributions across domains.

Prepare `data/binary_features.pt` with binary labels `0/1`. Stratified sampling ensures that each training batch contains both positive and negative samples.

```python
import torch
from torch import nn
from lm_monitor import openworldauc_loss

# Use CUDA when available and keep the model and each batch on the same device.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load local features and labels into CPU memory before moving individual batches.
data = torch.load("data/binary_features.pt", map_location="cpu", weights_only=True)
# features has shape [N, D] with float values; labels contains N integer class IDs.
features, labels = data["features"].float(), data["labels"].long()
# Check that each feature vector has one label and that the tensor dimensions match.
assert features.ndim == 2 and labels.shape == (features.size(0),)
# Require both label 0 and label 1 observations for pairwise ranking.
assert torch.equal(torch.unique(labels), torch.tensor([0, 1]))

# Store positive and negative indices separately to construct batches with both classes.
positive = torch.where(labels == 1)[0]
negative = torch.where(labels == 0)[0]
# Produce one real-valued score per input for relative positive-negative ranking.
ranker = nn.Linear(features.size(1), 1).to(device)
optimizer = torch.optim.AdamW(ranker.parameters(), lr=1e-3)
ranker.train()

# Perform 100 updates by sampling existing observations without generating features.
for step in range(100):
    # Sample 32 observations per class with replacement to keep both labels in every batch.
    pos_idx = positive[torch.randint(positive.numel(), (32,))]
    neg_idx = negative[torch.randint(negative.numel(), (32,))]
    # Concatenate class indices and use the same indices to select features and labels.
    idx = torch.cat([pos_idx, neg_idx])
    inputs, targets = features[idx].to(device), labels[idx].to(device)

    # Clear gradients from the previous batch to prevent unintended accumulation.
    optimizer.zero_grad(set_to_none=True)
    # Keep raw scores without sigmoid; squeeze removes the trailing singleton dimension.
    scores = ranker(inputs).squeeze(-1)
    # Compute a differentiable ranking loss from positive-negative score differences.
    loss = openworldauc_loss(scores, targets)
    # Compute gradients of the current loss with respect to trainable parameters.
    loss.backward()
    # Update the parameters managed by the optimizer using their gradients.
    optimizer.step()

# Report the final update loss; compute ROC-AUC separately.
print({"ranking_train_loss": loss.item()})
```

The return value is a ranking surrogate loss. Compute ROC-AUC separately from labels and predicted scores on a validation set. Optional `pos_weight` and `neg_weight` arguments correspond to the positive and negative samples within the batch. To target a particular region of a performance curve, configure sample selection or weighting in the trainer.

### 4. DirMixE: Distribution-Agnostic Mixture-of-Experts Fine-Tuning

DirMixE implements distribution-agnostic mixture-of-experts fine-tuning in Step 2. After Step 5 confirms a class-distribution change, updated training-set statistics feed the next fine-tuning cycle.

`dirmixe_loss` incorporates class frequencies into multi-expert training and accounts for both aggregated predictions and per-expert classification losses. It supports fine-tuning under different class distributions. The external model and trainer configure the expert architecture and aggregation rule.

Load training data from `data/train_features.pt`. Three expert classifiers share the input features, and class frequencies are computed from the full training set.

```python
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from lm_monitor import dirmixe_loss

# Use CUDA when available and keep the model and each batch on the same device.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load local features and labels into CPU memory before moving individual batches.
data = torch.load("data/train_features.pt", map_location="cpu", weights_only=True)
# features has shape [N, D] with float values; labels contains N integer class IDs.
features, labels = data["features"].float(), data["labels"].long()
num_features = features.shape[1]
num_classes = int(labels.max().item()) + 1
# Require contiguous class IDs starting at zero to match classifier output indices.
assert torch.equal(torch.unique(labels), torch.arange(num_classes))
# Shuffle observations and load batches of up to 64 samples.
loader = DataLoader(TensorDataset(features, labels), batch_size=64, shuffle=True)

# Three experts receive the same features while maintaining separate classifier parameters.
experts = nn.ModuleList([
    nn.Linear(num_features, num_classes) for _ in range(3)
]).to(device)
# Compute class frequencies from the full training set, not from a batch or validation labels.
class_counts = torch.bincount(labels, minlength=num_classes).to(device)
# Use one optimizer for all experts to optimize their joint objective.
optimizer = torch.optim.AdamW(experts.parameters(), lr=1e-3)

experts.train()
for inputs, targets in loader:
    # Move features and labels to the model device before computing predictions and loss.
    inputs, targets = inputs.to(device), targets.to(device)
    # Clear gradients from the previous batch to prevent unintended accumulation.
    optimizer.zero_grad(set_to_none=True)
    # Stack outputs as [B, E, C] for batch size, number of experts, and number of classes.
    expert_logits = torch.stack([head(inputs) for head in experts], dim=1)
    # Pass expert outputs, labels, and class frequencies to the distribution-adaptation objective.
    loss = dirmixe_loss(expert_logits, targets, class_counts)
    # Compute gradients of the current loss with respect to trainable parameters.
    loss.backward()
    # Update the parameters managed by the optimizer using their gradients.
    optimizer.step()

# Switch experts to evaluation mode without further parameter updates.
experts.eval()
with torch.no_grad():
    # Compute predictions on a training batch; use held-out data for performance evaluation.
    inputs, _ = next(iter(loader))
    logits = torch.stack([head(inputs.to(device)) for head in experts], dim=1)
    # Adjust scores using training class frequencies; clamp_min prevents taking log of zero.
    adjusted_logits = logits - class_counts.clamp_min(1).float().log()
    # Average scores across experts, then select the class with the highest score.
    predictions = adjusted_logits.mean(dim=1).argmax(dim=-1)
print(predictions.cpu().tolist())
```

Keep expert aggregation and class-adjustment rules fixed when comparing models, and evaluate them on the same validation data.

### 5. Focal-SAM: Distribution-Robust SAM Optimization

Focal-SAM implements distribution-robust SAM optimization in Step 2 and can also support additional training in Step 5. The updated model receives new grouped-performance and semantic-score baselines.

Focal-SAM introduces parameter-neighborhood perturbations into classification training and uses multiple forward and backward stages to compute updates. `FocalSAM` provides the optimizer interface, while `focal_sam_step` encapsulates a batch update and replaces the standard update step in a classification loop.

Load training data from `data/train_features.pt`. The classifier consists of two linear layers and ReLU, and `focal_sam_step` performs a three-stage update for each batch.

```python
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from lm_monitor import FocalSAM, focal_sam_step

# Use CUDA when available and keep the model and each batch on the same device.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load local features and labels into CPU memory before moving individual batches.
data = torch.load("data/train_features.pt", map_location="cpu", weights_only=True)
# features has shape [N, D] with float values; labels contains N integer class IDs.
features, labels = data["features"].float(), data["labels"].long()
num_classes = int(labels.max().item()) + 1
# Require contiguous class IDs starting at zero to match classifier output indices.
assert torch.equal(torch.unique(labels), torch.arange(num_classes))
# Shuffle observations and load batches of up to 64 samples.
loader = DataLoader(TensorDataset(features, labels), batch_size=64, shuffle=True)

# Use a classifier without Dropout or BatchNorm to keep repeated forward passes consistent.
model = nn.Sequential(
    nn.Linear(features.shape[1], 64),
    nn.ReLU(),
    nn.Linear(64, num_classes),
).to(device)
# Use SGD as the base optimizer; rho controls the parameter-neighborhood perturbation radius.
optimizer = FocalSAM(
    model.parameters(), torch.optim.SGD,
    lr=0.01, momentum=0.9, rho=0.05,
)

model.train()
total_loss = 0.0
for inputs, targets in loader:
    # Move features and labels to the model device before computing predictions and loss.
    inputs, targets = inputs.to(device), targets.to(device)
    # Clear the previous batch gradients before delegating the full update to focal_sam_step.
    optimizer.zero_grad()
    # Run all three forward/backward stages and update parameters without an extra backward or step call.
    result = focal_sam_step(model, optimizer, inputs, targets)
    # Weight batch losses by sample count so a smaller final batch does not distort the mean.
    total_loss += result["loss"].item() * inputs.size(0)

# Report the returned mean training loss; evaluate updated performance with a fresh forward pass.
print({"train_loss": total_loss / len(loader.dataset)})
```

Run a fresh forward pass to evaluate the updated model. For networks with stochastic or stateful layers, manage their behavior consistently across multiple forward passes.

### 6. BlackMirror: Response-Consistency-Based Anomaly Monitoring

BlackMirror provides condition labels for response-consistency-based anomaly monitoring in Step 3, linking model performance to domains and input subsets. Tagged records proceed to Step 4 or Step 5.

`check_prompt_trigger` and `keyword_attack_score` provide deterministic text-condition matching and labels. These labels can be associated with predictions recorded by an external evaluator to analyze response behavior under different input conditions. The interfaces produce condition labels; the evaluator computes task metrics such as correctness and response consistency.

Evaluation records contain `prompt` and `correct` fields for the input text and prediction correctness. Group records by domain marker, then calculate sample counts and accuracy for each group.

```python
from lm_monitor import check_prompt_trigger, keyword_attack_score

# prompt stores the input text; correct stores prediction correctness from an external evaluator.
records = [
    {"prompt": "[DOMAIN:finance] Summarize the quarterly report.", "correct": True},
    {"prompt": "[DOMAIN:finance] Explain the revenue change.", "correct": False},
    {"prompt": "[DOMAIN:science] Summarize the experiment.", "correct": True},
]
# Define an input condition with a domain marker, task marker, or other fixed text.
trigger = "[DOMAIN:finance]"
# Track sample counts and correct predictions separately for matching and nonmatching inputs.
groups = {
    True: {"count": 0, "correct": 0},
    False: {"count": 0, "correct": 0},
}

for record in records:
    # Match against input text without using prediction outcomes to assign groups.
    matched = check_prompt_trigger(record["prompt"], trigger)
    # Attach condition metadata to the original record for linking scores and reassessment results.
    record["condition_metadata"] = keyword_attack_score(
        record["prompt"], trigger, target="finance-subset"
    )
    # Accumulate group counts and convert boolean correctness to zero or one.
    groups[matched]["count"] += 1
    groups[matched]["correct"] += int(record["correct"])

# Compute group accuracy and retain counts to contextualize fluctuations in small groups.
for matched, values in groups.items():
    count = values["count"]
    print({
        "condition_matched": matched,
        "count": count,
        # Return None for empty groups instead of reporting zero accuracy.
        "accuracy": values["correct"] / count if count else None,
    })
```

Add time-window and task labels to compare group performance across periods and route subsets requiring reassessment into the correction workflow.

### 7. InterNeg: Confidence-Guided Anomalous Response Correction

InterNeg supplies scores for confidence-guided anomalous response correction in Step 4, uses the Step 2 baseline to select candidates, and passes scores to the application response-handling workflow.

`interneg_score` scores an input image by its relative matching to known and negative text semantics. It provides a confidence signal for analyzing semantic coverage and informing response handling. Comparing current scores with the reference period identifies candidates for reassessment, response correction, or targeted additional training.

Prepare `data/monitor_features.pt` with four floating-point tensors. Extract all features with the same vision-language encoder and apply the same preprocessing to both image groups.

| Field | Shape | Description |
| --- | --- | --- |
| `reference_images` | `[N_ref, D]` | Image features from in-domain validation data |
| `current_images` | `[N_cur, D]` | Image features from the current data window |
| `id_text` | `[K, D]` | Text features for known semantics |
| `negative_text` | `[M, D]` | Text features for negative semantics |

```python
import torch
from lm_monitor import interneg_score

# Load reference and current image features plus two text banks sharing the embedding dimension.
bank = torch.load("data/monitor_features.pt", map_location="cpu", weights_only=True)
# Known text describes covered semantics; negative text provides a relative matching reference.
known_text = bank["id_text"].float()
negative_text = bank["negative_text"].float()

# Score for monitoring without computing gradients or updating model parameters.
@torch.no_grad()
def score_window(image_features):
    # Process up to 256 image features per batch and concatenate scores in input order.
    return torch.cat([
        interneg_score(
            # Use the same text banks and temperature for reference and current windows to keep scores comparable.
            batch.float(), known_text, negative_text, temperature=0.1
        )
        for batch in image_features.split(256)
    ])

# Score both windows separately and use the reference window to establish the baseline.
reference_scores = score_window(bank["reference_images"])
current_scores = score_window(bank["current_images"])

# Set the threshold from the reference 95th percentile without fitting it to the current window.
threshold = torch.quantile(reference_scores, 0.95)
# Flag samples above the reference threshold for reassessment with labels and task metrics.
needs_review = current_scores > threshold
# Retain current-window indices for retrieving original inputs and condition labels.
review_indices = torch.where(needs_review)[0]

# Report reference and current exceedance rates with candidate counts for window comparisons.
print({
    "threshold": threshold.item(),
    "reference_exceedance": (reference_scores > threshold).float().mean().item(),
    "current_exceedance": needs_review.float().mean().item(),
    "current_mean_score": current_scores.mean().item(),
    "review_count": review_indices.numel(),
})
# Pass candidate indices to application review, response handling, or additional training.
print({"review_indices": review_indices.tolist()})
```

Candidate indices link scores to the original inputs. The application performs response correction based on reassessment results. Analyze score changes together with ground-truth labels and task metrics, and rebuild the reference baseline after changing the negative text set or temperature.

## License

LM Monitor is released under the MIT License.
