Metadata-Version: 2.4
Name: mindforge-kb
Version: 0.2.0
Summary: A local semantic memory engine that transforms AI conversation transcripts into a structured, queryable knowledge base.
Author: Accelerated Industries
License: BUSL-1.1
Project-URL: Homepage, https://github.com/AcceleratedIndustries/MindForge
Project-URL: Repository, https://github.com/AcceleratedIndustries/MindForge
Project-URL: Issues, https://github.com/AcceleratedIndustries/MindForge/issues
Project-URL: Documentation, https://github.com/AcceleratedIndustries/MindForge/blob/main/README.md
Keywords: knowledge-base,semantic-memory,llm,mcp,knowledge-graph,rag,agents
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: networkx>=3.4.2
Requires-Dist: pyyaml>=6.0
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=5.4.1; extra == "embeddings"
Requires-Dist: faiss-cpu>=1.13.2; extra == "embeddings"
Requires-Dist: numpy>=1.24.0; extra == "embeddings"
Provides-Extra: mcp
Requires-Dist: mcp>=1.27.0; extra == "mcp"
Provides-Extra: eval
Requires-Dist: pyyaml>=6.0; extra == "eval"
Provides-Extra: all
Requires-Dist: mindforge-kb[embeddings,eval,mcp]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-asyncio>=1.0; extra == "dev"
Requires-Dist: pytest-cov>=7.1.0; extra == "dev"
Requires-Dist: pyyaml>=6.0; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: mypy>=1.13; extra == "dev"
Requires-Dist: hypothesis>=6; extra == "dev"
Requires-Dist: bandit[toml]>=1.7; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: types-networkx; extra == "dev"
Dynamic: license-file

# MindForge

**Transform messy AI conversations into a structured, queryable knowledge base.**

MindForge is a local-first semantic memory engine. Feed it raw conversation transcripts and it distills them into atomic, interlinked concepts -- complete with a knowledge graph, wiki-style links, and optional vector search. Think of it as a second brain that actually organizes itself.

---

## Why MindForge?

Every conversation with an AI produces knowledge. Most of it vanishes into scroll history.

MindForge captures that knowledge and turns it into something you can **navigate, query, and build on**:

- **Concepts, not conversations** -- Each idea becomes its own clean Markdown file
- **Relationships are explicit** -- Wiki-style `[[links]]` and typed edges (`uses`, `depends_on`, `enables`)
- **Knowledge graph included** -- Visualize how concepts connect
- **Semantic search ready** -- Find concepts by meaning, not just keywords
- **LLM-powered extraction** -- Optionally use Ollama or any OpenAI-compatible API for dramatically better results
- **100% local** -- No cloud required. Your knowledge stays yours.

---

## Install

Pick the path that matches you:

### Python users (recommended)

```bash
uv tool install mindforge-kb
# or
pipx install mindforge-kb
```

With optional embeddings for semantic search:

```bash
uv tool install 'mindforge-kb[embeddings]'
```

### Homebrew (macOS / Linux)

```bash
brew tap acceleratedindustries/mindforge
brew install mindforge
```

### Single binary (no Python required)

Download the latest binary for your platform from [GitHub Releases](https://github.com/AcceleratedIndustries/MindForge/releases):

```bash
curl -L https://github.com/AcceleratedIndustries/MindForge/releases/latest/download/mindforge-macos-arm64 -o mindforge
chmod +x mindforge
./mindforge --help
```

### From source

```bash
git clone https://github.com/AcceleratedIndustries/MindForge.git
cd MindForge
pip install -e ".[dev]"
```

## Quick Start

```bash
# Run on your transcripts
mindforge ingest --input path/to/transcripts --output output

# Ask questions
mindforge query "How does semantic search work?"

# See what you've built
mindforge stats
```

**With LLM extraction** (recommended for best results):

```bash
# Using local Ollama
mindforge ingest --input transcripts/ --llm --llm-model llama3.2

# Using OpenAI
mindforge ingest --input transcripts/ --llm --llm-provider openai --llm-api-key sk-...

# Using any OpenAI-compatible endpoint (vLLM, LM Studio, Together, etc.)
mindforge ingest --input transcripts/ --llm --llm-provider openai \
    --llm-base-url http://localhost:8000 --llm-model my-model
```

---

## What It Produces

Given a directory of conversation transcripts, MindForge outputs:

### Concept Files (Markdown)

Each concept gets its own file with YAML frontmatter, a clean definition, explanation, key insights, and wiki-style links to related concepts:

```markdown
---
title: "KV Cache"
slug: "kv-cache"
tags: [transformers, inference, optimization]
confidence: 0.90
---

# KV Cache

## Definition

KV Cache is a mechanism that stores the Key and Value matrices from the
attention computation of previously processed tokens, avoiding redundant
recomputation during autoregressive generation.

## Key Insights

- Trades memory for computation -- critical trade-off in production LLM serving
- Techniques like Multi-Query Attention reduce KV cache size

## Related Concepts

- [[Vector Embeddings]]
- [[Attention Mechanism]]
```

### Knowledge Graph (JSON)

A NetworkX-powered directed graph with typed edges, exportable as JSON:

```json
{
  "nodes": [{"id": "kv-cache", "label": "KV Cache"}],
  "edges": [{"source": "rag", "target": "semantic-search", "type": "uses"}]
}
```

### Embeddings Index (Optional)

FAISS-backed vector index for semantic search over your concepts.

---

## The Pipeline

```
Transcripts --> Parse --> Chunk --> Extract --> Deduplicate --> Distill --> Link --> Graph
                                     |                                      |
                                     +-- Heuristic (patterns, headings)     +-- Wiki-links
                                     +-- LLM (Ollama / OpenAI) [optional]   +-- Typed relationships
```

| Stage | What it does |
|-------|-------------|
| **Parse** | Multi-format transcript parser (role-prefixed, heading-style, separators) |
| **Chunk** | Semantic chunking that respects paragraphs, headings, and code blocks |
| **Extract** | Identify concepts via definition patterns, heading analysis, keyword frequency -- or LLM |
| **Deduplicate** | Merge near-duplicates by slug matching and Jaccard similarity |
| **Distill** | Clean conversational fluff, extract insights, build structured definitions |
| **Link** | Detect relationships via co-occurrence, keyword overlap, and structural patterns |
| **Graph** | Build a NetworkX knowledge graph, export as JSON |
| **Index** | Optional FAISS + sentence-transformers for semantic queries |

---

## Project Structure

```
mindforge/
├── cli.py                  # CLI: ingest / query / stats
├── config.py               # Central configuration
├── pipeline.py             # 6-stage pipeline orchestrator
├── ingestion/
│   ├── parser.py           # Multi-format transcript parser
│   ├── chunker.py          # Semantic chunking
│   └── extractor.py        # Heuristic concept extraction
├── llm/
│   ├── client.py           # Ollama + OpenAI HTTP client (stdlib only)
│   ├── extractor.py        # LLM-based concept extraction
│   └── distiller.py        # LLM-aware concept distillation
├── mcp/
│   └── server.py           # MCP server (JSON-RPC over stdio)
├── distillation/
│   ├── concept.py          # Data models (Concept, Relationship, ConceptStore)
│   ├── deduplicator.py     # Similarity-based deduplication
│   ├── distiller.py        # Raw -> clean concept transformation
│   └── renderer.py         # Concept -> Markdown with frontmatter
├── linking/
│   └── linker.py           # Relationship detection + wiki-links
├── graph/
│   └── builder.py          # NetworkX graph + JSON export
├── embeddings/
│   └── index.py            # FAISS + sentence-transformers index
├── query/
│   └── engine.py           # Keyword + semantic search
└── utils/
    └── text.py             # Slugify, similarity, keyword extraction
```

---

## Installation

**Core** (no external dependencies beyond NetworkX):

```bash
pip install -e .
```

**With semantic search**:

```bash
pip install -e ".[embeddings]"
```

**For development**:

```bash
pip install -e ".[dev]"
pytest
```

---

## Transcript Formats

MindForge accepts Markdown or plain text files. Drop them in a directory and point `--input` at it.

Supported conversation formats:

- **Role-prefixed**: `User: ...` / `Assistant: ...`
- **Heading-style**: `## User` / `## Assistant`
- **Separator-based**: Turns separated by `---`
- **Plain text**: Treated as a single knowledge document

---

## Heuristic vs LLM Extraction

MindForge works in two modes:

| | Heuristic (default) | LLM-assisted (`--llm`) |
|---|---|---|
| **Speed** | Instant | Depends on model |
| **Dependencies** | None | Ollama or API |
| **Quality** | Good for well-structured transcripts | Excellent for any text |
| **Relationships** | Detected via patterns | Explicitly identified by the LLM |
| **Fallback** | N/A | Automatic fallback to heuristic |

When `--llm` is enabled, MindForge runs **both** extractors and merges the results. LLM-identified concepts take priority, and any unique heuristic findings are added in. If the LLM server is unreachable, it falls back to heuristic-only with a clear message.

---

## Relationship Types

MindForge tracks eight types of concept relationships:

| Type | Meaning | Example |
|------|---------|---------|
| `uses` | A uses B | RAG **uses** Semantic Search |
| `depends_on` | A requires B | Semantic Search **depends on** Embeddings |
| `enables` | A makes B possible | Embeddings **enables** Similarity Search |
| `improves` | A enhances B | Hybrid Search **improves** Recall |
| `part_of` | A is a component of B | HNSW **part of** Vector Database |
| `example_of` | A is an instance of B | Qdrant **example of** Vector Database |
| `contrasts_with` | A differs from B | Keyword Search **contrasts with** Semantic Search |
| `related_to` | General association | KV Cache **related to** Attention Mechanism |

---

## MCP Server (AI Agent Interface)

MindForge includes an MCP (Model Context Protocol) server that lets external AI agents query your knowledge base as a tool.

```bash
# Start the MCP server
mindforge mcp --output path/to/knowledge-base
```

**Available tools:**

| Tool | Description |
|------|-------------|
| `search` | Natural language search across all concepts |
| `get_concept` | Get full details of a concept by name or slug |
| `list_concepts` | List all concepts, optionally filtered by tag |
| `get_neighbors` | Get related concepts via the knowledge graph |
| `get_stats` | Knowledge base statistics and most central concepts |

**Claude Desktop configuration** (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "mindforge": {
      "command": "mindforge",
      "args": ["mcp", "--output", "/path/to/your/output"]
    }
  }
}
```

---

## Roadmap

- [x] **MCP server interface** -- Expose the knowledge base as a tool server for external AI agents
- [ ] **Incremental ingestion** -- Content hashing to skip already-processed transcripts
- [ ] **Concept versioning** -- Track how concepts evolve across ingestion runs
- [ ] **Confidence decay** -- Unreinforced concepts fade over time, surfacing review candidates
- [ ] **Auto-refactoring** -- Merge or split concepts as the knowledge base grows
- [ ] **Graph visualization** -- Interactive web UI for exploring the knowledge graph

---

## License

Business Source License 1.1 (BUSL-1.1). See [LICENSE](LICENSE) for details.

In short: free for any use, including commercial use, except offering MindForge as a hosted or managed service with functionality substantially similar to a paid offering by Accelerated Industries. Each release auto-converts to the Apache License 2.0 on April 21, 2028.
