Metadata-Version: 2.4
Name: repo-knowledge-graph
Version: 0.2.0
Summary: Clone a GitHub repo and emit an RDF knowledge graph plus AI context graph JSON.
Author: repo-knowledge-graph contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/your-org/repo-knowledge-graph
Project-URL: Repository, https://github.com/your-org/repo-knowledge-graph
Project-URL: Issues, https://github.com/your-org/repo-knowledge-graph/issues
Keywords: github,knowledge-graph,rag,code-intelligence,ai-coding
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Version Control :: Git
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rdflib>=7.0.0
Provides-Extra: dev
Requires-Dist: build>=1.2.0; extra == "dev"
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: ruff>=0.3.0; extra == "dev"
Requires-Dist: twine>=5.1.0; extra == "dev"
Provides-Extra: experiment
Requires-Dist: openai>=1.40.0; extra == "experiment"
Requires-Dist: tiktoken>=0.7.0; extra == "experiment"
Provides-Extra: agents
Requires-Dist: openai>=1.40.0; extra == "agents"
Provides-Extra: embeddings
Requires-Dist: openai>=1.40.0; extra == "embeddings"
Provides-Extra: embeddings-local
Requires-Dist: sentence-transformers>=2.2.0; extra == "embeddings-local"
Provides-Extra: all
Requires-Dist: openai>=1.40.0; extra == "all"
Requires-Dist: tiktoken>=0.7.0; extra == "all"
Requires-Dist: sentence-transformers>=2.2.0; extra == "all"
Dynamic: license-file

# repo-knowledge-graph

> **Turn any GitHub repository into a structured knowledge graph — in seconds — so AI tools, RAG pipelines, and onboarding assistants can understand your codebase without reading every file.**

---

## Why this accelerator exists

Every AI coding tool, onboarding assistant, or RAG system faces the same cold-start problem: it needs to **understand a codebase before it can help with it**. The conventional approach is to dump raw files into a context window, hope the model navigates the noise, and repeat for every new repo or new engineer.

`repo-knowledge-graph` eliminates that cold start. It takes a single GitHub URL and produces a **machine-readable map of the repository** — files, languages, classes, functions, imports, dependencies, and directory structure — in both a **semantic RDF graph** and a **LLM-optimised JSON graph**. The outputs are immediately consumable by:

- **AI agents and Copilots** (via `context_chunks.jsonl` or the Python API)
- **RAG / vector search pipelines** (each JSONL chunk is self-contained and has a stable `id`)
- **Graph databases** (Turtle / JSON-LD loads directly into any SPARQL / RDF store)
- **Code intelligence dashboards** (the manifest is a structured file index with language counts and dependency inventory)
- **Onboarding docs generators** (the README snippet + dependency nodes give instant project overview)

---

## Value and time savings

| Activity | Without this accelerator | With this accelerator |
|----------|--------------------------|----------------------|
| **AI context prep for a new repo** | Engineer manually curates files, writes summaries — 2–4 hours | Run one CLI command — **under 30 seconds** |
| **RAG ingestion pipeline** | Custom crawlers, chunkers, and deduplication — 1–2 weeks dev | Drop `context_chunks.jsonl` straight into your vector DB loader — **ready immediately** |
| **Onboarding a new developer** | "Read the code, ask Slack" — days to get mental model | Share `manifest.json` + repo graph; structured summary of every module and dependency — **minutes** |
| **Dependency audit / compliance** | Manual inspection of each `package.json` / `composer.json` — hours | All declared packages extracted as typed edges in the graph — **automatic** |
| **Reproducible AI answers** | AI tool has no idea which version of the code it answered about | Every graph is stamped with **commit SHA + branch** — fully reproducible |
| **Multi-language codebases** | Different scripts needed per language | Single command handles Python, PHP, JS, TS, Go, Java, Rust, and more |

### Dynamic, need-sized context windows (especially for large repos)

You do **not** need the entire repository in the model’s context. After generation, `context_graph.json`, `manifest.json`, and `context_chunks.jsonl` act as a **retrieval index plus graph**: at inference time you can assemble **only** the chunks that match the current task (feature area, package, path, or developer question), instead of loading millions of lines into one prompt.

**In short:** That model is correct — **dynamic, need-sized context windows at inference time**, not “everything in context” — **as long as** you treat the outputs as a retrieval index + graph, **plan re-generation or incremental updates** when you need freshness against the latest commit, and **plug in a retrieval/orchestration layer** on top (vector search, graph traversal, or an agent that fetches nodes by id). The graphs themselves are pinned to the scanned **commit SHA**; query-time selection is dynamic; repo freshness is a separate pipeline concern.

---

## What it produces

Running `repo-kg <github-url>` on [an-prog/sample_billing_app](https://github.com/an-prog/sample_billing_app) — a PHP/JS billing system — produces graph artifacts and, optionally, a wiki in under 10 seconds:

```
kg_out/
├── knowledge_graph.ttl       ← RDF/Turtle (semantic graph, triple stores, SPARQL)
├── knowledge_graph.jsonld    ← JSON-LD (linked data, semantic web tooling)
├── context_graph.json        ← Full LLM context graph  (nodes + typed edges)
├── manifest.json             ← Lightweight file index  (RAG, dashboards, search)
├── context_chunks.jsonl      ← Per-entity JSONL chunks (one per line, RAG-ready)
└── wiki/                     ← Karpathy-style markdown knowledge base
  ├── index.md
  ├── overview.md
  ├── schema.md
  ├── log.md
  ├── concepts/
  ├── entities/
  └── queries/
```

**Sample run output:**

```
Wrote outputs to kg_out/
  Graph: 3,165 nodes, 3,164 edges
  Files: 2,576 (PHP/JS/JSON/YAML/…)
  Dependencies: 2 manifest(s)
  Wiki pages: 143 (kg_out/wiki)
  Commit: 68fcb3787f2dcbb42631c177e77042c5acc675ea  branch: master
```

### The Knowledge Graph (`knowledge_graph.ttl` / `.jsonld`)

The RDF graph uses stable, widely-recognised vocabularies:

| Vocabulary | Used for |
|------------|----------|
| [W3C PROV-O](https://www.w3.org/TR/prov-o/) | Provenance: `wasDerivedFrom`, derivation chain |
| [Schema.org](https://schema.org/SoftwareSourceCode) | `SoftwareSourceCode`, `codeRepository`, `name` |
| `code:` (custom, open namespace) | `Repository`, `SourceFile`, `ClassDeclaration`, `FunctionDeclaration`, `dependsOn`, `declares`, `contains`, `gitCommitSHA`, `gitBranch` |

This makes it **interoperable** with any SPARQL endpoint, RDF triplestore (Apache Jena, Oxigraph, GraphDB), or linked-data pipeline — no custom schema lock-in.

### The Context Graph (`context_graph.json`)

A plain JSON document with `nodes` and `edges` arrays, designed for LLM-friendly consumption:

```json
{
  "meta": {
    "schema_version": 2,
    "source_url": "https://github.com/an-prog/sample_billing_app",
    "commit_sha": "68fcb3787f2dcbb42631c177e77042c5acc675ea",
    "branch": "master",
    "github": { "owner": "an-prog", "repo": "sample_billing_app" }
  },
  "nodes": [
    {
      "id": "repo:an-prog/sample_billing_app",
      "type": "repository",
      "summary": "GitHub repository an-prog/sample_billing_app. README: BillRun is Open-Source Billing..."
    },
    {
      "id": "dep:composer:mongodb/mongodb",
      "type": "dependency",
      "name": "mongodb/mongodb",
      "ecosystem": "composer",
      "version_constraint": "^1.4",
      "kind": "runtime"
    },
    {
      "id": "file:application/controllers/Api.php",
      "type": "source_file",
      "language_hint": ".php",
      "summary": "Source file `application/controllers/Api.php` (.php)."
    }
  ],
  "edges": [
    { "source": "repo:an-prog/sample_billing_app", "target": "dep:composer:mongodb/mongodb", "relation": "dependsOn" },
    { "source": "repo:an-prog/sample_billing_app", "target": "file:application/controllers/Api.php", "relation": "contains" }
  ]
}
```

**Edge types:**

| `relation` | Meaning |
|------------|---------|
| `contains` | repository/directory → file |
| `declares` | file → class/function/method |
| `imports` | Python module → imported name (proper node, not a dangling string) |
| `dependsOn` | repository → dependency set (manifest) |
| `dep_has_package` | dependency set → individual package |

### The Manifest (`manifest.json`)

A **lightweight file index** — just the repository node, list of files with their language, and stats. Ideal for dashboards, search indexes, and initial context loads where you don't need the full graph.

```json
{
  "stats": {
    "total_nodes": 3165,
    "total_edges": 3164,
    "file_count": 3152,
    "symbol_count": 0,
    "dependency_count": 10
  }
}
```

### JSONL Chunks (`context_chunks.jsonl`)

**One JSON object per line**. Each chunk is a self-contained, retrievable unit:

```jsonl
{"id": "repo:an-prog/sample_billing_app", "type": "repository", "summary": "GitHub repository ...", "meta": {...}}
{"id": "file:application/controllers/Api.php", "type": "source_file", "path": "application/controllers/Api.php", "language": ".php", "summary": "...", "meta": {...}}
```

This is the **primary RAG ingestion format**: load each line as a document into your vector database with `id` as the key. No custom parsing required.

---

## Key features

- **Single command, any public GitHub repo** — HTTPS or SSH URL, optional branch/tag
- **Commit-pinned provenance** — every graph includes `commit_sha`, `short_sha`, `branch`, and `tag` so AI answers are reproducible and auditable
- **Multi-language file graph** — Python (deep: classes/functions/imports), PHP, JavaScript, TypeScript, Go, Java, Kotlin, Rust, C/C++ at file granularity; PHP/JS/TS symbols with optional tree-sitter
- **Dependency extraction** — automatically parses `composer.json`, `package.json`, `pyproject.toml`, `requirements*.txt`, `go.mod`, `pom.xml` into typed `dependsOn` edges
- **Five output formats** — RDF Turtle, JSON-LD, full context graph, lightweight manifest, RAG-ready JSONL chunks
- **Karpathy-style wiki generation** — add `--wiki` to generate a structured markdown wiki for architecture memory and AI retrieval
- **Wiki lifecycle operations** — `repo-kg-wiki init | ingest | query | lint` for maintaining and validating wiki quality
- **Agentic orchestration support** — `KGBuildAgent` can run end-to-end graph + wiki builds with optional LLM enrichment hooks
- **README-boosted summaries** — the top-level README is embedded in the repository node summary for instant project context
- **Configurable scan policy** — YAML/JSON config for include/exclude globs, max file size, vendor/node_modules toggles
- **Token budget control** — `--max-chunk-tokens N` truncates JSONL chunk summaries to fit LLM context windows
- **Python API** — `build_graph(url, options)` for programmatic use in notebooks, CI, or agent tool wrappers
- **100% open source** — RDFLib (BSD-3), system `git`, stdlib only; no paid APIs, no vendor lock-in

---

## Installation

**Requirements:** Python 3.10+, `git` on PATH.

```bash
pip install repo-knowledge-graph
```

Install with OpenAI-powered agent extras:

```bash
pip install "repo-knowledge-graph[agents]"
```

Or from source:

```bash
git clone https://github.com/your-org/repo-knowledge-graph
cd repo-knowledge-graph
pip install -e ".[dev]"
```

**Optional: tree-sitter for PHP/JS/TS symbol extraction**

```bash
pip install tree-sitter tree-sitter-php tree-sitter-javascript tree-sitter-typescript
```

Without these, PHP/JS/TS files still appear as file-level nodes in the graph.

## Quickstart for testers

The productization step for external evaluation is the benchmark bundle command. It creates one canonical folder with KG artifacts, packed prompt contexts, a JSON result file, and a shareable markdown summary.

```bash
pip install repo-knowledge-graph

cat > feature_spec.md <<'EOF'
Add validation for stock location capacity and cover it with tests.
EOF

repo-kg benchmark "https://github.com/owner/repo" \
  --feature-file feature_spec.md \
  --output-dir benchmark_out
```

What you get:

```text
benchmark_out/
├── benchmark_result.json
├── benchmark_result.schema.json
├── kg/
├── prompts/
└── reports/
```

Use [docs/user_guide.md](docs/user_guide.md) as the handout for testers. The canonical result schema is packaged at [src/repo_knowledge_graph/schemas/benchmark_result.schema.json](src/repo_knowledge_graph/schemas/benchmark_result.schema.json).

---

## Usage

### CLI

```bash
# Basic: clone, scan, and write all outputs to ./kg_out
repo-kg "https://github.com/owner/repo"

# Custom output directory
repo-kg "https://github.com/owner/repo" -o ./my_output

# Specific branch
repo-kg "https://github.com/owner/repo/tree/develop"

# SSH URL
repo-kg "git@github.com:owner/repo.git"

# Skip cloning — scan a local repo
repo-kg "https://github.com/owner/repo" --clone-dir /path/to/local/clone

# Include directory nodes + truncate chunks at ~1000 tokens
repo-kg "https://github.com/owner/repo" --include-dirs --max-chunk-tokens 1000

# Generate wiki along with graph artifacts
repo-kg "https://github.com/owner/repo" --wiki

# Customize wiki output behavior
repo-kg "https://github.com/owner/repo" --wiki --wiki-dir-name wiki_docs --wiki-overwrite-existing

# Custom scan policy from a config file
repo-kg "https://github.com/owner/repo" --config scan_config.yaml

# Build a tester-ready benchmark bundle
repo-kg benchmark "https://github.com/owner/repo" --feature-file feature_spec.md -o benchmark_out
```

### Benchmark bundle output

The benchmark entrypoint writes a single canonical layout:

| Path | Purpose |
|------|---------|
| `benchmark_result.json` | Machine-readable comparison result for collection and analysis |
| `benchmark_result.schema.json` | JSON schema for validating the result bundle |
| `kg/` | Full KG build artifacts for repo structure and debugging |
| `prompts/` | Feature spec plus baseline and graph-guided packed contexts |
| `reports/benchmark_report.md` | Human-readable summary for testers to share |

The result JSON includes the benchmark input, repository metadata, KG stats, role budgets, baseline vs graph-guided estimates, retrieval diagnostics, and an `assessment.status` field of `improved`, `mixed`, or `degraded`.

### Wiki operations CLI

```bash
# Initialize wiki from a context graph
repo-kg-wiki init kg_out/context_graph.json kg_out/wiki

# Re-ingest latest graph changes
repo-kg-wiki ingest kg_out/context_graph.json kg_out/wiki

# Ask questions over wiki content
repo-kg-wiki query kg_out/wiki "How does authentication flow?"

# Validate wiki structure/frontmatter
repo-kg-wiki lint kg_out/wiki
```

**Example `scan_config.yaml`:**

```yaml
include_globs:
  - "src/**"
  - "lib/**"
exclude_globs:
  - "vendor/**"
  - "*.min.js"
max_file_bytes: 524288        # 512 KB
include_vendor: false
include_node_modules: false
tree_sitter_languages:
  - PHP
  - JavaScript
  - TypeScript
max_imports_per_file: 100
```

### Python API

```python
from pathlib import Path
from repo_knowledge_graph.api import build_graph, BuildOptions

result = build_graph(
    "https://github.com/owner/repo",
    options=BuildOptions(
        output_dir=Path("kg_out"),
        parse_deps=True,
        include_directories=True,
        max_chunk_tokens=1000,
      build_wiki=True,
      wiki_dir_name="wiki",
      wiki_overwrite_existing=False,
    ),
)

print(result.stats)
# BuildStats(nodes=3165, edges=3164, python_files=0, treesitter_files=0,
#            other_files=2576, dependency_sets=2, commit_sha='68fcb37...', wiki_pages=143)
print(result.wiki_dir)

# Access graph in memory — no disk required
nodes = result.context_graph["nodes"]
meta  = result.context_graph["meta"]
```

### Agentic orchestration API

```python
from pathlib import Path
from repo_knowledge_graph.agents.orchestrator import KGBuildAgent

agent = KGBuildAgent()
result = agent.build(
  "https://github.com/owner/repo",
  output_dir=Path("kg_out"),
  wiki=True,
)

print(result.plan)
print(result.wiki_dir)
print(result.build.stats)
```

---

## Output file reference

| File | Format | Best for |
|------|--------|----------|
| `knowledge_graph.ttl` | RDF/Turtle | SPARQL queries, triple stores (Jena, Oxigraph, GraphDB), compliance audit |
| `knowledge_graph.jsonld` | JSON-LD | Linked data pipelines, semantic web tooling, schema.org consumers |
| `context_graph.json` | JSON (schema v2) | Full in-memory graph traversal, agent tools, custom RAG loaders |
| `manifest.json` | JSON | Dashboards, search index bootstrapping, quick repo overview |
| `context_chunks.jsonl` | JSONL | Vector database ingestion (Pinecone, Weaviate, Chroma, pgvector), streaming RAG |
| `wiki/` | Markdown tree | Compounding architecture memory, retrieval substrate for agentic query workflows |

---

## Architecture

```
GitHub URL
    │
    ▼
github_url.py  ──── parse owner/repo/ref
    │
    ▼
clone_repo.py  ──── git clone --depth 1
    │
    ▼
git_provenance.py ── HEAD SHA · branch · tag
    │
    ▼
scan.py ─────────── walk files (ScanConfig: globs, max size, vendor policy)
    │           ┌── extract/python_ast.py   (stdlib ast)
    │           ├── extract/treesitter.py   (optional, PHP/JS/TS)
    │           └── extract/dependencies.py (composer/npm/pip/go/maven)
    │
    ├──▶ rdf_builder.py ──── knowledge_graph.ttl + .jsonld
    │
    └──▶ context_graph.py ── context_graph.json
              │
              └──▶ chunked_export.py ── manifest.json + context_chunks.jsonl
                          │
                          └──▶ agents/wiki_builder.py ── wiki/index.md + wiki/entities/* + wiki/concepts/*

Wiki runtime operations:
  repo-kg-wiki (agents/wiki_ops.py): init | ingest | query | lint
  KGBuildAgent (agents/orchestrator.py): end-to-end build orchestration
```

---

## Applicability

This accelerator is valuable for any team that:

- **Builds AI coding assistants or Copilots** and needs structured repo context beyond raw file retrieval
- **Runs RAG on codebases** and wants pre-chunked, structured, token-budget-aware documents
- **Onboards developers to unfamiliar repos** and wants AI-generated, structured overviews instead of "read the docs"
- **Does software due diligence or audits** and needs a machine-readable inventory of files, languages, and declared dependencies
- **Builds graph-native knowledge bases** with SPARQL / triplestore pipelines and wants code structure as linked data
- **Maintains large polyglot monorepos** (PHP, JS, Python, Go, Java) and needs a unified, vendor-agnostic structural view

---

## Open-source foundation

| Component | Library | License |
|-----------|---------|---------|
| RDF graph + Turtle/JSON-LD | [RDFLib](https://github.com/RDFLib/rdflib) | BSD-3-Clause |
| Python structure | `ast` (stdlib) | PSF |
| Multi-language symbols | [tree-sitter](https://tree-sitter.github.io/) (optional) | MIT |
| TOML parsing (Python 3.11+) | `tomllib` (stdlib) | PSF |
| Clone | `git` CLI | GPLv2 (system tool) |
| Vocabularies | PROV-O, Schema.org, custom `code:` | W3C / open |

No paid APIs. No vendor lock-in. All outputs are standard open formats.

---

## Django A/B experiment (max-pack vs graph-guided)

A reproducible harness compares **Scenario 1** (max-pack `django/**/*.py` up to ~380k tokens) vs **Scenario 2** (only `django/utils/*.py` from `manifest.json`) for the same feature spec, using **`gpt-5.4`** by default. See **[experiments/README.md](experiments/README.md)**. Install extras: `pip install -e ".[experiment]"`.

---

## Testing

- Core graph and scanner tests: [tests/test_context_graph.py](tests/test_context_graph.py), [tests/test_scan.py](tests/test_scan.py), [tests/test_deps.py](tests/test_deps.py), [tests/test_github_url.py](tests/test_github_url.py)
- Focused agent/wiki tests: [tests/test_agents.py](tests/test_agents.py)

Run all tests:

```bash
py -m pytest -q
```

Run focused agent tests:

```bash
py -m pytest tests/test_agents.py -q
```

---

## Roadmap

The following capabilities are identified as next priorities (see [assessment plan](.cursor/plans/kg_utility_assessment_6d7ca142.plan.md)):

- **Incremental / cached builds** — content-hash per file; rebuild only changed paths
- **GitHub API enrichment** — topics, description, default branch, language stats via REST (no token needed for public repos)
- **SHACL / OWL validation** — shapes for RDF graph integrity checks
- **Private repo support** — `GITHUB_TOKEN` / SSH key passthrough
- **Secret-aware export** — `.gitignore`-respecting mode to prevent accidental credential exposure in AI context

---

## License

MIT — see [LICENSE](LICENSE).
