Metadata-Version: 2.4
Name: codecortex-engine
Version: 2.0.1
Summary: Adaptive semantic intelligence engine for large-scale code understanding
License: MIT
Project-URL: Repository, https://github.com/NafeesMansoor/codecortex
Project-URL: Documentation, https://github.com/NafeesMansoor/codecortex#readme
Keywords: code-intelligence,semantic-analysis,graph,embeddings,retrieval,visualization
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tree-sitter<0.26,>=0.25
Requires-Dist: tree-sitter-python<1.0,>=0.23
Requires-Dist: tree-sitter-javascript<1.0,>=0.23
Requires-Dist: tree-sitter-typescript<1.0,>=0.23
Requires-Dist: tree-sitter-php<1.0,>=0.23
Requires-Dist: tree-sitter-go<1.0,>=0.23
Requires-Dist: tree-sitter-rust<1.0,>=0.23
Requires-Dist: tree-sitter-java<1.0,>=0.23
Requires-Dist: tree-sitter-c-sharp<1.0,>=0.23
Requires-Dist: tree-sitter-c<1.0,>=0.23
Requires-Dist: tree-sitter-cpp<1.0,>=0.23
Requires-Dist: tree-sitter-ruby<1.0,>=0.23
Requires-Dist: tree-sitter-kotlin<2,>=1.1
Requires-Dist: tree-sitter-swift<1.0,>=0.7
Requires-Dist: tree-sitter-scala<1.0,>=0.23
Requires-Dist: tree-sitter-lua<1.0,>=0.5
Requires-Dist: tree-sitter-bash<1.0,>=0.23
Requires-Dist: networkx>=3.1
Requires-Dist: numpy>=1.24
Requires-Dist: pyyaml>=6.0
Requires-Dist: certifi>=2023.7
Provides-Extra: embeddings
Requires-Dist: faiss-cpu>=1.7; extra == "embeddings"
Requires-Dist: sentence-transformers>=2.6; extra == "embeddings"
Requires-Dist: torch>=2.0; extra == "embeddings"
Provides-Extra: lsp
Requires-Dist: jedi>=0.19; extra == "lsp"
Requires-Dist: pygls>=1.0; extra == "lsp"
Provides-Extra: clustering
Requires-Dist: hdbscan>=0.8; extra == "clustering"
Requires-Dist: umap-learn>=0.5; extra == "clustering"
Requires-Dist: scikit-learn>=1.3; extra == "clustering"
Provides-Extra: all
Requires-Dist: codecortex[clustering,embeddings,lsp]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff<0.17,>=0.16; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Dynamic: license-file

# CodeCortex

**Adaptive semantic intelligence engine for large-scale code understanding.**

> Run `codecortex --version` for the installed version — see [CHANGELOG.md](CHANGELOG.md) for release history.

CodeCortex transforms source code into a living, queryable semantic graph — combining Code Property Graphs, real embedding models, approximate nearest-neighbor retrieval, intelligent graph traversal, and an interactive human-facing visualization layer into a unified platform purpose-built for production-scale repositories.

---

## What CodeCortex Is

CodeCortex is not a linter, not a static analyzer, and not a dependency graph visualizer. It is a **semantic intelligence infrastructure** layer that answers questions like:

- *Which components are structurally central to this codebase?*
- *What is the full blast radius if `UserService.authenticate` changes?*
- *Which functions are semantically similar to "authentication middleware"?*
- *What does the dataflow look like through this API endpoint?*
- *How are these modules architecturally coupled?*

It ingests multi-language source code and produces a compressed, semantically-enriched graph that supports sub-20 ms retrieval across repositories of 100k+ nodes — and renders as a fully interactive visual graph for human exploration.

---

## Core Architecture

```
Source Files (16 languages — see Supported Languages)
        │
        ▼
┌───────────────────────────────────┐
│   Tier 1 — Semantic Index         │  LSP · SCIP · Jedi · Tree-sitter
│   Symbols · APIs · Imports        │  ~O(files) build time
└──────────────────┬────────────────┘
                   │
                   ▼
┌───────────────────────────────────┐
│   Tier 2 — Hybrid Code Property   │  AST + CFG + DFG + Endpoints
│   Graph  (pruned, bounded)        │  60–80% node reduction
└──────────────────┬────────────────┘
                   │
                   ▼
┌───────────────────────────────────┐
│   Tier 3 — Embedding Layer        │  CodeBERT · local models · OpenAI
│   FAISS IVF · HNSW · cosine ANN  │  Batched · incremental · cached
└──────────────────┬────────────────┘
                   │
                   ▼
┌───────────────────────────────────┐
│   Clustering & Community          │  Louvain · HDBSCAN · adaptive
│   Semantic group labeling         │  Cluster confidence scoring
└──────────────────┬────────────────┘
                   │
                   ▼
┌───────────────────────────────────┐
│   Traversal & Ranking             │  Beam BFS · Personalized PageRank
│   Semantic-aware path scoring     │  Impact radius · Taint flow
└──────────────────┬────────────────┘
                   │
             ┌─────┴──────┐
             ▼            ▼
┌────────────────┐  ┌──────────────────────┐
│   Retrieval    │  │  Graph Visualization  │
│   Layer        │  │  Interactive Explorer │
│   AI-ready     │  │  Human-centric UI     │
│   context      │  │  Zero pipeline impact │
└────────────────┘  └──────────────────────┘
```

### Supported Languages

Every grammar ships as a wheel, so parsing works offline. `--languages`
accepts a language name, alias or extension (`py`, `rust`, `cs`, `tsx`).

| Language | Extensions | Extracted |
|---|---|---|
| Python | `.py` | functions, methods, classes, calls, imports, bases, tests, CFG/DFG |
| JavaScript | `.js .jsx .mjs .cjs` | functions (incl. arrow functions bound to names), classes, methods, calls, `import`/`require`/`import()`, `extends` |
| TypeScript | `.ts .tsx .mts .cts` | as JavaScript, plus interfaces, enums, type aliases, `implements` |
| PHP | `.php` | classes, interfaces, traits, enums, methods, functions, calls (incl. `$this->m()`, `A::m()`), `use`/grouped `use`/`require`, `extends`/`implements`/trait use |
| Go | `.go` | functions, receiver methods, structs, interfaces, calls, imports, `Test*` in `_test.go` |
| Rust | `.rs` | functions, `impl` methods, structs, enums, traits, calls, `use`, `impl Trait for`, `#[test]` |
| Java | `.java` | classes, records, interfaces, enums, methods, constructors, calls, imports, `extends`/`implements`, `@Test` |
| C# | `.cs` | classes, records, structs, interfaces, enums, methods, calls, `using`, base lists, `[Fact]`/`[Test]` |
| C | `.c .h` | functions, structs, unions, enums, calls, `#include` |
| C++ | `.cpp .cc .cxx .hpp .hh .hxx` | as C, plus classes, out-of-line `Class::method`, base classes |
| Ruby | `.rb .rake` | classes, modules, methods, calls, `require`, superclasses |
| Kotlin | `.kt .kts` | classes, objects, interfaces, enums, functions, calls, imports, supertypes |
| Swift | `.swift` | classes, structs, enums, protocols, extensions, functions, calls, imports, inheritance |
| Scala | `.scala .sc` | classes, objects, traits, functions, calls, imports, `extends` |
| Lua | `.lua` | functions, table functions (`M.f`, `M:g`), calls, `require` |
| Bash | `.sh .bash` | functions, commands, `source` |

CFG/DFG passes are tuned for Python, JavaScript/TypeScript and PHP; other
languages get the structural graph (definitions, calls, imports, inheritance).

### Key Design Decisions

| Decision | Rationale |
|----------|-----------|
| Hybrid CPG, not full-graph | Full CPG causes node explosion. Semantic filtering retains 20–40% of nodes while preserving retrieval quality. |
| AdjacencyCache (in-memory) | Eliminates per-hop SQL overhead. BFS latency drops from ~1400 ms to <1 ms. |
| Semantic-only centrality | Filtering out VARIABLE/PARAMETER/CALLSITE nodes gives 32× better PageRank discrimination. |
| On-demand CFG/DFG | Tier-1 index builds in seconds; deep CPG is only expanded for files touched by a query. |
| Incremental embedding updates | Only re-embed nodes whose source files changed (mtime-driven delta). |
| Visualization as a separate layer | Graph explorer consumes exported data — zero changes to pipeline internals. |

---

## Major Capabilities

### Semantic Retrieval
Natural language queries return ranked, structurally-aware results. Results are fused from embedding similarity, graph centrality, and cluster membership.

```python
from codecortex import CodeCortexPipeline

pipeline = CodeCortexPipeline.from_directory("/path/to/repo")
context = pipeline.query("authentication middleware", max_tokens=3000)
print(context)  # AI-ready context string
```

### Impact Analysis
Trace the full blast radius of any change — direct callers, transitive dependents, affected tests, and cross-file propagation.

```python
impact = pipeline.impact("UserService.authenticate", max_depth=4)
print(impact.to_dict())
# {root, direct_callers, affected_files, affected_tests, transitive_affected}
```

### Hotspot Detection
Identify structurally critical nodes by composite centrality: PageRank, betweenness, fan-in/fan-out, eigenvector.

```python
hotspots = pipeline.top_nodes(n=20)
```

### Interactive Graph Visualization
Launch a full-featured browser-based graph explorer. Navigate the codebase visually, inspect semantic clusters, trace impact paths, and identify hotspots — all without modifying a line of pipeline code.

```bash
codecortex visualize /path/to/repo
# Opens http://localhost:7979 in the browser
```

### Incremental Indexing
`codecortex query` and `explain` load the saved index from `.codecortex/` and
re-parse only files that changed since it was written. From Python, reindex a
single changed file without rebuilding the full graph:

```python
pipeline.reindex_file("/path/to/repo/auth/service.py")
```

---

## Interactive Graph Explorer

The visualization layer is a purpose-built human interface on top of the semantic graph. It runs as a local web server and opens automatically in the browser.

### Features

| Feature | Description |
|---------|-------------|
| Force / Hierarchy / Circle / Grid / Concentric layouts | Switch between graph layouts instantly |
| Node type legend | Click to show/hide nodes by kind (Function, Class, Endpoint, etc.) |
| Semantic cluster chips | Filter the graph to a single semantic community |
| Edge type legend | Understand CALLS, IMPORTS, INHERITS, CONTROLS, READS, WRITES |
| Hotspot highlighting | One-click to dim all non-hotspot nodes |
| Variable/Parameter toggle | Show or hide low-level structural noise |
| Centrality slider | Progressively filter out low-importance nodes |
| Symbol search | Fuzzy match by name, kind, or file path |
| Node detail panel | Click any node: file, line, centrality score, cluster, docstring, connections |
| Neighbor traversal | Click neighbors in the panel to jump to them |
| Minimap | Viewport overview with live update |
| Zoom / Pan / Fit | Full graph navigation controls |
| Keyboard shortcuts | `Ctrl/Cmd+F` search · `Esc` deselect |

### Launch the Explorer

```bash
# Basic launch
codecortex visualize .

# Point at any repository
codecortex visualize /path/to/your/repo

# Custom port, no auto-open
codecortex visualize . --port 8080 --no-open

# Limit to specific languages
codecortex visualize . --languages py ts
```

### What You Can Do in the Explorer

**Architecture discovery** — use the Hierarchy layout to see module structure from top to bottom.

**Hotspot review** — click "Hotspots" in the header to highlight the top 10% most-connected nodes. These are your highest-risk change points.

**Cluster exploration** — click a semantic cluster chip to isolate one community and understand what it does.

**Impact tracing** — click any node, inspect its connections in the detail panel, and jump through the call chain.

**Dependency audit** — filter to IMPORTS_FROM edges only to map inter-module dependencies cleanly.

**Onboarding** — point a new team member at the graph to let them explore the codebase structure visually before reading a single line of code.

---

## Repository Structure

```
codecortex/                 Main package + unified CLI
  __init__.py               Public API surface
  cli.py                    Unified CLI (index / query / explain / install / visualize)
  config.py                 YAML config loader
  index.py                  Index subcommand
  query.py                  Query subcommand
  explain.py                Explain subcommand
  integrate.py              install / uninstall for AI assistants
  hook.py                   Claude Code PreToolUse hook (hook-guard)

  core/                     Shared types and parser framework
    types.py                NodeKind, EdgeKind, NodeInfo, EdgeInfo
    parser_framework.py     LanguageParser ABC + ParserRegistry
    parsers/                Language parsers (16 languages)
      registry.py           The one list of parsers
      treesitter_parser.py  Table-driven tree-sitter parser
      languages.py          Go, Rust, Java, C#, C, C++, Ruby, Kotlin, …
      python_parser.py      javascript_parser.py  typescript_parser.py  php_parser.py

  graph/                    Code Property Graph
    schema.py               CPGNode, CPGEdge
    graph_store.py          SQLite-backed persistent store
    cpg_builder.py          AST → CPG ingestion
    cfg_builder.py          Control Flow Graph
    dfg_builder.py          Data Flow Graph
    endpoint_detector.py    HTTP API / route detection
    graph_pruner.py         Semantic node filtering (60–80% reduction)

  indexing/                 Semantic symbol indexing
    semantic_index.py       In-memory symbol lookup
    symbol_resolver.py      Cross-file name resolution
    symbol_cache.py         SQLite persistent symbol table
    reference_graph.py      Cross-file occurrence store
    jedi_provider.py        Python semantic analysis (Jedi)
    lsp_provider.py         JSON-RPC LSP client
    scip_provider.py        SCIP index reader
    graph_enricher.py       Upgrades unresolved CALLS edges

  embeddings/               Vector embedding layer
    code_embedder.py        Code → text normalization + rich context
    embedding_pipeline.py   CPG → FAISS index (batch, incremental)
    embedding_store.py      FAISS wrapper with cosine similarity
    provider_factory.py     lexical | local | openai | stub providers

  clustering/               Semantic community detection
    semantic_clusters.py    HDBSCAN + KMeans fallback
    louvain_clusterer.py    Graph-based Louvain community detection
    cluster_labeler.py      Token frequency + module dominance labels
    cluster_quality.py      Silhouette, cohesion, separation metrics
    cluster_store.py        SQLite cluster persistence

  traversal/                Graph traversal engine
    traversal_engine.py     AdjacencyCache, BFS, DFS, TaintFlow
    beam_traversal.py       Beam search with semantic scoring
    query_optimizer.py      Query plan optimization

  ranking/                  Centrality and ranking
    centrality_engine.py    PageRank, betweenness, closeness, eigenvector
    ppr_engine.py           Personalized PageRank (query-relative ranking)

  retrieval/                Retrieval orchestration
    retrieval_layer.py      Hybrid semantic + structural retrieval

  pipeline/                 End-to-end orchestration
    context_builder.py      CodeCortexPipeline (build → query → impact)
    state.py                Saved per-project index (.codecortex/)

  performance/              Incremental performance optimizations
    graph_snapshot.py       JSON graph persistence + mtime delta
    parallel_indexer.py     ThreadPoolExecutor multi-file indexing
    delta_embedder.py       Mtime-driven incremental re-embedding

  visualization/            Interactive graph explorer (human-facing)
    __init__.py
    graph_exporter.py       Pipeline state → visualization JSON
    server.py               Local HTTP server + browser launcher
    static/
      index.html            Full interactive graph UI (Cytoscape.js)

tests/                      Test suite
  test_phase1_core.py       … through test_phase10_performance.py
  test_languages.py         Per-language extraction
  test_saved_index.py       Saved index, install, hook
  benchmarks/               Performance benchmarks
    benchmark_runner.py     Composite score benchmark
    bench_v01.py            Stage-wise evaluation
    bench_v02.py            Per-phase architectural evaluation

codecortex.yaml             Default configuration
```

---

## Use with Your AI Assistant

CodeCortex is meant to be asked, not loaded: build the graph once, and the
assistant queries it when a question needs it instead of grepping and
reading files.

```bash
pipx install codecortex-engine

cd your-project
codecortex install          # writes the assistant instructions (Claude Code by default)
codecortex index .          # builds the graph once, saved to .codecortex/
```

`codecortex install` adds a short, marked `## codecortex` block to `CLAUDE.md`,
a `codecortex` skill under `.claude/skills/`, and a `PreToolUse` hook in
`.claude/settings.json`. When the project has a saved index and the assistant
is about to grep or read a source file, the hook adds a one-line note that
`codecortex query` / `explain` can answer more cheaply. It never blocks a
tool, speaks at most once per session for searches and once for reads, and
takes ~0.1 s. Skip it with `--no-hooks`. Use `--platform agents`
(AGENTS.md: Codex, OpenCode, Amp, Copilot), `--platform cursor`,
`--platform gemini`, or `--platform all`. The block tells the assistant to:

| When | Run |
|---|---|
| Before grepping or reading files for a codebase question | `codecortex query --query "<question>"` |
| To see one symbol's definition, callers, callees and base types | `codecortex explain <Symbol>` |
| Before changing a symbol | `codecortex query --impact <Qualified.name>` |

Each call reuses `.codecortex/` (graph, ranking, clusters and search vectors)
and re-parses and re-embeds only files whose size or mtime changed, so there
is nothing to keep running and nothing to remember to refresh. On a 636-file
repository a query or `explain` takes about a second. `codecortex index .` rebuilds everything (centrality and clusters are
reused across incremental refreshes and recomputed by a full index).
`.codecortex/` ignores itself, so it never appears in `git status`.

`codecortex uninstall` removes exactly the block and files it added
(`--purge` also deletes `.codecortex/`).

The default `lexical` embedding backend matches query words against symbol
names, kinds and modules with no model download; set `embedding_backend:
local` in `codecortex.yaml` (with the `embeddings` extra) for model-based
semantic search.

---

## Installation

### Prerequisites

- Python 3.10+
- pip / pip3

> **macOS note:** Homebrew Python enforces PEP 668 and blocks global `pip install`.
> Always install into a virtual environment (step 2 below).
> If `pip` is not found, use `pip3` or `python3 -m pip` instead.

### Local Development Setup

```bash
# 1. Clone the repository
git clone https://github.com/NafeesMansoor/codecortex.git
cd codecortex

# 2. Create a virtual environment (required on macOS Homebrew Python)
python3 -m venv .venv
source .venv/bin/activate         # Windows: .venv\Scripts\activate

# 3. Install core dependencies
pip install -r requirements.txt

# 4. Install in editable mode
pip install -e .

# 5. Optional: model embeddings + full semantic stack
pip install -e ".[all]"

# 6. Verify
pytest tests/ --ignore=tests/benchmarks
codecortex --help
```

### Production Setup

CodeCortex is distributed through GitHub Releases; install a specific,
immutable version rather than a branch.

**For the CLI, install it as a tool.** `pipx` (or `uv tool`) gives CodeCortex
its own virtualenv and puts only the `codecortex` command on your `PATH`, so
its dependencies never mix with a project's. To use the Python API, `pip
install` it into your application's environment instead; everything lives
under the single `codecortex` package, so it adds no other top-level names.

```bash
# Recommended: isolated tool install (~100 MB, all 16 grammars included)
pipx install codecortex-engine
# or: uv tool install codecortex-engine
# or a tagged release from GitHub:
#     pipx install "codecortex-engine @ git+https://github.com/NafeesMansoor/CodeCortex.git@v2.0.1"

# Optional model embeddings (sentence-transformers, torch, faiss) — large
pipx install "codecortex-engine[embeddings]"

# Or from a release artifact
pipx install ./codecortex_engine-2.0.1-py3-none-any.whl

# Verify
codecortex health
```

To upgrade a tool install, run `pipx upgrade codecortex-engine` (or `uv tool
upgrade codecortex-engine`). A tool installed before 2.0.1 was named
`codecortex`; replace it with `pipx uninstall codecortex && pipx install
codecortex-engine`.

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `CODECORTEX_EMBEDDING_PROVIDER` | `stub` | `stub` · `local` · `openai` |
| `OPENAI_API_KEY` | — | Required for `openai` provider |
| `CODECORTEX_CACHE_DIR` | `.codecortex/` | Graph snapshot and cache directory |
| `CODECORTEX_LOG_LEVEL` | `WARNING` | `DEBUG` · `INFO` · `WARNING` · `ERROR` |
| `CODECORTEX_NO_UPDATE_CHECK` | — | Set to `1` to disable all outbound update checks |
| `CODECORTEX_STATE_DIR` | `~/.codecortex/` | Update state and backups |
| `CODECORTEX_UPDATE_REPO` | `NafeesMansoor/CodeCortex` | Repository to check for releases |

---

## Versioning & Updates

CodeCortex follows Semantic Versioning, with the version defined once in
`codecortex/version.py` and read from there by the packaging metadata.

```bash
codecortex --version          # CodeCortex 2.0.1
codecortex version --check    # is a newer release available?
codecortex update             # back up, install, migrate, verify, roll back on failure
codecortex update --rollback  # restore the previous version
codecortex health             # verify this installation
```

Updates detect how CodeCortex was installed (git checkout or pip package), move
to an immutable release tag, preserve configuration, rebuild stale index caches,
and verify the result in a fresh interpreter before committing to it. An update
check never blocks startup and never sends information about your installation.

### Upgrading from 1.1.0 or earlier

`codecortex update` ships **with** 1.2.0, so an installation older than that has
no such command yet — those versions predate the updater entirely. Upgrade once
by hand; every release after this one is reachable with `codecortex update`.

Check what you have, then pick the line matching how it was installed:

```bash
codecortex --version    # or: pip show codecortex  (codecortex-engine from 2.0.1)
```

**Installed with pip:**

```bash
# Installed 2.0.0 or earlier with pip? The package was renamed in 2.0.1;
# remove the old name first so both don't claim the same files.
pip uninstall -y codecortex
pip install codecortex-engine
```

**Installed as a git checkout** (`git clone` + `pip install -e .`):

```bash
cd /path/to/CodeCortex
git fetch --tags origin
git checkout v2.0.1        # commit or stash local changes first
pip install -e .
```

Then confirm, and re-index once:

```bash
codecortex --version       # CodeCortex 2.0.1
codecortex health          # expect Status: HEALTHY
codecortex index .         # index caches from older versions are rebuilt
```

Your `codecortex.yaml` and any `.env` are left untouched by both routes.
Existing index caches carry no schema stamp, so the first run rebuilds them —
nothing is lost, because every cache is derived from your source tree.

Anything older than 1.2.0 upgrades directly; there are no staged upgrades.

Applications embedding CodeCortex should declare the range they support:

```python
from codecortex import require_version

require_version(minimum="1.1.0", below="2.0.0", consumer="my-service")
```

Full detail — channels, migrations, rollback, offline installs, and the release
process — is in [UPDATING.md](UPDATING.md). Release history is in
[CHANGELOG.md](CHANGELOG.md).

---

## Quick Start

### Index a Repository

```bash
# Fast structural index
codecortex index .

# Full CPG with embeddings and clustering
codecortex index . --mode cpg --enable-embeddings --enable-clustering

# Hybrid: Tier-1 at build time, CFG/DFG on demand
codecortex index . --on-demand-cfg --on-demand-dfg

# The index is saved to .codecortex/ for later queries; skip that with
codecortex index . --no-save

# Also export a JSON snapshot
codecortex index . --save-snapshot snapshots/graph.json

# Limit languages
codecortex index . --languages py ts

# Exclude extra project paths, on top of the built-in defaults
codecortex index . --languages php --exclude storage/ bootstrap/cache/
```

> **Dependency directories are skipped automatically.** Virtualenvs (`venv/`,
> `.venv/`), `node_modules/`, `vendor/`, `site-packages/`, `__pycache__/`,
> build output and VCS metadata are pruned before the walker descends into
> them, and the repository `.gitignore` is honoured. Pass
> `--no-default-excludes` / `--no-gitignore` to opt out, or set `exclude_dirs`
> in `codecortex.yaml` to replace the built-in list.

### Query

```bash
# Natural language query
codecortex query --query "authentication middleware" --target .

# Impact analysis
codecortex query --impact UserService.authenticate --target .

# One symbol: definition, callers, callees, base types
codecortex explain UserService.authenticate --target .

# Ignore the saved index and rebuild it
codecortex query --query "authentication middleware" --rebuild

# JSON output
codecortex query --query "HTTP routing" --json --target .

# With centrality ranking and larger context budget
codecortex query \
  --query "database connection pooling" \
  --enable-centrality-ranking \
  --max-tokens 5000 \
  --target .
```

### Visualize

```bash
# Launch graph explorer (opens browser automatically)
codecortex visualize .

# Custom port
codecortex visualize /path/to/repo --port 8080

# Headless (server only, no browser open)
codecortex visualize . --no-open

# Limit to Python and TypeScript
codecortex visualize . --languages py ts
```

### Python API

```python
from codecortex import CodeCortexPipeline, PipelineConfig

config = PipelineConfig(
    embedding_backend="local",
    enable_clustering=True,
    use_louvain=True,
    use_beam_traversal=True,
    use_ppr=True,
    use_graph_pruning=True,
)

pipeline = CodeCortexPipeline.from_directory("/path/to/repo", config=config)

# Semantic query
context = pipeline.query("authentication flow", max_tokens=3000)

# Ranked results
results = pipeline.retrieve("database models")
for r in results:
    print(r.qualified_name, r.score)

# Impact analysis
impact = pipeline.impact("UserService.authenticate", max_depth=5)

# Hotspot detection
top = pipeline.top_nodes(n=20)

# Incremental update
pipeline.reindex_file("/path/to/repo/auth/service.py")

# Export for visualization
from codecortex.visualization import export_pipeline

graph_json = export_pipeline(pipeline, project_name="my-repo")
```

---

## Configuration

`codecortex.yaml` in your project root:

```yaml
codecortex:
  # Graph construction
  enable_cpg: true
  on_demand_cfg: false
  on_demand_dfg: false

  # Embedding provider: stub | local | openai
  embedding_backend: stub
  embedding_dimensions: 384

  # Clustering
  enable_clustering: true
  min_cluster_size: 5

  # Retrieval
  retrieval_top_k: 20
  token_budget: 4000
  min_semantic_score: 0.0

  # Traversal
  bfs_depth: 3
  max_nodes_per_query: 500

  # Centrality
  centrality_ranking: true

  # Paths excluded from indexing (relative to the indexed root)
  # Useful for Laravel/Node projects with large vendor/node_modules trees
  exclude_paths:
    - vendor/
    - node_modules/
    - storage/
    - bootstrap/cache/
```

---

## Performance Modes

### Lightweight Mode
Fast index, structural retrieval only. Builds in <1s for 100 files.

```yaml
codecortex:
  enable_cpg: false
  enable_embeddings: false
  enable_clustering: false
  centrality_ranking: false
```

### Balanced Mode
Default. Full CPG with stub embeddings. Builds in ~8s for 60 files.

```yaml
codecortex:
  enable_cpg: true
  embedding_backend: stub
  enable_clustering: true
```

### Deep Semantic Analysis Mode
Real code-aware embeddings, Louvain clustering, beam traversal, personalized PageRank. Best retrieval quality.

```yaml
codecortex:
  enable_cpg: true
  embedding_backend: local
  enable_clustering: true
  use_louvain: true
  use_beam_traversal: true
  use_ppr: true
  use_graph_pruning: true
```

Requires: `pip install "codecortex-engine[all]"`

### Enterprise Scale Mode
On-demand CPG expansion + graph pruning + parallel indexing + incremental updates. Designed for monorepos with 500k+ LOC.

```yaml
codecortex:
  on_demand_cfg: true
  on_demand_dfg: true
  use_graph_pruning: true
  embedding_backend: openai
  embedding_dimensions: 3072
```

---

## Benchmarking

```bash
# Run against any codebase
python tests/benchmarks/benchmark_runner.py --target /path/to/repo --phase "v1.0"

# Stage-wise evaluation
python tests/benchmarks/bench_v01.py --target /path/to/repo

# Per-feature architectural evaluation
python tests/benchmarks/bench_v02.py --target /path/to/repo

# Quiet mode (results appended to docs/results.md)
python tests/benchmarks/benchmark_runner.py --target . --quiet
```

### Composite Score Formula

```
CodeCortex Score = 0.25×retrieval + 0.25×graph + 0.20×semantic + 0.15×efficiency + 0.15×ranking
```

| Dimension | What it measures |
|-----------|-----------------|
| Retrieval quality | Results per query (structural proxy) |
| Graph quality | Edge/node density ratio |
| Semantic coverage | Nodes extracted per file |
| Efficiency | Index build speed |
| Ranking discrimination | PageRank standard deviation |

---

## Testing

```bash
# All 223 unit tests
pytest tests/ --ignore=tests/benchmarks

# With coverage
pytest tests/ --ignore=tests/benchmarks --cov=. --cov-report=term-missing

# Specific phase
pytest tests/test_phase3_cpg.py -v
```

---

## Developer Experience

### Debugging

```bash
CODECORTEX_LOG_LEVEL=DEBUG codecortex index .
```

### Profiling

```bash
python -m cProfile -o profile.out tests/benchmarks/benchmark_runner.py --target . --quiet
python -c "import pstats; p=pstats.Stats('profile.out'); p.sort_stats('cumulative'); p.print_stats(20)"
```

### Troubleshooting

**No nodes in visualization / empty graph**
Ensure the target directory contains supported source files (`.py`, `.js`, `.ts`, `.php`). Check `CODECORTEX_LOG_LEVEL=DEBUG codecortex visualize .` for parser errors.

**8000+ files indexed / graph polluted with framework internals (Laravel, Node)**
You are likely indexing `vendor/` or `node_modules/`. Target the application directory directly or use `--exclude`:
```bash
codecortex index /path/to/laravel --languages php --exclude vendor/ node_modules/
```
Or add `exclude_paths` to `codecortex.yaml` (see Configuration above).

**Clustering skipped — scikit-learn / hdbscan not installed**
Install the optional clustering extras:
```bash
pip install scikit-learn hdbscan umap-learn
```
HDBSCAN also requires real (non-stub) embeddings. Set `embedding_backend: local`.

**Clustering produces 0 clusters**
HDBSCAN requires real (non-stub) embeddings. Set `embedding_backend: local` and install sentence-transformers.

**Slow CPG build (>10s for 60 files)**
Enable graph pruning and on-demand CFG/DFG:
```yaml
use_graph_pruning: true
on_demand_cfg: true
on_demand_dfg: true
```

**Browser does not open**
Use `--no-open` and navigate manually to `http://localhost:7979`.

**`pip: command not found` on macOS**
Use `pip3` or `python3 -m pip`. macOS Homebrew Python does not symlink `pip`.

**`error: externally-managed-environment` (PEP 668)**
Create a virtualenv first: `python3 -m venv .venv && source .venv/bin/activate`, then run `pip install`.

---

## Extensibility

### Add a Language Parser

Most languages need no parsing code: subclass `TreeSitterParser`, name the
grammar node types, and add the class to `PARSER_CLASSES` in
`core/parsers/registry.py`. The pipeline, the parallel indexer and the
`--languages` filter all resolve parsers there.

```python
from codecortex.core.parsers.treesitter_parser import TreeSitterParser
from codecortex.core.types import EdgeKind, NodeKind


class ZigParser(TreeSitterParser):
    language = "zig"
    extensions = [".zig"]
    grammar = "zig"  # add the grammar wheel to core/parsers/grammars.py
    function_types = frozenset({"function_declaration"})
    type_kinds = {"struct_declaration": NodeKind.STRUCT}
    call_types = frozenset({"call_expression"})
    import_types = frozenset({"builtin_function"})
    base_clauses: dict[str, EdgeKind] = {}
```

Override a hook (`container`, `name_of`, `owner_of`, `callee_of`,
`imports_of`, `bases_of`, `is_test`) only where the grammar does not fit the
tables, and add a case to `tests/test_languages.py`.

### Add an Embedding Provider

```python
from codecortex.embeddings.provider_factory import EmbeddingProvider
import numpy as np


class MyProvider(EmbeddingProvider):
    def embed(self, texts: list[str]) -> np.ndarray: ...
```

### Export Graph Programmatically

```python
from codecortex.visualization import export_pipeline
import json

pipeline = CodeCortexPipeline.from_directory(".")
data = export_pipeline(pipeline, project_name="my-project")
json.dump(data, open("graph.json", "w"), indent=2)
```

---

## License

MIT

---

## Author

**Nafees Mansoor**
