Metadata-Version: 2.4
Name: karta-ng
Version: 2.4.0
Summary: karta-ng (Karta next-gen): identify and match open-source libraries inside binaries. Structural function fingerprinting and location-driven matching for malware and firmware analysis.
Author-email: Mahmoud Abdelkader <mabdelkader@gmail.com>
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: elementals
Requires-Dist: click
Requires-Dist: scikit-learn>=1.5.0
Requires-Dist: ida-domain>=0.5.1
Provides-Extra: speedups
Requires-Dist: Cython>=3.0.0; extra == "speedups"
Provides-Extra: dev
Requires-Dist: setuptools>=77.0.1; extra == "dev"
Requires-Dist: libcst>=1.0.0; extra == "dev"
Requires-Dist: import-linter>=2.0; extra == "dev"
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: coverage[toml]>=7.0.0; extra == "dev"
Requires-Dist: ruff>=0.12; extra == "dev"
Requires-Dist: pydocstyle; extra == "dev"

# karta-ng — Karta next-gen, for IDA Pro 9.1+

<img src="https://github.com/mahmoudimus/karta-ng/blob/main/assets/karta-ng-logo.png?raw=true" width="100px" height="100px" alt="karta-ng logo" /> [![karta-ng tests](https://github.com/mahmoudimus/karta-ng/actions/workflows/python.yml/badge.svg)](https://github.com/mahmoudimus/karta-ng/actions/workflows/python.yml) [![coverage](https://codecov.io/gh/mahmoudimus/karta-ng/branch/main/graph/badge.svg)](https://codecov.io/gh/mahmoudimus/karta-ng)

"Karta" (Russian for "Map") identifies and matches open-source libraries inside a binary. It uses a location-driven technique that scales to very large binaries (>200,000 functions) with almost no impact on performance, because the matching cost depends on K — the number of functions in the open source — rather than N, the size of the binary, and usually N >> K.

karta-ng is a maintained continuation of that work: IDA 9.x and Python 3.13 support, an `ida-domain`-backed analysis layer, a headless entry point that runs the same matcher core without opening a form, relocation-independent structural signatures, and a much larger library-seeker catalogue.

## Table of contents

- [Why you would use this](#why-you-would-use-this)
- [Installation](#installation)
  - [Quick install](#quick-install)
  - [Install with hcli](#install-with-hcli)
  - [Need to find your plugin directory?](#need-to-find-your-plugin-directory)
  - [Where is my default user directory?](#where-is-my-default-user-directory)
- [Requirements](#requirements)
- [Usage](#usage)
  - [Identifier](#identifier)
  - [Matcher](#matcher)
  - [Headless](#headless)
  - [Generating a configuration](#generating-a-configuration)
- [How it works](#how-it-works)
- [What is new over upstream Karta](#what-is-new-over-upstream-karta)
- [Configuration schema](#configuration-schema)
- [Development](#development)
  - [Tests](#tests)
  - [Architectural gates](#architectural-gates)
  - [Versioning](#versioning)
  - [Contributing](#contributing)
- [Acknowledgements](#acknowledgements)
- [Contact](#contact)

## Why you would use this

1. Identifying which open sources (and which versions) a binary uses, when hunting for a useful 1-day.
2. Matching the symbols of supported open sources to speed up malware reverse engineering.
3. Matching those symbols in a binary or firmware image when searching for 0-days in the proprietary code around them.

## Installation

### Quick install

```bash
pip install -e .                # from a clone
```

`ida-domain` — the supported Hex-Rays abstraction — comes with it: it is the only disassembler backend, which is why IDA 9.1 is the floor.

Then deploy the plugin into IDA:

```bash
python -m karta_ng.installers.ida_installer
```

The installer detects your IDA installation and copies the plugin entry point into its `plugins/` directory. Restart IDA afterwards; the plugin registers **Karta NG: Match libraries** under `Edit > Plugins`.

### Install with hcli

[`hcli`](https://hcli.docs.hex-rays.com/) is Hex-Rays' command-line tool. Install it once:

```bash
curl -LsSf https://hcli.docs.hex-rays.com/install | sh        # macOS/Linux
iwr -useb https://hcli.docs.hex-rays.com/install.ps1 | iex    # Windows (PowerShell)
```

This repository ships an `ida-plugin.json` descriptor that is validated against the IDA Plugin Repository schema on every CI run, so it installs through `hcli` once published. Requires IDA 9.1+.

### Need to find your plugin directory?

From IDA's Python console:

```python
import idaapi, os; print(os.path.join(idaapi.get_user_idadir(), "plugins"))
```

### Where is my default user directory?

- Windows: `%APPDATA%\Hex-Rays\IDA Pro`
- macOS: `~/.idapro`
- Linux: `~/.idapro`

## Requirements

| | |
|---|---|
| IDA Pro | 9.1, 9.2, 9.3, 9.4 |
| Python | 3.10+ (IDA 9.x bundles 3.13; CI exercises that interpreter) |
| Required | `elementals`, `click`, `scikit-learn`, `ida-domain` >= 0.5.1 |
| Optional | `Cython` for native speedups |

`sark` is **not** required. The disassembler layer runs on `ida-domain`; the sark-shaped shim that replaced it (`thumbs_up/ida_compat.py`) now lives inside `thumbs_up/`, its only consumer.

## Usage

### Identifier

The identifier fingerprints which supported open sources are present and which versions they are, so you do not reverse the same library twice. It writes `<input>_libraries.txt` beside the database — the same place and naming convention the matcher uses for `<input>.analysis`.

karta-ng currently registers **152** seekers, spanning the upstream set (OpenSSL, libpng, libjpeg, NetSNMP, zlib, libtiff, libxml2, gSOAP, OpenSSH, and more) plus categories the upstream project never covered: crypto, compression, network, multimedia, database, math, runtime, game engines, GUI toolkits, IoT, ML, containers, parsing, cloud SDKs, logging, web, graphics, security, and codecs.

### Matcher

Once a library and version are identified, point Karta at a matching `.json` configuration and it resolves the open source's functions — and the external functions it calls, such as `memcpy`, `fread`, or `zlib_inflate` — onto the functions in your binary.

In the GUI: **Edit > Plugins > Karta NG: Match libraries**.

Six ready-made configurations ship in `configs/`: OpenSSL 1.0.1j, libpng 1.2.29, libtiff 4.0.8 (Linux and Windows), and zlib 1.2.3 / 1.2.11.

### Headless

The same matcher core runs without opening a single form, through `idapro`:

```bash
# match, write <input>.analysis (JSON) beside the database, touch nothing else
python -m karta_ng.karta_headless --config configs/ sample.bin

# opt in to renaming the matched functions and saving the IDB
python -m karta_ng.karta_headless --config configs/ --apply-names sample.bin
```

The report carries `approxLibs`, `versions.{original,testedAgainst}`, per-library match statistics, and a per-function ledger: `function_matches`, `unmatched_used_functions`, `structural_signature_candidates`, and `applied_names`.

Name import is opt-in by design — the default path never mutates your database.

### Generating a configuration

To match a library version nobody has published a config for, compile that version and point the analyzer at its objects:

```bash
karta-analyze-src <lib-name> <lib-version> -C 8 <source-dir> <archive>
```

`-C` sets how many `idapro` worker processes run in parallel. The run is resumable: existing per-file state files are reused. Output is `<lib>_<version>[_windows].json`.

## How it works

Identification and matching are separate phases joined only by JSON configuration files on disk.

The matching itself is **location-driven**. Rather than fingerprinting each function independently, Karta locates the compiled *files* and matches functions by their original order within each file. Unique constants and strings act as anchors that pin a small number of certain matches; confidence then propagates outward through the call graph and across file boundaries. A speculative branch that turns out contradictory is unwound rather than accepted.

karta-ng adds two relocation-independent signals on top:

- **Instruction mnemonics** — the normalized opcode sequence of a function. Static linking rewrites branch and call operands but not the mnemonic flow, so a unique instruction shape works as a final post-anchor confirmation pass.
- **CFG block topology** — per-basic-block predecessor and successor counts plus a terminal category (`exit`, `linear`, `branch`, `fanout`). Deliberately low-weight supporting evidence, so Karta's conservative scoring behaviour is preserved.

## What is new over upstream Karta

- **Far more seekers**, across 30+ categories rather than the original handful.
- **IDA 9.x and Python 3.13** — the UI layer follows the 9.x move to `ida_kernwin` and falls back to `idaapi` where it is unavailable.
- **`ida-domain` backend** — the analysis layer prefers the supported Hex-Rays Domain API and falls back to raw `idaapi` when it is unavailable.
- **Headless surfaces** — `karta_headless` for matching, `karta-analyze-src` for config generation with concurrent `idapro` workers, neither needing an `idat` subprocess.
- **Forced matching** (`get_nearby_version`) — when the exact version config is missing, match against the numerically closest available version, weighted toward earlier components.
- **Structural signatures and CFG topology**, described above.
- **Opt-in name import** — `--apply-names` renames only the matched functions.
- **`sark` removed.**

## Configuration schema

`karta-analyze-src` emits one record per source function with a stable tag set, guarded by `tests/unit/test_schema_roundtrip.py`:

`Function Name`, `Instruction Count`, `Stack Frame Size`, `Hash`, `Is Static`, `Numeric Consts`, `Strings`, `Calls`, `Unknown Functions`, `Unknown Globals`, `Code Block Sizes`, `Instruction Mnemonics`, `CFG Block Topology`, `Call Order`

The schema is **additive only**. Configurations generated before `Instruction Mnemonics` and `CFG Block Topology` existed remain valid and simply contribute no topology score.

> **Note for the DontStarveLuaJIT2 consumer:** `KartaConfig.cpp` parses `isStatic` / `Unknown Consts`, but every Karta lineage emits `Is Static` / `Unknown Globals`. That mismatch lives in the consumer; karta-ng emits the upstream-compatible schema.

## Development

### Tests

Unit tests are pure Python and need no IDA:

```bash
PYTHONPATH=src python -m pytest tests/unit/ -q
```

System tests need IDA (or `idapro`) and skip cleanly without it. The docker runner handles the whole environment:

```bash
tools/scripts/run_system_tests_docker.sh system
tools/scripts/run_system_tests_docker.sh system -- -k headless -v   # args after -- go to pytest
```

A green run is 31 passed: 23 unit and 8 system. CI runs the suite against IDA 9.4 on arm64 and IDA 9.1 on amd64.

### Architectural gates

Two boundaries are machine-enforced rather than merely documented — the portable layers (`config`, `core`, `libs`) hold the fingerprinting and matching logic and must stay IDA-free, receiving the disassembler by dependency injection:

```bash
sg scan --config sgconfig.yml   # ast-grep: no print(), no mocks in system tests, no live IDA in portable core
lint-imports                     # import-linter contracts
```

Both run in `.githooks/pre-commit`. Enable once per clone:

```bash
git config core.hooksPath .githooks
```

### Versioning

`__version__` in `src/karta_ng/__init__.py` is the single source of truth. `tools/sync_plugin_version.py` copies it into `ida-plugin.json` — both the manifest version and its exact `karta-ng==VERSION` dependency — and the pre-commit hook stages the result.

### Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request

New library seekers are the easiest contribution: subclass `Seeker` in `src/karta_ng/libs/`, implement `search_lib` and `identify_versions`, and call `MySeeker.register(MySeeker.NAME, MySeeker)` at the bottom of the module. `libs/__init__.py` walks the package, so being on disk is what makes it active — `len(lib_factory.get_lib_factory())` is the ground truth.

Read the `Seeker` docstring first. A seeker produces the `(library, version)` pair that names `<lib>_<version>.json`, the configuration the matcher resolves onto the binary — it is a lookup key, not a detector. It is worth writing only if identifying the library could reduce the reversing workload, which needs the code to be statically linked *into* the binary and the source to be obtainable and compilable. Packers, system DLLs and bare algorithms fail that test; 204 seekers were deleted in 2026-08 for failing it, and the docstring says why so it does not happen again.

Configuration files for library versions not yet covered are just as welcome.

## Acknowledgements

Karta was created by Eyal Itkin ([@EyalItkin](https://twitter.com/EyalItkin)) with the support of Check Point Research, and originally lived at [CheckPointSW/Karta](https://github.com/CheckPointSW/Karta). The original background reading is still the best explanation of the technique:

- [Karta — Matching Open Sources in Binaries](https://research.checkpoint.com/karta-matching-open-sources-in-binaries/)
- [Thumbs Up: Using Machine Learning to Improve IDA's Analysis](https://research.checkpoint.com/thumbs-up-using-machine-learning-to-improve-idas-analysis)
- [Upstream documentation](https://karta.readthedocs.io/)

Eyal stepped away from maintaining the project on leaving Check Point Research. It was later forked to [mitre/Karta](https://github.com/mitre/Karta) by MalwareFrank in May 2025 and updated for IDA Pro 9, with additional IDA 9.1 and Python 3.13 work by seifreed. karta-ng continues from that lineage.

## Contact

ping me on x [@mahmoudimus](https://x.com/mahmoudimus) or you may contact me from any one of the addresses on [mahmoudimus.com](https://mahmoudimus.com).
