Metadata-Version: 2.4
Name: indexgap
Version: 1.5.1
Summary: Lint your programmatic SEO pipeline — from keywords to indexed pages. No dependencies.
License: MIT
Project-URL: Homepage, https://github.com/borisowlexa2010-star/IndexGap
Project-URL: Changelog, https://github.com/borisowlexa2010-star/IndexGap/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/borisowlexa2010-star/IndexGap/issues
Keywords: seo,programmatic-seo,sitemap,indexnow,aeo,geo,search-console,site-audit,lint
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Site Management
Classifier: Topic :: Text Processing :: Markup :: HTML
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# indexgap

[![tests](https://img.shields.io/github/actions/workflow/status/borisowlexa2010-star/IndexGap/tests.yml?branch=main&label=tests)](https://github.com/borisowlexa2010-star/IndexGap/actions/workflows/tests.yml)
[![pypi](https://img.shields.io/pypi/v/indexgap)](https://pypi.org/project/indexgap/)
[![python](https://img.shields.io/pypi/pyversions/indexgap)](https://pypi.org/project/indexgap/)
[![dependencies](https://img.shields.io/badge/dependencies-none-brightgreen)](https://github.com/borisowlexa2010-star/IndexGap/blob/main/pyproject.toml)
[![license](https://img.shields.io/badge/license-MIT-green)](https://github.com/borisowlexa2010-star/IndexGap/blob/main/LICENSE)

**Lint your programmatic SEO pipeline — from keywords to indexed pages.**

> [Русская версия](README.ru.md)

You generated three thousand pages from a dataset. They're on disk, they're in
the sitemap, and there's no traffic. `indexgap` tells you where they were lost.

Pure Python 3.9+ standard library. No dependencies, no API keys, no paid
subscriptions. Nothing leaves your machine without an explicit flag.

---

## Why this exists

Plenty of tools audit *a page*. A generated pipeline breaks differently —
systematically and quietly:

* a page is in the sitemap, but no internal link points to it, so the crawler
  never arrives;
* three hundred pages differ by five words, and the search engine keeps one;
* the model wrote "12 years in business" and "3,500 orders" — neither number
  exists in any row of your data, but it reads convincingly;
* `lastmod` equals the build date on every page, so it means nothing;
* two keywords with the same intent produced two competing pages.

None of this is visible page by page. Each one looks fine on its own.

---

## Install

```bash
pipx install indexgap
```

`pipx` keeps the tool in its own environment and still puts `indexgap` on your
PATH — `pip install` into a system Python is refused outright on most current
distributions. If you do not have it: `python3 -m pip install --user pipx`.
To work from the `main` branch instead: `git clone` the repository and
`pip install -e .`

Then, once per project:

```bash
cd ~/projects/my-site
indexgap init
```

`init` reads the project and records what makes it different: where the pages
are, the site URL, the content type, the dataset. It installs skills into
`.claude/skills/` so your coding agent picks them up on its own, and adds the
working files to `.gitignore`.

Nothing project-specific is ever copied between projects — it is detected
fresh each time. **The IndexNow key in particular is never carried over:**
it is bound to one domain by a file at that site's root, and a borrowed key
returns 403. `indexgap init --key` mints a new one for this project.

After that, the daily command is just:

```bash
indexgap check
```

---

## Commands

```bash
indexgap init                       # install into this project
indexgap plan keywords.csv          # audit the keyword set before generating
indexgap check                      # everything local: text, structure, machine-readability
indexgap brief --write              # turn the findings into work orders beside the pages
indexgap sitemap --out-dir ./public # sitemap with sharding and an honest lastmod
indexgap notify --key <your-key>    # tell IndexNow what actually changed
indexgap doctor --sitemap ./public/sitemap.xml --indexed gsc.csv
indexgap portfolio projects.json    # every site you own, in one run
indexgap profiles                   # what the content-type presets change
indexgap cite --domain example.com  # do AI answers cite you? (your own API keys)
```

Output is English by default, Russian with `--lang ru` (or `INDEXGAP_LANG`, or
your system locale). That includes the HTML report, every finding description,
`--help`, and the skills `indexgap init` installs into your project.

Understands both built HTML and Markdown sources with frontmatter. Input file
encoding is detected, not assumed: UTF-16, BOM, and the cp1251 CSVs that
Russian Excel produces all read correctly. Exports are read as they come —
CSV with any delimiter, XLSX straight from Ahrefs or Semrush without
re-saving, JSON, NDJSON, an XML sitemap, or a plain list of URLs — and the
keyword column is found whether the export calls it `Keyword`, `Фраза`,
`Запрос` or `Search Term`.

---

## What it checks

**Before generating** — exact duplicate keys, slug collisions and *same-intent*
keys. That last one matters most: two keywords meaning the same thing will
produce two competing pages, and it is cheaper never to create the second.

**After generating — the text**

| Check | Why it matters |
|---|---|
| **Numbers absent from your data** | prices, terms and counts are verified against the source row. A number that appears nowhere in the dataset is never forgiven, no matter how many pages repeat it |
| Identical heading skeletons | different words, same structure — a stamping tell |
| Identical opening sentences | the second tell |
| Leftover brief, `status: draft` | unfinished pages never reach the sitemap or the IndexNow queue |
| "Click here" anchors | a link with no meaning in it |

**After generating — the structure**

Near-duplicates (exact pairwise below 400 pages, MinHash + LSH above), share of
unique text measured *by bigrams* rather than words, thin pages, orphans,
click depth, `noindex`, `nosnippet`, foreign canonicals, missing or duplicate
H1, duplicate titles and descriptions.

**Machine readability for AI search**

Snippet controls (`nosnippet`, `max-snippet:0`), `robots.txt` rules for
OAI-SearchBot, PerplexityBot, ClaudeBot, GPTBot and Google-Extended (they are
not interchangeable — blocking OAI-SearchBot removes you from ChatGPT search
answers, while Google-Extended does not affect AI Overviews), empty JS shells,
a direct answer in the first paragraph, question-shaped subheadings, valid
JSON-LD that matches the visible text, machine-readable dates and author.

**At publish time** — sitemap sharding past 45,000 URLs and a `lastmod` that
changes only when the text, title or description changed. Editing one menu item
does not mark the whole site as modified. IndexNow sends only what changed.

---

## From findings to work orders

A report answers "what is wrong with me". `indexgap brief` answers "what do I
do" — it lays the same findings out as markdown files next to the pages they
belong to, each with an imperative fix, the thresholds the fix has to meet, and
the dataset row as the only permitted source of numbers.

```bash
indexgap brief --dataset keywords.csv        # dry run: says what it would write
indexgap brief --dataset keywords.csv --write
```

On the live 2,970-page catalogue the check reports 6,074 findings and `brief`
writes 966 work orders. The difference is three placement rules, and they are
the whole point:

* **A property of the template gets one brief, not 2,919.** `no-question-headings`
  fired on 2,919 of 2,970 pages. That is one edit to one template, and 2,919
  identical tasks would bury the 51 pages that actually differ. The same holds
  per language: a finding on every page of the Chinese version is 10% of the
  site but still one template fix.
* **Near-duplicates are fixed as a group.** 588 duplicate findings are 72
  groups. One page out of a group of 17 cannot be fixed alone — the brief goes
  to the group: keep one, pull the rest apart by intent, and never link them to
  each other.
* **robots.txt, markup and hreflang clusters belong to the site**, not to a
  page, and live in their own file.

`--limit` (50 by default) writes the heaviest pages first; 892 briefs is not a
task list, it is a second report. Nothing is created without `--write`.

**The package writes no text, on purpose.** Its central check compares the
numbers on a page against the dataset row that produced it. If the package
supplied those numbers itself, the check would be checking its own output and
would always be green. So `brief` states the task and stops; a person or an
agent writes the words.

---

## The funnel

The reason to install it at all:

```
Generated                 58
Indexable                 56   (−2: noindex or a canonical pointing elsewhere)
In sitemap                56
In at least one index     28   (−28: the engine knows the URL and didn't add it)

Why pages are not indexed:
    7  orphaned or unreachable  → link them from hub pages
   12  deeper than the click budget → move them up
    9  no local explanation → check status in Search Console
```

Index data comes from ordinary exports — no API keys. A webmaster panel is the
direct source (Search Console, Bing Webmaster Tools, Yandex.Webmaster, Naver,
Seznam), but not everyone has one, so exports from **Ahrefs, Semrush, Serpstat,
Moz, Screaming Frog, Sitebulb, JetOctopus, OnCrawl, Netpeak, GA4, Matomo,
Plausible** — or a plain list of URLs — are read too, in CSV, XLSX, JSON,
NDJSON or XML.

They are not interchangeable, and the tool refuses to pretend otherwise. A
panel answers "does the engine know this page". Analytics proves a page is
indexed, but only for pages someone actually visited — its silence proves
nothing. A crawler proves reachability, not indexation. Ahrefs and Semrush are
*their* index, not Google's. So the funnel step is renamed to match the
evidence — "at least in one index" versus "known to a third-party service" —
and when a crawler export sits next to a panel, the report says out loud that
the step count is higher than real indexation. Engine-vs-engine comparison
runs over panels only.

**Ask for all the panels you have, not just Google.** A page missing *everywhere* is a technical problem. A page missing
*only in one engine* was crawled and accepted by the others, which makes it a
quality or speed question that technical fixes rarely solve. Without the split
the two look identical and people fix the wrong thing.

One honest caveat the tool states out loud: the Search Console "Pages" export
is an *impressions* report, not an index report. A page that is indexed but has
no impressions won't appear in it, so on a young site the funnel overstates
losses.

---

## Multilingual and multi-region

A programmatic pipeline is usually multilingual, and that breaks in ways a
single-language site never does.

Text volume and title/description length are measured by each page's **script**,
in display width — not by one language detected for the whole project. That
distinction is not academic: on a live 2,970-page catalogue in ten languages the
project language came out as `en`, every Chinese page was counted as English,
and all 174 `thin` findings were false. Width also handles mixed strings, which
is what real pages are: those Chinese titles are 43% Han and the rest Latin.

Anchor length is judged only in scripts where length means something. All 914
`vague-anchor` findings on that same site were false — "यमन" (Yemen) and "হোম"
(Home) are three characters and complete words.

**hreflang** is checked as a cluster: a missing self-reference, a one-way link
(Google discards the whole cluster rather than counting it partly), an alternate
that is noindex or canonicalised elsewhere, a canonical that leaves the language
and cancels the cluster, a missing `x-default`, and language codes where a
country code was meant — `uk` is Ukrainian, not the United Kingdom. On a
monolingual site none of this runs.

And `en-us` versus `en-gb` is **not** a duplicate. Those pages are legitimately
near-identical; "keep one and set a canonical" would delete the regional
version. Pairs inside one hreflang cluster are excluded from the duplicate
check and reported separately.

---

## Content-type profiles

Thresholds differ across content types by substance, not taste. 250 words is
normal for a guide and absurd for an event card; fact-checking is meaningless
where there is no dataset at all.

| Profile | For | What changes |
|---|---|---|
| `catalog` | pages generated from data rows | fact-checking is primary; duplicate threshold 0.80; thin under 250 words |
| `events` | listings, schedules, venues | threshold 0.88 — two dates of one tour are legitimately similar; adds `stale-event` |
| `ugc` | feeds, threads, reviews | fact-checking is switched off *and says so*; threshold 0.92 |
| `product` | dozens of landing pages, not thousands | duplicates aren't the issue; the AI-readability checks are |

`stale-event` catches an event whose date has passed while the page stays open
to indexing. That isn't a traffic problem, it's a trust problem: someone drives
to a concert that no longer exists.

For `ugc`, silence is not a clean bill of health — it means there was nothing to
check against, and the tool prints that as a line rather than leaving you to
assume.

---

## Portfolio

Separate reports answer "what's wrong with this site". A portfolio answers the
question you can't see in them: **what breaks the same way everywhere.**

```
  · visa       2933 pages  critical  412  [catalog]
  · events      840 pages  critical   61  [events]
  · feed       5100 pages  critical    0  [ugc]
  · product      34 pages  critical    2  [product]

Shared problems, as a share of each project's pages:
  orphan        3 projects: visa 41%, events 38%, product 35%
  same-opening  2 projects: visa 62%, events 55%
```

Shares, not counts: a hundred findings across three thousand pages and ten
across twenty are the same disease at different volumes. One project failing
doesn't stop the run — it becomes a line in the report.

---

## What it does *not* claim

It does not promise citations in AI search, and it says so in the output.
Ahrefs, across 75,000 brands, found AI visibility correlates most with mentions
*off* your site (0.66–0.74) and with page count at **0.19** — which is exactly
what a programmatic pipeline produces. 76% of AI Overview citations come from
pages already ranking in the classic top 10.

So: machine readability is a necessary condition and this tool's job. Getting
cited is decided by work outside your files, and that is not a code problem.

`indexgap cite` measures the thermometer, and says so: it asks Perplexity, the
OpenAI Responses API, the Gemini API and Grok a set of real questions and counts
how often your domain comes back in the sources. It reports a share of runs, not
yes/no, because these answers are not deterministic. And it never says "ChatGPT
cites you" — what it measured is the API, which OpenAI's own docs describe as
behaving differently from the product. It is the only command that needs API
keys, it is off by default, and it sends nothing without `--send`.

No `llms.txt` generator either. Google has stated it does not support it and
has no plans to; no engine has confirmed using it for ranking. Generating a
file nobody reads is a ritual, not a feature.

It also doesn't write or rewrite content, check rankings, or call paid services.
`indexgap brief` formulates the task and stops there — on purpose. The central
check compares the numbers on a page against the dataset row that produced it;
if the package supplied those numbers itself, the check would be checking its
own output and would always be green.
Rejecting keywords and acting on contested findings is always confirmed by a human.

And it does not replace Search Console — it leads you there. If the tool says
one thing and Search Console says another, Search Console is right.

---

## Tests

```bash
python3 -m unittest discover -s tests
```

301 scenarios. Each one is a reproduced defect found by two waves of adversarial
review and one run against six live sites, plus the behaviour of profiles,
portfolio and project installation. The rule: a finding without a test comes back.

## Calibrated on live sites, not fixtures

Thresholds are not guesses. They were checked against six production sites —
7,149 sitemap URLs, 5,041 pages fetched and parsed — and the run changed the
tool in three ways:

* **One cause, not four findings.** Two of the six sites served every page as an
  empty JavaScript shell. The tool reported 1,099 `js-shell` *and* 1,099
  `low-uniqueness` *and* 1,098 `orphan` — one disease counted four times. Checks
  that need text or links are now skipped on a shell, and the run says how many
  shells there were.
* **Duplicates are groups, not pages.** 588 near-duplicate pages turned out to be
  72 groups, the largest holding 24. "Rewrite 588 pages" is a sentence;
  "untangle 72 topics" is a task.
* **A finding on every page is a template property.** `vague-anchor` fired on
  2,970 of 2,970 pages — the culprits were a language switcher (`中文`) and a
  social link (`VK`), short in characters and perfectly informative. Anchor
  length is now measured in the right unit, and any code that hits ≥90% of pages
  is labelled as something to fix once in the template.

---

## For agents

`SKILL.md` at the root is the overview. `indexgap/skills/` holds four
stage-specific skills — `indexgap-plan`, `indexgap-review`, `indexgap-publish`,
`indexgap-portfolio` — which `indexgap init` copies into your project's
`.claude/skills/`. A narrow skill fires more accurately than a broad one:
the agent sees only the commands and finding codes for the stage it's in.

Codex users: `indexgap init --agents` writes a marked block into `AGENTS.md`.

---

MIT.
