Metadata-Version: 2.4
Name: samepage
Version: 0.1.1
Summary: MIQPS-style URL normalization for content deduplication.
Author-email: Mark Shervey <markshervey@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/marksher/samepage
Project-URL: Repository, https://github.com/marksher/samepage
Project-URL: Issues, https://github.com/marksher/samepage/issues
Keywords: url,normalization,canonicalization,deduplication,miqps,tracking
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Filters
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Dynamic: license-file

# samepage

_Last updated: 2026-07-24_

MIQPS-style URL normalization for content deduplication.

> _Are these two URLs the same page?_

`samepage` answers that question. It strips tracking parameters, canonicalizes
host/scheme/path/fragment, and preserves the parameters that actually identify
content — so two URLs that point at the same page hash to the same string.

## Credit

This library is a faithful adaptation of Pinterest's published approach,
**MIQPS** (Most Informative Query Parameter Strategy):

> [Smarter URL normalization at scale: how MIQPS powers content deduplication
> at Pinterest](https://medium.com/pinterest-engineering/smarter-url-normalization-at-scale-how-miqps-powers-content-deduplication-at-pinterest-4aa42e807d7d)
> — Pinterest Engineering

The four-layer design, the "fail-safe / conservative default" rule, and the
distinction between *neutral* and *non-neutral* parameters all come from that
post. All credit for the strategy goes to the Pinterest team. This package is
an independent open-source implementation; it is **not** affiliated with or
endorsed by Pinterest.

## Install

```bash
pip install git+https://github.com/marksher/samepage.git
```

(PyPI release coming once the API has stabilized in production use.)

## Quick start

```python
from samepage import normalize_url

normalize_url("https://www.example.com/p?utm_source=fb&id=42")
# -> 'https://www.example.com/p?id=42'

normalize_url("https://www.youtube.com/watch?v=abc123&utm_source=share&fbclid=x")
# -> 'https://www.youtube.com/watch?v=abc123'

# Two semantically-equal URLs hash to the same string:
a = normalize_url("https://EXAMPLE.com/article?b=2&a=1#anchor")
b = normalize_url("https://example.com/article?a=1&b=2")
assert a == b   # 'https://example.com/article?a=1&b=2'
```

## How it works

`samepage` is a per-URL filter that decides, for each query parameter, whether
to keep or drop it. The decision uses four layers:

| Layer | What it does | Example |
|------:|--------------|---------|
| **1. Allowlist** | Per-domain "always keep" content keys | `?v=…` on youtube.com, `?k=…` on amazon.com |
| **2. Blocklist** | Universal tracking-param strip (set + regex) | `utm_*`, `fbclid`, `gclid`, `ref`, `igshid`, `mkt_tok`, … |
| **3. Learned set** | Optional, per-domain non-neutral params discovered offline | `{"weirdsite.com": {"page_id"}}` |
| **4. Default** | Conservative fallback — **keep** | Unknown param on unknown domain |

The blocklist runs **before** the allowlist, so a too-broad allowlist entry
can't accidentally let `fbclid` through.

In addition to the parameter filter, `samepage` applies the URL hygiene any
serious dedupe scheme needs:

- lowercase scheme and host
- drop `:80` for `http`, `:443` for `https`
- collapse `//` runs in the path; strip a single trailing slash on non-root paths
- strip the fragment (`#…`) — fragments never reach the server
- sort surviving query parameters alphabetically (so equivalent URLs hash identically)

## Custom configuration

Most callers just want `normalize_url`. If you need to inject your own config,
use the class directly:

```python
from samepage import MIQPSNormalizer

# Add a tracking param the default set doesn't know about:
norm = MIQPSNormalizer(extra_blocklist=frozenset({"sessionid"}))
norm.normalize("https://example.com/p?id=1&sessionid=abc")
# -> 'https://example.com/p?id=1'

# Pin a param as content for a specific host:
norm = MIQPSNormalizer(learned={"weirdsite.com": frozenset({"page_id"})})
norm.normalize("https://weirdsite.com/x?page_id=42&utm_source=fb")
# -> 'https://weirdsite.com/x?page_id=42'
```

## Plugging in a learned non-neutral set

The Pinterest paper describes an *offline rendering pipeline*: for each
domain, render URLs with and without each query parameter, hash the visual
output, and build a per-domain set of parameters that change the rendered
page (the "non-neutral" set).

`samepage` exposes that hook on `MIQPSNormalizer(learned=…)` but does **not**
ship a renderer. If you have one, the contract is:

```python
learned: dict[host_suffix: str, frozenset[param_name: str]]
```

`host_suffix` matches by suffix, so an entry for `example.com` covers
`api.foo.example.com`. Build the dict however you like — headless Chrome,
Playwright, screenshot diffing, perceptual hashing — and pass it in.

## What's intentionally not here (yet)

- **A learned non-neutral set out of the box.** Producing it requires
  Pinterest-scale infrastructure. The hook is there; the data isn't.
- **PSL-aware host parsing.** `samepage` matches host *suffixes* by string,
  not by registrable domain via the [Public Suffix List](https://publicsuffix.org/).
  In practice this is fine; if you need true PSL handling, wrap the call.
- **PyPI release.** Install via `pip install git+…` for now.

## Development

```bash
git clone https://github.com/marksher/samepage.git
cd samepage
pip install -e ".[test]"
pytest
```

34 tests cover query stripping, the per-domain allowlist, host/scheme
normalization, path/fragment hygiene, dedupe pairs, edge cases, and custom
config injection.

## License

MIT — see [LICENSE](LICENSE).
