Metadata-Version: 2.5
Name: equate
Version: 0.0.26
Summary: Tools for matching things: featurize, compare, and match collections of objects (fuzzy correspondence, not exact equality).
Project-URL: Homepage, https://github.com/thorwhalen/equate
Project-URL: Changelog, https://github.com/thorwhalen/equate/blob/master/CHANGELOG.md
Author: thorwhalen
License-Expression: MIT
License-File: LICENSE
Keywords: assignment,entity-resolution,joining,matching,record-linkage,similarity
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: numpy
Requires-Dist: scipy
Provides-Extra: active
Requires-Dist: scikit-activeml; extra == 'active'
Provides-Extra: ann
Requires-Dist: hnswlib; extra == 'ann'
Provides-Extra: api
Requires-Dist: cohere; extra == 'api'
Requires-Dist: openai; extra == 'api'
Provides-Extra: benchmarks
Provides-Extra: bio
Requires-Dist: biopython; extra == 'bio'
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: sentence-transformers; extra == 'embeddings'
Provides-Extra: eval
Requires-Dist: scikit-learn; extra == 'eval'
Provides-Extra: examples
Requires-Dist: pandas; extra == 'examples'
Requires-Dist: py2store; extra == 'examples'
Requires-Dist: requests; extra == 'examples'
Provides-Extra: fuzzy
Requires-Dist: rapidfuzz; extra == 'fuzzy'
Provides-Extra: graph
Requires-Dist: networkx; extra == 'graph'
Provides-Extra: graph-kernels
Requires-Dist: grakel; extra == 'graph-kernels'
Provides-Extra: graph-matching
Requires-Dist: pygmtools; extra == 'graph-matching'
Provides-Extra: grub
Requires-Dist: grub; extra == 'grub'
Provides-Extra: llm
Requires-Dist: openai; extra == 'llm'
Provides-Extra: lsh
Requires-Dist: datasketch; extra == 'lsh'
Provides-Extra: ot
Requires-Dist: pot; extra == 'ot'
Provides-Extra: phonetic
Requires-Dist: jellyfish; extra == 'phonetic'
Provides-Extra: service
Requires-Dist: anyio; extra == 'service'
Requires-Dist: fastapi; extra == 'service'
Requires-Dist: pydantic; extra == 'service'
Requires-Dist: qh; extra == 'service'
Requires-Dist: uvicorn; extra == 'service'
Provides-Extra: sklearn
Requires-Dist: scikit-learn; extra == 'sklearn'
Provides-Extra: timeseries
Requires-Dist: dtaidistance; extra == 'timeseries'
Requires-Dist: tslearn; extra == 'timeseries'
Description-Content-Type: text/markdown

# equate

Match things that have no shared identifier: names with typos, product listings from two shops, records from two databases, time series, graphs, the columns of two tables. `equate` decides which items *correspond* and returns those correspondences as scored pairs.

```bash
pip install equate
```

```python
from equate import match

m = match(
    ["apple pie", "banana split", "cherry tart"],
    ["cherry tarte", "aple pie", "banana split"],
)
dict(m.labeled_pairs())
# {'apple pie': 'aple pie', 'banana split': 'banana split', 'cherry tart': 'cherry tarte'}
m.scores  # [0.76, 1.0, 0.84]
```

`match(A, B)` works with no configuration: it featurizes both sides (character n-gram TF-IDF), scores every pair (cosine), and finds the best one-to-one assignment (scipy's linear assignment). Every step is swappable with one keyword, and nothing heavy is installed unless you ask for it.

## Matching, a stage at a time

Matching factors into three stages: **featurize → compare → match**. Each takes a registered name or any callable.

```python
match(A, B, compare="jaro_winkler")  # score raw pairs with a string comparator instead
match(A, B, featurize="sbert")  # dense embeddings (pip install 'equate[embeddings]')
match(
    A, B, block="sorted_neighborhood"
)  # score only candidate pairs (sparse), for scale
match(A, B, how="greedy")  # or 'assign' (default), 'stable', 'soft'
match(A, B, compare="levenshtein_distance")  # a distance: minimized automatically
```

- **Records work as they are.** A list of dicts is read through its values, so `match(people_a, people_b)` needs no flattening.
- **Results are structured.** A `Matching` iterates as `(i, j)` pairs (`dict(m)` works), and has `.scores`, `.labeled_pairs()` and, for `how='soft'`, the transport `.plan`.
- **Strategies are registries** (`equate.featurizers`, `comparators`, `blockers`, `matchers`, `clusterers`), so a strategy of your own is one `register` call, not a subclass.
- **`equate.score_matrix(A, B, ...)`** gives the scored `ScoreMatrix` without matching, for when you want to match later or several ways.

## Deduplicate one collection, resolve several

Matching two collections is the common case; grouping is the general one. Transitive closure is a choice you make (`cluster='connected_components'` or `'correlation'`), never a side effect.

```python
from equate import dedupe, resolve

dedupe(["Jon Smith", "Jon Smyth", "Kate Doe", "Kate  Doe"], threshold=0.8).groups()
# {0: [0, 1], 1: [2, 3]}
resolve(["Jon Smith", "Kate Doe"], ["John Smith"], ["Kate Do"], threshold=0.8).groups()
# {0: [0, 2], 1: [1, 3]}   (indices into the pooled collections)
```

`equate.cluster.classify` adds a Fellegi-Sunter three-way decision (match / possible / non-match) with unsupervised EM.

## Tables: join keys, near-duplicate columns, completion

Columns are objects too. Each is featurized as its set of values and compared by overlap, so no header names are needed.

```python
from equate import find_join_keys, complete_table

orders = {"customer": ["C-001", "C-002", "C-001"], "city": [None, "Lyon", None]}
customers = {"cid": ["C-002", "C-001"], "city": ["Lille", "Paris"]}

find_join_keys(orders, customers)[
    0
]  # ColumnMatch(left='customer', right='cid', score=1.0, ...)
complete_table(orders, customers).table
# {'customer': ['C-001', 'C-002', 'C-001'], 'city': ['Paris', 'Lyon', 'Paris']}
```

`complete_table` discovers the key (directional *containment*: how much of the target column lies in the source's), aligns rows by fuzzy key matching (so `'c001'` meets `'C-001'`), and fills only the missing cells. `find_near_duplicate_columns(table)` finds redundant columns. Tables are column dicts, lists of row dicts, or anything DataFrame-like.

## Matching sequences: `compare='dtw'` and friends

To match a collection of *sequences* (time series, token lists, DNA), compare each pair by aligning its elements, then match the collections as usual.

```python
from equate import match

A = [[0, 1, 2, 3, 2], [5, 5, 6, 7, 7]]
B = [[5, 6, 6, 7, 7], [0, 1, 1, 2, 3, 2]]
dict(match(A, B, compare="dtw"))  # {0: 1, 1: 0}
```

The comparators are `'dtw'` (exact DTW in a Sakoe-Chiba band), `'subsequence_dtw'`, `'soft_dtw'`, `'needleman_wunsch'` and `'smith_waterman'`. A *distance* comparator such as `'dtw'` is minimized automatically, and `sense=` overrides that. `equate.compare.sequence.dtw.align(a, b)` returns an `Alignment(score, path)` with the element-level correspondence. Faster engines are optional: `pip install 'equate[timeseries]'` / `'equate[bio]'`, then pass `backend=`.

## Matching graphs: `equate.align`

To align the *nodes* of two graphs, use `align`. A node's match depends on how its neighbours match, so this is a quadratic problem, not a flat one.

```python
from equate import align

G1 = {"a": ["b"], "b": ["a", "c"], "c": ["b"]}
G2 = {"x": ["y"], "y": ["x", "z"], "z": ["y"]}
dict(align(G1, G2).labeled_pairs())  # {'a': 'x', 'b': 'y', 'c': 'z'}
```

The default is IsoRank, built in. Also built in: `how='final'` (IsoRank gated by node attributes, from `label=`), `'regal'` (REGAL/xNetMF structural embeddings) and `'cone'` (CONE-Align: per-graph embeddings aligned by Procrustes and optimal transport). `how='gw'` is Gromov-Wasserstein optimal transport (`equate[ot]`), and `'rrwm'` uses pygmtools (`equate[graph-matching]`). Graphs can be networkx graphs, `{node: neighbours}` mappings, or adjacency matrices. To match collections *of* graphs, use `featurize='wl'` (Weisfeiler-Lehman).

## Reviewing matches: the HTTP service and the review UI

`equate.service.MatchService` exposes matching as JSON-in/JSON-out operations: a stateless `match`, and review *sessions* that score two collections once, then take confirm / reject / reassign edits and active-learning labels, re-solving the assignment from the cached scores after each one.

```python
from equate.service import MatchService

svc = MatchService()
view = svc.open_session(["Jon Smith", "Kate Doe"], ["Kate Do", "John Smith"])
svc.edit(view.session_id, [{"action": "reject", "item": 0, "candidate": 1}])
svc.next_query(view.session_id)  # the pair whose label would help most
```

`pip install 'equate[service]'` adds the HTTP API (built with [qh](https://github.com/i2mint/qh)): `python -m equate.service serve`, then `POST /match`, `POST /sessions`, `GET|POST /sessions/{id}/candidates/{item}`, `POST /sessions/{id}/edits`, `POST /sessions/{id}/label`. `equate.service.mk_app(service)` returns the FastAPI app to mount elsewhere. `python -m equate.service serve ui=ui/dist` also serves the built review UI from the same process. Sessions live in any `MutableMapping` (`MatchService(sessions=...)`), so a `dol` store persists them. Use one when running several server processes (`uvicorn --workers N`): the default in-memory store is per process. `open_session(..., floor=0.5)` sets a score floor: a pair below it is never proposed, so rejecting an item's only good candidate leaves the item unmatched instead of handing it a poor one.

The review UI (React, [zodal](https://github.com/i2mint/zodal) + shadcn) is in [`ui/`](ui/README.md): a review queue ordered by uncertainty, side-by-side field diffs, confirm / reject / reassign with optimistic updates, and a labelling widget.

## Measuring a matcher: `equate.evaluate`

Dependency-light metrics that return precision **and** recall (a `Scores`), never F1 alone, plus a harness that scores a matcher (pairwise) on unseen-entity splits and reports the cost and base rate next to every number. The standard suites (Abt-Buy, Amazon-Google, DBLP-ACM, WDC Products) are a `load_benchmark` away, below.

```python
from equate.evaluate import pairwise_prf, bcubed, pairs_to_labels, benchmark_matcher

pairwise_prf(predicted_pairs, gold_pairs)  # Scores(precision, recall, f1, support)
bcubed(
    pairs_to_labels(predicted_pairs, n=n),  # B³, the default cluster metric
    pairs_to_labels(gold_pairs, n=n),
)
print(
    benchmark_matcher(on=["toy_names"]).to_markdown()
)  # equate.match, 3 seeds, entity splits
```

A single run (a benchmark's canonical split) reports a bootstrap confidence interval instead of a seed spread: `F1 0.628 [0.588, 0.670]` on WDC Products, from resampling the test *entities* (`ci='auto'`; pass a level to get it on every row). Also there: `gmd` (generalized merge distance: the cost of the splits and merges between two clusterings, with pluggable costs), `ceaf`, `cluster_metrics` (ARI and V-measure use scikit-learn when installed, numpy otherwise; `pip install 'equate[eval]'`), `blocking_metrics`, `pr_auc`, `ranking_metrics`, `select_threshold` (pass validation data), `entity_split` and `assert_no_leakage`. Benchmarks are a registry (`equate.evaluate.benchmarks`), so a new dataset is a registered loader.

The standard suites download on first use, into `~/.local/share/equate/benchmarks` (or `$EQUATE_DATA_DIR`), never into the package:

```python
from equate.evaluate import benchmark_matcher, load_benchmark

ds = load_benchmark("amazon_google")  # Leipzig original: complete gold, CC BY 4.0
ds.meta["difficulty"]  # literature context + measured linearity
benchmark_matcher(on=["dblp_acm", "wdc_products"]).to_markdown()
```

`abt_buy`, `amazon_google` and `dblp_acm` take `source='leipzig'` (default; the complete gold mapping) or `source='deepmatcher'` (DeepMatcher's preprocessed tables, with its canonical labelled pairs in `ds.pairs`). `wdc_products` takes `corner_cases=80|50|20`, `unseen=0|50|100` and `dev_size='small'|'medium'|'large'`, defaulting to the hardest generalization variant, and carries WDC's own train/valid/test split, which `benchmark_matcher` uses (`split='auto'`). Every download is checked against a pinned SHA-256. `meta['difficulty_profile']['linearity']` is the best F1 a single TF-IDF cosine threshold reaches over all pairs: near 1, the benchmark cannot tell matchers apart.

## Optional extras

The core needs only numpy and scipy. Everything heavier is an extra, imported only when used, with an install hint if it is missing:

| Extra | Brings | For |
|---|---|---|
| `fuzzy`, `phonetic` | rapidfuzz, jellyfish | fast string comparators |
| `embeddings`, `api` | sentence-transformers; openai, cohere | dense text featurizers |
| `ann`, `lsh` | hnswlib, datasketch | blocking at scale |
| `graph`, `ot` | networkx, POT | max-weight / Kuhn-Munkres matchers; soft (optimal transport) matching |
| `graph-kernels`, `graph-matching` | grakel, pygmtools | graph featurizers and quadratic matchers |
| `timeseries`, `bio` | dtaidistance, tslearn; biopython | faster sequence-alignment engines |
| `eval` | scikit-learn | ARI / V-measure (numpy fallback otherwise) |
| `active`, `llm` | scikit-activeml; openai | active learning; LLM re-ranking |
| `service` | qh, fastapi, pydantic, uvicorn | the HTTP service |
| `benchmarks` | (nothing yet) | the standard benchmark loaders |
| `grub`, `sklearn` | grub; scikit-learn | the legacy featurizers of the original API |
| `examples`, `dev` | pandas, py2store, requests; pytest, ruff | the example scripts; development |

## Design

`equate` owns the orchestration and wraps mature libraries at each stage. The reasoning, and a cited survey of the field behind every decision, is in [`docs/research/`](docs/research/README.md). Start with [`00-taxonomy-and-terminology.md`](docs/research/00-taxonomy-and-terminology.md) and the decision register [`11-design-decisions-and-open-questions.md`](docs/research/11-design-decisions-and-open-questions.md).

## The original API

The first versions of `equate` exposed the functions below, and they are all still importable, with the same signatures. One thing changed underneath: the default featurizer is now `equate`'s own character n-gram TF-IDF instead of `grub`'s word TF-IDF (#4), so similarity *scores* differ from older releases, while the matchings in these examples are the same. The redesigned stages are built from the same pieces: `similarity_matrix` is `score_matrix` with the TF-IDF default, and the `equate.util` matchers are now in the `matchers` registry. New code should prefer `match` / `score_matrix`. The old `grub`-based `equate.completion.similarity_matrix` still needs `pip install 'equate[grub]'`.

### Greedy matching with a score function: `match_greedily`

Say you have two sets of strings, and all you do is want to is match each element of
the "keys" set to an element of the "values" set (never reusing the same value for a 
different key), and say you know that a matching value string will differ from its 
key only by a few characters. In that case just do this:

```python
from equate import match_greedily

keys = ["apple", "banana", "carrot"]
values = ["car", "app", "carob", "cabana"]
dict(match_greedily(keys, values))
# == {'apple': 'app', 'banana': 'cabana', 'carrot': 'carob'}
```

Note that by default, `match_greedily` uses the edit-distance as its `score_func`, 
and you can specify your own custom `score_func`. But the matching is done "greedily". 
This means that at every step, the highest-score value will be taken as the match. 
Some times this is not ideal (typically, your matches validity will decline along with 
the number of keys matched). 

### A similarity matrix and a matcher: `match_keys_to_values`, `similarity_matrix`

Here's a more involved entry point:

```python
from equate import match_keys_to_values

keys = ["apple pie", "apple crumble", "banana split"]
values = ["american pie", "big apple", "american girl", "banana republic"]
dict(match_keys_to_values(keys, values))
# == {'apple pie': 'american pie',
# 'apple crumble': 'big apple',
# 'banana split': 'banana republic'}
```

The algorithm that gets you these matches is a bit more involved, and parametrizable.
This is how it works.

First, it computes a similarity matrix, and then applies a "matcher" that will 
search through this similarity matrix for an optimal matching, using any 
optimal search function you want (and of course, we provide a few standard ones).

The similarity matrix, if full, contains scores for every possible combination of 
keys and values. 
You can specify how to compute this similarity matrix, which means that if you 
don't want to compute the similarity for every combination, you don't need to. 
Usually, your similarity matrix will be a "sparse matrix", that is, only specify a 
few non-zero entries.

The default similarity matrix function uses a `obj_to_vect` function along with a 
vector similarity function `similarty_func` to compute what the `score_func` 
did in our first `match_greedily` function.
By default here, though, `similarity_matrix` will use methods that are more powerful 
than just an edit-distance. It will learn and use a "TfIdf" vectorization 
(a.k.a. embedding) and a cosine similarity function. 
As a result, you should get finer matchings.

```python
from equate import similarity_matrix

keys = ["apple pie", "apple crumble", "banana split"]
values = ["american pie", "big apple", "american girl", "banana republic"]
m = similarity_matrix(keys, values)
m.round(2).tolist()
# == [[0.32, 0.42, 0.01, 0.0], [0.04, 0.35, 0.01, 0.02], [0.03, 0.04, 0.02, 0.44]]
```

The `equate.util` module has a few optimal matching function you can use from 
there to extract the matching pairs from the matrix. 
At the time of writing this, we've implemented: 
`greedy_matching`, 
`hungarian_matching`, 
`maximal_matching`,
`stable_marriage_matching`, and 
`kuhn_munkres_matching`.

At the time of writing this, the default `matcher` used by `match_keys_to_values` is 
[hungarian_matching](https://en.wikipedia.org/wiki/Hungarian_algorithm).

## Worked example: in search of an import-to-package name matcher

Here, we'll go through an actual practical example of when you might want to match 
things: "Guessing" the pip install name from a pip package name, 
and other related analyses.

### The problem

Ever got an import error and wondered what the pip install package name was.

Say... 
```
ImportError: No module named skimage
```

But it ain't `pip install skimage` is it (well, it USED to not to, but you get the point...).
What you actually need to do to install (with `pip`) is:
```
pip install scikit-image
```

I would have guessed that!

So no, it's annoying. It shouldn't be allowed. And since it is, there should be an index out there to help out, right?

```
pip install --just-find-it-for-me skimage
```

Instead of just complaining, I thought I'd throw some code at it.
(I'll still complain though.)

Here's a solution: Ask the world (of semantic clouds -- otherwise known as "Google") about it...

### A (fun) solution


```python
import requests
import re
from collections import Counter

search_re = re.compile("(?<=pip install\W)[-\w]+")


def pkg_name_options(query):
    r = requests.get(
        "https://www.google.com/search", params={"q": f'python "pip install" {query}'}
    )
    if r.status_code == 200:
        return Counter(
            filter(lambda x: x != query, p.findall(r.content.decode("latin-1")))
        ).most_common()


def best_guess(query):
    t = pkg_name_options(query)
    if t:
        return t[0][0]
```


```python
>>> pkg_name_options('skimage')
[('scikit-image', 5),
 ('-e', 2),
 ('virtualenv', 1),
 ('scikit', 1),
 ('scikit-', 1),
 ('pillow', 1)]
```









```python
>>> best_guess('skimage')
'scikit-image'
```


Yay, it works!
With a sample of one!
Let's try two...


```python
>>> pkg_name_options('sklearn')
[('numpy', 3), ('scikit-learn', 2), ('-U', 2), ('scikit-', 1), ('scipy', 1)]
```




Okay, so it already fails. 

Sure, I could parse more carefully. I could dig into the webpages and get more scope. 

That'd be fun. 

But that's not very nice too Google (and probably is illegal, if anyone cares). 

What you'll find next is an attempt to look at the man in the mirror instead. Looking locally, where the packages actually are: In the site-packages folders...



### Extract, analyze and compare site-packages info names


```python
import pandas as pd
import numpy as np

from equate.examples.site_names import (
    DFLT_SITE_PKG_DIR,
    site_packages_info_df,
    print_n_null_elements_in_each_column_containing_at_least_one,
    Lidx,
)
```


```python
>>> DFLT_SITE_PKG_DIR
'~/.virtualenvs/382/lib/python3.8/site-packages'
```



```python
>>> data = site_packages_info_df()
>>> print(f"{data.shape}")
(303, 8)
>>> data
```






<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>dist_info_dirname</th>
      <th>info_kind</th>
      <th>dist_name</th>
      <th>most_frequent_record_dirname</th>
      <th>first_line_of_top_level_txt</th>
      <th>installer</th>
      <th>metadata_name</th>
      <th>pypi_url_name</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>xlrd-1.2.0.dist-info</td>
      <td>dist-info</td>
      <td>xlrd</td>
      <td>xlrd</td>
      <td>xlrd</td>
      <td>pip</td>
      <td>xlrd</td>
      <td>None</td>
    </tr>
    <tr>
      <th>1</th>
      <td>boltons-20.2.0.dist-info</td>
      <td>dist-info</td>
      <td>boltons</td>
      <td>boltons</td>
      <td>boltons</td>
      <td>pip</td>
      <td>boltons</td>
      <td>None</td>
    </tr>
    <tr>
      <th>2</th>
      <td>appdirs-1.4.3.dist-info</td>
      <td>dist-info</td>
      <td>appdirs</td>
      <td>appdirs</td>
      <td>appdirs</td>
      <td>pip</td>
      <td>appdirs</td>
      <td>None</td>
    </tr>
    <tr>
      <th>3</th>
      <td>yapf-0.29.0.dist-info</td>
      <td>dist-info</td>
      <td>yapf</td>
      <td>yapftests</td>
      <td>yapf</td>
      <td>pip</td>
      <td>yapf</td>
      <td>None</td>
    </tr>
    <tr>
      <th>4</th>
      <td>cmudict-0.4.4.dist-info</td>
      <td>dist-info</td>
      <td>cmudict</td>
      <td>cmudict</td>
      <td>cmudict</td>
      <td>pip</td>
      <td>cmudict</td>
      <td>None</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>298</th>
      <td>simplegeneric-0.8.1.dist-info</td>
      <td>dist-info</td>
      <td>simplegeneric</td>
      <td>simplegeneric</td>
      <td>simplegeneric</td>
      <td>pip</td>
      <td>simplegeneric</td>
      <td>None</td>
    </tr>
    <tr>
      <th>299</th>
      <td>plotly-4.6.0.dist-info</td>
      <td>dist-info</td>
      <td>plotly</td>
      <td>plotly</td>
      <td>_plotly_future_</td>
      <td>pip</td>
      <td>plotly</td>
      <td>None</td>
    </tr>
    <tr>
      <th>300</th>
      <td>rsa-3.4.2.dist-info</td>
      <td>dist-info</td>
      <td>rsa</td>
      <td>rsa</td>
      <td>rsa</td>
      <td>pip</td>
      <td>rsa</td>
      <td>None</td>
    </tr>
    <tr>
      <th>301</th>
      <td>backcall-0.1.0.dist-info</td>
      <td>dist-info</td>
      <td>backcall</td>
      <td>backcall</td>
      <td>backcall</td>
      <td>pip</td>
      <td>backcall</td>
      <td>None</td>
    </tr>
    <tr>
      <th>302</th>
      <td>cantools-33.1.1.dist-info</td>
      <td>dist-info</td>
      <td>cantools</td>
      <td>cantools</td>
      <td>cantools</td>
      <td>pip</td>
      <td>cantools</td>
      <td>None</td>
    </tr>
  </tbody>
</table>
<p>303 rows × 8 columns</p>
</div>



```python
>>> print_n_null_elements_in_each_column_containing_at_least_one(data)
most_frequent_record_dirname:	1 null values
first_line_of_top_level_txt:	6 null values
installer:	32 null values
metadata_name:	1 null values
pypi_url_name:	255 null values
```


```python
>>> lidx = Lidx(data)
>>> df = data[lidx.no_nans]
>>> print(f"no nan df: {len(df)=}")
no_nans: 302
equal: 187
dash_underscore_eq: 220
('equal', 'dash_underscore_eq'): 186
```


```python
>>> lidx = Lidx(df)
>>> lidx.print_diagnosis()
no_nans: 302
equal: 187
dash_underscore_eq: 220
('equal', 'dash_underscore_eq'): 186
```





```python
>>> lidx = Lidx(df, 'first_line_of_top_level_txt')
>>> lidx.print_diagnosis()
no_nans: 297
equal: 182
dash_underscore_eq: 214
('equal', 'dash_underscore_eq'): 181
```



```python
>>> t = Lidx(df, 'most_frequent_record_dirname')
>>> tt = Lidx(df, 'first_line_of_top_level_txt')
>>> sum(t.equal | tt.equal)
199
```


```python
>> sum(t.dash_underscore_eq | tt.dash_underscore_eq)
233
```



