Metadata-Version: 2.4
Name: privyscope-ja
Version: 0.1.4
Summary: privyscope-ja — Japanese language pack for the privyscope PII engine
Author: privyscope Core Team
License: Apache-2.0
Project-URL: Homepage, https://github.com/zafrem/privyscope-ja
Project-URL: Documentation, https://github.com/zafrem/privyscope-ja/blob/main/README.md
Keywords: pii,ner,redaction,privacy,japanese,onnx
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: privyscope>=0.1.3
Provides-Extra: train
Requires-Dist: privyscope[train]>=0.1.0; extra == "train"
Requires-Dist: fugashi>=1.3; extra == "train"
Requires-Dist: unidic-lite>=1.0.8; extra == "train"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# privyscope-ja

**Japanese language pack** for [privyscope](https://github.com/zafrem/privyscope),
a two-stage hybrid PII detection & masking engine.

This is a thin **plugin**: it ships only the Japanese data (`regex_rules.yaml`,
`entity_config.yaml`) and registers itself with the `privyscope` core via the
`privyscope.languages` entry point. The engine and the `privyscope` command live
in the core, which is pulled in automatically.

Japanese mixes three scripts — kanji (漢字), hiragana (ひらがな) and katakana
(カタカナ) — with no inter-word spaces, so the pack advertises all three scripts
and leaves personal-name detection to the NER stage rather than regex.

## Install

```bash
pip install privyscope-ja      # pulls in the `privyscope` core automatically
```

## Quickstart

```bash
privyscope redact "田中さんの電話は090-1234-5678、メールは taro@example.co.jp"
# → "<PER>さんの電話は<PHONE>、メールは <EMAIL>"
```

```python
from privyscope import Privyscope
eng = Privyscope.from_pretrained(lang="ja")
eng.redact("田中さんの電話は090-1234-5678").masked_text   # "<PER>さんの電話は<PHONE>"
```

## How it works

A **two-stage hybrid pipeline**, results merged via union:

1. **Regex filter** — structurally obvious PII: Japanese mobile
   (`0[789]0-xxxx-xxxx`, `+81`), landlines, My Number (個人番号, 12-digit with
   check digit), passport, driver's license, Corporate Number (法人番号),
   Japanese bank accounts, email, and credential secrets.
2. **ONNX NER** — a BIOES token classifier with a constrained Viterbi decoder for
   contextual PII (names, addresses, private dates). Weights download from
   Hugging Face Hub (`zafrem/privyscope-ja`) on first use.

Inference is **ONNX Runtime only — no PyTorch at runtime**. The base encoder
tokenizes with MeCab, but that dependency is needed only for training/fine-tuning.

## Entities

8 base types — `PER`, `PHONE`, `ID_NUM`, `EMAIL`, `LOC`, `BANK`, `DATE`,
`SECRET` — plus extended `BIZ_NUM` (Corporate Number 法人番号) and
`DRIVER_LICENSE`, caught by regex.

## Model & performance

- **Base encoder** — `tohoku-nlp/bert-base-japanese-v3` → BIOES head → Viterbi
  decoder.
- **Runtime artifact** — INT8-quantized ONNX, max sequence length 256, downloaded
  from the Hub with a SHA-256 `checksum.txt` for integrity.
- **Accuracy** — run `privyscope eval --lang ja your_val.jsonl` on your own
  labelled data; numbers will be published here once the held-out evaluation is
  finalized.

## Limitations

- Not an anonymization/compliance guarantee; use as one layer of
  privacy-by-design, with human review for sensitive workflows.
- Personal names are contextual (NER only): recall depends on the fine-tuned
  model, which is trained across kanji/hiragana/katakana renderings.

## License

Apache-2.0. See [LICENSE](LICENSE).

Stage-1 regex patterns are compiled from the
[pii-pattern-engine](https://github.com/zafrem/pii-pattern-engine) ruleset. Most rules carry
a **verification function** — a checksum or dictionary validator (`jp_my_number_valid`, `luhn`) that a match
must pass before it is redacted, so a value that merely looks like an ID is left alone.

Regex-only extended entities: `BIZ_NUM · DRIVER_LICENSE · CRYPTO · IP · DEVICE · URL`.

> `privyscope_ja/regex_rules.yaml` is **generated** by `scripts/gen_regex_rules.py` and is overwritten
> on every build — edit the mapping in that script, not the YAML. See [CONTRIBUTING](CONTRIBUTING.md).
