Metadata-Version: 2.5
Name: wikiq
Version: 0.3.0
Summary: Convert MediaWiki XML database dumps into tabular datasets for research
Project-URL: Homepage, https://wiki.communitydata.science/
Project-URL: Repository, https://gitea.communitydata.science/collective/wikiq
Author: Benjamin Mako Hill, Nathan TeBlunthuis, Will Beason, Sohyeon Hwang, Kaylea Champion
License-Expression: GPL-3.0-or-later
License-File: COPYING
Keywords: dump,mediawiki,research,revision,wiki,wikipedia,xml
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.11
Requires-Dist: deltas>=0.7.0
Requires-Dist: more-itertools>=10.7.0
Requires-Dist: mwparserfromhell>=0.6.0
Requires-Dist: mwpersistence>=0.2.4
Requires-Dist: mwreverts>=0.1.5
Requires-Dist: mwtypes>=0.4.0
Requires-Dist: mwxml>=0.3.8
Requires-Dist: pyarrow>=20.0.0
Requires-Dist: sortedcontainers>=2.4.0
Provides-Extra: dev
Requires-Dist: pandas>=2.1.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.0.0; extra == 'dev'
Requires-Dist: pytest-benchmark>=5.1.0; extra == 'dev'
Requires-Dist: pytest>=8.4.1; extra == 'dev'
Provides-Extra: legacy
Requires-Dist: mediawiki-utilities>=0.4.18; extra == 'legacy'
Description-Content-Type: text/markdown

# wikiq

wikiq converts MediaWiki XML database dumps—the "history" dumps that
include every revision of every page—into tabular datasets for research.
It is a command line program that produces one row per revision with
metadata such as the page, namespace, timestamp, editor, text size, and
revert status, plus a range of optional computed columns.

## Installation

wikiq requires Python 3.11 or later. Install it from PyPI:

    pip install wikiq

Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.

That is everything most uses need, and none of it requires a compiler.
Two further features carry heavier dependencies, so they are kept out of
the base install.

### Diffs and wikidiff2 persistence

`--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
PyPI and compiles a C++ extension, so it needs a C++ compiler and
libthai:

    pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'

The other persistence methods, including the default `-p sequence`, do
not need it.

### Legacy persistence

`-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects:

    pip install 'wikiq[legacy]'

wikiq tells you which of these to install if you use an option that
needs one.

## Usage

    wikiq dump.xml.7z -o output/

Each output row describes one revision. The output format follows the
`-o` argument: a path ending in `.jsonl` produces JSON Lines, anything
else produces tab-separated values. With no dump file argument, wikiq
reads XML on stdin and writes to stdout.

The `title` and `namespace` columns are page-level identity values as
of the time of export: a moved page's entire history carries its final
title, and namespace derives from that title. Redirect status, by
contrast, is determined per revision from each revision's own text: the
`revision_is_redirect` and `revision_redirect_target` columns record
whether a revision's text begins with a redirect directive and where it
points. `#REDIRECT` is recognized on every wiki; wikis that also use a
localized keyword can add it with `--redirect-aliases` (for example,
`--redirect-aliases OMDIRIGERING` for Swedish wikis).

The most commonly useful options (`wikiq --help` describes them all):

- `-n ID` limits output to a namespace, and can be given more than once.
  `-rr N` sets how many prior edits are checked when detecting reverts.
- `--collapse-user` collapses each sequence of consecutive edits by the
  same user into a single row, which can address problems with text
  persistence measures.
- `-p [METHOD]` computes content persistence measures for each revision:
  persistent token revisions, tokens added, and tokens removed. The
  available methods are `sequence` (the default), `segment` (robust to
  content moves but slower), `wikidiff2` (like segment, using the
  wikidiff2 diff engine), and `legacy` (the behavior of older research
  projects). Persistence is the slowest thing wikiq computes.
- `-d` outputs a structured diff for each revision; `-t` outputs the
  full revision text.
- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
  `--headings` parse each revision's wikitext and add a column with the
  extracted elements.
- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
  and reports the matches in a column named by the label; repeat the
  pair for multiple patterns. `-CP`/`-CPl` do the same for edit
  summaries. Adding `-RPc` or `-CPc` reports the number of matches
  instead of the matched text.
- `--resume` continues an interrupted run from the last complete line of
  an existing JSONL output file. Combined with `--time-limit HOURS`,
  this supports processing very large dumps in bounded chunks.
- `--print-schema` prints a Spark-compatible JSON schema for the
  configured output and exits.

## Tests

From a clone of the repository:

    git clone https://gitea.communitydata.science/collective/wikiq.git
    cd wikiq
    pip install -e '.[dev]'
    pytest test/

Run the suite from the root—some tests open fixture files by paths
relative to it. The full suite processes several real dumps from
`test/dumps/`; expected outputs live in `test/baseline_output/`.

Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
optional dependency is absent, so a base install reports skips rather
than failures.

## Authors

wikiq is written and maintained by members of the [Community Data
Science Collective](https://wiki.communitydata.science/). Contributors
include Benjamin Mako Hill, Nathan TeBlunthuis, Will Beason, Sohyeon
Hwang, and Kaylea Champion. It builds on earlier versions of the tool
written in Python and in C++ by Benjamin Mako Hill.

## License

wikiq is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your
option) any later version. See [COPYING](COPYING) for the full text.
