Metadata-Version: 2.1
Name: topic-benchmark
Version: 0.6.0
Summary: CLI suite for benchmarking topic models
License: MIT
Author: Márton Kardos
Author-email: power.up1163@gmail.com
Requires-Python: >=3.9,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: catalogue (>=2.0.0,<3.0.0)
Requires-Dist: datasets (>=2.18.0,<3.0.0)
Requires-Dist: gensim (>=4.3.2,<5.0.0)
Requires-Dist: hdbscan (>=0.8.0,<0.9.0)
Requires-Dist: numpy (>=1.23.0)
Requires-Dist: pandas (>=2.1.0,<3.0.0)
Requires-Dist: pyro-ppl (>=1.8.0,<2.0.0)
Requires-Dist: radicli (>=0.0.25,<0.0.26)
Requires-Dist: rich (>=13.6.0,<14.0.0)
Requires-Dist: scikit-learn (>=1.2.0)
Requires-Dist: scipy (>=1.10.0,<1.11.0)
Requires-Dist: sentence-transformers (>=3.0.0)
Requires-Dist: torch (>=2.1.0,<3.0.0)
Requires-Dist: turftopic (>=0.5.3)
Requires-Dist: umap-learn (>=0.5.0,<0.6.0)
Description-Content-Type: text/markdown

# topic-benchmark
Command Line Interface for benchmarking topic models.

The package contains `catalogue` registries for all models, datasets and metrics for model evaluation,
along with scripts for producing tables and figures for the S3 paper.

## Usage

### Installation

You can install the package from PyPI.

```bash
pip install topic-benchmark

```

### Commands

#### `run`

Run the benchmark. Defaults to running all models with the benchmark used in Kardos et al. (2024).

```bash
python3 -m topic_benchmark run
```

| Argument               | Short Flag | Description                                                                                              | Type                                   | Default         |
|------------------------|------------|----------------------------------------------------------------------------------------------------------|----------------------------------------|-----------------|
| `--out_dir OUT_DIR`    | `-o`       | Output directory for the results.                                                                        | `str`                                  | `results/`      |
| `--encoders ENCODERS`  | `-e`       | Which encoders should be used for conducting runs?                                                       | `str`                                  | `None`          |
| `--models MODELS`      | `-m`       | What subsection of models should the benchmark be run on.                                                | `Optional[list[str], NoneType]`        | `None`          |
| `--datasets DATASETS`  | `-d`       | What datasets should the models be evaluated on.                                                         | `Optional[list[str], NoneType]`        | `None`          |
| `--metrics METRICS`    | `-t`       | What metrics should the models be evaluated on.                                                          | `Optional[list[str], NoneType]`        | `None`          |
| `--seeds SEEDS`        | `-s`       | What seeds should the models be evaluated on.                                                            | `Optional[list[int], NoneType]`        | `None`          |


### Push to hub

Push results to a HuggingFace repository.

```bash
python3 -m topic_benchmark push_to_hub "your_user/your_repo"
```

| Argument          | Description                                            | Type  | Default    |
|-------------------|--------------------------------------------------------|-------|------------|
| `hf_repo`         | HuggingFace repository to push results to.             | `str` | N/A        |
| `results_folder`  | Folder containing results for all embedding models.    | `str` | `results/` |

## Reproducing $S^3$ paper results
Result files to all runs in the $S^3$ publication can be found in the `results/` folder in the repository.
To reproduce the results reported in our paper, please do the following.

First, install this package by running the following command:

```bash
pip install topic-benchmark
python3 -m topic-benchmark run -o results/
```

The results for each embedding model will be found in the `results` folder (unless a value for `--out_file` is explicitly passed).

To produce figures and tables in the paper, you can use the scripts in the  `scripts/s3_paper/` folder.

