Metadata-Version: 2.1
Name: aepo
Version: 0.1.4
Summary: Annotation Efficient Preference Optimization
Home-page: https://github.com/CyberAgentAILab/annotation-efficient-po
Author: Yuu Jinnai
Author-email: ddyuudd@gmail.com
Requires-Python: >=3.9,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.9
Requires-Dist: datasets (>=2.10.0,<3.0.0)
Requires-Dist: evaluate (>=0.4.2,<0.5.0)
Requires-Dist: llm-blender (>=0.0.2,<0.0.3)
Requires-Dist: numpy (>=1.26.4,<2.0.0)
Requires-Dist: pandas (>=2.0.0,<3.0.0)
Requires-Dist: torch (>=2.3.0,<3.0.0)
Requires-Dist: tqdm (>=4.66.4,<5.0.0)
Requires-Dist: transformers (>=4.40.2,<5.0.0)
Project-URL: Documentation, https://CyberAgentAILab.github.io/annotation-efficient-po
Project-URL: Repository, https://github.com/CyberAgentAILab/annotation-efficient-po
Description-Content-Type: text/markdown

## Annotation-Efficient Preference Optimization

This repository implements the Annotation-Efficient Preference Optimization (AEPO) algorithm.

The code is tested on Ubuntu 20.04 using Python 3.9 and CUDA 11.0 (Docker image nvidia/cuda:11.0.3-cudnn8-devel-ubuntu20.04).

## Install

You can install aepo via pip.
```
pip install aepo
```

Source install is available too. Clone this repository and run `pip install .`.
```
git clone git@github.com:CyberAgentAILab/annotation-efficient-po.git
cd annotation-efficient-po
pip install .
```


## Usage

The command line interface is available.
The input dataset can be csv file or a dataset uploaded to Huggingface Hub.
The dataset should have a column named *prompt* or *instruction*. aepo recognize it as the user prompt given to the system and the rest of the columns to be the responses generated by the system.

I prepared an example dataset in `dataset/alpaca_samples.csv`.
The csv file includes 128 responses generated by [HuggingFaceH4/mistral-7b-sft-beta](https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta) for each instruction of the `alpaca_human_preference` split of [tatsu-lab/alpaca_farm](https://huggingface.co/datasets/tatsu-lab/alpaca_eval).
You can try aepo using this dataset with the following command:

```
aepo dataset/alpaca_samples.csv --num_responses 8 --num_annotations 2 --num_instructions 10
```

`--num_responses` is the number of input responses you use. The dataset has to have responses larger than or equal to `--num_responses`. `--num_annotations` is the number of responses after the subsampling process. It is also the number of times the reward model is queried per instruction.

### Example: Running AEPO

You can generate a pair of responses for each instruction using aepo using the following command.

```
aepo dataset/alpaca_samples.csv --num_responses 8 --num_annotations 2 --num_instructions 10
```

To subsample four responses for e.g., [LiPO](https://arxiv.org/abs/2402.01878v1), set `--num_annotations` to four.

```
aepo dataset/alpaca_samples.csv --num_responses 8 --num_annotations 4 --num_instructions 10
```

### Example: Running West-of-N over 8 samples
[West-of-N](https://arxiv.org/abs/2401.12086) is a strategy to pick the Best-of-N as the chosen response, and Worst-of-N as a rejected response. It is shown to be effective for DPO and reward modeling.
You can run West-of-N using this package by setting `--num_annotations` == `--num_responses`.

```
aepo dataset/alpaca_samples.csv --num_responses 8 --num_annotations 8 --num_instructions 10
```

This command will generate a dataset with 8 responses, ranked by their rewards. If you only need the best and worst of the N samples, then use `--west_of_n` option.

```
aepo dataset/alpaca_samples.csv --num_responses 8 --num_annotations 8 --num_instructions 10 --west_of_n
```

This will pick the best and worst responses as the chosen and rejected. The rest of the responses are discarded.
It would be useful to construct a pairwise preference dataset.

## Reference

TBA. Yuu Jinnai and Honda Ukyo. Annotation-Efficient Preference Optimization for Language Model Alignment, 2024.

## Contact
For any questions, feel free to raise an issue or contact me at jinnai_yu@cyberagent.co.jp.

