Metadata-Version: 2.1
Name: VeriScore
Version: 1.0.1
Summary: Pip package for Verifact
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: spacy
Requires-Dist: openai
Requires-Dist: anthropic
Requires-Dist: tiktoken
Requires-Dist: tqdmunsloth

# VeriScore
This is introduction for pip package of VeriScore. It have two type of method to extract claims by 
1) prompting (Implemented)
2) fine-tuned model (in progress)

VeriScore consists of three parts 1) `claim extraction` 2) `evidence searching` and 3) `claim verification`.
We provide an end-to-end pipeline to obtain the VeriScore, along with each of its components individually.
## Install
1. Make a new Python 3.6+ environment using `virtualenv` or `conda`.
2. Install `veriscore` pacakge using `pip`
3. Download `en_core_web_sm` using `spacy` library
```
pip install --upgrade veriscore
python -m spacy download en_core_web_sm
```

## Setup environment before running code
1. Download `prompt` folder that have txt file of prompt template (you can see `prompt` folder VeriScore's repository)
2. Set OpenAI or Claude API key to environment variable of `bash` for prompting approach
```
export OPENAI_API_KEY_PERSONAL={your_openai_api_key}
epport CLAUDE_API_KEY={your_claude_api_key}
```
3. Set SERPER API key to environment variable of `bash` for searching evidence
```
export SERPER_KEY_PRIVATE={your_serper_api_key}
```

## Running VeriScore using a command line
This is an end-to-end pipeline for running VeriScore.
```
 python3 -m veriscore.veriscore --data_dir {data_dir} --input_file {input_file} --model_name_extraction {model_name_extraction} --model_name_verification {model_name_verification}
```
* `data_dir`: Directory containing input data. `./data` by default.
* `input_file`: Name of input data file. It should be `jsonl` format where each line contains
    * `question`: query to ask
    * `response`: generated response from of `question`
    * `model`: name of model generate response
    * `prompt_source`: name of dataset provide `question` like FreshQA
* `model_name_extraction`: Name of model used for claim extraction. `gpt-4-0125-preview` by default.
* `model_name_verification`: Name of model used for claim verification. `gpt-4o` by default.

Other optional flags:

* `output_dir`: Directory for saving ouptut data. `./data` by default.
* `cache_dir`: Directory for saving cache data. `./data/cache` by default.
* `label_n`: This is type of label for claim verification. It could be `2` (binary) ro `3` (trinary)
    * `2`: query to ask binary `supported` and `unsupported`
    * `3`: query to ask trinay `labels—supported`, `contradicted`, and `inconclusive`
* `search_res_num`: A Hyperparameter for number of search result. `5` by default.

Saving output: 
`input_file_name` is file name removed `jsonl` from `—-input_file`
* `extracted claims` will be saved to `output_dir/claims_{input_file_name}.jsonl`.
* `searched evidence` will be saved to `output_dir/evidence_{input_file_name}.jsonl`.
* `verified claims` will be saved to `output_dir/model_output/verification_{input_file_name}.jsonl`.


## Running individual part using a command line
`Claim extraction`:
```
 python3 -m veriscore.extract_claims --data_dir {data_dir} --input_file {input_file} --model_name {model_name} 
```
* `input_file`: Name of input data file. It should be `jsonl` format where each line contains
    * `question`: query to ask
    * `response`: generated response from of `question`
    * `model`: name of model generate response
    * `prompt_source`: name of dataset provide `question` like FreshQA
* `model_name`: Name of model used for claim extraction. `gpt-4-0125-preview` by default.
output:
```dictionary
 {
  "question": question.strip(),
  "prompt_source": prompt_source,
  "response": response.strip(),
  "prompt_tok_cnt": prompt_tok_cnt,
  "response_tok_cnt": response_tok_cnt,
  "model": model,
  "abstained": False,  
  "claim_list": claim_list,
  "all_claim_lst": all_claim_lst
 }
```
`Evidence searching`:
```
 python3 -m veriscore.retrieval_evidence --data_dir {data_dir} --input_file {input_file}
```
* `input_file`: Name of input data file. It should be `jsonl` format where each line contains the keys of the output dictionary from the `Claim extraction`.
output:
```dictionary
 {
  ...
  "claim_snippets_dict": dictionary for claim and list of searched evidence. each evidence have dictionary of {"title": title, "snippet": snippet, "link": link}
 }
```

`Claim verification`:
```
 python3 -m veriscore.verify_claims --data_dir {data_dir} --input_file {input_file} --model_name {model_name}
```
* `input_file`: Name of input data file. It should be `jsonl` format where each line contains the keys of the output dictionary from the `Evidence searching`.
output:
```dictionary
 {
  ...
  "response": result of claim verification
 }
```
