Metadata-Version: 2.1
Name: faithfulness
Version: 0.0.3
Summary: A library to evaluate faithfulness (factual correctness) of abstractive summaries.
Home-page: https://github.com/bigabig/faithfulness
Author: Tim Fischer
Author-email: tim.s.fischer96@googlemail.com
License: UNKNOWN
Project-URL: Bug Tracker, https://github.com/bigabig/faithfulness/issues
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE

# Faithfulness

An easy-to-use library to evaluate faithfulness (factual correctness) of abstractive summaries. Faithfulness is computed by comparing the summary with the original source document.

This library includes multiple faithfulness metrics based on:
- BERTScore
- Entailment 
- Question Generation & Question Answering framework (QGQA)
- Named Entity Recognition (NER)
- Open Information Extraction (OpenIE)
- Semantic Role Labeling (SRL)
- Sentence Similarity (SentSim)

## Installation

1. `$ conda create -n my_project python=3.8` This creates a new virtual environment for your project with conda. You can activate it with `$ conda activate my_project`.
2. `$ conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch` Please install PyTorch by following the instructions [here](https://pytorch.org/get-started/locally/). Make sure to install the CUDA variant that matches the CUDA version of your GPU. 
3. `$ pip install faithfulness` This installs the faithfulness library and it's dependencies. Read more about the dependencies [below](#user-content-dependencies).

All faithfulness metrics are model-based. Some models have to be installed manually:
- Download the SRL model [here](https://storage.googleapis.com/allennlp-public-models/structured-prediction-srl-bert.2020.12.15.tar.gz) and save it in your project. e.g. __/models/srl_model.tar.gz__
- Download a spacy model: `$ python -m spacy download en_core_web_sm`
- Download CoreNLP: `import stanza && stanza.install_corenlp()`

## Usage
```
from faithfulness import QGQA

qgqa = QGQA()
summary = "Lorem ipsum dolor sit amet"
source = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam ..."
faithfulness, info = qgqa.score(summary, source)
```

More examples can be found [here](https://github.com/bigabig/faithfulness/examples/).

## Metric evaluation
We evaluated all faithfulness metrics by correlating them with human judgements on the XSUM dataset ([link](https://github.com/google-research-datasets/xsum_hallucination_annotations)).
You will soon be able to read more about the evaluation in our paper. ([Master's thesis](https://www.inf.uni-hamburg.de/en/inst/ab/lt/teaching/theses/completed-theses/2021-ma-timfischer.pdf))

| Method     | Pearson (r) | Spearman (p) |
|------------|-------------|--------------|
| BERTScore  | 0.501       | 0.486        |
| Entailment | 0.366       | 0.422        |
| SentSim    | 0.392       | 0.389        |
| SRL        | 0.393       | 0.377        |
| NER        | 0.252       | 0.259        |
| QGQA       | 0.252       | 0.258        |
| OpenIE     | 0.169       | 0.185        |

## Dependencies

By running `$ pip install faithfulness` you will install this library as well as the following dependencies:
- [transformers](https://huggingface.co/transformers/)
- [spaCy](https://spacy.io/) (used for Entailment, NER, QGQA, SentSim, SRL)
- [Stanza](https://stanfordnlp.github.io/stanza/) (used for OpenIE)
- [AllenNLP](https://allennlp.org/) (used for SRL)
- [SentenceTransformers](https://www.sbert.net/) (used for NER, OpenIE, QGQA, SentSim, SRL)

## Troubleshooting
There are currently problems when installing allennlp and jsonnet. If you encounter "_Building wheel for jsonnet (setup.py) ... error_" during the installation please try:
```
apt-get install make 
apt-get install g++ 
```
or install jsonnet before installing this library
```
conda install -c conda-forge jsonnet
pip install faithfulness
```

