Metadata-Version: 2.1
Name: payment-text-parser
Version: 0.0.9
Summary: Parser for entity/address free text (based on libpostal/spacy)
Home-page: https://gitlab.com/alpina-analytics/payment_text_parser.git
Author: Pierre Oberholzer
Author-email: pierre.oberholzer@alpina-analytics.com
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: GNU Affero General Public License v3
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
Requires-Dist: absl-py (==0.7.1)
Requires-Dist: appnope (==0.1.0)
Requires-Dist: astor (==0.8.0)
Requires-Dist: attrs (==19.1.0)
Requires-Dist: backcall (==0.1.0)
Requires-Dist: beautifulsoup4 (==4.8.0)
Requires-Dist: blis (==0.2.4)
Requires-Dist: ccy (==1.0.0)
Requires-Dist: certifi (==2019.6.16)
Requires-Dist: chardet (==3.0.4)
Requires-Dist: Click (==7.0)
Requires-Dist: corenlp-protobuf (==3.8.0)
Requires-Dist: cycler (==0.10.0)
Requires-Dist: cymem (==2.0.2)
Requires-Dist: decorator (==4.4.0)
Requires-Dist: Faker (==2.0.1)
Requires-Dist: Flask (==1.1.1)
Requires-Dist: Flask-Cors (==3.0.8)
Requires-Dist: gast (==0.2.2)
Requires-Dist: grpcio (==1.22.0)
Requires-Dist: h5py (==2.9.0)
Requires-Dist: idna (==2.8)
Requires-Dist: imageio (==2.5.0)
Requires-Dist: ipython (==7.7.0)
Requires-Dist: ipython-genutils (==0.2.0)
Requires-Dist: itsdangerous (==1.1.0)
Requires-Dist: jedi (==0.14.1)
Requires-Dist: Jinja2 (==2.10.1)
Requires-Dist: joblib (==0.13.2)
Requires-Dist: jsonschema (==3.0.2)
Requires-Dist: Keras (==2.2.4)
Requires-Dist: Keras-Applications (==1.0.8)
Requires-Dist: Keras-Preprocessing (==1.1.0)
Requires-Dist: kiwisolver (==1.1.0)
Requires-Dist: langdetect (==1.0.7)
Requires-Dist: Markdown (==3.1.1)
Requires-Dist: MarkupSafe (==1.1.1)
Requires-Dist: mock (==3.0.5)
Requires-Dist: murmurhash (==1.0.2)
Requires-Dist: names-dataset (==1.9.0)
Requires-Dist: networkx (==2.3)
Requires-Dist: numpy (==1.17.0)
Requires-Dist: pandas (==0.24.0)
Requires-Dist: parso (==0.5.1)
Requires-Dist: pexpect (==4.7.0)
Requires-Dist: pickleshare (==0.7.5)
Requires-Dist: Pillow (==6.1.0)
Requires-Dist: plac (==0.9.6)
Requires-Dist: postal (==1.1.7)
Requires-Dist: preshed (==2.0.1)
Requires-Dist: prompt-toolkit (==2.0.9)
Requires-Dist: protobuf (==3.9.0)
Requires-Dist: ptyprocess (==0.6.0)
Requires-Dist: public (==2019.4.13)
Requires-Dist: pycountry (==19.7.15)
Requires-Dist: Pygments (==2.4.2)
Requires-Dist: pyparsing (==2.4.2)
Requires-Dist: pyrsistent (==0.15.4)
Requires-Dist: python-dateutil (==2.8.0)
Requires-Dist: pytz (==2019.2)
Requires-Dist: PyWavelets (==1.0.3)
Requires-Dist: PyYAML (==5.1.2)
Requires-Dist: query-string (==2019.4.13)
Requires-Dist: requests (==2.22.0)
Requires-Dist: scikit-image (==0.15.0)
Requires-Dist: scikit-learn (==0.21.3)
Requires-Dist: scipy (==1.3.0)
Requires-Dist: six (==1.12.0)
Requires-Dist: soupsieve (==1.9.2)
Requires-Dist: spacy (==2.1.4)
Requires-Dist: srsly (==0.0.7)
Requires-Dist: stanford-corenlp (==3.9.2)
Requires-Dist: style (==1.1.0)
Requires-Dist: tensorboard (==1.13.1)
Requires-Dist: tensorflow (==1.13.1)
Requires-Dist: tensorflow-estimator (==1.13.0)
Requires-Dist: termcolor (==1.1.0)
Requires-Dist: thinc (==7.0.8)
Requires-Dist: torch (==1.1.0)
Requires-Dist: tqdm (==4.32.2)
Requires-Dist: traitlets (==4.3.2)
Requires-Dist: Unidecode (==1.1.1)
Requires-Dist: update (==0.0.1)
Requires-Dist: urllib3 (==1.25.3)
Requires-Dist: wasabi (==0.2.2)
Requires-Dist: wcwidth (==0.1.7)
Requires-Dist: Werkzeug (==0.15.5)

# Payment text parser

### Description


Inputs a text string and parses entity/address free text input to:
- Flag entity fields ('ORG' for companies, 'PER' for individuals, 'PER_ORG' for uncertain decision) <= Based on [Spacy](https://spacy.io/)
- Flag address components of the address ('house number', 'town', 'country', ...) <= Based on [Libpostal](https://github.com/openvenues/libpostal)
- Flag other fields (i.e. not entity nor address) with POS tags ('NE', 'ADJ', 'NN', ..) <= Based on [CoreNLP](https://stanfordnlp.github.io/CoreNLP/)

More generally, the package includes following features:
- A data generator for entity/address fields and free text fields (based on open data)
- A model distinguishing entity/address field from free text (using Keras/TensorFlow and CoreNLP) to apply dedicated heuristics
- A series of cleaning_postprocessing steps including true case recognition (CoreNLP)
- A parser of entity/address/other fields as described above using re-trained vanilla Spacy model (based on labeled open data)
- Simple heuristics and metrics applied after the parsing to improve accuracy
- Part-of-speech (POS) tagging of the remaining flags (Spacy and/or CoreNLP) for downstream processing

This package is specifically intended to be used together with the upstream Swiftflow pipeline that parses
all fields from the SWIFT MT messages, including the entity/address and free text fields, which are decisive for inter-banking
transactional communication.

### Installation

The package uses essentally Libpostal and Spacy.
Also, it uses Keras on Tensorflow to recognize if the text input is a free text or an entity/address text.

#### Pre-requiste: Libpostal

Refer to [Libpostal installation](https://github.com/openvenues/libpostal).
Once Libpostal is installed, the Python binder `postal` will be installed as part
of the package with pip (see below)

#### Payment_text_parser

The other dependencies, including [Spacy](https://spacy.io/), will be installed via `pip` on the present package:

#### Create environment

One recommends to use Python 3.7.

Native Python:

```
/usr/local/bin/python3 -m venv <my_env>
source <my_env>/bin/activate`
```

Conda:

``
conda create --name <my_env> python=3.7`
conda activate <my_env>
``

##### From pip

```
pip install payment-text-parser --use-feature=2020-resolver
python -m spacy download de_core_news_sm
```

##### From git

```
pip install git+https://gitlab.com/alpina-analytics/payment_text_parser.git
python -m spacy download de_core_news_sm
```

#### From requirements.txt

```
git clone https://gitlab.com/alpina-analytics/payment_text_parser.git
cd payment_text_parser
pip install -r requirements.txt
python -m spacy download de_core_news_sm
export PYTHONPATH=$(pwd)
```

### Usage

#### Script

```
from payment_text_parser.entity_extractor.entity_extractor import ExtractorClass
e = ExtractorClass(text)
d_res = e.d_res
```

#### Webserver

```
# Launch
python main.py

# Test
curl -H "Content-type: application/json" -X POST http://127.0.0.1:5000/parse -d '{"text":"John Deere Les Abues 2 75000 Paris"}'
```

#### Optional : start Stanford NLP server

Required if :
- Field type detection enabled by `ExtractorClass(text,check_field_type=True)`
- POS-tagging of rest fields enabled by `ExtractorClass(text,create_nlp_tags_rest_text=True)`
If not started, an warning message will be prompted, however full processing can still take place.

CoreNLP server can be started as follow:

```
cd ./core_nlp/stanford-corenlp-full-2018-10-05
java -Xmx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer \
-serverProperties StanfordCoreNLP-german.properties \
-preload tokenize,ssplit,pos,parse \
-status_port 9000  -port 9000 -timeout 15000
```

### References

#### Spacy

https://spacy.io/

#### Libpostal

https://github.com/openvenues/libpostal

#### CoreNLP

https://stackoverflow.com/questions/33259191/installing-libicu-dev-on-mac
https://stackoverflow.com/questions/50217214/import-error-for-icu-in-mac-and-ubuntu-although-pyicu-is-installed-correctly/50364835#50364835
https://www.khalidalnajjar.com/setup-use-stanford-corenlp-server-python/

