Metadata-Version: 2.4
Name: cefrpy
Version: 1.0.2
Summary: Python package for analyzing words based on the CEFR level.
Author: Maksym Bielikov
Author-email: maximax6767@gmail.com
Project-URL: Source, https://github.com/Maximax67/cefrpy
Project-URL: Bug Tracker, https://github.com/Maximax67/cefrpy/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# cefrpy

![PyPI - License](https://img.shields.io/pypi/l/cefrpy)
![PyPI - Version](https://img.shields.io/pypi/v/cefrpy)
![PyPI - Downloads](https://img.shields.io/pypi/dm/cefrpy)
[![Hugging Face Space](https://img.shields.io/badge/Hugging%20Face-Space-4CA6A7?logo=huggingface&style=flat)](https://huggingface.co/spaces/Maximax67/cefrpy-demo)
[![Unit Tests](https://github.com/Maximax67/cefrpy/actions/workflows/unittest.yml/badge.svg)](https://github.com/Maximax67/cefrpy/actions/workflows/unittest.yml)

## About cefrpy

The cefrpy python module offers a comprehensive toolkit for analyzing linguistic data based on the Common European Framework of Reference for Languages (CEFR).

Documentation: <https://maximax67.github.io/cefrpy/>

HuggingFace demo: <https://huggingface.co/spaces/Maximax67/cefrpy-demo>

## Installation

You can install `cefrpy` for Python >= 3.6 via pip:

```bash
pip install cefrpy
```

## Usage examples

### Getting part of speech level of a word

```py
from cefrpy import CEFRAnalyzer

analyzer = CEFRAnalyzer()

word = "happy"
pos_tag = "JJ"  # Adjective
level = analyzer.get_word_pos_level_float(word, pos_tag)
if level is not None:
    print(f"The level of '{word}' as a {pos_tag} is: {level}")
else:
    print(f"Level not found for '{word}' a {pos_tag}.")


# You can also get the level in CEFR format
cefr_level = analyzer.get_word_pos_level_CEFR(word, pos_tag)
if cefr_level is not None:
    print(f"The CEFR level of '{word}' as a {pos_tag} is: {cefr_level}")
else:
    print(f"CEFR level not found for '{word}' as a {pos_tag}.")
```

### Getting average level of a word

```py
from cefrpy import CEFRAnalyzer

analyzer = CEFRAnalyzer()

word = "supremacy"
average_level = analyzer.get_average_word_level_float(word)
if average_level is not None:
    print(f"The average level of '{word}' is: {average_level}")
else:
    print(f"Average level not found for '{word}'.")


# You can also get the average level in CEFR format
cefr_average_level = analyzer.get_average_word_level_CEFR(word)
if cefr_average_level is not None:
    print(f"The CEFR average level of '{word}' is: {cefr_average_level}")
else:
    print(f"CEFR average level not found for '{word}'.")
```

### Recommended usage with [spaCy](https://spacy.io)

#### Import spacy and load model

```py
import spacy

from cefrpy import CEFRSpaCyAnalyzer, CEFRLevel

nlp = spacy.load("en_core_web_sm")
```

#### Analyze any text

Optionally supports custom abbreviation mapping and exclusion of specific [spaCy](https://spacy.io) entity types (such as names of people, languages, countries, cities, etc.) from CEFR level matching.

```py
# Source: ChatGPT 3.5
text = """
In the heart of every forest, a hidden world thrives among the towering trees. Trees, 
those silent giants, are more than just passive observers of nature's drama; they are 
active participants in an intricate dance of life.

Did you know that trees communicate with each other? It's not through words or gestures 
like ours, but rather through a complex network of fungi that connect their roots 
underground. This network, often called the "wood wide web," allows trees to share 
nutrients, water, and even warnings about potential threats.

But trees are not just generous benefactors; they are also masters of adaptation. Take 
the mighty sequoias, for example, towering giants that have stood the test of time for 
thousands of years. These giants have evolved thick, fire-resistant bark to withstand 
the frequent wildfires of their native California.

And speaking of longevity, did you know that some trees have been around for centuries, 
witnessing history unfold? The ancient bristlecone pines of the American West, for 
instance, can live for over 5,000 years, making them some of the oldest living organisms 
on Earth.

So the next time you find yourself wandering through a forest, take a moment to appreciate 
the remarkable world of trees. They may seem like silent spectators, but their lives are 
full of fascinating stories waiting to be discovered.
"""

ABBREVIATION_MAPPING = {
    "'m": "am",
    "'s": "is",
    "'re": "are",
    "'ve": "have",
    "'d": "had",
    "n't": "not",
    "'ll": "will"
}

# Optional. List of all possible spaCY entity types:
# 'CARDINAL', 'DATE', 'EVENT', 'FAC', 'GPE', 'LANGUAGE', 'LAW', 'LOC', 'MONEY',
# 'NORP', 'ORDINAL', 'ORG', 'PERCENT', 'PERSON', 'PRODUCT', 'QUANTITY', 'TIME',
# 'WORK_OF_ART'
ENTITY_TYPES_TO_SKIP_CEFR = {
    'QUANTITY', 'MONEY', 'LANGUAGE', 'LAW',
    'WORK_OF_ART', 'PRODUCT', 'GPE',
    'ORG', 'FAC', 'PERSON'
}

doc = nlp(text)
text_analyzer = CEFRSpaCyAnalyzer(entity_types_to_skip=ENTITY_TYPES_TO_SKIP_CEFR, abbreviation_mapping=ABBREVIATION_MAPPING)
tokens = text_analyzer.analyze_doc(doc)

print('-' * 55)
print(f' {"WORD".ljust(26)}\tPOS\tLEVEL\tCEFR')
print('-' * 55)
for token in tokens:
    word, pos, is_skipped, level, _, _ = token
    print(f'{word.ljust(26)}\t{pos}\t{"Skip" if is_skipped else "{:.2f}".format(level)}\t{CEFRLevel(round(level)) if level else None}')

```

Result (truncated):

```text
-------------------------------------------------------
 WORD                       POS LEVEL CEFR
-------------------------------------------------------
                           _SP Skip None
In                         IN 1.00 A1
the                        DT 1.00 A1
heart                      NN 1.00 A1
of                         IN 1.00 A1
every                      DT 1.00 A1
forest                     NN 2.00 A2
,                          , Skip None
a                          DT 1.00 A1
hidden                     JJ 3.00 B1
world                      NN 1.00 A1
thrives                    VBZ 5.86 C2
among                      IN 2.00 A2
the                        DT 1.00 A1
towering                   VBG 1.00 A1
trees                      NNS 1.00 A1
.                          . Skip None
Trees                      NNS 1.00 A1
,                          , Skip None
                           _SP Skip None
those                      DT 1.00 A1
silent                     JJ 3.00 B1
```

#### Get more statistical information

1. Filter tokens by level:

    ```py
    def filter_for_desired_level(level_tokens: list[tuple[str, str, bool, float, int, int]],
                                min_level: float | int = 1.0, max_level: float | int = 6.0
                                ) -> set[tuple[str, str, bool, float, int, int]]:
        filtered_tokens = set()
        for token in level_tokens:
            level = token[3]

            if level and level >= min_level and level <= max_level:
                filtered_tokens.add(token)

        return filtered_tokens


    # You can also set min/max level as an int or float in range from 1 to 6
    desired_min_level = CEFRLevel.C1
    desired_level_words_set = filter_for_desired_level(tokens, min_level=int(desired_min_level))

    desired_level_words_list = list(desired_level_words_set)
    desired_level_words_list.sort()

    print(f'\tWords with level {desired_min_level} and higher: {len(desired_level_words_list)}')
    for word_data in desired_level_words_list:
        word, pos, _, level, _, _ = word_data
        print(f"{word.ljust(26)} {pos.ljust(6)} {'{:.2f}'.format(level).ljust(6)} {CEFRLevel(round(level))}")
    ```

    ```text
    Words with level B2 and higher: 16
    benefactors                NNS    6.00   C2
    bristlecone                NN     6.00   C2
    evolved                    VBN    4.00   B2
    fungi                      NNS    5.20   C1
    living                     NN     4.00   B2
    longevity                  NN     5.96   C2
    masters                    NNS    4.00   B2
    mighty                     JJ     4.00   B2
    observers                  NNS    4.00   B2
    pines                      NNS    4.00   B2
    potential                  JJ     4.00   B2
    sequoias                   NNS    6.00   C2
    thrives                    VBZ    5.86   C2
    underground                RB     4.00   B2
    wildfires                  NNS    6.00   C2
    withstand                  VB     5.12   C1
    ```

2. Get CEFR statistic of the text:

    ```py
    def get_word_level_count_statistic(level_tokens: list[tuple[str, str, bool, float, int, int]]) -> list[int]:
        difficulty_levels_count = [0] * 6
        for token in level_tokens:
            level = token[3]
            if not level:
                continue

            level_round = round(level)
            difficulty_levels_count[level_round - 1] += 1

        return difficulty_levels_count

    difficulty_levels_count = get_word_level_count_statistic(tokens)
    print('CEFR statistic (total words):')
    for i in range(1, 7):
        print(f'{CEFRLevel(i)}: {difficulty_levels_count[i - 1]}')
    ```

    ```text
    CEFR statistic (total words):
    A1: 136
    A2: 36
    B1: 27
    B2: 11
    C1: 2
    C2: 6
    ```

3. Get CEFR statistic for unique words in the text:

    ```py
    def get_word_level_count_statistic_unique(level_tokens: list[tuple[str, str, bool, float, int, int]]) -> list[int]:
        processed_word_pos_set = set()
        difficulty_levels_count = [0] * 6
        for token in level_tokens:
            level = token[3]
            if not level:
                continue

            to_check_tuple = (token[0], token[1])
            if not to_check_tuple in processed_word_pos_set:
                level_round = round(token[3])
                difficulty_levels_count[level_round - 1] += 1
                processed_word_pos_set.add(to_check_tuple)

        return difficulty_levels_count


    difficulty_levels_count_unique = get_word_level_count_statistic_unique(tokens)
    print('CEFR statistic (unique words):')
    for i in range(1, 7):
        print(f'{CEFRLevel(i)}: {difficulty_levels_count_unique[i - 1]}')
    ```

    ```text
    CEFR statistic (unique words):
    A1: 77
    A2: 33
    B1: 23
    B2: 11
    C1: 2
    C2: 6
    ```

4. Get set of not found CEFR levels for words in text:

    ```py
    def get_not_found_words(level_tokens: list[tuple[str, str, bool, float, int, int]]) -> set[str]:
        not_found_words = set()
        for token in level_tokens:
            if token[2]:
                continue

            if not token[3]:
                not_found_words.add(token[0])

        return not_found_words


    not_found_words_set = get_not_found_words(tokens)
    not_found_words_list = list(not_found_words_set)
    not_found_words_list.sort()

    print('Not found words:', len(not_found_words_list))
    if len(not_found_words_list):
        print('\n'.join(not_found_words_list))
    ```

    ```text
    Not found words: 0
    ```

## Additional features

### Get all possible part-of-speech tags for a word

```py
from cefrpy import CEFRAnalyzer

analyzer = CEFRAnalyzer()

print(analyzer.get_all_pos_for_word("test")) # [<POSTag.JJ: 4>, <POSTag.NN: 8>, <POSTag.VB: 19>]
print(analyzer.get_all_pos_for_word_as_str("test")) # ['JJ', 'NN']

# {<POSTag.JJ: 4>: <CEFRLevel.A2: 2>, <POSTag.NN: 8>: <CEFRLevel.A1: 1>, <POSTag.VB: 19>: <CEFRLevel.B2: 4>}
print(analyzer.get_pos_level_dict_for_word("test"))

# {'JJ': 2.5, 'NN': 1.0, 'VB': 4.0}
print(analyzer.get_pos_level_dict_for_word("test", pos_tag_as_string=True, word_level_as_float=True))
```

### Checking if a word exists in the database

```py
from cefrpy import CEFRAnalyzer

analyzer = CEFRAnalyzer()

word = "apple"
if analyzer.is_word_in_database(word):
    print(f"'{word}' exists in the database.")
else:
    print(f"'{word}' does not exist in the database.")
```

### Checking if a word with a specific part-of-speech exists in the database

```py
from cefrpy import CEFRAnalyzer

analyzer = CEFRAnalyzer()

word = "run"
pos_tag = "VB"  # Verb
if analyzer.is_word_pos_id_database(word, pos_tag):
    print(f"'{word}' with part of speech '{pos_tag}' exists in the database.")
else:
    print(f"'{word}' with part of speech '{pos_tag}' does not exist in the database.")
```

### POSTag usage examples

```py
from cefrpy import POSTag

# Get list of all part-of-speech tag names
print(POSTag.get_all_tags()) # ['CC', 'CD', 'DT', ...]

# Print total tags
print(POSTag.get_total_tags()) # 28

# Get description for a tag
print(POSTag.get_description_by_tag_name('NN')) # Noun, singular or mass

tag = POSTag.VB
print(tag)                          # VB
print(POSTag.get_description(tag))  # Verb, base form
print(int(tag))                     # 19 (unique tag id)
print(tag == POSTag.NN)             # False
```

### CEFRLevel usage examples

```py
from cefrpy import CEFRLevel

level = CEFRLevel.A1
print(level)            # A1
print(int(level))       # 1

level2 = CEFRLevel.C2
print(level2)           # C2
print(int(level2))      # 6

# You can perform any comparisons:
print(level2 > level)   # True
print(level2 == level)  # False

print(CEFRLevel.from_str("B1") == CEFRLevel.B1) # True
print(CEFRLevel.from_str("B1") == CEFRLevel(3)) # True
```

### Yields CEFRAnalyzer methods

For every example you should import and initialize `CEFRAnalyzer`:

```py
from cefrpy import CEFRAnalyzer

analyzer = CEFRAnalyzer()
```

#### Iterating over words with a specific length (alphabetical order)

```py
iteration_limit = 10
word_list = []
for word in analyzer.yield_words_with_length(6):
    if iteration_limit == 0:
        break
    word_list.append(word)
    iteration_limit -= 1

# ['aaberg', 'aachen', 'aahing', 'aargau', 'aarhus', 'abacus', 'abadan', 'abadia', 'abakan', 'abaris']
print(word_list)
```

#### Iterating over words with a specific length (reversed alphabetical order)

```py
iteration_limit = 10
word_list = []
for word in analyzer.yield_words_with_length(6, reverse_order=True):
    if iteration_limit == 0:
        break
    word_list.append(word)
    iteration_limit -= 1

# ['zymase', 'zygote', 'zygoma', 'zydeco', 'zwolle', 'zwicky', 'zuzana', 'zusman', 'zurvan', 'zurich']
print(word_list)
```

#### Iterating over words in alphabetical order

```py
iteration_limit = 10
word_list = []
for word in analyzer.yield_words():
    if iteration_limit == 0:
        break
    word_list.append(word)
    iteration_limit -= 1

# ['a', 'aa', 'aaa', 'aaaa', 'aaas', 'aaberg', 'aachen', 'aae', 'aaee', 'aaf']
print(word_list)
```

#### Iterating over words with their pos in alphabetical order with word length priority ascending

```py
iteration_limit = 6
word_pos_list = []
for word, pos_tag in analyzer.yield_word_pos(word_length_sort=True):
    if iteration_limit == 0:
        break
    word_pos_list.append((word, pos_tag))
    iteration_limit -= 1

# [('a', <POSTag.DT: 2>), ('a', <POSTag.IN: 3>), ('a', <POSTag.JJ: 4>), ('a', <POSTag.NN: 8>), ('a', <POSTag.VB: 19>), ('b', <POSTag.JJ: 4>)]
print(word_pos_list)
```

#### Iterating over words with their pos as str and levels as float in reversed alphabetical order with word length priority descending

```py
iteration_limit = 3
word_pos_list = []
for word, pos_tag, level in analyzer.yield_word_pos_level(word_length_sort=True, reverse_order=True, pos_tag_as_string=True, word_level_as_float=True):
    if iteration_limit == 0:
        break
    word_pos_list.append((word, pos_tag, level))
    iteration_limit -= 1

# [('demethylchlortetracycline', 'NN', 6.0), ('electrocardiographically', 'RB', 6.0), ('polytetrafluoroethylene', 'NN', 6.0)]
print(word_pos_list)
```

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Acknowledgments

I would like to acknowledge the contributions of the following resources. I used them to create my initial SQLite version [Words-CEFR-Dataset](https://github.com/Maximax67/Words-CEFR-Dataset):

- [Spacy](https://spacy.io/)
- [CEFR-J](https://cefr-j.org/)
- [LemmInflect](https://github.com/bjascob/LemmInflect)
- [The Google Books Ngram Viewer (used 1-grams dataset, version 20200217)](https://books.google.com/ngrams/)
- [List of pos tags form Penn Treebank Project](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html)

Also I used these resources to create my [valid English words list](https://github.com/Maximax67/English-Valid-Words):

- [Word list by infochimps (archived)](https://web.archive.org/web/20131118073324/https://www.infochimps.com/datasets/word-list-350000-simple-english-words-excel-readable)
- [English words github repo by dwyl](https://github.com/dwyl/english-words)
- [NLTK (Natural Language Toolkit)](https://www.nltk.org/)
- [WordNet](https://wordnet.princeton.edu/)

MIT License

Copyright (c) 2024 Bielikov Maksym

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
