INNER CODE UNIT · Python

preprocess_training_data

dragondrop-cloud/cloud-concierge · nlpengine/main.py:85

def preprocess_training_data(nlp: spacy.Language, category_docs: dict) -> List[Example]:
    """Creates all examples needed to train the text classification model."""
    list_of_examples = []

    # List of all categories
    categories = list(category_docs.keys())

    for category, doc in category_docs.items():
        current_category_examples = _create_examples_from_doc(
            nlp=nlp,
            current_cat=category,
            doc=doc,
            categories=categories,
        )
        list_of_examples.extend(current_category_examples)

    return list_of_examples

View source record →