INNER CODE UNIT · Python

deep_tokenize_in_vocab

R1j1t/contextualSpellCheck · contextualSpellCheck/contextualSpellCheck.py:585

    def deep_tokenize_in_vocab(self, text):
        """Check if the token contains punctuations
            if char is punctuation then check in vocab
            check rest of the word in vocab
            if both in vocab return False

        Args:
            text (str): Text to tokenize again for punct

        Returns:
            Bool: True if both punct and rest of the word
                 in vocab
        """
        text_len = len(text)
        sub_tokens = []
        pre_puct_position = -1
        for char_position in range(text_len):
            if unicodedata.category(text[char_position]).startswith("P"):

View source record →