INNER CODE UNIT · Python

is_punct

Koziev/NLP_Datasets · Samples/sort_samples_by_kenlm.py:25

def is_punct(word):
    return len(word)>0 and unicodedata.category(word[0]) == 'Po'


def is_num(s):
    if len(s)>0:
        if s[0] in ('-', '+'):
            return s[1:].isdigit()
        return s.isdigit()
    else:
        return False


def prepare_word(w):
    if is_num(w):
        return u'_num_'
    else:
        return w.lower()

View source record →