INNER CODE UNIT · Python

is_num

Koziev/NLP_Datasets · Samples/sort_samples_by_kenlm.py:29

def is_num(s):
    if len(s)>0:
        if s[0] in ('-', '+'):
            return s[1:].isdigit()
        return s.isdigit()
    else:
        return False


def prepare_word(w):
    if is_num(w):
        return u'_num_'
    else:
        return w.lower()


def prepare4lm(tokens):
    s2 = unicode.join(u' ', [prepare_word(t) for t in tokens if not is_punct(t) and len(t)>0 ])

View source record →