INNER CODE UNIT · Python

tiktoken_len

junruxiong/IncarnaMind · docs2db.py:46

def tiktoken_len(text: str):
    """Calculate the token length of a given text string using TikToken.

    Args:
        text (str): The text to be tokenized.

    Returns:
        int: The length of the tokenized text.
    """
    tokens = tokenizer.encode(text, disallowed_special=())

    return len(tokens)


def string2md5(text: str):
    """Convert a string to its MD5 hash.

    Args:

View source record →