INNER CODE UNIT · Python

main

HIT-SCIR/Chinese-Mixtral-8x7B · data/preprocess_datasets.py:61

def main(
        ds_name: str,
        tokenizer_name_or_path: str,
        sequence_length: int = 2048,
        cache_dir: str = "./hf-cache",
):
    tokenizer = transformers.AutoTokenizer.from_pretrained(
        tokenizer_name_or_path,
        cache_dir=cache_dir,
        model_max_length=sequence_length,
        padding_side="right",
        use_fast=True,
    )
    if tokenizer.pad_token is None:
        tokenizer.add_special_tokens(special_tokens_dict=dict(pad_token="</s>"))

    with open("./data/datasets.toml", "rb") as f:
        ds_info = tomli.load(f)

View source record →