INNER CODE UNIT · Python

total_length

HIT-SCIR/Chinese-Mixtral-8x7B · data/preprocess_datasets.py:22

            total_length = (total_length // sequence_length) * sequence_length
        # Split by chunks of block_size.
        result = {
            k: [t[i: i + sequence_length] for i in range(0, total_length, sequence_length)]
            for k, t in concatenated_examples.items()
        }
        result["labels"] = result["input_ids"].copy()
        return result

    num_workers = multiprocessing.cpu_count()

    raw_datasets = load_dataset(
        "json",
        data_files=data_path,
        split="train",
        cache_dir=cache_dir,
    )

View source record →