INNER CODE UNIT · Python

main

HIT-SCIR/Chinese-Mixtral-8x7B · data/parquet2jsonl.py:21

def main(
    target_jsonl: str = "./data/en/DKYoon-SlimPajama-6B.jsonl",
    root: str = "./data/en/DKYoon-SlimPajama-6B/data",
):
    root = Path(root)
    target_jsonl = Path(target_jsonl)

    files = os.listdir(root)

    with open(target_jsonl, "w") as f:
        for file in tqdm(files):
            objs = parse_file(root / file)
            for obj_str in list(objs):
                try:
                    obj = json.loads(obj_str)
                    f.write(f"{json.dumps(obj, ensure_ascii=False)}\n")
                except:
                    print("json parse error")

View source record →