INNER CODE UNIT · Python

read_header

FareedKhan-dev/kimi-k3-in-c · tools/budget.py:77

def read_header(path: str) -> dict:
    """Return one shard's tensor index.

    A safetensors file starts with an 8-byte little-endian length followed by that many
    bytes of JSON. Only those bytes are read, so this touches a few hundred KB of a
    17 GB shard.

    The length is bounds-checked before it is used as a read size: it comes from a file
    that may have been fetched from a mirror, and an absurd value would otherwise turn
    into an absurd allocation.
    """
    with open(path, "rb") as f:
        raw = f.read(8)
        if len(raw) != 8:
            raise ValueError("%s: too short to be a safetensors file" % path)
        n = struct.unpack("<Q", raw)[0]
        if not 0 < n < (1 << 30):
            raise ValueError("%s: implausible header length %d" % (path, n))

View source record →