INNER CODE UNIT · Python

scale

Extraltodeus/ComfyUI-AutomaticCFG · experimental_temperature.py:30

        scale = dim_head ** -0.5

        h = heads
        q, k, v = map(
            lambda t: t.unsqueeze(3)
            .reshape(b, -1, heads, dim_head)
            .permute(0, 2, 1, 3)
            .reshape(b * heads, -1, dim_head)
            .contiguous(),
            (q, k, v),
        )

        # force cast to fp32 to avoid overflowing
        if attn_precision == torch.float32:
            sim = einsum('b i d, b j d -> b i j', q.float(), k.float()) * scale
        else:
            sim = einsum('b i d, b j d -> b i j', q, k) * scale

View source record →