INNER CODE UNIT · Python

lr

lucidrains/x-transformers · train_discover_reversal.py:69

    lr: float = 3e-4
):
    bos_id = num_fillers
    pad_id = num_fillers + 1
    num_tokens = num_fillers + 2

    device = default_device()
    torch.manual_seed(seed)

    decoder = TransformerWrapper(
        num_tokens = num_tokens,
        max_seq_len = 16,
        emb_dropout = 0.05,
        attn_layers = Decoder(
            dim = dim,
            depth = depth,
            heads = heads,
            attn_dropout = 0.05,

View source record →