INNER CODE UNIT · Python

params_before

Guitaricet/relora · torchrun_main.py:529

    params_before = sum(p.numel() for p in model.parameters())

    if args.use_peft:
        need_linear_weight = (
            args.relora is not None
            or args.force_keep_original
            or args.warmed_up_model is not None
        )
        logger.info(f"Wrapping model with LoRA ({need_linear_weight=})")

        # target modules should define all linear layers from transformer block
        # "attn" and "mlp" are used in LLaMA
        # "attention" and "mlp" are used in Pythia
        model = ReLoRaModel(
            model,
            r=args.lora_r,
            lora_alpha=args.lora_alpha,
            lora_dropout=0.1,

View source record →