INNER CODE UNIT · Python

make_inputs_require_grad

HIT-SCIR/Chinese-Mixtral-8x7B · train.py:250

            def make_inputs_require_grad(module, input, output):
                output.requires_grad_(True)

            model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)

        model = get_peft_model(model, peft_config)

        # To resume from existing adapter
        # from peft import PeftModel
        # model = PeftModel.from_pretrained(model, "/path/to/adapter/checkpoint", is_trainable=True)

        for name, param in model.named_parameters():
            if "lora" in name:
                param.data = param.data.to(torch.bfloat16)

        print_rank_0("=========================================")
        model.print_trainable_parameters()
        print_rank_0("=========================================")

View source record →