INNER CODE UNIT · Python

outputs

KodCode-AI/kodcode · pipeline/completion_open_model.py:142

        outputs = llm.generate(**inputs,
                tokenizer=tokenizer, 
                do_sample=gen_do_sample, 
                temperature=args.temperature if gen_do_sample else None, # To avoid temperature` (=0) has to be a strictly positive float
                top_p=args.top_p,
                repetition_penalty=args.repetition_penalty, 
                max_length=args.max_tokens,
                )
        outputs = tokenizer.batch_decode(outputs[i][len(inputs[i]):] for i in range(len(outputs)))
        # Setting stop tokens seems not working for Gemma, so we manually truncate the outputs
        for i, completion in enumerate(outputs):
            for stop_token in stop_tokens:
                if stop_token in completion:
                    outputs[i] = completion[:completion.index(stop_token)]

    for i, item in enumerate(batch):
        message = item["messages"]
        if args.engine == "vllm":

View source record →