INNER CODE UNIT · Python

max_model_len

KodCode-AI/kodcode · pipeline/completion_open_model.py:232

            max_model_len = args.max_model_len, # limited by kv-cache 
            tensor_parallel_size = args.tensor_parallel_size,
            gpu_memory_utilization = args.gpu_memory_utilization
        )

        params = SamplingParams(
            max_tokens=args.max_tokens,
            temperature=args.temperature,
            top_p=args.top_p,
            repetition_penalty=args.repetition_penalty,
            stop_token_ids=stop_token_ids,
        )

        tokenizer = AutoTokenizer.from_pretrained(args.model_path)
    elif args.engine == "hf":
        print("Start Hugging Face engine...")
        params = None
        # Load the model and tokenizer

View source record →