INNER CODE UNIT · Python

Item

CerebriumAI/examples · 1-getting-started/2-using-cerebrium-secrets/main.py:35

class Item(BaseModel):
    prompt: str
    cutoff_len: Optional[int] = 256
    temperature: Optional[float] = 0.1
    top_p: Optional[float] = 0.75
    top_k: Optional[float] = 40
    num_beams: Optional[int] = 4
    max_new_tokens: Optional[int] = 256


base_model_name = "meta-llama/Llama-2-7b-hf"  # Hugging Face Model Id
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = LlamaTokenizer.from_pretrained(base_model_name)

View source record →