INNER CODE UNIT · Python
load_local_llm
junruxiong/IncarnaMind · toolkit/local_llm.py:138
def load_local_llm(
model_id: str,
model_basename: str,
temperature: float,
max_tokens: int,
device_type: str,
):
"""
Select a model for text generation using the HuggingFace library.
If you are running this for the first time, it will download a model for you.
subsequent runs will use the model from the disk.
Args:
device_type (str): Type of device to use, e.g., "cuda" for GPU or "cpu" for CPU.
model_id (str): Identifier of the model to load from HuggingFace's model hub.
model_basename (str, optional): Basename of the model if using quantized models.
Defaults to None.