INNER CODE UNIT · Python
estimate_gpu_memory_requirement
ai-infra-curriculum/ai-infra-engineer-learning · projects/project-103-llm-deployment/src/llm/server.py:423
def estimate_gpu_memory_requirement(
model_name: str,
quantization: Optional[str] = None
) -> float:
"""
Estimate GPU memory requirement for a model.
Args:
model_name: Hugging Face model name
quantization: Quantization method (awq, gptq, 8bit, 4bit)
Returns:
Estimated memory in GB
TODO: Implement estimation logic:
1. Get number of parameters from model config
2. Calculate memory based on precision:
- FP16: 2 bytes per parameter