INNER CODE UNIT · Python

get_gpu_stats

ai-infra-curriculum/ai-infra-engineer-learning · projects/project-103-llm-deployment/src/llm/server.py:297

    def get_gpu_stats(self) -> Dict[str, float]:
        """
        Get current GPU utilization statistics.

        Returns:
            Dictionary with GPU metrics

        TODO: Implement GPU monitoring:
        1. Query CUDA memory usage (allocated, cached, free)
        2. Get GPU utilization percentage
        3. Get GPU temperature
        4. Calculate KV cache usage

        Hints:
        - Use torch.cuda.memory_allocated()
        - Use torch.cuda.memory_reserved()
        - Use pynvml for detailed GPU stats
        """

View source record →