INNER CODE UNIT · Python
batch_generate
ai-infra-curriculum/ai-infra-engineer-learning · projects/project-103-llm-deployment/src/llm/server.py:245
async def batch_generate(
self,
prompts: List[str],
**generation_kwargs
) -> List[str]:
"""
Generate text for multiple prompts in a batch.
Args:
prompts: List of input prompts
**generation_kwargs: Generation parameters
Returns:
List of generated texts
TODO: Implement batch processing:
1. Create sampling params for all requests
2. Submit all prompts to vLLM engine