What is continuous batching and why is it important for local model serving?
Continuous batching or dynamic batching is an optimization technique where new inference requests are inserted into the processing batch as soon as any existing request completes, rather than waiting for the entire batch to finish. In traditional static batching, all requests in a batch must complete before any results are returned, and new requests wait for the next batch cycle. Continuous batching maximizes GPU utilization by keeping the processing pipeline full. This is particularly important for local model serving where hardware resources are limited. vLLM and TGI implement continuous batching, achieving 2-5x higher throughput compared to static batching. For offline deployments serving multiple users from a single GPU, continuous batching makes the difference between responsive and sluggish service.