Continuous Batching Inference Server
A minimal serving engine that demonstrates the scheduling ideas behind high throughput LLM inference.
Overview
This project implements an iteration level scheduling loop for variable length requests and makes the tradeoffs behind continuous batching visible in a compact codebase.
Selected results
- Added dynamic request scheduling and iteration level batching.
- Managed a bounded KV cache budget under changing sequence lengths.
- Implemented preemption for lower priority sequences under memory pressure.
- Documented why sequential serving leaves accelerator capacity unused.