FIELD NOTES TOPIC
queuing theory
1 note, newest first.
Adaptive Inference Batching using Policy Gradients
The paper investigates using reinforcement learning (RL) to create adaptive batching and routing policies for inference serving systems that can handle bursty, heterogeneous workloads. The research finds that in…
