FIELD NOTES TOPIC
multi-resource decisions
1 note, newest first.
Adaptive Inference Batching using Policy Gradients
The paper investigates using reinforcement learning (RL) to create adaptive batching and routing policies for inference serving systems that can handle bursty, heterogeneous workloads. The research finds that in…
