FIELD NOTES TOPIC
routing policies
1 note, newest first.
Adaptive Inference Batching using Policy Gradients
The research proposes using reinforcement learning (RL) to create adaptive batching and routing policies for inference serving systems, which can handle bursty, heterogeneous workloads more efficiently than traditional…
