FIELD NOTES TOPIC
adaptive batching
2 notes, newest first.
Adaptive Inference Batching using Policy Gradients
The research proposes using reinforcement learning (RL) to create adaptive batching and routing policies for inference serving systems, which can handle bursty, heterogeneous workloads more efficiently than traditional…
Adaptive Inference Batching using Policy Gradients
The paper investigates using reinforcement learning (RL) to create adaptive batching and routing policies for inference serving systems that can handle bursty, heterogeneous workloads. The research finds that in…
