FIELD NOTES TOPIC
scheduling
3 notes, newest first.
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
The paper proposes SMetric, a new scheduling method for Large Language Models (LLMs) in serving agents. The method aims to balance the session-centric workload of agents and increase Key-Value (KV$) reuse by…
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
The paper 'SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling' presents a new approach to scheduling Language Model (LLM) requests for AI agents, rather than humans. The…
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
The paper proposes a new LLM (Language Model) scheduling method called SMETRIC for serving agents. This method aims to balance the workload between instances by focusing on the first request of each agent session and…
