Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

Source

arXiv

arxiv.org/abs/2607.08565v1

Read the source

Source

Plain-Language Brief

The paper proposes a new method for scheduling Language Model (LLM) requests called SMetric, which aims to improve the performance of serving agents in terms of tokens per second (TPS). The current methods prioritize routing requests to instances that cache the needed Key-Value pairs (KV$), leading to overloading a few instances and leaving others idle. However, SMetric balances the load by routing each session's first request for load balance and subsequent requests in a cache-aware manner, preserving both load balance and local reuse while keeping demand on the global tier low. This leads to an improvement in TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation compared to state-of-the-art schedulers.

What the Document Says

  • The paper discusses LLM scheduling for agentic serving, where requests are issued by agents instead of humans.
  • The shift in workload makes the cluster's TPS the primary goal and relaxes per-token latency requirements.
  • Requests share a high percentage of KV$-reuse, exceeding 80% in a production trace from BAILIAN compared to 54-62% in chat.
  • Existing schedulers prioritize routing requests to instances caching their KV$, but this leads to overloading a few instances and leaving others idle, thus capping TPS.
  • The paper presents two key insights: balancing load need not sacrifice all KV$ reuse due to the global-tier KV$ store and by utilizing the workload's intra-session locality, balancing a small fraction of requests is enough to balance the cluster without sacrificing most KV$ reuse on local instances.
  • SMETRIC realizes these insights with balanced session-centric scheduling, which routes each session's first request purely for load balance and subsequent requests in a cache-aware manner.
  • Using the session turn information as the scheduling metric is deliberate, as it can be derived efficiently and accurately from user inputs alone, keeping the scheduler clean and stateless.

Key Concepts Mentioned

  • LLM scheduling
  • agentic serving
  • tokens per second (TPS)
  • Key-Value pairs (KV$)

Claims Made by the Document

  • Existing LLM schedulers overly prioritize routing requests to instances caching their KV$, leading to overloading and idle instances.
  • By balancing a small fraction of requests, the cluster can be balanced without sacrificing most KV$ reuse on local instances.
  • SMETRIC improves TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation compared to state-of-the-art schedulers.

Evidence Used

  • The paper presents research results from two real-world traces.
  • The improvements in TPS are provided as numerical data.

Methods / Approach

  • Systematic study of request scheduling for agents on two real-world traces
  • Balanced session-centric scheduling with SMETRIC

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans. This shifts the workload in two ways: (1) agents act only on complete responses, making the cluster's tokens per second (TPS) the primary goal and relaxing--not eliminating--per-token latency requirements; and (2) requests share much of their KV$-reuse exceeds 80% of request tokens in a production trace from BAILIAN, versus 54-62% in chat. This paper first contributes a systematic study of request scheduling for agents on two real-world traces. We find that to increase KV$ reuse, existing schedulers overly prioritize routing requests to instances caching their KV$, overloading a few while leaving the rest idle, capping TPS. We thus present two key insights: (1) load balance need not sacrifice all KV$ reuse, thanks to the global-tier KV$ store and (2) by utilizing the workload's intra-session locality, balancing a small fraction of requests--the first request in each agent session--suffices to balance the cluster without sacrificing most KV$ reuse on local instances. SMETRIC realizes these insights with balanced session-centric scheduling: it routes each session's first request purely for load balance and its follow-up requests in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using the session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from the user inputs alone, so the scheduler stays clean and stateless. SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation over state-of-the-art schedulers, also with a better per-token latency.