Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

Source

arXiv

arxiv.org/abs/2607.08565v1

Read the source

Source

Plain-Language Brief

The paper proposes SMetric, a new scheduling method for Large Language Models (LLMs) in serving agents. The method aims to balance the session-centric workload of agents and increase Key-Value (KV$) reuse by prioritizing load balance and utilizing locality within agent sessions. SMetric improves the Tokens Per Second (TPS) of the cluster by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation compared to state-of-the-art schedulers.

What the Document Says

  • The paper discusses the need for rethinking LLM scheduling for serving agents.
  • Agents act on complete responses, making TPS the primary goal.
  • Requests issued by agents share much KV$-reuse with 80% in a production trace from BAILIAN versus 54-62% in chat.
  • The paper presents two insights: load balance doesn't need to sacrifice all KV$ reuse and utilizing the workload's intra-session locality can balance the cluster.
  • SMETRIC is a new scheduling method that realizes these insights with balanced session-centric scheduling.
  • SMETRIC routes each session's first request for load balance and follow-up requests in a cache-aware manner.

Key Concepts Mentioned

  • LLM scheduling
  • serving agents
  • Tokens Per Second (TPS)
  • Key-Value (KV$) reuse
  • session-centric scheduling
  • SMetric

Claims Made by the Document

  • Existing LLM schedulers overly prioritize routing requests to instances caching their KV$, overloading a few while leaving the rest idle.
  • By utilizing the workload's intra-session locality, balancing a small fraction of requests improves the cluster without sacrificing most KV$ reuse on local instances.

Evidence Used

  • Data from two real-world traces is used to study request scheduling for agents.
  • Results show that SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans. This shifts the workload in two ways: (1) agents act only on complete responses, making the cluster's tokens per second (TPS) the primary goal and relaxing--not eliminating--per-token latency requirements; and (2) requests share much of their KV$-reuse exceeds 80% of request tokens in a production trace from BAILIAN, versus 54-62% in chat. This paper first contributes a systematic study of request scheduling for agents on two real-world traces. We find that to increase KV$ reuse, existing schedulers overly prioritize routing requests to instances caching their KV$, overloading a few while leaving the rest idle, capping TPS. We thus present two key insights: (1) load balance need not sacrifice all KV$ reuse, thanks to the global-tier KV$ store and (2) by utilizing the workload's intra-session locality, balancing a small fraction of requests--the first request in each agent session--suffices to balance the cluster without sacrificing most KV$ reuse on local instances. SMETRIC realizes these insights with balanced session-centric scheduling: it routes each session's first request purely for load balance and its follow-up requests in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using the session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from the user inputs alone, so the scheduler stays clean and stateless. SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation over state-of-the-art schedulers, also with a better per-token latency.