Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

Source

arXiv

arxiv.org/abs/2607.08565v1

Read the source

Source

Plain-Language Brief

The paper 'SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling' presents a new approach to scheduling Language Model (LLM) requests for AI agents, rather than humans. The proposed method, called SMETRIC, aims to balance the workload by prioritizing load balance and local reuse of Key-Value (KV$) data, while keeping demand on the global tier low. SMETRIC improves cluster Throughput Per Second (TPS) by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation compared to state-of-the-art schedulers.

What the Document Says

  • The paper discusses the importance of LLM scheduling for serving AI agents, as requests issued by agents differ from those issued by humans.
  • Existing LLM scheduling designs may not be effective for agentic serving due to changes in workload and request reuse rates.
  • The authors conduct a study on two real-world traces to analyze request scheduling for agents.
  • Overloading a few instances while leaving the rest idle is found to be an issue with existing schedulers due to prioritizing KV$ reuse.
  • Two key insights are presented: load balance need not sacrifice all KV$ reuse and utilizing intra-session locality can balance the cluster without sacrificing most KV$ reuse on local instances.
  • SMETRIC is a new approach that realizes these insights with balanced session-centric scheduling, routing each session's first request for load balance and follow-up requests in a cache-aware manner.
  • Using the session turn information as the scheduling metric is deliberate because it can be derived efficiently and accurately from user inputs.
  • SMETRIC improves cluster TPS and offers better per-token latency compared to state-of-the-art schedulers.

Key Concepts Mentioned

  • LLM scheduling
  • AI agents
  • Key-Value (KV$) reuse
  • Load balance
  • Throughput Per Second (TPS)
  • SMETRIC

Claims Made by the Document

  • Existing LLM scheduling designs may not be effective for agentic serving.
  • Overloading a few instances while leaving the rest idle is an issue with existing schedulers.
  • Load balance need not sacrifice all KV$ reuse.
  • Utilizing intra-session locality can balance the cluster without sacrificing most KV$ reuse on local instances.

Evidence Used

  • The study on two real-world traces.
  • Findings from the study.
  • Comparison with state-of-the-art schedulers.

Methods / Approach

  • Systematic study of request scheduling for agents
  • Comparative analysis with state-of-the-art schedulers

Limitations Stated by the Authors

Unknown

Limitations Noted by the Agent

Agent interpretation — not from the source.

Not stated in source

Open Questions

  • What are the specific algorithms or techniques used in SMETRIC?
  • How does SMETRIC handle dynamic changes in workload?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper doesn't provide details on the specific algorithms or techniques used in SMETRIC.
  • It's unclear how SMETRIC handles dynamic changes in workload.

Raw Abstract / Excerpt

LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans. This shifts the workload in two ways: (1) agents act only on complete responses, making the cluster's tokens per second (TPS) the primary goal and relaxing--not eliminating--per-token latency requirements; and (2) requests share much of their KV$-reuse exceeds 80% of request tokens in a production trace from BAILIAN, versus 54-62% in chat. This paper first contributes a systematic study of request scheduling for agents on two real-world traces. We find that to increase KV$ reuse, existing schedulers overly prioritize routing requests to instances caching their KV$, overloading a few while leaving the rest idle, capping TPS. We thus present two key insights: (1) load balance need not sacrifice all KV$ reuse, thanks to the global-tier KV$ store and (2) by utilizing the workload's intra-session locality, balancing a small fraction of requests--the first request in each agent session--suffices to balance the cluster without sacrificing most KV$ reuse on local instances. SMETRIC realizes these insights with balanced session-centric scheduling: it routes each session's first request purely for load balance and its follow-up requests in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using the session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from the user inputs alone, so the scheduler stays clean and stateless. SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation over state-of-the-art schedulers, also with a better per-token latency.