SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jiahao Wang, Kaizhan Lin, Kaixi Zhang, Jinbo Han, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Rong Chen, Haibo Chen
• Venue: arXiv: cs.DC
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08565v1
• PDF: https://arxiv.org/pdf/2607.08565v1
• DOI: UNKNOWN

Plain-Language Brief

The paper proposes a new LLM (Language Model) scheduling method called SMETRIC for serving agents. This method aims to balance the workload between instances by focusing on the first request of each agent session and using cache-aware routing for subsequent requests. The goal is to increase KV$ reuse, improve cluster TPS, and maintain a good per-token latency.

What the Document Says

• The paper discusses LLM scheduling for serving agents instead of humans.
• Agents act only on complete responses, making TPS the primary goal.
• Existing schedulers prioritize routing requests to instances caching their KV$, overloading a few while leaving the rest idle.
• The paper presents two key insights: load balance need not sacrifice all KV$ reuse and utilizing the workload's intra-session locality can help balance the cluster.
• SMETRIC is proposed as a method that realizes these insights with balanced session-centric scheduling.
• SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation and prefill TPS by 2-34% under disaggregation compared to state-of-the-art schedulers.

Key Concepts Mentioned

• LLM scheduling
• serving agents
• KV$-reuse
• load balance
• global tier
• cache-aware routing
• SMETRIC

Claims Made by the Document

• Existing LLM schedulers overload a few instances while leaving the rest idle.
• Load balancing need not sacrifice all KV$ reuse.
• Utilizing the workload's intra-session locality can help balance the cluster.

Evidence Used

• The paper provides results from two real-world traces to support its claims.
• It presents experimental improvements in TPS and per-token latency for SMETRIC compared to state-of-the-art schedulers.

Methods / Approach

• Balanced session-centric scheduling
• Using the session turn information as the scheduling metric

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The data used in the study is not specified, and the potential impact of different types of requests or workloads on SMETRIC's performance is unknown.

Open Questions

• How does SMETRIC perform under different types of requests or workloads?
• What are the specific details of the real-world traces used in the study?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper does not provide information on the specific details of the real-world traces used in the study, so further investigation is required to understand their characteristics.

Raw Abstract / Excerpt

LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans. This shifts the workload in two ways: (1) agents act only on complete responses, making the cluster's tokens per second (TPS) the primary goal and relaxing--not eliminating--per-token latency requirements; and (2) requests share much of their KV$-reuse exceeds 80% of request tokens in a production trace from BAILIAN, versus 54-62% in chat. This paper first contributes a systematic study of request scheduling for agents on two real-world traces. We find that to increase KV$ reuse, existing schedulers overly prioritize routing requests to instances caching their KV$, overloading a few while leaving the rest idle, capping TPS. We thus present two key insights: (1) load balance need not sacrifice all KV$ reuse, thanks to the global-tier KV$ store and (2) by utilizing the workload's intra-session locality, balancing a small fraction of requests--the first request in each agent session--suffices to balance the cluster without sacrificing most KV$ reuse on local instances. SMETRIC realizes these insights with balanced session-centric scheduling: it routes each session's first request purely for load balance and its follow-up requests in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using the session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from the user inputs alone, so the scheduler stays clean and stateless. SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation over state-of-the-art schedulers, also with a better per-token latency.