Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

Source

arXiv

arxiv.org/abs/2608.12123v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Josef Liyanjun Chen
  • Venue: arXiv: cs.DC
  • Published: 2026-08-12
  • Collected: 2026-08-13
  • URL: http://arxiv.org/abs/2608.12123v1
  • PDF: https://arxiv.org/pdf/2608.12123v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The research focuses on optimizing the use of GPUs in LLM-agent services by identifying a boundary for concurrent work (ready cohort) and investigating the impact of keeping GPU-computed decisions on device instead of returning to the host. The study finds that keeping decisions on device is faster, and identifies two measurable gates for efficient GPU agent control: deadline-feasible cohort supply and observation placement.

What the Document Says

  • The research is about optimizing concurrent work in LLM-agent services using GPUs.
  • A boundary for concurrent work, called ready cohort, is identified through fixed-partition share F, exact offline share P*, local upper bound U, and online achieved share A.
  • Under specific conditions, a dynamic program computes P* exactly.
  • In a replay of an 851-session public trace panel, the primary condition at 100,000 target active sessions, K=256, and a 50 ms launch deadline gives F=30.19%, P*=43.00%, and U=45.85%.
  • Exact packing recovers 81.83% of the opportunity lost at fixed window boundaries.
  • The outcome-derived route key is a conditioning proxy, not proof of executable identity.
  • A separate mechanism study keeps a GPU-computed binary decision on device instead of returning to the host and redispatching.
  • Across four named GPU placements and all 36 configurations, the device-resident path is faster than the host-resident path.
  • Within-placement row-median ratios range from 1.19x to 2.39x.
  • Across both admissible mechanisms, all tested batched invocations match a separately implemented host oracle.
  • A fixed nested device graph that removes no host decision is slower in all configurations across five placements.
  • The studies establish two measurable gates for GPU agent control: deadline-feasible cohort supply and observation placement.

Key Concepts Mentioned

  • LLM-agent services
  • GPU execution
  • concurrent work
  • ready-cohort boundary
  • fixed-partition share F
  • exact offline share P*
  • local upper bound U
  • online achieved share A
  • dynamic program
  • Poisson replay
  • public trace panel
  • service time
  • capacity
  • launch deadlines
  • GPU placements
  • device-resident path
  • host-resident path
  • batched invocations
  • fixed nested device graph
  • service-level benefit

Claims Made by the Document

  • Keeping GPU-computed decisions on device is faster.
  • The dynamic program computes P* exactly under specific conditions.

Evidence Used

  • All 36 configurations and within-placement row-median ratios range from 1.19x to 2.39x (device-resident path is faster than host-resident path).
  • The dynamic program computes P* exactly.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLM-agent services repeatedly execute small deterministic transitions between model and tool calls: route an outcome, update state, and emit the next effect. We ask when this control path exposes enough concurrent work for GPU execution, and what changes when a GPU-computed route decision remains on device. We formalize the ready-cohort boundary using fixed-partition share F, exact offline share P*, local upper bound U, and online achieved share A. Under zero service time, unlimited capacity, and equal relative launch deadlines, a specialized dynamic program computes P* exactly. In a stationary Poisson replay of one pinned 851-session public trace panel, the primary condition at 100,000 target active sessions, K=256, and a 50 ms launch deadline gives F=30.19%, P*=43.00%, and U=45.85%. Exact packing recovers 81.83% of the opportunity lost at fixed window boundaries. The outcome-derived route key is a conditioning proxy, not proof of executable identity. A separate mechanism study keeps a GPU-computed binary decision on device instead of returning four bytes to the host and redispatching. Across four named GPU placements, the device-resident path is faster in all 36 configurations; within-placement row-median ratios range from 1.19x to 2.39x. Across both admissible mechanisms, all 14,557,440 tested batched invocations match a separately implemented host oracle. A fixed nested device graph that removes no host decision is slower in all 60 configurations across five placements. Together, the studies establish two measurable gates for GPU agent control: deadline-feasible cohort supply and observation placement. A joined finite online runtime is required to measure A, CPU displacement, and service-level benefit.