Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Josef Liyanjun Chen
• Venue: arXiv: cs.DC
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12123v1
• PDF: https://arxiv.org/pdf/2608.12123v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper discusses a method for determining when it's beneficial to execute small deterministic transitions between model and tool calls on GPUs, and presents a dynamic program that computes the optimal amount of work for GPU execution under certain conditions. The study also investigates the impact of keeping GPU-computed route decisions on the device itself instead of sending them back to the host for redispatching, finding that this approach is faster in all tested configurations.

What the Document Says

• The paper discusses a method for determining when small deterministic transitions between model and tool calls can be executed on GPUs (LLM-agent control).
• A dynamic program is introduced to compute the optimal amount of work for GPU execution under certain conditions (zero service time, unlimited capacity, equal relative launch deadlines).
• In a Poisson replay of one pinned 851-session public trace panel, the primary condition at 100,000 target active sessions, K=256, and a 50 ms launch deadline gives F=30.19%, P*=43.00%, and U=45.85%.
• Exact packing recovers 81.83% of the opportunity lost at fixed window boundaries.
• The outcome-derived route key is a conditioning proxy, not proof of executable identity.
• A separate mechanism study keeps a GPU-computed binary decision on device instead of returning four bytes to the host and redispatching, finding that this approach is faster in all tested configurations.
• Across both admissible mechanisms, all batched invocations match a separately implemented host oracle.
• A fixed nested device graph that removes no host decision is slower in all tested configurations across five placements.

Key Concepts Mentioned

• LLM-agent control
• GPU execution
• dynamic program
• Poisson replay
• fixed window boundaries
• outcome-derived route key
• admissible mechanisms
• batched invocations
• host oracle
• nested device graph

Claims Made by the Document

• A dynamic program can compute the optimal amount of work for GPU execution under certain conditions.
• Keeping a GPU-computed binary decision on device instead of returning it to the host and redispatching is faster in all tested configurations.

Evidence Used

• The dynamic program computation is described in the paper.
• Test results showing the speed difference between device-resident and hosted decisions are provided in the paper.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLM-agent services repeatedly execute small deterministic transitions between model and tool calls: route an outcome, update state, and emit the next effect. We ask when this control path exposes enough concurrent work for GPU execution, and what changes when a GPU-computed route decision remains on device. We formalize the ready-cohort boundary using fixed-partition share F, exact offline share P*, local upper bound U, and online achieved share A. Under zero service time, unlimited capacity, and equal relative launch deadlines, a specialized dynamic program computes P* exactly. In a stationary Poisson replay of one pinned 851-session public trace panel, the primary condition at 100,000 target active sessions, K=256, and a 50 ms launch deadline gives F=30.19%, P*=43.00%, and U=45.85%. Exact packing recovers 81.83% of the opportunity lost at fixed window boundaries. The outcome-derived route key is a conditioning proxy, not proof of executable identity. A separate mechanism study keeps a GPU-computed binary decision on device instead of returning four bytes to the host and redispatching. Across four named GPU placements, the device-resident path is faster in all 36 configurations; within-placement row-median ratios range from 1.19x to 2.39x. Across both admissible mechanisms, all 14,557,440 tested batched invocations match a separately implemented host oracle. A fixed nested device graph that removes no host decision is slower in all 60 configurations across five placements. Together, the studies establish two measurable gates for GPU agent control: deadline-feasible cohort supply and observation placement. A joined finite online runtime is required to measure A, CPU displacement, and service-level benefit.