Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

Source

arXiv

arxiv.org/abs/2608.19147v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Tate Berenbaum, Muthaiah Venkatachalam
  • Venue: arXiv: cs.DC
  • Published: 2026-08-19
  • Collected: 2026-08-20
  • URL: http://arxiv.org/abs/2608.19147v1
  • PDF: https://arxiv.org/pdf/2608.19147v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The authors propose a method to utilize multiple Intel AI PCs with integrated GPUs and NPUs, even when they have insufficient memory for large models like a 70B-parameter LLM. The method involves breaking the model down into smaller parts (shards) and distributing these shards across different AI PCs, which then work together over a network to serve the model. Three techniques are used to make this method fast: recovering the speed of the unsplit model, leveraging speculative decoding on stateful OpenVINO models, and interleaving user requests across stages. The method is shown to improve performance, with a two-node setup serving two concurrent users at 1.79 times the single-user throughput of the unsplit model, and scaling up to serve a single user with a 70B model on four Lunar Lake AI PCs.

What the Document Says

  • Modern Intel AI PCs have integrated GPUs and NPUs with 16+ GB of unified memory but are often idle.
  • A large model such as a 70B-parameter LLM cannot fit into the memory of a single AI PC.
  • The authors propose using pipeline parallelism to serve models beyond the capability of any single AI PC by splitting the model into per-stage shards, each pre-compiled into an OpenVINO graph.
  • Every machine runs one shard and passes activations to the next.
  • Three techniques are used to make this fast: recovering the speed of the unsplit model, leveraging speculative decoding on stateful OpenVINO models, and interleaving user requests across stages.
  • The method improves performance, with a two-node setup serving two concurrent users at 1.79 times the single-user throughput of the unsplit model.
  • The method scales up to serve a single user with a 70B model on four Lunar Lake AI PCs.

Key Concepts Mentioned

  • Intel AI PC
  • 70B-parameter LLM
  • pipeline parallelism
  • OpenVINO graph
  • shards
  • IndirectKVCache fusion
  • beam_idx Gather
  • speculative decoding
  • stateful OpenVINO models
  • micro-batching

Claims Made by the Document

  • A pipeline of multiple AI PCs can serve models beyond the capability of any single one.
  • The proposed method improves performance.
  • The proposed method scales up to a 70B model.

Evidence Used

  • The authors demonstrate the performance improvements through benchmark logs and reproduction scripts.

Methods / Approach

  • Pipeline parallelism
  • Pre-compiling shards into OpenVINO graphs
  • Injecting beam_idx Gather
  • IndirectKVCache fusion
  • Speculative decoding
  • Micro-batching

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The method requires the use of Intel AI PCs and may not be applicable to other hardware configurations.

Open Questions

  • What is the performance impact of the proposed method on different models or hardware configurations beyond those mentioned in the paper?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The authors do not specify any potential limitations in their paper, but as an assistant I can suggest that the method's performance may vary based on the specific model and hardware configuration.

Raw Abstract / Excerpt

Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of AIPCs, working together over an ordinary network, can serve models beyond the capability of any single one. We use pipeline parallelism: a model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so that every machine runs one shard and passes activations to the next. Three techniques make this fast enough to be useful. First, we recover the speed of the unsplit model: a naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization, and injecting a beam_idx Gather into each shard triggers that optimization (the IndirectKVCache fusion) and brings the shards to parity. Second, we leverage speculative decoding on stateful OpenVINO models. Third, the pipeline serves several users at once by interleaving their requests across the stages, each request carrying its own cache (micro-batching). Together, a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput of the unsplit model on the same hardware, and the gap widens under simulated wide-area latency. The same design scales to a 70B model that no single fleet member can hold: a four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed, with output token-for-token identical to the same four-node pipeline decoding without speculation. Code, raw benchmark logs, and reproduction scripts ship as a self-contained package at https://github.com/labscommunity/pipeline-sharded-inference-paper (in the top-level reproduction/ directory).