Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note4 min read

Long-Context Fine-Tuning with Limited VRAM

Source

arXiv

arxiv.org/abs/2607.15105v1

Read the source

Source

Plain-Language Brief

Researchers introduce Hierarchical Global Attention (HGA) to reduce memory usage during long training sequences in large language models. They use segment-wise backpropagation, tiered KV storage, and detach older data from VRAM to fit more tokens on a Quadro RTX 5000 card. Results show that HGA can handle up to 16,384 tokens with only 15.28 GB peak VRAM, whereas dense training fails at 4,096 tokens on the same card. The authors also mention that at a shared 2K training length, HGA-trained and dense-trained adapters perform comparably in terms of natural language understanding (nat), but HGA is already faster in terms of tokens processed per second. They plan to develop an optimized production-grade serving implementation for HGA.

What the Document Says

  • Hierarchical Global Attention (HGA) and segment-wise backpropagation are used with tiered KV storage in large language models to reduce memory usage during long training sequences.
  • Only the active segment remains differentiable in VRAM, while older data is detached into RAM or NVMe, allowing more tokens to be processed on a Quadro RTX 5000 card.
  • The researchers tested their method on Qwen3-8B with 4-bit QLoRA and PG19. Dense training fails at 4,096 tokens but fits 2,048 tokens, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM.
  • Under evaluation, the same adapter runs through 131,072 tokens on a Quadro RTX 5000 card.
  • At a shared 2K training length, HGA-trained and dense-trained adapters perform comparably in terms of natural language understanding (nat), but HGA is faster in terms of tokens processed per second.
  • HGA keeps the attended historical set per token approximately constant while dense work per token grows, leading to a wider lead as context grows.
  • The authors plan to develop an optimized production-grade serving implementation for HGA.

Key Concepts Mentioned

  • Large language models
  • Hierarchical Global Attention (HGA)
  • Segment-wise backpropagation
  • Tiered KV storage
  • VRAM
  • Quadro RTX 5000 card
  • 4-bit QLoRA
  • PG19
  • Natural language understanding (nat)

Claims Made by the Document

  • HGA and segment-wise backpropagation with tiered KV storage can reduce memory usage during long training sequences in large language models.
  • The use of HGA allows for the processing of more tokens on a Quadro RTX 5000 card.
  • At a shared 2K training length, HGA-trained adapters are faster than dense-trained adapters.

Evidence Used

  • Combining Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage
  • Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, allowing more tokens to be processed.
  • The authors tested their method on Qwen3-8B with 4-bit QLoRA and PG19. Dense training fails at 4,096 tokens but fits 2,048 tokens, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM.
  • Under evaluation, the same adapter runs through 131,072 tokens on a Quadro RTX 5000 card.
  • At a shared 2K training length, HGA-trained and dense-trained adapters perform comparably in terms of natural language understanding (nat), but HGA is faster in terms of tokens processed per second.

Methods / Approach

  • Combining Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage
  • Detaching older data from VRAM to fit more tokens on a Quadro RTX 5000 card

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive. We combine Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage. Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, and HGA loads a bounded set of exact historical tokens for each query block. On Qwen3-8B with 4-bit QLoRA and PG19, dense training on a 16 GB Quadro RTX 5000 fits 2,048 tokens but fails at 4,096, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM. Under evaluation the same adapter runs through 131,072 tokens on this card; VRAM is not constant but grows gently with the resident chunk summaries, so RAM and NVMe capacity set the practical limit beyond these lengths. At the shared 2K training length, HGA-trained and dense-trained adapters obtain 2.7405 and 2.7383 nat under the same dense-attention readout, while the stock model obtains 2.9541. At this boundary HGA training is already marginally faster (217.75 vs. 207.02 tokens/s), and the HGA-to-dense throughput ratio improves from 1K to 2K; because HGA keeps the attended historical set per token approximately constant while dense work per token grows, we expect this lead to widen as context grows. Dense attention is used for the main quality and retrieval comparisons so that they measure the learned weights and remain compatible with standard generation frameworks. HGA can also be used for retrieval and generation; an optimized production-grade serving implementation is under development.