AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, Shelby Heinecke
• Venue: arXiv: cs.LG
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12307v1
• PDF: https://arxiv.org/pdf/2608.12307v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The study presents a new approach for transferring capabilities of large AI models to smaller ones at test time instead of during training. This method involves using a stronger model to create inference-time harnesses that help a weaker model perform tasks more reliably without any parameter updates. The paper focuses on four Theory-of-Mind benchmarks and demonstrates that this test-time capability transfer significantly improves the average performance of the target model, nearly doubling it from 0.49 to 0.91. The key factors contributing to the gains are offloading unstable model reasoning into deterministic code, specific routing for each benchmark, and enforcing strict answer formats.

What the Document Says

• The study focuses on test-time capability transfer from larger models (builders) to smaller ones (target models) using inference-time harnesses.
• Four Theory-of-Mind benchmarks are used for the study.
• Builders use a validation set of 5% of the data to refine harnesses over multiple rounds before evaluating on the full test set.
• Empirical results show that this approach significantly improves target-model performance, nearly doubling it from 0.49 to 0.91.
• The gains are primarily due to offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement.
• Builder-model reasoning effort affects harness quality positively.
• Platform effects are modest compared to the builder model's capability.
• Weaker target models receive the largest gains from this approach.

Key Concepts Mentioned

• AI distillation
• Test-time capability transfer
• Inference-time harnesses
• Theory-of-Mind benchmarks

Claims Made by the Document

• Test-time capability transfer is effective and can nearly double the average performance of target models.
• Offloading unstable model reasoning, specific routing, and strict answer format enforcement contribute significantly to gains.

Evidence Used

• Empirical results showing a significant improvement in target-model performance from 0.49 to 0.91.
• Analysis of factors contributing to the gains.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.