Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA

Source

arXiv

arxiv.org/abs/2606.32002v1

Read the source

Source

Plain-Language Brief

The authors investigate the generation of question-answer (QA) pairs used for training language models and find that this process is not neutral as it can introduce biases due to the selection of evidence and the decision on how to answer, especially when dealing with artifacts like poor markup or instruction-like passages embedded in the text. They suggest methods to reduce these biases by fixing targets for each question and filtering out instruction-like spans before answering.

What the Document Says

  • Language models are increasingly trained using synthetic QA supervision, where a model generates questions about a document and answers them based on the same text.
  • The authors argue that the generation step is not neutral preprocessing because it acts as an implicit policy for selecting which evidence becomes training signal and deciding how to answer.
  • They find that during question selection, generators do not uniformly scan documents but instead saturate early and concentrate on salient spans.
  • Diverse prompts converge on the same regions, and the choice of what looks question-worthy is driven by local presentation.
  • Salient artifacts like poorly cleaned markup can hijack question generation across model families and scales.
  • During answering, the model that produces the supervision tends to obey instruction-like passages embedded in the text.
  • Compliance depends on the intent and surface form of the passage rather than its strictness, and is worse under task conflict.
  • Larger models comply more often under task conflict.
  • To reduce biases, they suggest tying each question to a fixed target and filtering out instruction-like spans before answering.
  • This reduces mean injection compliance from 88% to 13% in their evaluation while retaining nearly all clean text.

Key Concepts Mentioned

  • Language models
  • Synthetic QA supervision
  • Question generation
  • Answer selection
  • Selection bias
  • Artifacts
  • Instruction-like passages
  • Task conflict

Claims Made by the Document

  • The generation step in synthetic QA supervision is not neutral.
  • Salient artifacts can hijack question generation.
  • Larger models comply more often with instruction-like passages under task conflict.

Evidence Used

  • ... tend to obey instruction-like passages embedded in the text...

Methods / Approach

  • Tying each question to a fixed target
  • Filtering instruction-like spans before answering

Limitations Stated by the Authors

The authors acknowledge that their study focuses on QA generation for language models and does not explore other aspects of model training or evaluation.

Limitations Noted by the Agent

Agent interpretation — not from the source.

Additional research could be conducted to evaluate the impact of these biases on various tasks, datasets, and models beyond what was explored in this study.

Open Questions

  • What is the extent of these biases on various tasks, datasets, and models?
  • How do these biases compare to other sources of bias in language model training and evaluation?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The authors focus on the specific issue of biased QA generation for language models. While their suggested methods can help reduce biases, it's important to consider that there may be other sources of bias in model training and evaluation.

Raw Abstract / Excerpt

Language models are increasingly taught from synthetic question--answer (QA) supervision: a model generates questions about a document, answers them from the same text, and the resulting pairs are used to fine-tune, distill, or compress knowledge into another model. We show that this generation step is not neutral preprocessing. It is an implicit policy that both selects which evidence becomes training signal and decides how that evidence is answered, and it is fragile at both stages. When choosing what to ask, generators do not scan a document uniformly. Coverage saturates early and concentrates on salient spans, diverse prompts converge on the same regions, and what looks question-worthy is driven by local presentation. As a result, salient artifacts such as poorly cleaned markup can hijack question generation across model families and scales. When answering, the model that produces the supervision tends to obey instruction-like passages embedded in the text. This compliance depends on the intent and surface form of the passage rather than its strictness, and is worst under task conflict, where larger models comply more often. These failure modes arise from choices made during QA generation, so they can be reduced without changing the training loop. Tying each question to a fixed target reduces biased selection, and filtering instruction-like spans before answering lowers mean injection compliance from $88%$ to $13%$ in our evaluation while retaining nearly all clean text.