Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations

Source

arXiv

arxiv.org/abs/2607.19231v1

Read the source

Source

Plain-Language Brief

The study aimed to replicate an earlier finding about lower label agreement in natural language inference (NLI) when using non-upward monotonicity operators, specifically in the ChaosNLI dataset. The current research found no evidence to support this boundary in the unselected populations that ChaosNLI was drawn from: SNLI and MultiNLI development sets. The effects observed were far below the smallest effect size of interest (0.10).

What the Document Says

  • The study replicated an earlier finding about lower label agreement in NLI using non-upward monotonicity operators in ChaosNLI.
  • The replication was conducted on the unselected populations that ChaosNLI was drawn from: SNLI and MultiNLI development sets.
  • No evidence was found to support the earlier negative boundary.
  • All seven contrasts returned a positive Cliff's delta.
  • Only one significant confirmatory contrast had an opposite sign to the registration.
  • Effects were far below the smallest effect size of interest (0.10).
  • Robustness checks supported the measurement.

Key Concepts Mentioned

  • natural language inference (NLI)
  • ChaosNLI
  • SNLI
  • MultiNLI development sets
  • non-upward monotonicity operators
  • Cliff's delta
  • label agreement

Claims Made by the Document

  • The earlier finding about lower label agreement in NLI using non-upward monotonicity operators holds for selected populations.
  • The earlier negative boundary is not a population-level property.

Evidence Used

  • The preregistered replication failed to find the expected lower label agreement for non-upward items.

Methods / Approach

  • Preregistered replication of an earlier study
  • Use of the same operator tagger
  • Four-level ordinal agreement outcome

Limitations Stated by the Authors

  • The findings are based on a single preregistered replication.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The robustness checks were performed but the generalizability of the results to other datasets or methods is not guaranteed.

Open Questions

  • What are the implications of these findings for the interpretation and use of selected re-annotation resources in NLI studies?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study does not mention if the earlier work on HLV (human label variation) in NLI was conducted on the same datasets.

Raw Abstract / Excerpt

Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disagreement level. An earlier study (arXiv:2607.15870) found that hypotheses containing non-upward monotonicity operators showed lower label agreement in ChaosNLI (Cliff's delta = -0.284), which is restricted to items whose majority label carries exactly three of five votes. We preregistered a replication of this boundary in the unselected populations that ChaosNLI was drawn from: the SNLI and MultiNLI development sets, using the same operator tagger and a four-level ordinal agreement outcome. The registered prediction fails. All seven contrasts return a positive Cliff's delta (non-upward items agree slightly more, not less), the only significant confirmatory contrast has the opposite sign to the registration, and every effect is far below our smallest effect size of interest (0.10). Robustness checks support the measurement: simulated tagger misclassification shrinks the effects rather than manufacturing them, and a manual re-tagging audit reaches four-class agreement of 0.875 on a fresh 200-item sample. We conclude that the earlier negative boundary is plausibly a structure conditional on low-agreement selection rather than a population-level property, and that HLV structure claims built on selected re-annotation resources should state their selection conditional explicitly.