Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models

Source facts

Pinocchio is an external calibrator for estimating response correctness from black-box language-model APIs without logits, weights, internal states, or fine-tuning access. The preprint reports a single-forward-pass estimate, transfer experiments across unseen models, and a lightweight text-only checkpoint.

Analysis

The proposal targets a practical deployment gap: many commercial APIs do not expose the signals required by traditional uncertainty estimation. External calibration could provide a model-agnostic reliability layer for high-stakes applications, subject to careful shift and calibration testing.

Uncertainty/Doubt

• This is an arXiv preprint; reported AUROC and cross-organization transfer were not independently reproduced here.