Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
11 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on uncertainty
11 notes
Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict
The preprint evaluates epistemic humility through three trajectory-level behaviors: Identify, Solve, and Escalate. It tests controlled and naturally occurring conflicts between parametric knowledge and retrieved…
Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
The preprint audits whether harmful refusal is a single measurable attribute in HELM Safety. It reports saturation in three candidate datasets and uses multidimensional item-response theory plus differential item…
HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment
The preprint uses hidden states before generation and at the end of an answer to predict correctness for local-model deployment. HARISSA fine-tunes both states and uses a single policy to choose among increasingly…
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
The preprint evaluates a decision-only judge as an economical first pass and escalates low-confidence cases to stronger evaluation. It compares the judge with 16 generative and reward-model judges using blinded human…
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Pinocchio is an external calibrator for estimating response correctness from black-box language-model APIs without logits, weights, internal states, or fine-tuning access. The preprint reports a single-forward-pass…
Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation
The protocol proposes three actions: refuse when a clean reference stream or matched runtime comparison lacks support; decompose execution, false admission, and structural hypotheses instead of collapsing them into…
PosteriorBench: From Point Estimates to Posterior Matching in Evaluating Generative Inverse Solvers
PosteriorBench evaluates whether generative inverse solvers recover full solution distributions rather than only one plausible reconstruction. It covers four physics-based inverse problems and uses reference posteriors…
In-Context Robot Learning with VLM Agents
The paper introduces GPT-Policy, a framework combining a context compiler, a vision-language model that proposes robot-tool actions, and a constrained controller that verifies and executes actions. The authors report…
ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation
ENCP rescales conformal-prediction scores for variable-length, dependent navigation episodes and calibrates one maximum score per episode. The abstract reports coverage evaluations across four navigation policies and…
When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control
The paper proposes Chain-of-Self-Questioning, a prompt-only framework that makes answer commitment conditional on an explicit assessment of whether the model has enough information. The abstract reports evaluations on…
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
The researchers evaluate the performance of uncertainty estimation methods across 22 languages in high-, mid-, and low-resource settings using two human-curated question and answer datasets. They compare open and…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
