HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment

Source facts

The preprint uses hidden states before generation and at the end of an answer to predict correctness for local-model deployment. HARISSA fine-tunes both states and uses a single policy to choose among increasingly expensive answer paths, skip paths predicted to fail, and defer answers predicted to be wrong. The abstract reports accuracy within one point of chain-of-thought at 2.7 times lower latency on a single-model device, plus comparisons with FrugalGPT and Self-REF cascades.

Analysis

The architecture combines efficiency control with a safety deferral policy, preserving local privacy and cost advantages for easier queries while escalating or abstaining on harder ones.

Uncertainty/Doubt

• This is an arXiv preprint. The hidden-state correctness signal, task mix, and reported latency/deferral comparisons were not independently reproduced here.