Interpretability
26 notes, newest first. Includes the vault tags interpretability, explainability, explainable ai.
Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs
Across 18 base-instruct model pairs spanning Qwen, Llama, and Gemma, the authors report that harmfulness is more readily decoded from attention updates while MLP updates contribute more strongly to harm-selective…
External Observers May See More Clearly: Cross-Model Span-Level Hallucination Detection in Large Language Models via Hidden State Probing
The preprint presents a hidden-state method for identifying the onset and continuation of hallucinated spans rather than making only token-level binary judgments. It proposes an external observer model that probes the…
Topological Attribution Distance (TAD): Revealing Segment-Level RAG Influence on LLM Output Geometry for Incident Log Analysis
Researchers propose a method called Topological Attribution Distance (TAD) to help identify which parts of cyber incident logs are critical in influencing the outputs of Large Language Models (LLMs) used in…
Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information
The paper presents a new method called 'Rule of Thumb' (RoT) for explaining how Artificial Intelligence systems make decisions. RoT identifies the most relevant features of an AI system to predict its behavior for a…
Multimodal Model Diffing for Feature Discovery and Control
Researchers developed a framework called MMDiff to help understand, control, and audit the behavior of Multimodal Large Language Models (MLLMs). The framework identifies features that are altered by multimodal…
Challenges in Evaluating Explanation Methods for Static and Evolving Data
The paper discusses the limitations in evaluating Explainable Artificial Intelligence (XAI) systems, specifically focusing on the DetoxAI image recognition system. It highlights the need for better evaluation methods…
Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review
A study investigates how the level of explanation in AI-assisted code reviews affects developer trust and agreement with AI recommendations. The results show that full explanations lead to higher perceived trust but…
Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review
A study investigates the impact of Explainable AI (XAI) on developers' trust in AI-assisted code reviews. The research compared three LLM-based code review systems with varying levels of XAI support. Results show that…
LEX-EC: A Lexical Evidence-Channel Audit Framework for Zero-Shot LLM Personality Classification in Black-Box Settings
Researchers introduce LEX-EC, a framework for analyzing the interpretability of large language models in assigning personality labels from text. The framework identifies and separates marginal distribution effects from…
M$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data
M$^3$-Gen is a new framework that generates synthetic gene expression profiles using both clinical metadata and histopathology images. The method leverages a Generative Adversarial Network to create biologically…
Explaining Process Control Optimisation Recommendations via GradientSHAP and Implicit Differentiation
Researchers propose a method to explain optimisation recommendations for industrial processes using Shapley Additive exPlanations (SHAP), Implicit Function Theorem, and Large Language Models (LLM). The new approach…
Explaining Process Control Optimisation Recommendations via GradientSHAP and Implicit Differentiation
Researchers have developed a method to explain optimization recommendations for industrial processes using GradientSHAP and Implicit Differentiation. This approach aims to bridge the trust gap between engineers who…
Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation
Researchers have proposed a self-evolving framework to improve the quality of depression symptom annotations in AI systems for mental health research. The framework combines large language models with human expertise…
Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation
Researchers propose a self-evolving annotation framework for Major Depressive Disorder (MDD) that combines large language model (LLM)-assisted labeling with expert verification. The goal is to create explainable…
Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation
Researchers have proposed a new framework to improve the quality of annotations in depression-related datasets for artificial intelligence systems in mental health research. The framework uses a large language model…
AIMO Interpretability Challenge
The AIMO Interpretability Challenge is a competition proposed by researchers aimed at distinguishing robust from spurious reasoning in advanced mathematical language models. The challenge focuses on evaluating the…
Explaining Reinforcement Learning Agents via Inductive Logic Programming
This research aims to improve Reinforcement Learning (RL) policies by making them more transparent and interpretable. It proposes objective and planning-oriented metrics for policy explainability in RL settings using…
Evaluating RE Practices for Explainability: Synthesizing Insights from Daimler Truck into an Explainable RE Framework Proposal
A study investigates the limitations of current Requirements Engineering (RE) practices in supporting explainability requirements, particularly within industrial contexts. The research focuses on Daimler Truck and…
Evaluating RE Practices for Explainability: Synthesizing Insights from Daimler Truck into an Explainable RE Framework Proposal
This research paper reports early findings from an ongoing study investigating how requirements engineering (RE) practices support explainability in AI-based systems, particularly in the industrial context of Daimler…
X-FEMR: A Token-level Explainable Approach for Electronic Health Records Foundation Models using Transformer-based Models
The authors propose a new approach to make Electronic Health Records Foundation Models (FEMRs) more transparent by identifying the most influential tokens in the model's predictions. This is done using a…
FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learning
The paper introduces FedXDS, a method that uses explainable AI (XAI) to help mitigate performance degradation in federated learning caused by statistical heterogeneity between client data distributions. The approach…
Scientific Explanations in Health Sciences: Causality, Trust, and Epistemic Adequacy
This research paper discusses the need for better understanding and transparency in Artificial Intelligence (AI) used in healthcare, focusing on the explanatory process of Machine Learning models. It highlights that…
TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models
Researchers propose a multi-agent system (MAS) called TreeAgent to automate bias labeling in forestry via expert decision trees and Vision-Language Models (VLMs). The MAS aims to reduce the need for human annotation…
TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models
Researchers propose a system called TreeAgent that uses artificial intelligence (AI) to classify tree heights in forestry remote sensing. The system combines decision trees created by experts with Vision-Language…
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Researchers have developed a new system called BINEVAL for evaluating large language models (LLMs). This system breaks down the evaluation criteria into binary questions, making it easier to understand the results and…
Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis
The researchers have developed a new method for diagnosing depression using deep learning models and facial videos. Their approach provides interpretable explanations for the model's decisions, which can help health…
