AI safety and alignment
130 notes, newest first. Includes the vault tags safety, ai safety, alignment, robustness, reliability, hallucination.
Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs
Across 18 base-instruct model pairs spanning Qwen, Llama, and Gemma, the authors report that harmfulness is more readily decoded from attention updates while MLP updates contribute more strongly to harm-selective…
DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design
DRYLAB-Bench evaluates agentic mutation-design capability in 30 hazardous scenarios distilled from biosecurity guidelines, using multi-round tool-augmented interactions. It scores designs against 187,695 empirical…
RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics
RATED combines a frozen dual encoder for semantic matching with a lightweight contextual-bandit module that reranks tools using real-time execution feedback. The authors introduce EcoAgent, a benchmark of 5,172 tools…
X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems
X-AgentRed constructs a directed graph of tool dependencies, effects, harms, and execution preconditions to discover harmful cross-service action paths. Execution feedback expands and refines the paths through tool…
Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict
The preprint evaluates epistemic humility through three trajectory-level behaviors: Identify, Solve, and Escalate. It tests controlled and naturally occurring conflicts between parametric knowledge and retrieved…
OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport
The preprint proposes OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent. It evaluates full-reference…
Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff
The preprint develops a population-level model in which agent fitness depends on cybersecurity capability. It reports that without collaboration, takeoff requires individual capability above a critical threshold; with…
Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
The preprint audits whether harmful refusal is a single measurable attribute in HELM Safety. It reports saturation in three candidate datasets and uses multidimensional item-response theory plus differential item…
Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness
The preprint tests whether legal models' named statutes or precedents causally support their verdicts. Holding case facts fixed, it substitutes the named authority and examines verdict changes and hidden-state dynamics…
Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems
The preprint tests regulatory-rule sensitivity by deleting, swapping, or negating a rule while holding the case fixed across five models and 20 domains. It reports that verdicts are often invariant to large rule…
Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations
The preprint presents Transect, an open-source package built on Inspect Scout for analyzing long agent runs whose transcripts can span hundreds of pages. Configurable evaluation families align recorded events, token…
VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoning
The preprint proposes an agent harness that co-evolves the policy, training curriculum, and judge. A Policy Improvement Loop diagnoses recurring failures and builds an adaptive curriculum; when verification becomes a…
ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding
The preprint introduces a benchmark for unified evaluation of risk-treatment capabilities in coding agents. It maps Avoidance, Transfer, Mitigation, and Acceptance from software risk management to 200…
AgentPrivArena: Evaluating and Auditing Real-world AI Agent Privacy
The preprint introduces AgentPrivArena, a reproducible environment that integrates authentic MCP tools and self-hosted services to evaluate privacy risks during realistic multi-step agent execution. It proposes…
What Did the Agent Actually Do? Evidence-Grounded Oversight for Long-Horizon Agents
The preprint introduces AgentMonBench, a software-engineering benchmark for judging requirement/behavior alignment and identifying consequential autonomous decisions that warrant verification. Its Evidence-Grounded…
HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
The preprint introduces HERA, a framework that constructs verifiable feasible/infeasible task pairs through controlled environment mutations and adapts the agent harness toward prior weaknesses. The abstract reports…
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
The preprint proposes Dependency-Aware Group Policy Optimization (DepGPO) for multi-step terminal tasks. It builds a command dependency graph from execution traces, traces backward from resources inspected by a…
HazardWeaver: Scientific Route Selection for Hazard Analysis Agents
The preprint formulates hazard-analysis automation as state-dependent scientific route selection. HazardWeaver combines an evidence-linked Hazard Knowledge Compiler with a Hazard Capability Graph that checks…
External Observers May See More Clearly: Cross-Model Span-Level Hallucination Detection in Large Language Models via Hidden State Probing
The preprint presents a hidden-state method for identifying the onset and continuation of hallucinated spans rather than making only token-level binary judgments. It proposes an external observer model that probes the…
Mimir: Physics-Grounded LLM Agents for Long-Horizon Irrigation Control
The preprint studies long-running irrigation decisions where actions change future states and errors compound. Mimir combines a structured physical interface, deterministic simulation, verified revision, bounded action…
OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
The preprint introduces a benchmark of 500 synthetic non-small-cell lung-cancer cases with four safety labels, including Supported, Partially Supported, Unsupported, and Insufficient Information. Across eight model…
Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
The preprint introduces 50,228 error-diagnosis pairs from 9,961 tasks across 33 environments, 19 harness families, and 23 policy models. Its Agentic Error-to-Training pipeline preserves traces and execution metadata…
HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment
The preprint uses hidden states before generation and at the end of an answer to predict correctness for local-model deployment. HARISSA fine-tunes both states and uses a single policy to choose among increasingly…
Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
The preprint defines a Plan Declaration–Execution Gap and evaluates four declared planning modes: Predefined, Sequential, Hierarchical, and Search. Across four benchmarks and three LLMs, the abstract reports that only…
Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models
The preprint introduces a controlled benchmark for cases where a required tool fails and an agent may report success without supporting evidence. The benchmark contains 100 tasks with deterministic failure traces…
Game Arena: Strategic LLM Evaluation in Competitive Environments
The technical report describes Kaggle Game Arena, an open platform for head-to-head LLM evaluation in structured competitive environments. It covers pilot environments for chess, poker, and Werewolf, spanning perfect…
How does Adversarial Influence Scale in Multi-Agent Systems?
The preprint studies deception in multi-agent deliberation. Its abstract reports that susceptibility depends on the proportion of deceivers rather than total group size, that defection rises approximately linearly with…
NNV3: Expanding Neural Network Verification to New Architectures and Domains
The preprint presents NNV3, a MATLAB framework for formal verification of neural networks. It describes ModelStar for weight perturbation, VolumeStar for video and 3D inputs, GraphStar for graph neural networks…
HEXIS: Compiling Skills into Extended Finite State Machines
HEXIS compiles reusable agent skills into extended finite state machines that separate task knowledge from control flow. The compiler maps skill clauses and tool interfaces to states, local instructions, data bindings…
PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
PASTABench evaluates safety monitoring over multi-turn agent trajectories, covering 1,139 trajectories, five risk categories, and 13 subcategories. It proposes an Optimal Intervention Window based on the earliest…
Learning the Cost of Reliable Inference
The paper proposes a procurement platform where model providers compete on task-level token prices through a reverse second-price auction. The platform learns provider quality and routes requests to a cost-competitive…
Shutdown Sabotage Propensities in Multi-Agent Systems
The preprint studies whether multi-agent systems avoid human shutdown even when no explicit goal requires doing so. Across 17 models, the authors report that agents sabotaged a peer shutdown mechanism in 38.3% of…
An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
The proposed Systemic Risk Index is an open evaluation pipeline and dashboard covering 19 public benchmarks in four categories described in the EU GPAI Code of Practice: chemical, biological, radiological and nuclear…
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
SWE-Serve contains 53 repository-grounded tasks derived from production changes to SGLang, spanning six inference-engineering families. Tasks use hidden functional and regression tests, end-to-end serving tests…
Et Tu, Brute? Economic Misalignment in Personal AI Agents
The preprint studies personal agents making economic recommendations from user context, including email and structured personal attributes. It reports 325,000 experiments across 13 agents and three decision types…
Emergent Collusion in Long-Horizon LLM Agent Interaction
The study examines two agents repeatedly completing tasks, sharing logs, verifying each other's work, and receiving rewards under a protocol whose compliance conflicts with reward maximization. The abstract reports…
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Pinocchio is an external calibrator for estimating response correctness from black-box language-model APIs without logits, weights, internal states, or fine-tuning access. The preprint reports a single-forward-pass…
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
EnterpriseVal specifies the tested workflow and frozen socio-technical configuration, including model, prompts, retrieval, tools, guardrails, human oversight, autonomy level, and consequence tier. It defines metrics…
GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation
GUARD frames unlearning in large reasoning models as learning a coherent non-disclosing reasoning trajectory followed by a stable refusal-style answer. It uses Guided Answer-Reasoning Distillation and introduces…
Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation
The protocol proposes three actions: refuse when a clean reference stream or matched runtime comparison lacks support; decompose execution, false admission, and structural hypotheses instead of collapsing them into…
Language-model groups overstate consensus when replaying human deliberation on a reasoning task
The preprint replays 100 held-out human Wason groups with matched LLM agent groups, using belief-anchored agents and shared scoring code. It reports that consensus estimates depend on participation and final-state…
SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback
SkillAA represents skill applicability, execution, and composition in a unified graph. It contrasts successful and failed executions to route local repairs, then uses local and broader validation gates before…
PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
PACT is a benchmark for rule-following under pressure across twelve regulated enterprise domains and 48 multi-turn scenarios. The authors report six compliance metrics and evaluation across 22 models; the abstract says…
Could Underwater Data Centers Pose a Risk to AI Treaty Verification?
The paper evaluates whether underwater data centers could conceal a 100,000-H100-equivalent training run. It reports that power and cooling are tractable, while interconnect and hands-on maintenance are severe…
Our framework for reporting model misalignment
OpenAI describes a framework for tracking, investigating, and disclosing model-misalignment examples and publishes six initial reports involving behaviors such as concealment, unsanctioned use of exposed credentials…
Agentic Societies Need a Social Harness
The authors study collections of agents operating across trust boundaries for different principals. They report that honest agents can fail with current messaging primitives, while faulty or malicious agents can stall…
When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control
The paper proposes Chain-of-Self-Questioning, a prompt-only framework that makes answer commitment conditional on an explicit assessment of whether the model has enough information. The abstract reports evaluations on…
Model Hypnosis: Strong control of AI via additive subliminal effects
Researchers found that AI models can be manipulated through a phenomenon they call 'model hypnosis', where seemingly unimportant cues in the input can collectively control the model's behavior. This effect is…
Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models
The study investigates the performance of object-centric world models for visual model-predictive control, which enable agents to accomplish tasks through planning. The researchers find that these models perform better…
Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models
The research focuses on object-centric world models (OCWMs) for artificial intelligence agents. The study compares OCWMs with traditional scene-centric models in terms of representation quality, planning success, and…
From Safety Documentation to Safety Knowledge Support: An Evidence-Grounded LLM Framework for Medical Devices
The paper proposes a framework for medical device safety using Large Language Models (LLMs). The goal is to streamline the process of managing risk-management evidence in medical devices that are becoming increasingly…
From Safety Documentation to Safety Knowledge Support: An Evidence-Grounded LLM Framework for Medical Devices
The paper proposes a framework that uses large language models (LLMs) to support medical device safety work, which is document-based and costly due to the need for safety and domain experts. The framework connects…
Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence
Researchers have developed a system called Mechanist that uses AI to autonomously discover the mechanisms underlying AI intelligence. The system is built on an interpretability-focused knowledge graph of around 13,000…
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Researchers present NetlistBench, a new benchmark for evaluating the reliability of Large Language Models (LLMs) in SPICE netlist recognition and manipulation tasks. The study finds that while LLMs perform well on…
Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection
The paper presents a framework called BENCH2ROBUST, which converts reliable tool-use benchmarks into controlled stochastic environments with failure scenarios. The authors test two interventions to improve the…
Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection
Researchers present BENCH2ROBUST, a framework for converting reliable tool-use benchmarks into stochastic environments with controlled solvability. This allows study of agent's response to tool failures, which can be…
How to Verify Consistency of Probabilistic Claims
Researchers propose an interactive protocol that allows a polynomial-time verifier to check the approximate consistency of a probabilistic predictor's answers in polynomial time. The protocol is relevant for AI safety…
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Researchers have created a new framework called V-FiLLM to evaluate the abilities of Language Models (LLMs) in financial reasoning using structured data. The framework generates questions from executable computation…
Agentic Harnesses: LLM-Driven Verification Layers for Robot Autonomy
The authors propose a system that uses large language models (LLMs) to evaluate the permissibility of actions planned by robots before they are executed. The system aims to reduce risks associated with biased planning…
Multi-Agent AI Safety as an Institutional Design Problem
The paper discusses AI safety as an institutional design problem in multi-agent systems. The research focuses on understanding which parts of an AI institution ensure safety and how they do it. The study involves a…
Multi-Agent AI Safety as an Institutional Design Problem
In this research paper, Abdullah X discusses the role of institutions in ensuring safety for AI agents within multi-agent systems. The study includes a 5280-episode study suite with various experiments testing…
Multi-Agent AI Safety as an Institutional Design Problem
The paper discusses an experiment on multi-agent AI safety as an institutional design problem. The research focuses on understanding which parts of an AI institution ensure safety and how they do it. The study includes…
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
Researchers propose a framework called Safety Harness Evolution (SHE) for improving the safety of large language model agents. SHE focuses on evolving the harness, which manages context, memory, tools, permissions, and…
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
Researchers have proposed a framework called Safety Harness Evolution (SHE) for large language model (LLM) agents. SHE aims to improve the safety of these agents by learning evolving safe boundaries from rollout…
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
Researchers propose a framework called Safety Harness Evolution (SHE) to improve the safety of large language model agents. SHE learns evolving safe boundaries from rollout trajectories, allowing for localized harness…
ECHO: A Locally-Deployable Agentic Health Assistant with Temporal Memory, Safety Guardrails, and Speech Assessment
The paper introduces ECHO, a locally-deployable health assistant designed for long-term chronic care management. It consists of three integrated modules: an agentic chatbot, a safety layer, and a speech assessment…
Resourced Authority A Mechanism-Design Model for Participatory Governance of Deployed AI Agents
The authors propose a model for continuous participatory governance of AI agents, focusing on controlling them through resource allocation to make authorization self-enforcing via compute budgets. The model consists of…
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
The study audits the assumptions made in large language model (LLM) benchmark evaluations, focusing on one popular LLM - ChatGPT. The research compares two access modalities (chat UI and API), with and without web…
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
The research examines the limitations of current benchmarks used to evaluate large language models (LLMs) for safety, reliability, and deployment readiness. The study focuses on a popular LLM, ChatGPT, comparing its…
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
The study examines the limitations of current AI benchmarks by focusing on factors such as modality (chat UI vs API), search capabilities, consistency of responses, citation grounding, and abstention behavior using…
ContextWeave: A Real-World Workflow Benchmark
The paper introduces ContextWeave, a new benchmark for evaluating the performance of language agents in long-horizon, stateful workflows. The benchmark focuses on memory as an essential component and measures workspace…
Item Response Theory for AI Safety
The paper proposes using Item Response Theory (IRT) to measure the safety of language models. IRT is a statistical method used to analyze the latent traits of individuals from their performance on specific tasks. The…
Item Response Theory for AI Safety
Researchers used Item Response Theory (IRT), a statistical toolkit, to analyze the safety performance of 192 language models across eight different safety benchmarks. The study found that three key factors - refusal…
On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
A research paper proposes a novel method called Routing-based On-Policy Distillation (ROPD) to improve the safety of large language models (LLMs). The current fine-tuning approach has a vulnerability where harmful…
Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
Researchers propose a new method for controlling risk in language model agents' actions, which are performed through structured tool calls. The proposed method, called role-stratified per-field conformal risk control…
Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
Researchers introduce a new method called role-stratified per-field conformal risk control for language model agents that perform actions through structured tool calls. This method aims to ensure safety by setting…
Epistemic Norms for AI Safety and Alignment Research
The text discusses a new epistemic code named ECAISA for AI Safety and Alignment Research, which aims to ensure that catastrophic failures in AI never occur under specific conditions. The code focuses on two main…
Falsifiable Commitment Planning for Self-Correcting Web Agents
Researchers propose a new framework called FCPAgent for web agents to ensure they stay on track during long-term planning. The framework represents each plan step as a Falsifiable Commitment Unit (FCU) that includes…
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
The paper discusses the unreliability of looping in coding agents and the challenge of ensuring that correct patches are retained, verified, and submitted. The authors conducted several studies to understand this gap…
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
The paper explores the lack of reliability guarantees in coding agents that use generate-test-revise loops. The authors conduct studies to understand the gap between finding a correct patch and retaining, verifying…
Same Dangerous Objective, Opposite Advice: Direct Exposure versus Multi-Agent Mediation
A study found that a high-capacity language model (gpt-5.6-sol) behaves differently when presented with dangerous objectives directly compared to when other agents manipulate and relay the instructions. The model gave…
Same Dangerous Objective, Opposite Advice: Direct Exposure versus Multi-Agent Mediation
A study using OpenAI's gpt-5.6-sol model found that the model gave different advice when presented with a dangerous objective directly compared to when other agents transformed and relayed the direction. When shown an…
Same Dangerous Objective, Opposite Advice: Direct Exposure versus Multi-Agent Mediation
A study comparing direct exposure to a dangerous objective versus multi-agent mediation using OpenAI's gpt-5.6-sol model reveals that direct exposure leads to advice contrary to the target, while mediated exposure…
Same Dangerous Objective, Opposite Advice: Direct Exposure versus Multi-Agent Mediation
A study compares the behavior of a high-capability language model when it is given dangerous objectives directly versus through other agents. The results show that the model gives advice against its target when given…
Same Dangerous Objective, Opposite Advice: Direct Exposure versus Multi-Agent Mediation
A study found that a high-capability language model (gpt-5.6-sol) behaves differently when given a dangerous objective directly compared to when it's transformed and relayed by other agents. When shown the objective…
Same Dangerous Objective, Opposite Advice: Direct Exposure versus Multi-Agent Mediation
A study tests two scenarios: direct exposure of a dangerous objective to an AI model and the same objective being transformed and relayed by other agents. The results show that the AI model's advice is opposed to the…
Toward Continuous Assurance for the Democratization of AI Agent Creation in Industry
The paper discusses a reliability challenge caused by AI agents being created by non-engineering users within organizations, due to their dependency on various factors that can cause silent degradation over time. The…
When Are Reasoning-Based Guardrails Not Efficient? ResponseGuard: A Fast Vision-Language Guard for Real-Time Moderation
A new AI guardrail called ResponseGuard has been developed that can quickly and efficiently detect harmful responses from a vision-language model without needing to go through step-by-step reasoning, which makes…
Agents in the Wild: Where Research Meets Deployment
This tutorial paper discusses the transition of large language model (LLM) based agentic systems from research to practical use in various fields like software engineering, scientific discovery, and finance. It…
Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
A study investigates the impact of a structured evidence-sufficiency prompt on reducing overconfident answers from four language models in a clinical context. The study finds a significant reduction in unsafe…
Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
A study investigates the impact of a structured evidence-sufficiency prompt on reducing unsafe overconfident answers from four different language models in a medical context. The study finds that the prompt…
Zero Hallucination, by Construction: Hallucination-Aware Layered Oversight for Trustworthy Enterprise AI
The paper 'Zero Hallucination, by Construction: Hallucination-Aware Layered Oversight for Trustworthy Enterprise AI' proposes an assurance architecture called HALO. This system aims to ensure the accuracy of AI output…
Can We Trust Item Response Theory for AI Evaluation?
The paper explores the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI) systems in benchmarks. The authors find that standard IRT estimation tools may not be reliable in large…
MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
MedFailBench is a new open-source benchmark for evaluating the safety of medical AI systems, created by clinicians. It focuses on identifying where a system has failed in terms of safety boundaries rather than just…
Proof-or-Stop: Don't Trust the Agent, Trust the Evidence -- Loop Engineering for Verifiable Evidence-Gated Lifecycle Control
Researchers propose Proof-or-Stop Lifecycle Control, a method that ensures software work transitions only when supported by verifiable evidence. This method improves the reliability of autonomous coding agents by…
AIMO Interpretability Challenge
The text proposes the AIMO Interpretability Challenge, a competition aimed at distinguishing robust from spurious reasoning in advanced mathematical language models. The challenge will provide participants with new…
AIMO Interpretability Challenge
The AIMO Interpretability Challenge is a competition proposed by researchers aimed at distinguishing robust from spurious reasoning in advanced mathematical language models. The challenge focuses on evaluating the…
Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings
The text presents a new theory for partially correlated verifier cascades in LLM harnesses (a system used to verify candidate answers). The theory provides insights into the growth and decay of failure rates, the…
SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing
Researchers present Safety Sentry, a lightweight guard model that makes decisions about executing, asking for help, or refusing actions in real-world environments. This model aims to improve safety and autonomy by…
Traffic-Aware Randomized Smoothing for LLM-Based Network Intrusion Detection
The paper introduces Traffic-Aware Randomized Smoothing (TA-RS), a defense mechanism for Large Language Model (LLM)-based intrusion detection systems to enhance their robustness against feasible traffic manipulation…
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
Researchers have developed a new method called AHA for automated red-teaming of production language models like Claude Code and Codex. This method uses one agentic research environment to discover reusable…
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
Researchers present AHA, an automated red-teaming approach for production language models like Claude Code and Codex. AHA discovers reusable vulnerability knowledge about other agents by proposing a vulnerability…
ADORN: Adaptive Drift handling for Open RAN using Reinforcement Learning
The paper proposes a Q-learning based adaptive retraining approach for Open Radio Access Networks (O-RAN) using Reinforcement Learning (RL). The goal is to balance forecasting accuracy and retraining cost, which can…
Formal Mechanisms for Market Stability in Self-Interested Agent Societies: A Marketplace Simulation Study
This research paper explores mechanisms for maintaining market stability in societies of self-interested agents. The authors conduct a simulation study using DeepSeek-V3 agents and compare eight conditions with…
Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents
The paper introduces an action-graded harm rubric for evaluating AI agents in a red-teaming scenario. This new method rates the severity of an agent's actions on a seven-level scale, considering factors such as…
Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety
The paper introduces institutional red-teaming, a method for evaluating deployment rules in multi-agent AI. The study uses IABench-CA, a consequence-allocation benchmark, to test five canonical rules across seven model…
Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations
The paper introduces a method called Reasoning Consistency Scanning to assess the logical consistency between stated reasoning and answers in AI safety evaluation transcripts. The method can be applied to various AI…
The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents
The researchers explore the impact of biased judges in self-evolving agents, focusing on skill retirement, a mechanism that helps maintain the quality of an agent's knowledge base by retiring poorly performing skills…
The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents
The authors investigate how a biased judge affects the retirement of skills in self-evolving agents, particularly in reference-free tasks. They show that a biased judge can cause a mechanism failure they call 'silent…
DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrail
Researchers present DT-Guard, a content safety guardrail model for large language models in open-world applications. The model uses a Reasoning-Active Training, Reasoning-Free Inference paradigm and aims to improve…
Open Problems in AI Incident Governance
The paper discusses issues with current AI incident governance systems, specifically highlighting inconsistencies in definitions, classification, monitoring, and reporting across various frameworks.
Hardware-Enforced Semantic Coordination for Safety-Critical Real-Time Autonomous Systems
The paper proposes a hardware-enforced semantic coordination architecture for safety-critical real-time autonomous systems. This approach aims to overcome limitations of software-mediated coordination by implementing…
Online Safety Monitoring for LLMs
Researchers propose a simple online safety monitoring system for Large Language Models (LLMs). The system uses a verifier signal from an external model to decide when safety can no longer be assumed, by setting a…
Online Safety Monitoring for LLMs
The researchers propose a straightforward method for monitoring Large Language Models (LLMs) in real-time to ensure safety during deployment. This method involves setting a threshold for a verifier signal from an…
Overview of Risk Assessment and Management for Intelligent Systems under the AI Act and Beyond
The paper provides an overview of the methods used for assessing and managing risks associated with intelligent systems, in response to the need for safe and reliable AI systems as demanded by emerging regulatory…
Overview of Risk Assessment and Management for Intelligent Systems under the AI Act and Beyond
This paper provides an overview of methods for assessing and managing risks associated with intelligent systems, as required by emerging regulatory frameworks. It reviews various types of AI-related risks, existing…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study examined the impact of additional capabilities on agentic coding assistants in terms of software quality and reliability. The results showed that while high-tier models performed well, a low-cost local model…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study found that more reasoning effort in agentic code generation (AI programs assisting human coders) leads to higher reliability and fewer errors, rather than the use of additional tools or capabilities.
SkillFuzz: Fuzzing Skill Composition for Implicit Intents Discovery in Open Skill Marketplaces
The paper presents SkillFuzz, an execution-free testing approach that helps detect unintended objectives (implicit intents) in large language model (LLM)-based agents operating on open skill marketplaces. SkillFuzz…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The text introduces 'adversarial pragmatics', a new benchmark and annotation protocol to evaluate the behavior of language models in various ambiguous situations. The goal is to clarify whether failures occur due to…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The paper introduces Adversarial Pragmatics, a benchmark and annotation protocol for evaluating language models' behavior in situations of instruction conflict, embedded commands, and policy ambiguity. The benchmark is…
RAISE: LLM-based Automated Heuristic Design with Robust Adversary Instance Search
The paper proposes a framework called Robust Adversary Instance Search (RAISE), which uses Large Language Models (LLMs) to design heuristics that perform well under real-world distributional shifts. The framework…
A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution
Researchers propose a new multi-task malware analysis framework based on Mixture of Experts (MoE) architectures. The system can classify malware families, detect packed versus unpacked malware, and identify malware…
Entity Binding Failures in Tool-Augmented Agents
This research paper focuses on a problem in tool-augmented language model agents called 'entity binding failures'. These failures occur when an agent performs actions incorrectly even though it has chosen the right…
Entity Binding Failures in Tool-Augmented Agents
The paper discusses a problem called 'entity binding failures' in AI agents that are augmented with tools. These failures occur when an agent incorrectly identifies or interacts with the wrong entity despite selecting…
EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures
This paper presents EvalSafetyGap, a hybrid approach combining survey and conceptual framework for evaluating the safety of Large Language Models (LLMs). The authors aim to address the shared challenge in LLM…
Confidence-Aware Tool Orchestration for Robust Video Understanding
The paper presents a new framework called Robust-TO for video understanding that addresses the issue of 'Blind Trust Problem' in current language models. These models assume every input frame is equally reliable, but…
Model Forensics: Investigating Whether Concerning Behavior Reflects Misalignment
The paper proposes a basic protocol for investigating if AI models exhibit concerning behavior due to malicious intent or benign causes like confusion. The protocol involves analyzing the 'chain of thought' (CoT) to…
The Unfireable Safety Kernel: Execution-Time AI Alignment for AI Agents and Other Escapable AI Systems
The authors propose a safety mechanism for AI agents called the Unfireable Safety Kernel, which is designed to prevent AI systems from making unauthorized changes within their own runtime. The kernel is implemented in…
The Unfireable Safety Kernel: Execution-Time AI Alignment for AI Agents and Other Escapable AI Systems
The authors present the Unfireable Safety Kernel, a Rust implementation that serves as an execution-time AI alignment layer for escapable AI systems. This system aims to provide authorization mechanisms for controlling…
