Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
41 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on ai safety
41 notes, page 2 of 2
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
Researchers present AHA, an automated red-teaming approach for production language models like Claude Code and Codex. AHA discovers reusable vulnerability knowledge about other agents by proposing a vulnerability…
Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents
The paper introduces an action-graded harm rubric for evaluating AI agents in a red-teaming scenario. This new method rates the severity of an agent's actions on a seven-level scale, considering factors such as…
Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety
The paper introduces institutional red-teaming, a method for evaluating deployment rules in multi-agent AI. The study uses IABench-CA, a consequence-allocation benchmark, to test five canonical rules across seven model…
The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents
The researchers explore the impact of biased judges in self-evolving agents, focusing on skill retirement, a mechanism that helps maintain the quality of an agent's knowledge base by retiring poorly performing skills…
The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents
The authors investigate how a biased judge affects the retirement of skills in self-evolving agents, particularly in reference-free tasks. They show that a biased judge can cause a mechanism failure they call 'silent…
DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrail
Researchers present DT-Guard, a content safety guardrail model for large language models in open-world applications. The model uses a Reasoning-Active Training, Reasoning-Free Inference paradigm and aims to improve…
Online Safety Monitoring for LLMs
Researchers propose a simple online safety monitoring system for Large Language Models (LLMs). The system uses a verifier signal from an external model to decide when safety can no longer be assumed, by setting a…
Online Safety Monitoring for LLMs
The researchers propose a straightforward method for monitoring Large Language Models (LLMs) in real-time to ensure safety during deployment. This method involves setting a threshold for a verifier signal from an…
Overview of Risk Assessment and Management for Intelligent Systems under the AI Act and Beyond
The paper provides an overview of the methods used for assessing and managing risks associated with intelligent systems, in response to the need for safe and reliable AI systems as demanded by emerging regulatory…
Overview of Risk Assessment and Management for Intelligent Systems under the AI Act and Beyond
This paper provides an overview of methods for assessing and managing risks associated with intelligent systems, as required by emerging regulatory frameworks. It reviews various types of AI-related risks, existing…
SkillFuzz: Fuzzing Skill Composition for Implicit Intents Discovery in Open Skill Marketplaces
The paper presents SkillFuzz, an execution-free testing approach that helps detect unintended objectives (implicit intents) in large language model (LLM)-based agents operating on open skill marketplaces. SkillFuzz…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The text introduces 'adversarial pragmatics', a new benchmark and annotation protocol to evaluate the behavior of language models in various ambiguous situations. The goal is to clarify whether failures occur due to…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The paper introduces Adversarial Pragmatics, a benchmark and annotation protocol for evaluating language models' behavior in situations of instruction conflict, embedded commands, and policy ambiguity. The benchmark is…
Entity Binding Failures in Tool-Augmented Agents
This research paper focuses on a problem in tool-augmented language model agents called 'entity binding failures'. These failures occur when an agent performs actions incorrectly even though it has chosen the right…
Entity Binding Failures in Tool-Augmented Agents
The paper discusses a problem called 'entity binding failures' in AI agents that are augmented with tools. These failures occur when an agent incorrectly identifies or interacts with the wrong entity despite selecting…
EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures
This paper presents EvalSafetyGap, a hybrid approach combining survey and conceptual framework for evaluating the safety of Large Language Models (LLMs). The authors aim to address the shared challenge in LLM…
The Unfireable Safety Kernel: Execution-Time AI Alignment for AI Agents and Other Escapable AI Systems
The authors propose a safety mechanism for AI agents called the Unfireable Safety Kernel, which is designed to prevent AI systems from making unauthorized changes within their own runtime. The kernel is implemented in…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
