Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
104 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on language models
104 notes, page 3 of 5
Tracing Agentic Failure from the Flow of Success
The paper proposes a new method called OAT (One-Class Attribution for Tracing) for identifying errors in language models without relying on expensive or difficult-to-collect data. This method is faster and more…
Win by Silence: Deletion Non-Monotonicity, Autonomous Exploitation, and Typed-State Gating in LLM Plan Evaluation
The study examines how plan evaluators may inadvertently reward less explicit plans generated by Language Model (LLM). The research focuses on a staged expected-value scorer for LLM-produced venture routes and…
Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
A research study presents an approach to understanding and mitigating bias in large language models (LLMs) used as judges, focusing on the hidden state of these models rather than just their input-output level. The…
Metacognition in LLMs: Foundations, Progress, and Opportunities
This paper provides a comprehensive overview of metacognition in Large Language Models (LLMs), highlighting the importance of this aspect for effective learning, problem solving, and communication. It discusses recent…
Game Theory Driven Multi-Agent Framework Mitigates Language Model Hallucination
Researchers present G-Frame, an adaptive multi-agent framework that reduces hallucinations in lightweight large language models used in rule-based scientific domains. The framework is based on Bayesian and team game…
Game Theory Driven Multi-Agent Framework Mitigates Language Model Hallucination
Researchers present G-Frame, a new framework that uses artificial intelligence (AI) to create a high-quality dataset and train a model called OmniChem. This model performs as well as another AI model (GPT 4o mini) on…
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
The paper proposes a new LLM (Language Model) scheduling method called SMETRIC for serving agents. This method aims to balance the workload between instances by focusing on the first request of each agent session and…
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
The authors of this paper introduce a new method to evaluate the effects of post-training quantization on large language models, as this technique is commonly used to deploy models in resource-limited settings. The…
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
The paper discusses the issue of language models providing incorrect or unanswerable responses. The author proposes two separate axes for abstention: one for answers the model would get wrong, and another for questions…
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
A study finds that large language models often struggle with two types of errors: answering questions incorrectly and attempting to answer unanswerable or misleading questions. The researchers propose a method to…
An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery
The researchers propose an experimental design framework to systematically evaluate the autonomous model discovery behavior of large language model coding agents, like Codex and Claude Code. The framework considers…
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Researchers have developed Spider 2.0-AIFunc, a benchmark for testing the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark contains 465 verified instances across…
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
The paper presents AgentGym2, a new evaluation framework designed to test language models in realistic, complex scenarios that mimic real-world deployments. It highlights that most existing benchmarks do not fully…
LLM-as-a-Verifier: A General-Purpose Verification Framework
The paper introduces LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback to improve the accuracy of agentic tasks. The framework scales across multiple dimensions for better…
MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolution
MetaSkill-Evolve is a two-timescale framework that allows self-improving language models to improve not only their task skills but also the improvement procedure itself. The authors claim that this recursive approach…
DRIFTLENS: Measuring Memory-Induced Reasoning Drift in Personalized Language Models
Researchers introduce DRIFTLENS, a framework to measure changes in the reasoning process used by personalized language models due to user attributes, preferences, and prior context. The study shows that these…
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Researchers have proposed a new method called Recursive Evidence Replay (RECONTEXT) to improve the ability of large language models to understand and reason over long contexts. This method doesn't require training…
AutoMem: Automated Learning of Memory as a Cognitive Skill
Researchers developed AutoMem, a framework for improving a language model's memory management skills in long-horizon games. By treating memory as a trainable skill, they allowed the model to decide how to manage its…
Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation
The paper introduces Distill to Detect (D2D), a method that helps identify hidden biases in language models used in high-stakes roles by amplifying these biases through cartridge distillation. The method is designed to…
Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation
Researchers have developed a method called Distill to Detect (D2D) to expose stealth biases in large language models used in high-stakes roles. These biases can favor certain entities, brands, or viewpoints and are…
Measuring the Gap Between Human and LLM Research Ideas
A study comparing the research ideas generated by large language models (LLMs) with those of human researchers. The authors build an evaluation framework to measure the gap between them using a two-axis taxonomy to…
Skills Are Not Islands: Measuring Dependency and Risk in Agent Skill Supply Chains
Researchers introduce Agent Skill Supply Chains (ASSCs) to manage the dependencies of Large Language Model (LLM) agents by characterizing their dependency graphs. The study finds that ASSCs have four structural…
A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problems
This research paper discusses the increased role of large language models in various systems beyond just text generation, and how this shift changes the security landscape. The authors systematize the existing…
Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision
The study examines how language models (LMs) generate self-explanations for their predictions, and finds that when training LMs to explain which aspects of their inputs influenced their behavior, they often produce…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
