evaluation metrics
7 notes, newest first.
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
The authors propose a reference-free framework to evaluate the quality of task-oriented conversational agent benchmarks, which are often used to assess these agents. The framework uses Large Language Models (LLMs) as…
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
OmniaBench is a new benchmark for evaluating the capabilities of general AI agents across diverse scenarios. It includes over 1,400 tasks with a hierarchical taxonomy of 90 level-1 and 354 level-2 domains based on app…
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
The authors of this paper introduce a new method to evaluate the effects of post-training quantization on large language models, as this technique is commonly used to deploy models in resource-limited settings. The…
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
The authors investigate how the quantization process used to deploy large language models in resource-limited settings can lead to unnoticed behavioral changes, even when task performance seems unaffected. They…
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
This research paper introduces a new evaluation metric called 'correctness agreement' for large language models that undergo post-training quantization, a method used to deploy these models in resource-constrained…
Two AI Metrics Diverged: Will it Make All the Difference?
The authors discuss how different metrics for evaluating AI capabilities can lead to varying conclusions about whether advanced AI will remain in the hands of a few powerful entities or become more accessible to…
Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
A study finds that using low-bit quantization to reduce the cost of large language models can lead to longer chains of thought, which offsets the expected per-token speedup in reasoning tasks. The study introduces a…
