FIELD NOTES TOPIC
performance metrics
2 notes, newest first.
Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems
The author argues that precision, rather than capability, should be the focus when comparing and measuring AI systems. Precision refers to how consistently an AI system performs the same task, while capability is where…
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
The study compares two access modalities of a large language model (LLM), ChatGPT's chat UI and OpenAI's API, with and without web search enabled. The research shows that the accuracy of responses varies between the…
