benchmark evaluation
5 notes, newest first.
TDD-Agent: Test-Driven Reasoning for Code Generation
Researchers introduce TDD-Agent, a system that uses the test-driven development (TDD) paradigm for code generation. The system first generates tests to clarify expected behaviors before implementation and then refines…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of vision-language models (VLMs) used to judge computer-using agents (CUAs). The study finds that even state-of-the-art VLMs have a…
Multimodal Pretraining for Generalizable EEG Representation Learning
Researchers have developed a multimodal EEG foundation model that can be used for various EEG-related tasks without being limited to specific datasets or tasks. The model uses a raw signal encoder, a time-frequency…
OpenForgeRL: Train Harness-native Agents in Any Environment
OpenForgeRL is an open-source framework for training AI agents using various complex inference harnesses in diverse environments. It allows for end-to-end training with open infrastructure and has been validated across…
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Researchers have developed Blind-Spots-Bench, a new benchmark to evaluate the limitations of AI systems by presenting tasks that are easy for humans but challenging for current AI models. The study reveals significant…
