ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

Source facts

ExplorationBench evaluates whether AI systems can acquire genuinely unfamiliar rules through exploration rather than recall. It uses two executable sandboxes, AlienCode and AlienLogic, with discovery targets, flawed manuals, environmental feedback, and dedicated tool-call schemas. The abstract reports evaluation of 10 AI systems: the strongest systems acquired and applied unfamiliar rules, while performance varied across trajectories and continued exploration could stall or reverse earlier gains.

Analysis

The benchmark operationalizes scientific exploration as hypothesis formation, experiment design, feedback use, and transfer to held-out tasks in environments that conflict with familiar knowledge. That makes exploration measurable without treating static factual recall as discovery.

Uncertainty/Doubt

• This is an arXiv preprint. The sandbox difficulty, discovery-target validity, system comparisons, and interpretation of stalled or reversed gains were not independently reproduced here.