Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

Source

arXiv

arxiv.org/abs/2607.05185v1

Read the source

Source

Plain-Language Brief

The paper introduces ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies in four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. The benchmark is unique because it has a hierarchical, explicit knowledge base for each game, allowing for fine-grained evaluation of an agent's reasoning capabilities.

What the Document Says

  • The paper discusses the challenge of compositional generalization in AI and mentions that there are few benchmarks, many focusing on linguistic tasks with lacking complex structures.
  • The authors introduce ClassicLogic, a new benchmark suite designed for evaluating an agent's ability to learn and compose problem-solving strategies.
  • ClassicLogic consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki.
  • The core innovation is a hierarchical, explicit knowledge base for each game that defines complex solving strategies as compositions of simpler ones.
  • This structure allows for fine-grained evaluation of an agent's reasoning capabilities.
  • The benchmark provides puzzles of increasing mathematically validated difficulty.
  • ClassicLogic is open-source and offers a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.

Key Concepts Mentioned

  • Compositional generalization
  • Artificial Intelligence
  • Knowledge base
  • Reasoning capabilities
  • Classic logic puzzles
  • Sudoku
  • KenKen
  • Kakuro
  • Futoshiki

Claims Made by the Document

  • There is a lack of benchmarks focusing on compositional generalization.
  • ClassicLogic offers a new benchmark suite to evaluate an agent's ability to learn and compose problem-solving strategies.
  • ClassicLogic provides puzzles of increasing mathematically validated difficulty.

Evidence Used

  • The text mentions the lack of benchmarks focusing on compositional generalization.
  • The authors introduce ClassicLogic as a solution to this problem.
  • The text describes that ClassicLogic provides puzzles of increasing mathematically validated difficulty.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.