ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Mahnoor Shahid, Hannes Rothe
• Venue: arXiv: cs.AI
• Published: 2026-07-06
• Collected: 2026-07-07
• URL: http://arxiv.org/abs/2607.05185v1
• PDF: https://arxiv.org/pdf/2607.05185v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies in four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. The benchmark includes a hierarchical, explicit knowledge base for each game that allows for fine-grained evaluation of an agent's reasoning capabilities.

What the Document Says

• The text discusses the challenge of compositional generalization in modern AI.
• It mentions four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki as part of a new benchmark suite called ClassicLogic.
• ClassicLogic's core innovation is a hierarchical, explicit knowledge base for each game.
• The knowledge base allows for fine-grained evaluation of an agent's reasoning capabilities.

Key Concepts Mentioned

• Compositional generalization
• Artificial Intelligence
• Knowledge-driven benchmark
• Classic Logic puzzles
• Sudoku
• KenKen
• Kakuro
• Futoshiki
• Hierarchical knowledge base
• Explicit strategies
• Neuro-symbolic AI

Claims Made by the Document

• Compositional generalization is a fundamental challenge for modern AI.

Evidence Used

• The text discusses the challenge of compositional generalization in modern AI.

Methods / Approach

• The introduction of ClassicLogic benchmark suite
• The use of a hierarchical, explicit knowledge base

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.