Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

Source

arXiv

arxiv.org/abs/2607.06229v1

Read the source

Source

Plain-Language Brief

Researchers introduce a new benchmark called Spider 2.0-AIFunc to test the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark includes 465 verified instances across 125 real-world databases covering six types of AI functions. The study evaluates ten state-of-the-art language models, finding that proprietary models perform better than open-source ones, with errors primarily due to predicate specification, schema grounding, and AI function parameterization.

What the Document Says

  • The text discusses a new benchmark called Spider 2.0-AIFunc for testing the ability of language models to generate AI-native SQL queries on large cloud data platforms.
  • The benchmark includes 465 verified instances across 125 real-world databases covering six types of AI functions.
  • It was constructed using an agent-based pipeline that rewrites source tasks into AI-native form, transforming target queries and refining natural language instructions.
  • The study evaluated ten state-of-the-art language models.
  • The strongest proprietary models reached 67-70% execution accuracy, while the best open-source model achieved 58.1%.
  • Errors were primarily due to predicate specification, schema grounding, and AI function parameterization.
  • Agent frameworks designed for traditional text-to-SQL challenges do not transfer effectively to AI-native SQL.
  • A minimal agent setup consistently matches or outperforms more elaborate alternatives.

Key Concepts Mentioned

  • Spider 2.0-AIFunc
  • text-to-SQL benchmark
  • AI-native SQL
  • language models
  • execution accuracy
  • proprietary models
  • open-source models
  • errors
  • predicate specification
  • schema grounding
  • AI function parameterization
  • agent frameworks

Claims Made by the Document

  • The new benchmark, Spider 2.0-AIFunc, tests the ability of language models to generate AI-native SQL queries.
  • The best open-source model achieved 58.1% execution accuracy.
  • Errors were primarily due to predicate specification, schema grounding, and AI function parameterization.

Evidence Used

  • The text describes the construction of a new benchmark called Spider 2.0-AIFunc for testing the ability of language models to generate AI-native SQL queries.
  • The study reports that the strongest proprietary models reached 67-70% execution accuracy and the best open-source model achieved 58.1%.

Methods / Approach

  • Using an agent-based pipeline to rewrite source tasks into AI-native form, transforming target queries, and refining natural language instructions

Limitations Stated by the Authors

The study did not discuss any limitations in the methodology or data used by the authors.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • Agent frameworks designed for traditional text-to-SQL challenges do not transfer effectively to AI-native SQL.

Open Questions

  • Why do agent frameworks designed for traditional text-to-SQL challenges not transfer effectively to AI-native SQL?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study found that a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting.

Raw Abstract / Excerpt

Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .