Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

Source

arXiv

arxiv.org/abs/2607.06229v1

Read the source

Source

Plain-Language Brief

Researchers have developed Spider 2.0-AIFunc, a benchmark for testing the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark contains 465 verified instances across various databases and evaluates ten state-of-the-art language models. Proprietary models perform better than open-source ones, but it's noted that traditional text-to-SQL strategies do not transfer effectively to AI-native SQL.

What the Document Says

  • Spider 2.0-AIFunc is a benchmark with 465 verified instances for testing the performance of language models in generating AI-native SQL queries on large cloud data platforms.
  • The benchmark covers six types of AI functions and consists of 125 real-world databases on the Snowflake platform.
  • Evaluation was conducted on ten state-of-the-art language models, with proprietary models performing better (67-70% execution accuracy) than open-source ones (58.1%).
  • Errors in predicate specification, schema grounding, and AI function parameterization were identified as the primary reasons for the performance gap.
  • It was observed that traditional text-to-SQL strategies do not effectively transfer to AI-native SQL, with a minimal agent setup often performing better than more elaborate alternatives.

Key Concepts Mentioned

  • Spider 2.0-AIFunc
  • benchmark
  • text-to-SQL
  • AI-native SQL
  • language models
  • proprietary models
  • open-source models
  • errors
  • predicate specification
  • schema grounding
  • AI function parameterization

Claims Made by the Document

  • Spider 2.0-AIFunc is a useful benchmark for testing the performance of language models in generating AI-native SQL queries.
  • Proprietary models outperform open-source ones on Spider 2.0-AIFunc.
  • Errors in predicate specification, schema grounding, and AI function parameterization impact the performance of language models.

Evidence Used

  • The text describes the purpose and design of Spider 2.0-AIFunc as a benchmark for testing language models.
  • The results of evaluation on various language models are provided in the text.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .