Source note3 min read
CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions
Source
Semantic Scholar
semanticscholar.org/paper/d5676076ac122b8767126e76cb3a1867e1e50…
Source
- Source name: Semantic Scholar
- Source kind: paper
- Authors / organization: Zishan Guo, Yufei Huang, D. Xiong
- Venue: Annual Meeting of the Association for Computational Linguistics
- Published: 2024
- Collected: 2026-06-28
- URL: https://www.semanticscholar.org/paper/d5676076ac122b8767126e76cb3a1867e1e50741
- PDF: UNKNOWN
- DOI: 10.18653/v1/2024.findings-acl.928
Plain-Language Brief
The authors propose CToolEval, a benchmark for evaluating the performance of large language models as agents in real-world Chinese API interactions. The benchmark includes 398 APIs from various apps and covers 14 domains. The evaluation framework simulates real-life scenarios to assess tool invocation ability and task completion ability. The authors evaluate 11 LLMs, finding that while GPT-3.5-turbo excels in tool invocation, Chinese LLMs struggle with issues like hallucination and lack of comprehensive tool understanding.
What the Document Says
- Assessment of large language models (LLMs) as agents for decision making and operational tasks is important for the development of LLM-as-agent service.
- The authors propose CToolEval, a benchmark for evaluating LLMs in Chinese societal applications using 398 APIs from 27 apps covering 14 domains.
- An evaluation framework is presented to simulate real-life scenarios and assess tool invocation ability and task completion ability.
- The experiments with CToolEval evaluate 11 LLMs, with findings that GPT-3.5-turbo excels in tool invocation but Chinese LLMs struggle with issues like hallucination and lack of comprehensive tool understanding.
- The findings suggest the need for refinement in decision-making capabilities of LLMs.
- CToolEval data and codes are released at https://github.com/tjunlp-lab/CToolEval.
Key Concepts Mentioned
- large language models
- LLM-as-agent service
- CToolEval benchmark
- API interactions
- tool invocation ability
- task completion ability
- GPT-3.5-turbo
- hallucination
- comprehensive tool understanding
- decision-making capabilities
Claims Made by the Document
- Assessing the capabilities of large language models as agents is crucial.
- CToolEval benchmark can evaluate LLMs in Chinese societal applications.
- GPT-3.5-turbo excels in tool invocation.
- Chinese LLMs struggle with issues like hallucination and lack of comprehensive tool understanding.
Evidence Used
- The text states that assessing the capabilities of large language models as agents is crucial for the development of LLM-as-agent service.
- The text describes CToolEval as a benchmark designed to evaluate LLMs in Chinese societal applications.
- The text mentions that GPT-3.5-turbo excels in tool invocation based on experimental results.
- The text states that Chinese LLMs struggle with issues like hallucination and lack of comprehensive tool understanding based on experimental results.
Methods / Approach
UNKNOWN
Limitations Stated by the Authors
UNKNOWN
Limitations Noted by the Agent
Agent interpretation — not from the source.
UNKNOWN
Open Questions
UNKNOWN
Notes for Future Q&A
May include interpretation. Labeled where it does.
UNKNOWN
Raw Abstract / Excerpt
Assessing the capabilities of large language models (LLMs) as agents in decision making and operational tasks is crucial for the development of LLM-as-agent service. We propose CToolEval, a benchmark designed to evaluate LLMs in the context of Chinese societal applications, featuring 398 APIs across 27 widely-used Apps (e.g., Apps for shopping, map, music, travel, etc.) that cover 14 domains. We further present an evaluation framework that simulates real-life scenarios, to facilitate the assessment of tool invocation ability of LLMs for tool learning and task completion ability for user interation. Our extensive experiments with CToolEval evaluate 11 LLMs, revealing that while GPT-3.5-turbo excels in tool invocation, Chinese LLMs usually struggle with issues like hallucination and a lack of comprehensive tool understanding. Our findings highlight the need for further refinement in decision-making capabilities of LLMs, offering insights into bridging the gap between current functionalities and agent-level performance. To promote further research for LLMs to fully act as reliable agents in complex, real-world situations, we release our data and codes at https: //github.com/tjunlp-lab/CToolEval .
