Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

Source

arXiv

arxiv.org/abs/2607.02432v1

Read the source

Source

Plain-Language Brief

A study evaluates four large language models (GPT, Claude Opus, Gemini, and GLM) to grade Linux/bash command responses for computing education. The study uses a four-level cognitive taxonomy that assesses the complexity and impact of the commands. Results show that Gemini 3.0 Pro with rubric-guided prompting had the highest agreement with human graders. The complexity of the questions was found to be a reliable predictor of the difficulty faced by these models in accurate grading.

What the Document Says

  • The text discusses an evaluation of four Large Language Models (GPT, Claude Opus, Gemini, and GLM) for grading short Linux/bash command responses in computing education.
  • A four-level cognitive taxonomy was adopted, combining cognitive complexity and operational impact, ranging from information retrieval (L1) to advanced system management (L4).
  • The models were tested on 1200 real responses from second-year Computer Engineering students, with three expert instructors providing independent grading.
  • Gemini 3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014).
  • As taxonomy level increased, agreement declined consistently, with the largest discrepancies at higher levels.
  • Across all models, rubric quality had a larger effect than provider choice, with structured prompts improving agreement.

Key Concepts Mentioned

  • Large Language Models
  • Linux/bash commands
  • Computing education
  • Four-level cognitive taxonomy
  • Gemini 3.0 Pro

Claims Made by the Document

  • Large Language Models can approximate expert judgment when grading short Linux/bash command responses.
  • Question complexity is a reliable predictor of the difficulty faced by these models in accurate grading.

Evidence Used

  • Results from testing four Large Language Models on 1200 real responses
  • Human-AI agreement measures (ICC(3,1), MAE, Bland-Altman bias)

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.