Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

Source

arXiv

arxiv.org/abs/2607.02432v1

Read the source

Source

Plain-Language Brief

The paper evaluates whether four large language models can approximate the judgement of experts when grading Linux/bash command responses from second-year Computer Engineering students. The study uses a four-level cognitive taxonomy to test the models' performance, and finds that Gemini~3.0 Pro with rubric-guided prompting achieves the highest agreement with human graders. The results suggest that the complexity of the questions is a reliable predictor of the difficulty faced by LLMs in grading accurately.

What the Document Says

  • The paper evaluates four Large Language Models (GPT, Claude Opus, Gemini, and GLM) for grading Linux/bash command responses from second-year Computer Engineering students.
  • A four-level cognitive taxonomy is used to test the models' performance, ranging from information retrieval to advanced system management.
  • The study tests the models with two prompt variants: a minimal baseline and a rubric-enhanced version.
  • 1200 real responses were graded by three expert instructors, independently.
  • Gemini~3.0 Pro with rubric-guided prompting achieved the highest agreement with human graders (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014).
  • As taxonomy level increased, agreement declined consistently.
  • Across all models, rubric quality had a larger effect than provider choice.
  • Structured prompts improved agreement consistently.

Key Concepts Mentioned

  • Large Language Models
  • Linux/bash command responses
  • Computer Engineering students
  • cognitive taxonomy
  • Gemini~3.0 Pro
  • rubric-guided prompting
  • agreement

Claims Made by the Document

  • The four Large Language Models can approximate expert judgement when grading Linux/bash command responses.
  • Gemini~3.0 Pro with rubric-guided prompting achieved the highest agreement with human graders.

Evidence Used

  • ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014

Methods / Approach

  • testing four LLMs on 1200 real responses from second-year Computer Engineering students
  • using a four-level cognitive taxonomy
  • two prompt variants: minimal baseline and rubric-enhanced version

Limitations Stated by the Authors

  • Agreement declined consistently as taxonomy level increased.
  • The largest discrepancies were at higher levels.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study only evaluates four Large Language Models. Other models may perform differently.

Open Questions

  • How do other LLMs compare to the ones tested in terms of grading accuracy?
  • Can these findings be applied to other programming languages or subjects?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The study provides a transferable evaluation protocol and prompt templates, but it does not specify how the evaluation was conducted beyond using structured prompts.

Raw Abstract / Excerpt

Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.