Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

Abstract: Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier LLMs (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.

• The paper introduces a four-level taxonomy integrating cognitive complexity with operational risk to evaluate Linux/bash exam responses using LLMs.
• It details a rigorous experimental design with 1,200 student responses and compares LLM outputs with expert human grading, highlighting significant gains with rubric enhancements.
• Results reveal LLMs perform well on basic tasks (L1/L2) but struggle with complex, context-dependent operations, emphasizing the need for human review for higher cognitive levels.

Automated Grading of Linux/Bash Exams Using LLMs: A Four-Level Taxonomy Approach

Experimental Design and Methodology

This paper presents a rigorous evaluation framework to assess the viability of LLMs as automated graders for Linux/bash examination responses. The methodology centers on a four-level cognitive taxonomy ("CogTax") combining an adapted Bloom’s taxonomy with an operational-impact dimension, capturing both conceptual complexity and the risk of system modification for each bash command or task. The study processes 1,200 real student responses to a midterm bash examination from an undergraduate operating systems course, with each response independently graded by three human experts using a standardized, rubric-driven protocol.

The research pipeline is meticulously structured as follows: (1) taxonomy-based classification of exam questions; (2) expert human grading to establish a high-reliability baseline, quantified by inter-rater agreement metrics; (3) systematic LLM grading using GPT-5.2, Claude Opus 4.6, Gemini 3.0 Pro, and GLM 5, under two prompt ablations—a minimal baseline and a rubric-enhanced variant; and (4) comprehensive analysis of human-LLM agreement using association (Pearson $r$, Spearman), absolute agreement (ICC), error (MAE), bias (Bland-Altman), and ordinal concordance (Weighted Kappa). This design directly benchmarks LLMs against expert reliability ceilings and exposes performance under both sparse and rich evaluation instructions.

Figure 1: The research methodology pipeline: taxonomy-driven question annotation, human grading baseline, LLM assessment with ablation on rubric provision, and statistical human-AI agreement analysis.

Figure 2: The four-level taxonomy (CogTax) classifies bash commands by cognitive complexity and operational impact, from information queries (L1) to advanced system management (L4).

Statistical Analysis and Human Baseline

Expert human grades exhibit exceptional inter-rater reliability (ICC(2,1)=0.949, Weighted Kappa=0.948, MAE=0.028, negligible bias), setting a strict standard for LLM evaluation. Human per-item score distributions are strongly bimodal, with concentrated peaks at zero and near-full marks, and sparse allocations in the midrange—reflecting the tendency for bash tasks to be either functionally correct or not, with partial credit reserved for demonstrable but incomplete conceptual mastery.

Figure 3: Frequency distribution of normalized item-level scores by human evaluators; strong bimodality at 0% and 90% bins.

Human graders exhibit a consistent drop in awarded scores across taxonomy levels, assigning highest marks to L1 (information retrieval) and L2 (basic manipulations) and exhibiting monotonic declines at L3 and L4 as cognitive complexity and irreversibility increase.

Figure 4: Distribution of assigned grade per taxonomy level—performance degrades from L1 to L4 in both mean and absolute terms.

LLM Performance and Prompt Ablation

Each LLM, under both minimal (V1) and rubric-provided (V2) prompt variants, is evaluated for agreement with the human consensus.

• Rubric provision (V2) yields consistent performance gains across all models. Gemini 3.0 Pro attains the highest overall agreement with humans under rubric provision (ICC(3,1)=0.888, MAE=0.10, bias=-0.014).
• Score distributions for all models mirror the bimodality seen in human grading, but differ in the density and assignment at perfection/zero with the rubric-augmented prompts producing a pronounced increase in high scores at the expense of mediocre ones. Notably, GPT-5.2 and Gemini both show a large spike in 90% marks under V2, while Claude’s gain is more at the 50% partial-credit region.

Figure 5: Score distributions for GPT-5.2 across V1/V2; rubric-guided prompting increases the prevalence of high scores.

Taxonomy-wise, LLM-human agreement is highest at L1/L2 and uniformly degrades with increasing complexity, with marked difficulty in evaluating structural (L3) and advanced management (L4) tasks. These findings are consistent with recent literature demonstrating that LLM grading reliability deteriorates with task complexity or the need for integrated reasoning [(Alonso-Carracedo et al., 30 Jun 2026), Emirtekin_2026].

• At L1/L2, most models achieve ICC values $>$0.85, indicating strong reliability for basic recall or procedural manipulation.
• At L3/L4, agreement drops (ICC $<$0.85) for all systems, with GPT plateauing at L4 even after rubric provision—indicating model limitations in reasoning about multi-step, contextually-linked operations.

Evaluation of Model and Prompt Effects

• Rubric quality has a larger impact on grading consistency than model selection: switching from minimal to rubric-enhanced prompts yields greater gains in ICC and MAE than switching between GPT, Claude, Gemini, or GLM.
• Gemini 3.0 Pro with rubric provision outperforms other models across all statistical metrics. Its bias is minimal, and its error is the lowest recorded, indicating strong suitability for scalable, semi-automated grading at lower taxonomy levels.
• All models exhibit substantial under-reporting of Distinction grades compared to humans, suggesting a tendency towards conservative or risk-averse grading in ambiguous or complex cases.
• Model-specific error analysis reveals that strict pattern matching (e.g., adherence to reference paths or filenames) can cause categorical grading failures, especially in cases where the question context is distributed or sequential. This limitation persists across both prompt variants and is particularly acute at L3/L4.

Figure 6: Assigned scores by taxonomy level for GPT-5.2, demonstrating systematic degradation at higher complexity regardless of prompt variant.

Implications and Theoretical Considerations

Taxonomy-based stratification emerges as a reliable predictor for the safe and effective delegation of assessment tasks to LLMs: L1/L2 items can be robustly automated, while L3/L4 require human intervention or hybrid review. The study also establishes that rubric engineering is a necessary prerequisite for successful automated grading pipelines; prompt refinement has a disproportionately positive effect compared to model choice, resonating with prior empirical work [Pathak_2025, Cipriano_2025].

A notable limitation is the inability of current LLMs to handle distributed cross-item context and to interpret student intent when superficial mismatches obscure underlying competence. These systems remain fundamentally reliant on rubric-specified, surface-level pattern recognition and lack the tacit pedagogical inference that characterizes expert educators. The significance of this gap is not marginal but structural: valid assessment—especially at higher taxonomy levels—depends on context-sensitive, interpretive reasoning that transcends explicit criteria.

For practical adoption, instructors and institutions should thus:

• Emphasize rubric precision and coverage for all items earmarked for automated grading.
• Limit fully automated workflows to retrieval and procedural items; employ AI-assisted pre-screening or triaging for more complex tasks.
• Recognize the continuing necessity for human review in authenticating advanced, integrative, or sequential responses.

From a theoretical perspective, the findings challenge the notion that scaling LLMs or prompt length alone can bridge pedagogical reasoning gaps; instead, advances may require architectural modifications supporting context tracking, process modeling, or explicit alignment with human grading heuristics.

Conclusion

A principled, taxonomy-based framework demonstrates that state-of-the-art LLMs—when guided by high-quality rubrics—can approximate expert grading of bash command-line tasks at information-retrieval and procedural levels, facilitating scalability in high-enrollment computing courses. However, LLMs’ reliability degrades as conceptual and operational complexity increases, and they fail to capture contextual dependencies and pedagogical nuances critical for advanced assessment.

The implications are clear: automated grading pipelines in technical education should be guided by cognitive taxonomy and rubric specificity, ensuring that only suitable tasks are delegated to LLMs while reserving expert human adjudication for high-complexity problems. Future research should focus on generalizing the taxonomy framework to other programming domains, optimizing rubric-to-model transferability, and exploring model innovations that close the reasoning gap exposed by complex technical assessments.

References: (2607.02432, Alonso-Carracedo et al., 30 Jun 2026), [Emirtekin_2026], [Pathak_2025], [Cipriano_2025].