Notation-Enhanced Rubrics for Image Feedback

• The paper introduces a framework where notation-enhanced rubrics integrate formal symbols and example prompts for consistent image evaluation.
• It details a pipeline that aggregates images with rubrics and employs deterministic VLM prompting to output discrete proficiency levels with rationale.
• The methodology demonstrates improved interpretability and grading accuracy across educational and technical domains, supported by quantitative metrics and modular adaptations.

Notation-Enhanced Rubrics for Image Feedback (NERIF) refer to a systematic framework for automating the scoring and feedback of student- or professional-generated images using vision-LLMs (VLMs), most notably large multimodal models such as GPT-4V. NERIF centers on integrating formal, compactly notated rubrics and example-based prompting to enable accurate, interpretable, and consistent assessment across multiple image classification tasks, with an emphasis on formative feedback and transparency. The methodology was first operationalized in the context of scientific modeling in education, and it now informs broader @@@@1@@@@ schemes across diagrammatic, artistic, and technical domains (Lee et al., 2023, Lee et al., 2023).

1. Formal Structure of Notation-Enhanced Rubrics

A NERIF scoring rubric employs a notation-centric schema to encode task-specific criteria, proficiency levels, and instructional guidance. The formal specification comprises:

• Proficiency levels $$L = \{\text{Beginning}, \text{Developing}, \text{Proficient}\}$$, or task-specific ordinal/multidimensional sets.
• Component criteria: Bullet-list notation for visual, structural, or symbolic features (e.g., "Label all refracted rays," "Arrowheads indicate motion").
• Explicit proficiency rules: Logical aggregation of components, such as:
  • Proficient: all components present
  • Developing: at least two but fewer than all components present
  • Beginning: one or zero components present
• Instructional notes: Heuristics or key detection cues, e.g., “Look for italic font in variable labels,” to mitigate ambiguity and guide model attention.
• Visual overlays: Color-coded or iconographic highlights on rubric sheets further communicate component mapping.

The rubric is embedded as a notational block within the model's prompt, typically in conjunction with a problem context and a matrix of few-shot exemplars. Each exemplar includes the original drawing, an annotated label, and a “rationale for proficiency” linked to specific rubric components (Lee et al., 2023, Lee et al., 2023).

2. NERIF Prompt Engineering and Pipeline

The pipeline operationalizes NERIF through a tightly structured visual question answering (VQA) process:

1. Image Aggregation: Task context, rubric, and few-shot exemplars are consolidated into a single high-resolution panel (e.g., ~3000×3500 px), or separate images if supported.
2. Prompt Construction: A text preamble encodes:
   • Role: “You are a science teacher scoring student work…”
   • Task: Explicit mapping of rubric and scoring expectation.
   • Notation-enhanced rubric: Formal definitions as per Section 1.
   • Randomly sampled examples per proficiency level with rationale.
   • Hyperparameters: temperature=0.0, top_p=0.01 (deterministic).
3. Model Inference: VLM receives the full prompt (image + text), processes rubric/criteria via OCR and visual parsing, and outputs a single discrete label per test image (“Beginning,” “Developing,” “Proficient”), together with a rationale (Lee et al., 2023).
4. Output Parsing and Evaluation: Model responses are collated; discrete class assignments are compared to gold-standard human labels via accuracy, precision, recall, F1, and quadratic-weighted kappa statistics (Lee et al., 2023).

The key innovation is the interleaving of strict notation-based rubric encoding, exemplary human-scored training cases, and minimal stochasticity to enforce consistency and interpretability in model outputs.

3. Quantitative and Qualitative Evaluation

Empirical studies using NERIF show that scoring accuracy of VLMs like GPT-4V on student-generated science diagrams achieves mean accuracy of 0.51 (SD=0.037), with substantial class-wise variation: $A_\text{Beginning}=0.65$, $A_\text{Developing}=0.62$, $A_\text{Proficient}=0.26$. Median quadratic-weighted kappa is 0.37–0.43 (“fair” to “moderate” agreement). Comparative evaluation demonstrates that Gemini Pro, under identical conditions, reaches only 0.30 accuracy and negative kappa, indicating performance below random chance for three-way classification (Lee et al., 2023).

Qualitatively, GPT-4V demonstrates robust retrieval of rubric context, effective chain-of-thought application of example rationales, and interpretable reasoning in its label assignments. Failure modes are observed for highly proficient or visually subtle constructs, often due to insufficiently specific rubric notes or ambiguous drawing conventions (Lee et al., 2023).

4. Rubric Design, Notation, and Multidimensionality

NERIF explicitly distinguishes itself from scalar-score-based or purely textual rubric approaches by emphasizing compact symbolic notation, multidimensionality, and domain specificity. In educational contexts, rubrics may involve up to four image components (e.g., state change, particle arrangement, labeling, motion indication), whereas professional or scientific diagrams, as in ProImage-Bench, incorporate a two-level rubric hierarchy: abstract criteria (e.g., “mathematical symbols present”) decompose to a variable number of binary checks (e.g., “is variable $x$ italicized?”) (Ni et al., 13 Dec 2025).

Recent work extends NERIF to a vector-valued attribute set $\mathcal{D} =\{\text{realism}$, deformation, imagination, color richness, color contrast, line combination, line texture, picture organization, transformation$\}$, with each $m \in \mathcal{D}$ scored on ordinal scales $y_m(x)\in\{1,\dots,5\}$, yielding formal multi-attribute feedback tuples $(x,\{(m, y_m, t_m)\}_{m\in\mathcal{D}})$ (Ye et al., 14 Dec 2025). This enables fine-grained, pedagogically aligned formative assessment.

5. Model Architectures and Adaptation for NERIF

Contemporary NERIF systems may deploy either direct VLM prompting (as in GPT-4V (Lee et al., 2023, Lee et al., 2023)) or parameter-efficient modular adaptation:

• Multi-LoRA Attribute-Specific Architecture: For each rubric dimension $m$, a separate low-rank adapter ($\Delta W_m = B_m A_m$) customizes the scoring head, supporting targeted, context-specific feedback (Ye et al., 14 Dec 2025).
• Regression-Aware Fine-Tuning (RAFT): Rather than vanilla regression, RAFT uses discrete Bayes-risk optimal inference and loss to enforce ordinal smoothness on rubric-aligned scales, achieving higher Pearson correlations (e.g., $r$ up to 0.653 with multi-LoRA+RAFT versus $r\approx0.468$ in zero-shot) (Ye et al., 14 Dec 2025).
• Notation Branching in Rubric Hierarchies: For technical and diagrammatic domains, a dedicated branch encodes symbolic/notation correctness with specialized binary checks and optionally OCR-assisted evaluation steps (Ni et al., 13 Dec 2025).

6. Scoring, Feedback Loops, and Automated Refinement

NERIF implementations support both summative and formative scoring. For discrete label tasks, model output is logically mapped to the notational criteria. In extended rubric frameworks (e.g., ProImage-Bench), failed binary checks may be automatically aggregated and fed back as editing instructions for iterative image refinement:

$$\text{Accuracy} = 1 - \frac{\sum_i e_i}{\sum_i N_i}, \quad \text{Score} = \frac{1}{M} \sum_{i=1}^M 0.5^{e_i}$$

Actionable feedback may be produced for each failed check (“Please correct the symbol for variable $v$…”) and provided as a loop for generator editing or learner intervention. For notation-sensitive tasks, scoring weights and penalty functions may preferentially penalize failures in symbolic accuracy, as guided by domain requirements (Ni et al., 13 Dec 2025).

7. Best Practices, Limitations, and Future Directions

For highest NERIF performance, empirical recommendations include:

• Minimize extraneous visual clutter; use isolated rubric/image panels where possible.
• Encapsulate rubric points as concise, symbolically denoted bullets with consistent labels.
• Provide balanced, fully annotated few-shot examples per proficiency level.
• Enforce deterministic inference (temperature=0) to reduce output variance.
• Report both overall and distance-sensitive metrics (accuracy, kappa).
• Iteratively refine rubric notes and example selection until performance stabilizes (Lee et al., 2023).

NERIF is currently limited by the visual resolution, text recognition fidelity, and domain adaptation of the underlying VLM. In scientific and technical diagramming, advanced rubric decomposition and high-weight penalties for notation errors are critical for maintaining semantic integrity (Ni et al., 13 Dec 2025). Future research will likely extend NERIF to more complex, multi-modal rubrics, interactive feedback, and fully automated iterative refinement, as well as integration of modular adapters and discrete inference objectives for broader generalization and pedagogical interpretability (Ye et al., 14 Dec 2025).