Using Large Language Models to Support High Volume Application Review for an Undergraduate Research Program

Abstract: Undergraduate research programs such as the Summer Undergraduate Research Fellowship (SURF) at Purdue University receive thousands of applications every year, requiring significant time and effort for program staff to evaluate each submission consistently and within tight timelines. This work-in-progress paper describes the development and initial deployment of a LLM-based tool to assist in the evaluation of approximately 1,200 student Statements of Purpose (SoPs) for the SURF 2026 cycle at Purdue University. The workflow utilizes OpenAI GPT models (GPT-4o, GPT-5-mini, and GPT-5.2) and uses a structured rubric across six subcategories, each scored on a 0-3 scale. A few SoPs, graded by program staff, were used to tune the model responses. The model prompt was designed to generate both numerical scores, rationales (including positive and negative aspects) and short excerpts from each submission. Using GPT-5.2, the full batch of 1,200 SoPs was processed in approximately 4.6 hours of compute time, averaging roughly 14 seconds per SoP (with per-SoP timing varying with SoP length, which ranged from 500 to 2,000 words). Notable differences in rubric adherence were observed across model versions, with GPT-5.2 adhering most closely. Disagreement in model scores was more pronounced for lower-scoring submissions. The LLM outputs replicated the role previously played by distributed human graders, providing the program coordinator with scored and rationale-annotated outputs for the entire applicant pool. The program coordinator then reviewed these outputs alongside each applicant's SoP, applying the same downstream office criteria used in prior SURF cycles, to produce a shortlist of strong candidates. This coordinator review was completed in approximately 4 hours, compared to the multi-week coordination effort required in prior program cycles.

• The paper presents a scalable LLM-based grading system using an objective rubric to evaluate 1,200 undergraduate SoPs in just 4.6 hours.
• It details iterative prompt engineering that mandates JSON outputs with textual evidence, drastically reducing inter-rater variability.
• The study compares model performance, revealing strong consensus for high-scoring submissions and greater score divergence for lower ones.

LLMs for High-Volume Application Review in Undergraduate Research Programs

Motivation and Context

The paper examines the deployment of LLMs (specifically GPT-5.2, GPT-5-mini, and GPT-4o) as a scalable solution for reviewing a high volume of undergraduate research program applications, focusing on Statements of Purpose (SoPs) for the Purdue SURF 2026 cycle (2606.05564). Traditional human review processes are operationally intensive, requiring multi-week coordination efforts and suffering from inter-rater variability. Recent literature highlights the potential of LLMs for rubric-based scoring given proper prompt engineering and calibration, with reliability approaching human-level agreement in certain contexts.

Rubric Development and Prompt Engineering

A key contribution is the iterative development of a structured, behavioral rubric comprising six dimensions across Passion, Clarity of Purpose, and Resilience, each scored on a 0–3 scale. Objective behavioral descriptors replaced earlier subjective, qualitative prompts, resulting in demonstrably improved rubric adherence by LLMs and facilitating rapid human verification. The prompt engineering mandated JSON schema outputs with both positive and negative rationales, sourced directly from applicants’ SoPs, specifically requiring citation of textual evidence to mitigate hallucination and enable quick human review. Few-shot calibration anchored model scoring to human judgment, with prompt refinements driven by observed output mismatches.

Computational Performance and Workflow Optimization

LLM grading was executed via the OpenAI API, processing approximately 1,200 SoPs in 4.6 hours of compute time, averaging 14 seconds per applicant using GPT-5.2, with a total cost of $25. This replaced weeks of distributed human grading with a process that allowed a program coordinator to complete review and candidate shortlisting in roughly 4 hours.

Figure 1: LLM grading latency per applicant using GPT-5.2, demonstrating scalable throughput for high-volume review.

The cited rationale requirement converted depth reading into rapid verification, drastically reducing review bottlenecks and operational overhead, without sacrificing final candidate pool quality. The tool outputs were strictly used for triage and guided review order; the human-in-the-loop remained central for final selection.

Model Comparison and Inter-Model Divergence

Model variant performance revealed significant differences in rubric fidelity and reliability. GPT-5.2 produced more uniform and rubric-anchored scores, particularly for low-performing submissions, whereas GPT-4o exhibited leniency and scoring inconsistency. GPT-5-mini was inadequate, lacking analytic depth. Score divergence across models correlated with applicant performance: high-scoring submissions yielded strong model consensus (mean divergence ≈2 points for total score ≥13), but divergence increased substantially (mean spread ≥4 points) for low-scoring submissions.

Figure 2: Inter-model score divergence plot, indicating increased disagreement for lower-scoring applicants and robust consensus at the top end.

This pattern aligns with studies showing LLM reliability is strongest when evaluative signals are explicit and weaker when evidence is ambiguous or absent, especially at the lower performance spectrum [mathew2026llms].

Practical Impact, Limitations, and Ethical Considerations

The LLM-based workflow eliminates most coordination and training overhead, but transferability depends on rubric specificity and prompt quality. Objective, behaviorally-anchored rubrics permit consistent scoring and expedite human verification. Crucially, cited textual evidence is the primary mechanism for bias detection and reliability assurance in a high-stakes context.

Key limitations include the absence of formal inter-rater reliability analysis in this deployment, lack of isolation between tool impact and downstream selection criteria, and potential LLM biases rooted in training data. Additional calibration at the low end of the scoring distribution may improve agreement, but formal fairness audits across demographic subgroups are necessary. Data privacy remains a core challenge, since SoPs often contain identifiable information difficult to redact programmatically; API deployment with contractual data retention exclusions is recommended, but principled solutions are pending.

Ethical guidance mandates persistent human-in-the-loop involvement, prohibiting fully automated selection and enabling recourse for applicants. The comparative fairness of LLM+human review versus human-only review is contingent on transparency, bias auditing, and iterative rubric refinement.

Implications and Future Directions

This deployment demonstrates that LLMs, provided explicit, objective rubrics and calibrated prompts, can serve as efficient, reliable triage tools in high-volume application review. The approach is readily adaptable to other domains in education or research administration (e.g., CVs, abstracts, reflections) as long as domain-specific scoring anchors are encoded.

Theoretical implications include the importance of prompt engineering and rubric design for constraining LLM behavior in evaluative tasks, as well as empirical evidence for model variability and its practical consequences. Future work will focus on systematic inter-rater agreement studies, improvement of low-end scoring reliability, web-based tool interfaces, and comprehensive fairness audits.

Conclusion

The paper presents a scalable LLM-powered workflow for structured, rubric-based application review in undergraduate research programs, replacing distributed human grading with a single coordinator workflow augmented by scored, rationale-annotated outputs. Explicit rubric construction, prompt engineering with evidentiary rationale, and mandatory human review collectively yield operational efficiency and maintain accountability. While LLMs are not yet a replacement for expert human judgment, their ability to serve as pre-screeners is robust when rubric-driven and properly calibrated. This paradigm is extensible to analogous high-throughput evaluative contexts in research and engineering education, contingent on continued bias mitigation and transparency.