An Empirical Evaluation of LLMs for Solving Offensive Security Challenges

Abstract: Capture The Flag (CTF) challenges are puzzles related to computer security scenarios. With the advent of LLMs, more and more CTF participants are using LLMs to understand and solve the challenges. However, so far no work has evaluated the effectiveness of LLMs in solving CTF challenges with a fully automated workflow. We develop two CTF-solving workflows, human-in-the-loop (HITL) and fully-automated, to examine the LLMs' ability to solve a selected set of CTF challenges, prompted with information about the question. We collect human contestants' results on the same set of questions, and find that LLMs achieve higher success rate than an average human participant. This work provides a comprehensive evaluation of the capability of LLMs in solving real world CTF challenges, from real competition to fully automated workflow. Our results provide references for applying LLMs in cybersecurity education and pave the way for systematic evaluation of offensive cybersecurity capabilities in LLMs.

• The paper demonstrates that ChatGPT and GPT-4 outperformed other models, with ChatGPT excelling in human-guided tasks and GPT-4 in fully-automated workflows.
• The study employs iterative human feedback and a Dockerized testing environment to standardize prompts and rigorously assess model performance in cybersecurity tasks.
• The findings indicate LLMs can notably supplement human expertise in offensive security, while highlighting critical areas for improved tool integration and error handling.

Empirical Evaluation of LLMs for Solving Offensive Security Challenges

This paper evaluates the performance of LLMs in solving Capture The Flag (CTF) challenges, emphasizing two workflows: human-in-the-loop (HITL) and fully-automated methods. The authors, Shao et al., meticulously compare the efficacy of six different LLMs, including well-known models like GPT-3.5, GPT-4, Bard, and Claude, alongside open-source alternatives such as DeepSeek Coder and Mixtral. The study aims to investigate the application and limitations of LLMs in cybersecurity education and offensive security tasks.

Core Methodologies

Human-in-the-loop (HITL) Workflow

The HITL workflow involves direct interaction between human participants and LLMs. Participants from the Cybersecurity Awareness Week (CSAW) at New York University (NYU) employed LLMs, mainly ChatGPT, to solve a curated set of CTF challenges. The study outlines how participants guided LLMs through iterative feedback, refining prompts based on the LLM outputs. This interaction mimicked real-world scenarios where dynamic problem-solving and prompt adjustment are crucial.

Fully-Automated Workflow

The fully-automated workflow evaluates the autonomous CTF-solving capabilities of LLMs. This involves initializing the LLMs with predefined prompts, relevant challenge descriptions, and necessary executable files. The LLM is expected to carry out flag validation autonomously, leveraging a Dockerized environment equipped with essential cybersecurity tools. The paper provides a systematic approach to standardizing prompts and assessing LLM outputs without human intervention.

Key Findings and Numerical Results

HITL Workflow

• ChatGPT (GPT-3.5 and GPT-4): Demonstrated a superior understanding and accuracy in solving CTF challenges compared to other models. In HITL experiments, ChatGPT solved several challenges through iterative feedback, highlighting its ability to reason and incorporate corrections effectively.
• Success Rates: Among the HITL participants, ChatGPT achieved a higher success rate, solving 11 out of 26 challenges without repetitive resets, and aligned closely with average-performing human CTF teams.

Fully-Automated Workflow

• Performance: GPT-4 outperformed other models in the fully-automated workflow, correctly solving 12 out of 21 evaluated challenges. GPT-3.5 solved 6 challenges, consistent with mean human performance in traditional CTF competitions.
• Failure Analysis: Predominant failures were due to empty solutions, faulty codes, and wrong flags. A significant proportion of errors stemmed from incorrect command line executions and import errors, indicative of the need for better context understanding and tool integration.

Implications and Future Work

Practical Implications

The study's results suggest that LLMs like ChatGPT can meaningfully supplement human efforts in CTF challenges, primarily through guided interactions. The HITL workflow effectively bridges the gap between autonomous reasoning and problem-solving finesse needed for complex cybersecurity tasks. This implies a growing role for LLMs in cybersecurity education, where they can act as supplementary teaching assistants for learners.

Theoretical Implications

The findings underscore the critical role of human feedback in enhancing LLM performance. While autonomous capabilities show promise, human expertise remains indispensable for complex, real-time problem-solving scenarios. This aligns with theories advocating for hybrid intelligence systems that combine human intuition with machine computation.

Speculations on Future AI Developments

Future developments in AI might focus on enhancing context comprehension and dynamic tool utilization within LLMs. Improved models could better integrate with specialized cybersecurity tools, reducing errors related to command execution and file handling. Furthermore, advancements in guardrail mechanisms can ensure ethical compliance without hindering the problem-solving capabilities of LLMs.

Conclusion

This empirical evaluation elucidates the potential and current limitations of LLMs in solving offensive security challenges. By comparing HITL and fully-automated workflows, the study provides valuable insights into enhancing LLM utility in cybersecurity domains. While LLMs like GPT-4 exhibit considerable promise, human expertise remains pivotal, advocating for a blended approach in leveraging AI capabilities effectively.