Demystifying RCE Vulnerabilities in LLM-Integrated Apps

Abstract: LLMs show promise in transforming software development, with a growing interest in integrating them into more intelligent apps. Frameworks like LangChain aid LLM-integrated app development, offering code execution utility/APIs for custom actions. However, these capabilities theoretically introduce Remote Code Execution (RCE) vulnerabilities, enabling remote code execution through prompt injections. No prior research systematically investigates these frameworks' RCE vulnerabilities or their impact on applications and exploitation consequences. Therefore, there is a huge research gap in this field. In this study, we propose LLMSmith to detect, validate and exploit the RCE vulnerabilities in LLM-integrated frameworks and apps. To achieve this goal, we develop two novel techniques, including 1) a lightweight static analysis to examine LLM integration mechanisms, and construct call chains to identify RCE vulnerabilities in frameworks; 2) a systematical prompt-based exploitation method to verify and exploit the found vulnerabilities in LLM-integrated apps. This technique involves various strategies to control LLM outputs, trigger RCE vulnerabilities and launch subsequent attacks. Our research has uncovered a total of 20 vulnerabilities in 11 LLM-integrated frameworks, comprising 19 RCE vulnerabilities and 1 arbitrary file read/write vulnerability. Of these, 17 have been confirmed by the framework developers, with 11 vulnerabilities being assigned CVE IDs. For the 51 apps potentially affected by RCE, we successfully executed attacks on 17 apps, 16 of which are vulnerable to RCE and 1 to SQL injection. Furthermore, we conduct a comprehensive analysis of these vulnerabilities and construct practical attacks to demonstrate the hazards in reality. Last, we propose several mitigation measures for both framework and app developers to counteract such attacks.

• The paper introduces LLMSmith, a static analysis tool that identifies RCE vulnerabilities by tracing unsafe API call chains in LLM frameworks.
• The paper combines white-box scanning and black-box searching to verify vulnerabilities in 51 real-world apps, uncovering 17 cases.
• The paper automates prompt-based exploitation, leading to 7 assigned CVE IDs and highlighting the urgent need for improved security measures.

Exploring Remote Code Execution Vulnerabilities in LLM-Integrated Applications

Introduction to LLM-Integrated Application Vulnerabilities

Recent advancements in LLMs have led to their widespread integration into web applications. However, this integration has introduced vulnerabilities, particularly Remote Code Execution (RCE) vulnerabilities, which allow attackers to execute arbitrary code on the application's server through prompt injections. Despite the critical nature of these vulnerabilities, there exists a notable gap in systematic investigations into their detection and mitigation in both frameworks and applications. This paper presents a pioneering effort to address this gap by introducing two novel strategies aimed at detecting potential RCE vulnerabilities in LLM-integration frameworks and verifying these vulnerabilities in real-world LLM-integrated web applications.

Detection and Verification Approaches

Vulnerable Framework API Detection

The paper introduces LLMSmith, a static analysis-based tool designed for the identification of RCE vulnerabilities within LLM-integrated frameworks. By scanning the source code, LLMSmith effectively extracts call chains leading from user APIs to potentially hazardous functions, enabling the discovery of vulnerabilities.

White-Box App Scanning and Black-Box App Searching

For real-world application testing, two methodologies are employed. The white-box scanning approach identifies and collects applications from GitHub repositories that use vulnerable APIs discovered by LLMSmith. The black-box searching method, on the other hand, relies on keyword identification to search for applications in various app markets, enlarging the scope of test subjects significantly.

Automated Prompt-Based Exploitation

LLMSmith automates the detection of vulnerabilities within applications through a sequence of pre-designed prompt injections that aim to trigger and verify the existence of RCE vulnerabilities systematically.

Experimental Evaluation and Results

The effectiveness of LLMSmith was evaluated on 6 LLM-integrated frameworks and 51 real-world applications, leading to the discovery of 13 vulnerabilities within the frameworks and the identification of 17 vulnerable applications. Notably, LLMSmith facilitated the assignment of 7 CVE IDs, underlining the critical nature of the uncovered vulnerabilities.

Implications and Future Directions

The findings highlight the urgent need for awareness and mitigation strategies among framework and application developers regarding RCE vulnerabilities. The paper not only advances the current understanding of security challenges in LLM-integrated applications but also sets a foundation for future research aimed at enhancing the security of such applications.

Concluding Remarks

This research underscores the potential risks associated with integrating LLMs into web applications, particularly the threat of RCE vulnerabilities. By introducing a comprehensive approach for the detection and verification of these vulnerabilities, the paper marks a significant step forward in the pursuit of secure LLM-integrated applications. Moving forward, it is imperative for both framework and application developers to prioritize the implementation of robust security measures to protect against such vulnerabilities.