Vital: Vulnerability-Oriented Symbolic Execution via Type-Unsafe Pointer-Guided Monte Carlo Tree Search

Abstract: How to find memory safety bugs efficiently when navigating a symbolic execution tree that suffers from path explosion? Existing solutions either adopt path search heuristics to maximize coverage rate or chopped symbolic execution to skip uninteresting code (i.e., manually labeled as vulnerability-unrelated) during path exploration. However, most existing search heuristics are not vulnerability-oriented, and manual labeling of irrelevant code-to-be-skipped relies heavily on prior expert knowledge, making it hard to detect vulnerabilities effectively in practice. This paper proposes Vital, a new vulnerability-oriented symbolic execution via type-unsafe pointer-guided Monte Carlo Tree Search (MCTS). A pointer that is type unsafe cannot be statically proven to be safely dereferenced without memory corruption. Our key hypothesis is that a path with more type unsafe pointers is more likely to contain vulnerabilities. Vital drives a guided MCTS to prioritize paths in the symbolic execution tree that contain a larger number of unsafe pointers and to effectively navigate the exploration-exploitation trade-off. We built Vital on top of KLEE and compared it with existing search strategies and chopped symbolic execution. In the former, the results demonstrate that Vital could cover up to 90.03% more unsafe pointers and detect up to 37.50% more unique memory errors. In the latter, the results show that Vital could achieve a speedup of up to 30x execution time and a reduction of up to 20x memory consumption on automatically detecting known vulnerabilities without prior expert knowledge.

• The paper presents a novel approach that biases symbolic execution towards paths with type-unsafe pointers for enhanced vulnerability detection.
• It integrates static type inference with a guided MCTS algorithm, achieving up to 37.50% more memory error detections than traditional methods.
• Experimental results show significant improvements in speed (up to 30x faster) and memory efficiency compared to existing chopped symbolic execution techniques.

Vital addresses the persistent challenge of path explosion in symbolic execution, specifically aiming to improve the efficiency of detecting memory safety vulnerabilities. Traditional symbolic execution path selection heuristics often prioritize maximizing code coverage, which does not necessarily correlate strongly with vulnerability discovery. Alternative approaches like chopped symbolic execution attempt to prune the search space by skipping code segments deemed irrelevant, but this typically requires significant manual effort and prior domain knowledge, limiting its applicability for finding unknown vulnerabilities. Vital proposes a vulnerability-oriented approach that automatically guides the search towards paths more likely to contain memory safety errors.

Rationale and Core Hypothesis

The central premise of Vital is that paths exercising a higher number of type unsafe pointers are statistically more likely to harbor memory safety vulnerabilities. The justification stems from the observation that spatial memory errors, such as buffer overflows, can fundamentally only occur during the dereference of pointers whose safety cannot be statically guaranteed.

Vital employs a static analysis phase, leveraging a type inference system derived from CCured, to classify pointers within the target program's LLVM bitcode. Pointers are categorized as:

1. [SAFE](https://www.emergentmind.com/topics/scale-adaptive-feature-enhancement-safe): Pointers whose usage is statically verifiable as memory safe (e.g., pointers confined within their allocated objects).
2. SEQ (Sequence): Pointers involved in arithmetic operations. Their safety is often context-dependent and difficult to ascertain statically.
3. DYN (Dynamic): Pointers resulting from type casts, particularly those involving casts between incompatible types or between pointers and integers.

Pointers classified as SEQ or DYN are collectively termed type unsafe. Vital operates under the hypothesis that prioritizing the exploration of execution paths that cover a greater number of these type unsafe pointer locations will lead to a higher rate of vulnerability discovery. Experimental validation within the paper supports this hypothesis, showing a Pearson correlation coefficient of 0.78 between the number of unique unsafe pointers covered and the number of unique memory errors detected across Coreutils benchmarks.

Methodology: Type-Unsafe Pointer-Guided MCTS

To exploit the unsafe pointer heuristic, Vital integrates a guided Monte Carlo Tree Search (MCTS) algorithm into the symbolic execution engine (specifically, KLEE). MCTS provides a framework for balancing exploration (investigating new paths) and exploitation (focusing on previously promising paths).

The implementation involves the following key components:

1. Static Type Inference: Prior to symbolic execution, a static analysis pass identifies all locations corresponding to the definition or use of SEQ or DYN pointers. These locations are stored in an unsafeSet. This analysis is reported to have minimal overhead (average 5 seconds per Coreutils utility).
2. MCTS State Selection: Vital replaces KLEE's default state scheduler with an MCTS-based scheduler (MCTSSearcher). The MCTS operates on the symbolic execution state tree. The standard MCTS cycle (Selection, Expansion, Simulation, Backpropagation) is adapted:

   • Selection: Traverses the existing MCTS search tree using the UCT (Upper Confidence bounds applied to Trees) formula to select a node for expansion. The UCT formula balances exploitation (nodes with high historical rewards) and exploration (less visited nodes):

     $UCT = \frac{w_i}{n_i} + c \sqrt{\frac{\ln N}{n_i}}$

     where $w_i$ is the total reward of node $i$, $n_i$ is the visit count of node $i$, $N$ is the total visit count of the parent node, and $c$ is the exploration constant.

   • Expansion: When a leaf node in the MCTS tree is reached, Vital needs to select which successor state (e.g., the true or false branch of a conditional) to explore next and add to the MCTS tree. This selection is guided by the ExpScore. For each potential child state, Vital identifies the set of basic blocks that are uniquely reachable only through that specific branch (determined via post-dominator analysis on the Control Flow Graph). The ExpScore is calculated as the count of unsafe pointers (from unsafeSet) located within these unique basic blocks. The child state with the higher ExpScore is prioritized for expansion, directly steering the search towards paths that expose new, potentially risky pointer operations.

   • Simulation (Playout): From the newly expanded node (representing a specific symbolic state), a simulation or "playout" is performed. This involves running the symbolic execution forward from that state but with forking disabled. At branch points dependent on symbolic values, a random concrete value is chosen to resolve the branch, effectively performing a randomized execution trace down one specific path until termination. The purpose is to quickly estimate the potential "value" (reward) of exploring deeper from the expanded state.

   • Reward Calculation: The reward ($F_{reward}$) for a simulation is calculated based on the number of unique unsafe pointers covered ($N_{unsafe}$) and the number of memory errors detected ($N_{error}$) during that specific simulation run:

     $$F_{reward} = \alpha \cdot N_{unsafe} + (1-\alpha) \cdot N_{error}$$

     (The paper uses $\alpha = 0.5$).

   • Simulation Optimization: To mitigate redundant simulations within loops that may not yield new insights, Vital incorporates an optimization. If repeated simulations (controlled by --optimization-degree) starting from a node within a loop fail to produce a higher reward than previously observed, further simulations from that node are skipped for a period.

   • Backpropagation: The calculated $F_{reward}$ from the simulation is propagated back up the MCTS tree path from the expanded node to the root, updating the total reward ($w_i$) and visit count ($n_i$) statistics for each node along the path. These updated statistics influence future selections via the UCT formula.

Experimental Evaluation

Vital's performance was evaluated against baseline KLEE search strategies and Chopped Symbolic Execution (Chopper).

1. Comparison with KLEE Baselines (RQ1): Tested on 75 GNU Coreutils programs with a 1-hour timeout.
   • Unsafe Pointer Coverage: Vital covered significantly more unique unsafe pointers than baseline strategies, achieving up to 90.03% more coverage compared to the nurs:icnt strategy.
   • Vulnerability Detection: Vital detected more unique memory errors, finding up to 37.50% more errors compared to the dfs strategy.
   • These results empirically support the correlation between unsafe pointer coverage and vulnerability detection.
2. Comparison with Chopped Symbolic Execution (RQ2): Tested on 6 known CVEs in the libtasn1 library. Chopper requires manual annotation of functions/lines to skip, representing prior expert knowledge.
   • Efficiency: Vital detected the vulnerabilities significantly faster, achieving speedups of up to 30x compared to various Chopper configurations (Chopper+Random, Chopper+DFS, Chopper+Coverage).
   • Memory Consumption: Vital exhibited substantially lower memory usage, with reductions of up to 20x compared to Chopper. Chopper's state recovery mechanisms for skipped functions were identified as a source of high memory overhead.
   • Automation: A key advantage highlighted is that Vital detected these known vulnerabilities without requiring any prior knowledge or manual annotation, unlike Chopper.
3. Ablation Study (RQ3): Experiments removing components of Vital (the unsafe pointer guidance in Expansion, the Simulation step, the simulation optimization) confirmed that the full configuration performed best in terms of both unsafe pointer coverage and error detection, validating the contribution of each component.

In conclusion, Vital presents a symbolic execution strategy that effectively biases the search towards potentially vulnerable program paths by prioritizing the coverage of type unsafe pointers. It achieves this through a novel integration of static type analysis and a guided MCTS algorithm. The experimental results demonstrate substantial improvements in both the rate of memory error detection and resource efficiency (time, memory) compared to traditional coverage-guided heuristics and knowledge-dependent chopped symbolic execution techniques, particularly highlighting its ability to operate automatically without requiring manual configuration based on prior vulnerability knowledge.