Incentives and Outcomes in Bug Bounties

Abstract: Bug bounty programs have contributed significantly to security in technology firms in the last decade, but little is known about the role of reward incentives in producing useful outcomes. We analyze incentives and outcomes in Google's Vulnerability Rewards Program (VRP), one of the world's largest bug bounty programs. We analyze the responsiveness of the quality and quantity of bugs received to changes in payments, focusing on a change in Google's reward amounts posted in July, 2024, in which reward amounts increased by up to 200% for the highest impact tier. Our empirical results show an increase in the volume of high-value bugs received after the reward increase, for which we also compute elasticities. We further break down the sources of this increase between veteran researchers and new researchers, showing that the reward increase both redirected the attention of veteran researchers and attracted new top security researchers into the program.

• The paper demonstrates that a 200% reward hike in Google’s VRP led to an average increase of 12.94 bug submissions per month, highlighting the effect of financial incentives.
• It employs Regression Discontinuity Design and Chow tests to uncover a low overall elasticity of 0.206, yet significant responsiveness in high-value bug categories.
• The study shows that veteran researchers predominantly drove high-value submissions while a small group of new entrants also achieved notable productivity.

Incentives and Outcomes in Bug Bounties

This paper analyzed the impact of financial incentives on the performance outcomes of Google’s Vulnerability Rewards Program (VRP), focusing on the substantial increase in rewards announced in July 2024. Through detailed empirical methods, it uncovered the responsiveness of bug submissions, notably high-value bugs, to reward enhancements, providing insights for crafting effective bug bounty programs.

Introduction

Bug bounties reward external security researchers for identifying software vulnerabilities, revealing dynamics between monetary incentives and researcher behavior. This paper utilized a dataset from Google’s VRP, particularly examining a reward hike up to 200% for high-tier bugs in July 2024. The research delved into bug submission volumes and types, offering elasticity computations and investigating veteran versus new researcher contributions, with significant implications for incentive-based program design.

Effects on Bug Submission Volumes

The analysis highlighted a notable increase in high-value bug submissions post-reward increase, with empirical methods such as Regression Discontinuity Design (RDD) and Chow tests confirming these findings:

Figure 1: All bugs per month received from 2023-2024.

The study observed an average rise of 12.94 bugs per month post-reward increase, though overall elasticity remained low (0.206), indicating moderate sensitivity to financial incentives. This pointed to the possible saturation of easier bug discoveries, supported by limited substitution effects towards more complex bug types.

High-Value Bug Dynamics

Emphasizing high-value categories (Tier 0, critical severity), a sharper increase in mean bug counts was noted, reflective of the program’s intention to prioritize critical vulnerabilities. Elasticities for these categories exceeded 1, demonstrating heightened responsiveness to increased rewards:

Figure 2: Results of OLS regressions under the RDD/RKD designs for all bugs in the treated program.

Contributions of Veteran vs. New Researchers

Key findings indicated that the reward change not only redirected efforts but also attracted new, highly productive researchers. Veteran researchers significantly contributed to the rise in high-value bugs, as seen in the proportion of these bugs from established participants post-increase:

Figure 3: Shares of bugs found by veteran researchers vs. new researchers in each six-month window.

Notably, a small cohort of new entrants achieved high productivity, challenging assumptions that higher entry numbers would drive increased output. Instead, a qualitative influx of proficient researchers, possibly from rival platforms, seemed to elevate productivity.

Conclusion

This study underscored the intricate interactions between incentive structures and researcher output in bug bounty programs. The elasticity findings for high-value submissions suggest potential strategic gains from continued reward adjustments targeting critical vulnerabilities. Future work can explore long-term retention, substitution effects across platforms, and the evolving role of AI in augmenting bug discovery.

These insights provide a framework for refining bug bounty programs, adding depth to discussions on leveraging financial incentives for enhanced security outcomes while considering the competitive landscape and researcher behaviors.