Incomplete Contracting and AI Alignment

Abstract: We suggest that the analysis of incomplete contracting developed by law and economics researchers can provide a useful framework for understanding the AI alignment problem and help to generate a systematic approach to finding solutions. We first provide an overview of the incomplete contracting literature and explore parallels between this work and the problem of AI alignment. As we emphasize, misalignment between principal and agent is a core focus of economic analysis. We highlight some technical results from the economics literature on incomplete contracts that may provide insights for AI alignment researchers. Our core contribution, however, is to bring to bear an insight that economists have been urged to absorb from legal scholars and other behavioral scientists: the fact that human contracting is supported by substantial amounts of external structure, such as generally available institutions (culture, law) that can supply implied terms to fill the gaps in incomplete contracts. We propose a research agenda for AI alignment work that focuses on the problem of how to build AI that can replicate the human cognitive processes that connect individual incomplete contracts with this supporting external structure.

• The paper reinterprets the AI alignment problem as analogous to incomplete contracting, offering an innovative framework to align AI actions with human values.
• It applies economic theories on property rights and multi-tasking to design reward systems that mitigate unintended AI behaviors.
• The study advocates embedding AI within normative structures, ensuring reward design accounts for complex human values and societal norms.

Incomplete Contracting and AI Alignment: Insights and Implications

The paper "Incomplete Contracting and AI Alignment" by Dylan Hadfield-Menell and Gillian K. Hadfield addresses a significant overlap between the longstanding principal-agent problem analyzed by economists and legal scholars and the emerging AI alignment problem faced by computer scientists. The authors propose leveraging economic theories of incomplete contracting to provide a structured framework for understanding and addressing AI alignment challenges.

AI Alignment and Incomplete Contracts

The fundamental AI alignment problem arises from discrepancies between specified reward functions and genuine human values and preferences. AI systems, particularly those using reinforcement learning, can exploit loopholes in their reward functions, resulting in unintended behaviors. This misalignment parallels the challenges in human principal-agent relationships, where incomplete contracts fail to specify rewards for every possible state or action. Economists have extensively studied incomplete contracts due to the inherent difficulties in specifying all contingencies with bounded rationality and unverifiable states.

The authors propose that AI alignment researchers can gain valuable insights from incomplete contracting by recognizing that misalignment is an inherent and unavoidable phenomenon. Reward misspecification in AI is analogous to contract incompleteness in human interactions. This analogy paves the way for systematically analyzing AI alignment using frameworks developed for incomplete contracts.

Insights from Incomplete Contracting

The paper suggests several key insights from the field of incomplete contracting that could be valuable for AI alignment research:

1. Property Rights and Reward Design: The allocation of property rights in economics helps optimize joint profits by assigning rights to those whose actions most impact joint outcomes. Similarly, AIs could benefit from reward designs that consider the global, interconnected nature of their tasks, not simply isolated objectives.
2. Multi-tasking and Measurement: In incomplete contracting, incentivizing across multiple tasks can impact effort allocation, leading to suboptimal task performance if one task is more measurable than others. This insight is pertinent to AI, particularly in cases where AIs are tasked with high-level goals comprising multiple, variably measurable sub-tasks.
3. Strategies and Incentives: Control rights and incentives can determine agent behavior in incomplete contracts. Analogous strategies could guide AI systems, especially in scenarios where they might engage in strategic behavior due to reward alignment imperfections.

The Role of Normative Frameworks

A major contribution of the paper is the advocacy for embedding AI systems within broader normative structures, akin to how legal and social frameworks buttress human contracts. For AI, rather than attempting to manually encode every human value into reward functions, it is proposed that AI systems learn to predict and align with human normative structures. This would involve developing technical tools enabling AIs to interpret and respond to implicit societal norms and sanctions.

Implications and Future Directions

By examining AI alignment through the lens of incomplete contracting, the paper posits that AI alignment requires more than technical sophistication in reward function design; it demands an understanding of and integration with human normative structures. This approach raises various research questions on how AIs can effectively learn from and interact with these normative systems.

The theoretical exploration into embedding AI systems within human-like normative environments suggests paves the way for developing robust, socially harmonized AI systems. The combination of economic insights and AI design posited in this paper is an important step towards addressing the intricate challenge of achieving alignment between AI actions and human values—an endeavor central to the safe and beneficial deployment of AI technologies.