Janus: a Playground for User-Involved Agentic Permission Management

Abstract: AI agents that autonomously execute tool calls on a user's behalf raise pressing questions about permission management: what role could users play, and what role should they play? Despite many proposed approaches, the user's role in agentic permission management remains under explored. We introduce Janus, a playground system for implementing and evaluating user-involved agentic permission management designs. Janus consists of two components: Janus-Core, a modular agentic system supporting a diverse spectrum of permission management designs, and Janus-Harness, an automated evaluation framework. Grounded in a conceptual model that identifies key design axes for user involvement, we implement six permission assistants spanning the design space and evaluate them across three scenarios and three synthetic responders. We demonstrate that user input is critical and can significantly strengthen privacy and security, that AI augmentation of user decisions can help reduce cognitive load, and that realistic user behavior including permission fatigue must be accounted for in system design. No single design performs optimally across all contexts, motivating a more principled and context-sensitive approach to deploying permission assistants in agentic systems. Janus is publicly available to support future investigation into this dimension of agentic system design.

• The paper presents Janus as a modular environment for evaluating user-driven and automated permission management in agentic AI systems.
• The study quantifies tradeoffs between high user involvement and automated risk thresholds using diverse permission assistant strategies.
• Results emphasize the need for compositional, context-sensitive designs that minimize misalignment and user fatigue while protecting critical data.

Janus: A Playground for User-Involved Agentic Permission Management

Introduction and Motivation

The proliferation of agentic AI systems capable of autonomous tool use introduces significant challenges in permission management, focusing particularly on the degree and modalities of user involvement. Modern agents can execute API calls, interact with critical user data (e.g., email, calendar, files), and exhibit flexible and unpredictable control flows. The risk surface encompasses direct attacks (e.g., prompt injection, hallucination-induced misbehavior), excessive privileges, and misalignment with user intent. Traditional security principles such as complete mediation and least privilege become increasingly complex to enforce due to the scale, ambiguity, and dynamic nature of agentic workflows.

"Janus: a Playground for User-Involved Agentic Permission Management" (2607.01510) presents Janus—a modular, extensible environment for evaluating permission management strategies in agentic systems, with a primary focus on varying models of user interaction. Janus enables empirical analysis by providing both a flexible core agentic system (Janus-Core) and a rigorous automated evaluation framework (Janus-Harness), supporting a spectrum of permission management strategies from fully manual to highly automated and AI-augmented.

Conceptual Model and Design Axes

Janus formalizes the runtime permission management process in agentic systems through a clear abstraction: the user issues tasks, the agent plans and proposes tool calls, and each call is evaluated by a permission management module against both explicit user policies and dynamic user interaction.

Figure 1: The flowchart encapsulates the modular runtime mediation in agentic systems where user involvement at various stages influences policy enforcement and decision points.

The authors identify a design space spanning five principal axes:

• Persistent policies: Use of global, deterministic allow/deny rules.
• Autonomy beyond policies: Degree of assistant initiative in approving/rejecting tool calls not addressed by policies.
• Level of user involvement: Frequency and context in which the user is prompted, both at configuration and runtime.
• Grounding: The contextual information (e.g., user instructions, historical actions, environmental state) leveraged in permission decisions.
• Policy evolution: Capacity for policy mutation and learning based on user input and observed context.

The Janus environment is specifically architected to accommodate and combine these strategies, supporting benchmarks across classic and state-of-the-art patterns seen in the literature.

System Architecture

Janus consists of two primary artifacts:

• Janus-Core: A modular agentic system structured around clear interfaces for agents, tools, a policy manager, and pluggable “permission assistants.” Permission assistants are responsible for mediation not handled by deterministic policies and can escalate to the user, apply machine-learned logic, or propose/modify persistent policy rules.

  Figure 2: Janus system overview with modularized cores for agent logic, tools, policy management, and permission assistant interfaces, supporting controlled experimentation.

• Janus-Harness (Evaluation Framework): An extensible, automatable evaluation framework supporting methodical scenario definition (including attack and ambiguous contexts), multiple synthetic oracles (always-accept, always-deny, alignment-aware), and fine-grained metric logging (attack/out-of-alignment rates, user interaction counts, success metrics). This enables controlled, repeatable experiments across a combinatorial set of scenarios, user behaviors, and assistant designs.

Permission Assistants: Implementation and Diversity

Janus implements six permission assistants at key points in the design space:

• risk_assessment: Uses an LLM to rate the risk of a tool call; escalates to the user only if above a chosen threshold.
• risk_assessment_autonomous: Similar to risk_assessment, but rejects (never escalates) risky actions outright.
• auto_approve: Naive baseline that approves all calls.
• user_confirmation: Escalates every tool call to the user.
• constitution: Compiles a semi-structured set of allow/escalate rules from natural language, escalating ambiguous or uncaptured actions.
• policy_suggestion: Proposes new policies in response to escalated events, enabling persistent policy evolution via LLM-generated conditions.

All assistants conform to a common source interface, facilitating rapid implementation (typically <500 lines per assistant) and systematic logging/analysis.

Experimental Insights

Hull Analysis: Security-Usability Tradeoff

The experiments systematically test each assistant under all scenarios and synthetic user behaviors, quantifying the tradeoff between user interaction load and prevention of attack/out-of-alignment tool calls.

Figure 3: Hull plot highlighting the tradeoffs for each permission assistant and synthetic responder behavior—lower right (fewer messages, fewer misalignments) is preferable but hard to achieve.

Key results demonstrate that:

• High user involvement (user_confirmation) is necessary for zero attack/out-of-alignment incidents with alignment-aware or always-deny synthetic responders.
• Automated or policy-driven approaches substantially reduce user prompts but universally exhibit higher misalignment rates—particularly under always-accept or non-vigilant user models.
• The introduction of persistent policy or constitution-driven systems creates risks of over-broad policy or latent errors, especially through misaligned initial configuration or unchecked escalation (as evidenced by nonzero attack rates in constitution-based approaches for certain default policies).

  Figure 4: Differential analysis for alignment-aware responder shows assistant efficacy varies strongly with scenario/context and the assistant’s runtime strategy.

Risk Tolerance and Non-Determinism

Varying risk thresholds in AI-augmented risk assessment assistants reveals a clear, parametric tradeoff: more permissive thresholds systematically reduce user burden but increase incident rates. Variance in experiment outcomes reflects both agent and model nondeterminism—highlighting an operational challenge in production environments.

Figure 5: Per-assistant grid shows the explicit relationship between risk threshold, user prompt rate, and attack/out-of-alignment tool call frequency across diverse scenarios.

Design Implications and Recommendations

The empirical findings of Janus motivate several recommendations for agentic system designers:

• Block at Source: Assistants must aim to automatically reject attack or misaligned actions, minimizing reliance on user vigilance, especially under high-fatigue or unengaged user populations.
• Careful Policy Evolution: Policy suggestions and constitutions require careful design, as a locally correct escalation or persistent policy can enable subsequent misalignment or attack without further user input.
• Scenario Diversity: Effective designs must be benchmarked across a range of attack, ambiguous, and policy-evolving scenarios, not only in idealized (alignment-aware) user settings.
• Address Nondeterminism: Designers should minimize sources of nondeterministic permission outcomes, as fluctuating outputs can degrade user trust and comprehension.
• Compositional Approaches: Singular permission management strategies are insufficient; robust agentic security likely requires defense-in-depth, combining policy, user interaction, and contextual analysis.
• Usability and Fatigue: Excessive user prompts quickly degrade both usability and security, motivating context-sensitive, selectively escalatory designs.

Practical and Theoretical Implications

Janus fills a critical gap by providing a modular, extensible testbed for evaluating permission management strategies in agentic systems under realistic and adversarial conditions. The authors' analysis illustrates that no universally optimal strategy emerges: context, user population, application domain, and threat model all impact which permission regime is appropriate.

Practically, Janus enables rapid prototyping and robust evaluation of new permission strategies, supporting informed deployment decisions and advancing the state of empirical agentic security research. Theoretically, Janus clarifies the dimensions that delineate permission management and demonstrates—through empirical hull analysis and systematic scenario manipulation—the inseparability of usability, security, and personalization concerns in agent-mediated workflows.

Future Directions

Key avenues for research include:

• Exploration of multi-agent and multi-user coordination contexts, where collaborative grounding and cross-user policy inference become critical.
• Integration with domain- and application-specific datasets to probe policy evolution and misalignment in verticals such as healthcare, finance, or workplace automation.
• Human subject studies measuring real user fatigue, response accuracy under realistic cognitive load, and long-term policy drift.
• Enhanced agentic compositionality—for example, integrating dynamic privilege separation and environment-based policy learning.

Conclusion

Janus provides a principled foundation for the study and advancement of user-involved permission management in agentic AI systems. Its modularity, empirical rigor, and focus on both user and system perspectives facilitate meaningful tradeoff analyses, supporting the design of context-sensitive, robust permission management components essential for the safe and effective deployment of autonomous AI agents.