PrivacyAssist: A User-Centric Agent Framework for Detecting Privacy Inconsistencies in Android Apps

Abstract: Mobile apps offer significant benefits, but their privacy protections often remain ineffective and confusing for users. While prior work mainly analyzes app privacy vulnerabilities, few approaches help users understand, set, and enforce their privacy preferences. This paper presents PrivacyAssist, a multi-agent LLM-based platform that detects inconsistencies between user-granted permissions and developers' declared sensitive data collection and sharing practices. Using Retrieval-Augmented Generation (RAG), PrivacyAssist provides concise explanations and real-time on-device warnings to support informed installation decisions. We evaluate PrivacyAssist with 200 users and 2,347 Android apps, finding that only 16% of apps are fully consistent between granted permissions and declared data practices.

• The paper presents PrivacyAssist, a dual-agent LLM framework that detects inconsistencies between declared data safety practices and actual runtime permissions in Android apps.
• It achieves 90% detection accuracy and significantly reduces false positives from 35% to 10% through a robust RAG-enabled analysis.
• The framework’s two-agent architecture combines on-device monitoring with server-side processing to deliver real-time, actionable privacy warnings to users.

PrivacyAssist: A User-Centric LLM Agent Framework for Detecting Privacy Inconsistencies in Android Apps

Motivation and Problem Landscape

Despite Android's dominant market position and adoption of privacy-centric mechanisms like Permission Model (PM) and Data Safety (DS) disclosures, end-user privacy in the Android ecosystem remains highly vulnerable. Existing app privacy policies and permission models tend to be insufficient, inconsistently enforced, and confusing to end-users, as they rely on developer self-reporting and non-verified declarations. Users' lack of technical understanding and transparency from app developers exacerbate this gap, resulting in frequent over-permissioning, data over-collection, and non-compliance with regulatory frameworks such as GDPR and CCPA.

The paper "PrivacyAssist: A User-Centric Agent Framework for Detecting Privacy Inconsistencies in Android Apps" (2604.23248) introduces a multi-agent, LLM-powered system—PrivacyAssist—to address this gap. Unlike prior approaches focusing mainly on developer-centric vulnerability analysis or static manifest examination, PrivacyAssist emphasizes user-centric, real-time detection of inconsistencies between permissions that users actually grant at runtime and the sensitive data practices declared by app developers.

PrivacyAssist System Architecture

PrivacyAssist adopts a distributed, two-agent model operating across smartphone and server endpoints to enable resource-efficient, real-time privacy analysis.

Figure 1: PrivacyAssist architecture.

On-device (Agent-1), the system passively monitors new app installs, extracts device and app identifiers, and queries granted permissions via the Android PackageManager API. Agent-2, server-side, collects the app’s DS declaration and functional description, forms the input context, and orchestrates privacy inconsistency analysis using an LLM augmented with Retrieval-Augmented Generation (RAG). Kafka-based secure messaging and MongoDB-backed persistent storage ensure responsive, isolated user experience and facilitate incremental knowledge accrual.

Key components of the system include:

• Agent-1: Runs as an Android foreground service, tracking real-time app installation and permission grants using the Broadcast Receiver model.
• Agent-2: Implements RAG-enabled analysis and summarization, retrieving declared app practices from both Google Play and APKCombo, and performing LLM-driven detection of inconsistencies.
• External Database (ED): Houses fine-grained definitions for all Android permissions, sensitive data categories, and operationalizes similarity-based retrieval for RAG.

Technical Features and Workflow

The workflow is composed of four principal stages:

1. Initialization: Unique device-based Kafka topics facilitate secure communication between the user device and central server agent.
2. Monitoring: On each app install, Agent-1 captures actual granted permissions, reflecting real runtime behavior rather than static manifest declarations.
3. Analysis: Agent-2 retrieves the app’s declared sensitive data practices, queries the external knowledge base for up-to-date definitions, and constructs a joint prompt (actual permissions vs. declared data access) for the LLM+RAG pipeline. The system operationalizes three categories of inconsistency:
   • Case 1: Permission is granted at runtime but not disclosed in DS.
   • Case 2: DS declares collection of sensitive data for which the app lacks corresponding user-granted permissions.
   • Case 3: Both Case 1 and Case 2 are jointly present.
4. Warning and Explanation: LLM-generated analysis is summarized for end-user consumption; concise, actionable warnings are sent back to the device and displayed as on-device notifications.

This pipeline leverages RAG not only to inject up-to-date, domain-specific context into the LLM but also to mitigate hallucination and token budget risks inherent in vanilla LLM completions. Inputs and outputs are standardized in JSON, optimizing contextual focus and downstream summarization. The summarization step decomposes and recombines LLM outputs via a MapReduce architecture to yield brief, plain-language risk explanations tailored to user comprehension.

Experimental Evaluation and Results

A substantial, two-country user study incorporating 200 participants and 2,347 distinct Android apps under real-world usage conditions benchmarks PrivacyAssist on technical efficacy, detection accuracy, runtime overhead, and user experience.

Key numerical results:

• Only 16.02% of apps were fully consistent between granted permissions and claimed DS, indicating a widespread compliance gap.
• Among the 1,971 apps with detected inconsistencies: 38.11% were Case 1, 28.56% Case 2, and 33.33% Case 3.
• Social category apps exhibited particularly high inconsistency rates (95.8%), followed by Games (88.8%) and Entertainment (87.8%).
• The most frequently omitted sensitive data categories were Personal (37.42%), Financial info (27.23%), Location (21.86%), and Photos/Videos (13.49%).

Evaluation of PrivacyAssist’s detection capability against manual audits across 100 apps showed a 90% accuracy rate for inconsistency and excess data detection. Notably, while the system produced false positives for multi-role apps with ambiguous feature-permission mappings, it did not miss any true inconsistent cases—a critical threshold for user safety applications. Ablation studies highlight the substantial reduction in false positives facilitated by RAG (35%→10%).

From a systems engineering perspective, Agent-1’s average per-day resource consumption is minimal (1.8% CPU, 10% battery), with typical warning latency of 1.5 minutes (first run) or 15 seconds (subsequent runs).

User experience findings:

• 100% of participants found PrivacyAssist either "very useful" or "helpful," with 73.1% indicating they would cancel app installations after warnings.
• Prior to PrivacyAssist, users rarely and inconsistently reviewed privacy information or DS disclosures, underscoring the practical value of automated, timely warnings.

Implications and Future Research Directions

PrivacyAssist situates itself as one of the few real-time, user-facing systems that critically bridge the gap between developer-declared privacy intentions and the actual permissions exercised by mobile apps. Its architecture demonstrates scalable, privacy-preserving deployment by keeping sensitive user data on-device and transmitting only minimal contextual information for analysis.

Practical implications include the system’s utility for raising user awareness, reducing inadvertent over-disclosure of personal data, and delivering enforceable, real-time privacy interventions. From a developer or regulatory perspective, the persistent detection of inconsistencies (over 83% of tested apps) underscores the need for stricter auditing of DS declarations and automated compliance verification at scale.

Future work will likely focus on embedding LLMs entirely on the mobile device to eliminate server-side dependencies and further minimize privacy risks. Additional directions include dynamic, personalized runtime permission enforcement and more robust cross-validation protocols for evaluating LLM-grounded explanations versus expert assessments.

Conclusion

PrivacyAssist demonstrates that LLM-based agents, augmented with domain-specific retrieval, can efficiently and accurately identify privacy declaration-permission inconsistencies in Android apps in a user-centric, real-time manner. Analysis of thousands of popular apps reveals widespread non-compliance, with practical deployment experiments confirming both strong technical performance and high user acceptance. These findings motivate further integration of LLM-powered agents into user-controlled, on-device privacy architectures and the continued evolution of transparency and compliance mechanisms in mobile ecosystems.