AlignDP: Hybrid Differential Privacy

• AlignDP is a hybrid differential privacy mechanism that partitions user data into rare events shielded by PAC indistinguishability and non-rare events privatized using RAPPOR.
• It employs effective zero-ε LDP for rare events and standard ε-LDP for frequent events, ensuring unbiased frequency estimation under strong privacy guarantees.
• The framework balances privacy and utility through rigorous theoretical bounds, empirical metrics, and global aggregation, making it ideal for secure LLM deployments.

AlignDP is a hybrid differential privacy (DP) mechanism developed to mitigate the risks posed by extraction, distillation, and unauthorized fine-tuning of LLMs. Distinct from post-hoc watermarking or monitoring strategies, AlignDP operates at the data interface by partitioning user data into rare and non-rare components, shielding rare events via PAC indistinguishability (effectively yielding zero-ε local DP) and privatizing non-rare events using RAPPOR. This two-tier framework enforces strong privacy guarantees while retaining statistical utility for frequent categories, with composition and budget constraints enforced by a global aggregator. The theoretical underpinnings establish limits on PAC extensions, tight bounds for RAPPOR estimation error, and utility trade-offs for each privacy regime (Gaikwad, 19 Dec 2025).

1. Two-Tier Architecture of AlignDP

Let each user record be $X = (X_1,\dots,X_d)$, with marginal distributions $\mu_i$ over their respective domains $\mathcal{D}_i$. Fixing a threshold $\alpha>0$, each field $i$ is partitioned as

$$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$

• Rare events ($x\in R_i$) are processed by a PAC indistinguishability shield. The mechanism $M$ outputs the symbol $x$, but only aggregate counts are released, bounded by a PAC-style indistinguishability parameter $\delta(n, \alpha)$.
• Non-rare events ($\mu_i$0) are encoded via $\mu_i$1-ary randomized response (RAPPOR). Each $\mu_i$2 is mapped to a one-hot vector $\mu_i$3, bits flipped independently with probability $\mu_i$4, yielding privatized vector $\mu_i$5 sent to the aggregator.

This architecture ensures that rare events are hidden with “effective zero–$\mu_i$6” LDP, while non-rare events support unbiased frequency estimation under standard LDP.

2. Formal Privacy Guarantees

PAC-Indistinguishability (Rare Events)

Define mechanism $\mu_i$7 for rare categories. $\mu_i$8 is said to satisfy PAC-indistinguishability with parameter $\mu_i$9 if, for any $\mathcal{D}_i$0 and any (possibly randomized) distinguisher $\mathcal{D}_i$1 observing $\mathcal{D}_i$2 outputs,

$\mathcal{D}_i$3

A Hoeffding-type bound yields

$\mathcal{D}_i$4

As $\mathcal{D}_i$5, this approaches $\mathcal{D}_i$6-DP, i.e., “zero–$\mathcal{D}_i$7” LDP for rare events.

Local Differential Privacy for Non-Rare Events (RAPPOR)

For non-rare $\mathcal{D}_i$8, the $\mathcal{D}_i$9-ary randomized response mechanism $\alpha>0$0 is $\alpha>0$1-LDP if

$\alpha>0$2

RAPPOR with bit-flip probability $\alpha>0$3 achieves

$\alpha>0$4

Each $\alpha>0$5-user aggregate yields, for each category $\alpha>0$6,

$\alpha>0$7

Resulting in unbiased estimates with variance $\alpha>0$8.

3. Fundamental Theoretical Results

Theorem 1: PAC Shielding of Rare Events

For $\alpha>0$9 with $i$0, $i$1 i.i.d. samples yield:

$i$2

No adversary can distinguish $i$3 from another rare value with advantage exceeding $i$4. This bound follows from Hoeffding's inequality applied to empirical frequencies and thresholding at $i$5.

Theorem 2: $i$6-LDP for RAPPOR

For non-rare categories, symmetric bit-flip RAPPOR with probability $i$7 satisfies

$i$8

Frequency estimators $i$9 are unbiased, with variance upper bound $$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$0.

Theorem 3: Global Composition

Aggregating up to $$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$1 RAPPOR reports, each with privacy loss $$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$2, yields:

$$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$3

(Basic composition.) For any $$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$4,

$$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$5

(Pinsker–type advanced composition).

PAC shielding does not compose beyond the rare domain. If $$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$6, the adversary’s distinguishing probability increases with $$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$7, requiring DP to control leakage.

4. Analysis of Utility–Privacy Trade-offs

• Non-Rare (RAPPOR): Mean-squared error per category:

$$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$8

With privacy budget $$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$9, set $x\in R_i$0; thus $x\in R_i$1, yielding

$x\in R_i$2

MSE decreases exponentially in $x\in R_i$3 and as $x\in R_i$4 with user count.

• Rare (PAC Shielding): Utility loss is the suppression of frequency estimation in $x\in R_i$5. Since $x\in R_i$6, the suppressed probability mass is at most $x\in R_i$7. For small $x\in R_i$8 (e.g., $x\in R_i$9), overall impact is minimal.
• Hybrid Choice: Reducing $M$0 lowers the suppressed mass but increases the proportion of categories privatized by RAPPOR, increasing estimation error. Typically, $M$1 is chosen small enough for $M$2 to remain modest, balancing the risk of leaking low-frequency identifiers and the noise introduced to moderately frequent events.

5. Empirical Performance and Metrics

Simulations with $M$3 users, $M$4 fields (each size $M$5), and threshold $M$6 yield:

Metric	Rare ($M$7)	Non-rare ($M$8)
Categories per field	$M$9	$x$0
MAE (est. freq.)	$x$1	matches MSE bound
Top-5 accuracy ($x$2)	n/a	$x$3
KL divergence ($x$4)	n/a	$x$5
Spearman's $x$6 ($x$7)	n/a	$x$8

PAC shielding keeps rare event estimates at noise floor (MAE $x$9), invariant to query repetition. Non-rare RAPPOR outputs (with $\delta(n, \alpha)$0, $\delta(n, \alpha)$1) are consistent with theoretical MSE bounds, decaying as $\delta(n, \alpha)$2. Repeated querying (up to 100) demonstrates that rare category estimation remains at noise floor, and non-rare recovery saturates at correlation coefficient $\delta(n, \alpha)$3. No repetition permits the adversary to breach the shield or exceed the RAPPOR noise ceiling.

6. Context and Significance in LLM Privacy

AlignDP introduces a principled interface-level defense for LLMs, contrasting with reactive watermarking or monitoring approaches. By enforcing PAC indistinguishability for rare values and LDP for frequent values, it ensures robust mitigation of low-frequency signal leakage—often the locus of identification risk—while supporting meaningful aggregate analytics. The systematic integration of two privacy regimes, composition-aware aggregation, and explicit utility analysis positions AlignDP as a primary candidate for data sharing and queryable LLM deployments under privacy constraints (Gaikwad, 19 Dec 2025).