Multi-Granularity MIA Evaluation

• Multi-granularity MIA evaluation is a detailed approach that dissects privacy risks by analyzing attack algorithms, models, and individual data points.
• It employs exposure-based metrics like MER and NMER to reveal heterogeneous risk profiles and identify ultra-vulnerable records in ML models.
• Hierarchical screening and multi-modal assessments enhance auditing, enabling precise defenses against membership inference across diverse applications.

Membership inference attacks (MIAs) seek to determine if a specific data point was used in the training of a machine learning model. Multi-granularity MIA evaluation systematically quantifies the privacy risks of ML models by disaggregating attack performance and vulnerability across distinct axes: attack algorithm, model, and data point. This granularity exposes limitations of traditional single-metric evaluation, reveals heterogeneous risk profiles, and enables more precise auditing of privacy leakage under realistic multipronged attacker scenarios. Recent research has established principled metrics, benchmarks, and methodology for multi-granularity evaluation, spanning classical classifiers, tree-based ensembles, LLMs, and vision-LLMs (LVLMs) (Conti et al., 2022, Preen et al., 13 Feb 2025, Miyamoto et al., 18 Oct 2025, Guépin et al., 2024, Puerto et al., 2024).

1. Taxonomy and Motivation for Multi-Granularity Evaluation

The necessity of multi-granularity evaluation arises because aggregate MIA performance (e.g., mean accuracy over the entire test set) obscures critical differences in vulnerability. Three principal “zoom-levels” are now established (Conti et al., 2022):

1. Attack-algorithm level: Performance is averaged over all target models and data points for each attack $a\in A$, exposing that attacks vary widely in strength even under identical conditions. For instance, accuracies can range from near-random ($0.51$) to strong ($0.85$), indicating that summary reporting risks underestimating attack success.
2. Model level: By fixing the model architecture and dataset, per-attack and per-point exposure is tracked. This detects particularly vulnerable model–dataset pairs (e.g., overfitted architectures).
3. Data-point level: For each data point $x_i$, evaluation is aggregated across all attacks and models, identifying a “long tail” of records consistently at risk, regardless of global averages.

These granularity distinctions are vital. They reveal, for example, that a single point may be reliably vulnerable to multiple attacks across diverse models even if overall attack performance is mediocre. Standard approaches ignoring this granularity can overlook “ultra-vulnerable” records (Conti et al., 2022, Guépin et al., 2024).

2. Formal Metrics and Aggregation Procedures

To consistently operationalize multi-granularity MIA evaluation, recent work defines exposure-based risk scores. For a fixed model $m$, attack $a$, and data point $x_i$ with true label $b_i$, the correctness indicator is $I_i^{(a,m)} = \mathbf{1}\{b_i'^{(a,m)} = b_i\}$, where $b_i'^{(a,m)}$ is the MIA outcome. Main metrics are:

• Member Exposure Rate (MER): For $x_i$ in train$(m)$, $$\mathrm{MER}_{(m)}(x_i) = \frac{1}{|A|}\sum_{a \in A} I_i^{(a, m)}$$
• Non-Member Exposure Rate (NMER): For $x_i$ in test$(m)$, $$\mathrm{NMER}_{(m)}(x_i) = \frac{1}{|A|}\sum_{a \in A} I_i^{(a, m)}$$
• Aggregate Across Models:
  • For all $m\in \mathcal{M}_i^+$ (where $x_i$ in train), $$\mathrm{AMER}(x_i) = \frac{1}{|\mathcal{M}_i^+|} \sum_{m\in \mathcal{M}_i^+} \mathrm{MER}_{(m)}(x_i)$$.
  • For all $m\in \mathcal{M}_i^-$ (where $x_i$ in test), $$\mathrm{ANMER}(x_i) = \frac{1}{|\mathcal{M}_i^-|}\sum_{m\in \mathcal{M}_i^-} \mathrm{NMER}_{(m)}(x_i)$$.

A composite vulnerability score $V_i$ is constructed as $$V_i = \max\{\mathrm{AMER}(x_i), \mathrm{ANMER}(x_i)\}$$ or as a convex combination, allowing focus on worst-case exposures (Conti et al., 2022). For reporting, exposure-rate curves (sorted AMER/ANMER over $i$) succinctly characterize population-wide risk and highlight the distributional “tail.”

3. Protocols and Empirical Foundations

Comprehensive empirical methodology underpins multi-granularity evaluations. Key protocol steps include:

• Target and shadow models: Multiple target models are trained on randomized splits (e.g., 20 per dataset on CIFAR-10, MNIST, PURCHASE-100) with corresponding shadow models for MIA calibration (Conti et al., 2022).
• Broad attack coverage: Diverse MIAs are deployed—classifier-based (SVM, XGBoost, MLP), threshold-based (posterior, cross-entropy, entropy metrics), “label-only” variants, and baseline “gap” attacks (Conti et al., 2022).
• Performance aggregation: Metrics are collected at attack, model, and data-point levels, enabling direct comparison and vulnerability attribution.
• Data subset selection: Specific protocols may isolate randomness to weight initialization for fixed target datasets (“specific evaluation”), or train across multiple pools for average-case reporting (“average evaluation”) (Guépin et al., 2024).

Controlled benchmarks for large models, such as OpenLVLM-MIA for LVLMs, carefully balance member vs. non-member distributions by matching temporal, domain, and statistical properties. Results on these setups demonstrate that when distributional bias is eliminated, MIA AUROC collapses to $0.5$, even for state-of-the-art attacks (Miyamoto et al., 18 Oct 2025).

4. Hierarchical and Efficient Evaluation for Specialized Models

Multi-granularity approaches extend beyond neural networks to tree-based models and ensembles. Recent work introduces hierarchical, two-stage screening:

• Ante-hoc hyperparameter analysis: Predicts “high-risk” hyperparameter configurations (e.g., max_depth, min_samples_leaf) before training. A risk score $R_{\text{hp}}(\theta)$ ranks vulnerability, with simple extracted rules (e.g., “if max_depth$>$7.5 and min_samples_leaf$\leq$7.5 then high-risk”) delivering $89-94\%$ accuracy for unseen datasets (Preen et al., 13 Feb 2025).
• Post-hoc structural analysis: After training, low-cost metrics (Degrees of Freedom, minimum leaf size, Class-Disclosure Risk) are thresholded to flag structural vulnerability. The structural risk indicator $R_{\text{struct}}(M)$ has high precision (few false positives, covers $>90\%$ of significant MIAs) (Preen et al., 13 Feb 2025).

This hierarchical filter efficiently reduces the number of expensive shadow-based attacks needed, while retaining nearly the full accuracy–privacy spectrum for practitioners.

Stage	Assessment Granularity	Key Output
Ante-hoc	Hyperparameter	High-risk $\theta$ filtered via rule-based risk scores
Post-hoc	Model structure	Structural metrics ($S_1$, $S_2$, $S_3$) and $R_{\text{struct}}$

5. Multi-Scale and Multi-Modal Evaluation in Modern Generative Models

Recent benchmarks for LLMs and LVLMs rigorously apply multi-granularity concepts in the context of high-dimensional, large-scale models.

• Textual granularity: In LLMs, attacks are evaluated at the sentence, paragraph, document, and corpus levels. While sentence-level AUROC hovers at random ($\approx0.5$), aggregation (e.g., over 500 documents) can yield AUROC exceeding $0.9$ for sources like ArXiv, revealing strong “compounding” effects in corpus-level MIA (Puerto et al., 2024).
• Calibration and statistical tests: Aggregated per-chunk MIA features (perplexity, compression, Min-K statistics) are input to classifiers and compared using t-tests or U-tests for corpus/document-level scoring.
• Distribution bias audits: For LVLMs, careful member/non-member balancing (via hashing, domain matching, and statistical tests like C2ST, MMD, FID) is essential to avoid spurious MIA success. Under controlled conditions, all tested attacks reduce to random performance, highlighting the risk of overestimating privacy leakage in unconstrained setups (Miyamoto et al., 18 Oct 2025).

6. Theoretical and Methodological Implications

Multi-granularity MIA analysis has illuminated key limitations of traditional average-case evaluation:

• Standard “average-over-datasets” protocols yield risk $R^\text{avg}_\varphi(x^*, D_\text{eval})$ that is the expectation of specific risks $R^\text{sp}_\varphi(x^*, D)$ (where $D$ is a particular dataset). High true risk under certain datasets can be obscured by averaging (Guépin et al., 2024).
• Empirically, many records are misclassified as low-risk by average-case protocols: e.g., $94\%$ of high-risk records on Adult+Synthpop fall into this category, with max absolute error $>0.1$ for $15\%$ of records.
• For auditing, both average-case and specific-case metrics must be reported, particularly to capture rare but extreme vulnerabilities or for released models trained on a fixed dataset.
• Vulnerability is not an immutable property of a datum: the identity of “most-exposed” points changes substantially with attack, model architecture, and split. The top-40 most-exposed records can shift $30–50\%$ across random splits (Conti et al., 2022), underscoring the context-dependence of privacy leakage.

7. Practical Recommendations and Outlook

The primary recommendations emerging from multi-granularity evaluation are:

• Always disaggregate privacy risk reporting by attack, model, and data-point. Visualize AMER/ANMER curves to identify tail risks and guide defenses (Conti et al., 2022).
• For tree-based models, apply inexpensive ante-hoc and post-hoc filters to efficiently prune high-risk configurations before resource-intensive attacks (Preen et al., 13 Feb 2025).
• When benchmarking on complex or multimodal models, audit for distribution bias and calibrate member/non-member splits using domain, time, and statistical matching (Miyamoto et al., 18 Oct 2025).
• Assess both average-case and specific-case risk, especially when releasing a particular trained model. For strong adversaries with extra dataset knowledge, specific-case risk can increase dramatically (Guépin et al., 2024).
• In legal and high-stakes applications (e.g., LLM memorization of copyrighted corpora), only document- or corpus-level MIA is currently effective. Paragraph-level signals alone are insufficient (Puerto et al., 2024).

The adoption of multi-granularity MIA evaluation, underpinned by exposure-based metrics and granular audit protocols, has established rigorous standards for quantifying and mitigating membership privacy risks in contemporary ML practice.