PromptSan: NSFW-Classifier Guided Prompt Sanitization

• The paper introduces PromptSan, a prompt-level safety framework that detoxifies harmful prompts while preserving benign generation via classifier-guided token editing.
• It employs two variants—PromptSan-Modify for inference-time token adjustments and PromptSan-Suffix for a universal safety suffix—to steer prompts to safer regions.
• Evaluation on Stable Diffusion v1.4 shows PromptSan significantly reduces NSFW detections while maintaining image quality, as evidenced by improved FID and CLIP similarity scores.

Searching arXiv for the cited papers to ground the article. NSFW-Classifier Guided Prompt Sanitization, or PromptSan, is a prompt-level safety framework for text-to-image generation that seeks to detoxify harmful prompts without altering the diffusion backbone or degrading benign generation capability. It is formulated around the idea that the prompt is the primary control signal of a text-to-image system, so sanitizing the prompt can reduce downstream harmful image synthesis while leaving the pretrained generator unchanged. The framework comprises two variants: PromptSan-Modify, which performs inference-time token-level prompt editing guided by a text NSFW classifier, and PromptSan-Suffix, which trains a universal safety suffix that is appended to harmful prompts and is optimized against both text and image NSFW classifiers (Xie et al., 23 Jun 2025).

1. Research setting and threat model

PromptSan is situated in the safety problem created by modern text-to-image systems such as Stable Diffusion, which can synthesize harmful content including pornography, violence, shocking content, and discrimination or abuse. The motivating claim is that removing unsafe knowledge from the model itself may harm generation quality, whereas prompt sanitization intervenes at the input layer and leaves the pretrained diffusion model unchanged (Xie et al., 23 Jun 2025).

This design choice is closely tied to the broader literature on adversarial prompting. SurrogatePrompt shows that substitution-based attacks can bypass Midjourney’s closed-source moderation layer by replacing high-risk prompt segments with surrogate expressions that preserve harmful meaning closely enough for the generator to remain unsafe, reporting an 88% prompt-bypass rate in its violent or political setup and 100% success for its adult-content prompt family in the intended sense (Ba et al., 2023). TokenProber reaches a similar conclusion from a differential-testing perspective, distinguishing between dirty words that preserve NSFW semantics and discrepant words that expose checker-model mismatch, and reports a 54%+ average increase in bypass rate over prior adversarial prompting baselines (Wang et al., 11 May 2025). In this context, PromptSan addresses a threat model in which small prompt modifications or appended suffixes can materially shift the safety behavior of a text-to-image pipeline.

The central premise is therefore not merely that harmful prompts exist, but that prompt-space manipulation is a first-class attack surface. This suggests that prompt-level defenses must reason about how classifiers localize harmful semantics and how sanitization can move prompts toward a safer region without collapsing benign utility.

2. Formalization and architectural design

PromptSan is defined over four components: a text encoder $E_{\text{text}}: T \to \mathbb{R}^d$, an image generator $G_{\text{image}}: \mathbb{R}^d \to I$, a text NSFW classifier $C_{\text{text}}: \mathbb{R}^d \to [0,1]$, and an image NSFW classifier $C_{\text{image}}: I \to [0,1]$. A prompt is written as

$T = (w_1,\ldots,w_n).$

The objective is to reduce the harmfulness probability of both the prompt representation and the generated image while preserving semantic usefulness (Xie et al., 23 Jun 2025).

The framework is explicitly split into two operational modes.

Variant	Stage	Core mechanism
PromptSan-Modify	Inference time	Identify harmful tokens via text-classifier gradients and optimize only those tokens
PromptSan-Suffix	Training time, then inference-time append	Learn a universal suffix optimized against text and image NSFW classifiers

The distinction is substantive. PromptSan-Modify is an instance-specific sanitizer: it edits a given prompt only if the text classifier judges it unsafe. PromptSan-Suffix is a reusable safety control: it learns a fixed suffix representation $S=(s_1,\ldots,s_m)\in\mathbb{R}^{m\times d}$ that can be concatenated to malicious prompts (Xie et al., 23 Jun 2025).

The broader design principle is “change the prompt, not the model.” In the paper’s framing, this yields a lightweight defense that avoids architecture changes and avoids fine-tuning the diffusion backbone. A plausible implication is that PromptSan is best understood as an input-conditioning defense rather than a generator-alignment method.

3. PromptSan-Modify

PromptSan-Modify is an inference-time editing procedure driven by a text NSFW classifier. For a prompt $T$, the text NSFW loss is defined as

$$L_{\text{text}}(T) = -\log\big(1 - C_{\text{text}}(E_{\text{text}}(T))\big).$$

For each token embedding $e_i = E_{\text{text}}(w_i)$, the method computes a gradient sensitivity

$$g_i = \left\| \nabla_{e_i} L_{\text{text}}(T) \right\|_\infty,\quad \forall i=1,\ldots,n,$$

and treats the most sensitive tokens as the most harmful or influential tokens (Xie et al., 23 Jun 2025).

The top-sensitive tokens are selected by

$G_{\text{image}}: \mathbb{R}^d \to I$0

A binary mask $G_{\text{image}}: \mathbb{R}^d \to I$1 is then formed so that only tokens in $G_{\text{image}}: \mathbb{R}^d \to I$2 are updated. The target label for the full prompt is set to the safe class, $G_{\text{image}}: \mathbb{R}^d \to I$3, and the global binary classification loss is

$G_{\text{image}}: \mathbb{R}^d \to I$4

The gradient is restricted by masking,

$G_{\text{image}}: \mathbb{R}^d \to I$5

or, equivalently at token level,

$G_{\text{image}}: \mathbb{R}^d \to I$6

The actual update rule is gradient descent on only the selected tokens:

$G_{\text{image}}: \mathbb{R}^d \to I$7

where $G_{\text{image}}: \mathbb{R}^d \to I$8 is the learning rate and $G_{\text{image}}: \mathbb{R}^d \to I$9 is the optimization step. The procedure stops after at most $C_{\text{text}}: \mathbb{R}^d \to [0,1]$0 iterations or once the prompt is classified as safe:

$C_{\text{text}}: \mathbb{R}^d \to [0,1]$1

The paper characterizes this as token-level adversarial sanitization: it uses the classifier not only for a binary decision, but also as a localization device that identifies which tokens should be nudged toward a safer representation (Xie et al., 23 Jun 2025).

In the reported implementation, PromptSan-Modify uses an inference learning rate of $C_{\text{text}}: \mathbb{R}^d \to [0,1]$2, performs 10 optimization steps, and selects the top-$C_{\text{text}}: \mathbb{R}^d \to [0,1]$3 fraction with $C_{\text{text}}: \mathbb{R}^d \to [0,1]$4 in the main setting. The ablation reports that $C_{\text{text}}: \mathbb{R}^d \to [0,1]$5 yields the best total NSFW reduction, with 43 total detections in the evaluation summarized by the paper, and larger $C_{\text{text}}: \mathbb{R}^d \to [0,1]$6 values perform worse (Xie et al., 23 Jun 2025). This supports the claim that only a small set of high-sensitivity tokens drives much of the harmful behavior.

4. PromptSan-Suffix

PromptSan-Suffix moves from per-prompt editing to universal prompt conditioning. It learns a suffix

$C_{\text{text}}: \mathbb{R}^d \to [0,1]$7

initialized as

$C_{\text{text}}: \mathbb{R}^d \to [0,1]$8

while the text encoder and image generator remain frozen (Xie et al., 23 Jun 2025). The learned suffix is intended to neutralize harmful intent when appended to malicious prompts.

The method imposes joint safety constraints on malicious prompts $C_{\text{text}}: \mathbb{R}^d \to [0,1]$9. At the image level, the expectation of the image classifier score on generated outputs should remain below an image safety threshold:

$C_{\text{image}}: I \to [0,1]$0

At the text level, the appended prompt should also remain below a text safety threshold:

$C_{\text{image}}: I \to [0,1]$1

Training proceeds in two stages within each update cycle. First, for a batch of malicious prompts, the suffix is appended, images are generated, and an image harmfulness loss is evaluated. The paper notes typographical inconsistencies in the printed formula, but the intended meaning is that an image NSFW classifier is used to assess harmfulness. The gradient sensitivity of each suffix token is

$C_{\text{image}}: I \to [0,1]$2

The top-$C_{\text{image}}: I \to [0,1]$3 sensitive suffix tokens are then selected:

$C_{\text{image}}: I \to [0,1]$4

Second, those selected suffix tokens are refined using the text NSFW classifier:

$C_{\text{image}}: I \to [0,1]$5

In effect, the image classifier identifies where the suffix matters most for visual safety, and the text classifier then regularizes those positions toward textual safety (Xie et al., 23 Jun 2025).

The reported implementation uses a suffix length of 20 tokens, AdamW with learning rate $C_{\text{image}}: I \to [0,1]$6, 100 text-classifier optimization steps, 15 image-classifier steps per iteration, and top-$C_{\text{image}}: I \to [0,1]$7 in the main configuration. The appendix reports that $C_{\text{image}}: I \to [0,1]$8 performs best, yielding 38 total NSFW detections, while $C_{\text{image}}: I \to [0,1]$9, $T = (w_1,\ldots,w_n).$0, and $T = (w_1,\ldots,w_n).$1 are worse (Xie et al., 23 Jun 2025). The paper also reports that prefix-based and suffix-based variants perform similarly, with a slight advantage for suffix insertion: prefix total 45 versus suffix total 38.

5. Evaluation, ablations, and reported performance

PromptSan is evaluated with Stable Diffusion v1.4 as the base model and compared against five concept-erasure or safety baselines: Concept Ablation (CA), Safe Latent Diffusion (SLD-M), Efficient Safe Diffusion (ESD-u), Unified Concept Editing (UCE), and Mass Concept Erasure (MACE). The main harmful-content benchmark is I2P, with 4,703 prompts across seven inappropriate categories. VISU is used for classifier training and category-specific harmful prompt handling, and COCO 30K is used to evaluate preservation of benign generation (Xie et al., 23 Jun 2025).

The primary safety evaluation counts residual NSFW detections with NudeNet at confidence threshold 0.6. On the summarized I2P nudity evaluation, the total NSFW detections are reported as follows: Stable Diffusion v1.4, 659; CA, 98; SLD-M, 369; ESD-u, 79; UCE, 371; MACE, 77; PromptSan-Modify, 43; and PromptSan-Suffix, 38 (Xie et al., 23 Jun 2025). In this comparison, PromptSan-Suffix attains the lowest total detection count, with PromptSan-Modify immediately behind.

For benign preservation, the paper reports FID and CLIP similarity on COCO 30K. The “Ours” entry is FID-30k 15.16 and CLIP-30k 30.70, compared with CA at 20.68 and 31.28, SLD-M at 20.92 and 30.38, ESD-u at 14.11 and 30.34, UCE at 16.09 and 31.29, and MACE at 13.42 and 29.41 (Xie et al., 23 Jun 2025). The stated interpretation is that PromptSan preserves image quality and text alignment at a level similar to the other safety methods, rather than paying for safety with severe degradation of benign outputs.

The paper also studies where sanitization should be applied within the diffusion process using DDIM with 100 steps. At $T = (w_1,\ldots,w_n).$2, PromptSan yields stronger safety but loses fine textures and color fidelity; at $T = (w_1,\ldots,w_n).$3, sanitization is too late to strongly suppress harmful concepts; the best balance is reported at $T = (w_1,\ldots,w_n).$4 (Xie et al., 23 Jun 2025). Additional ablations examine interaction between the two variants. Modify-then-Suffix yields total 47, which is somewhat effective but not better than the standalone methods, whereas Suffix-then-Modify yields 201 and is much weaker, which the authors attribute to suffix-induced perturbation of the text classifier used by the later modification stage.

The appendix further notes that the text classifier is a lightweight binary neural network with roughly 1.1M parameters operating on a 768-dimensional CLIP feature vector, and that separate text classifiers were trained for different NSFW categories (Xie et al., 23 Jun 2025). This implementation detail is important because it anchors PromptSan’s classifier-guided behavior in category-aware scoring rather than a single monolithic notion of harmfulness.

6. Relation to adjacent safety paradigms

PromptSan belongs to a broader family of prompt-side and conditioning-side defenses, but its operational locus differs from several neighboring approaches. GuardT2I reconstructs the latent meaning of a prompt by translating text-guidance embeddings back into natural language with a conditional LLM and then moderating the reconstruction with a verbalizer and a sentence-similarity checker; on its reported benchmarks it achieves average AUROC 95.58, AUPRC 94.63, and FPR@TPR95 18.39 (Yang et al., 2024). Embedding Sanitizer (ES) instead operates immediately after the text encoder and subtracts a learned toxic residual from the prompt embedding using token-level harmfulness scores, achieving 81.91% overall erasure rate on I2P and 95.04% on adversarial prompts (Qiu et al., 2024). P-Guard learns a universal safety soft prompt in the text embedding space and reports an average unsafe ratio of 5.84% with 1.39 s/image generation time (Yuan et al., 7 Jan 2025).

These methods differ from PromptSan in where safety supervision enters the pipeline. PromptSan-Modify uses a text NSFW classifier to localize and edit harmful tokens at inference time, while PromptSan-Suffix learns a classifier-guided universal suffix. GuardT2I performs latent-to-text interpretation rather than token editing (Yang et al., 2024). ES sanitizes embeddings through a learned residual erasure mechanism rather than through explicit classifier-guided token optimization (Qiu et al., 2024). P-Guard relies on a learned safety soft prompt rather than a suffix selected through image-gradient sensitivity and text-classifier refinement (Yuan et al., 7 Jan 2025).

PromptSan also differs from in-generation moderation. In-Generation Detection (IGD) treats the predicted diffusion noise $T = (w_1,\ldots,w_n).$5 as a semantic safety signal and feeds it to a lightweight binary classifier; if the classifier predicts NSFW, generation is terminated early. IGD reports 92.45% accuracy, 96.78% AUROC, and 16.78% FPR@TPR95 on its combined naive plus adversarial benchmark (Yang et al., 5 Aug 2025). Relative to PromptSan, IGD operates after text input but before final image output, whereas PromptSan modifies the prompt or prompt conditioning before generation begins. This suggests that PromptSan is naturally an upstream defense, while IGD is a runtime safety gate on the diffusion trajectory itself.

7. Limitations, scope, and significance

The reported limitations of PromptSan are concept-dependent inconsistency, weaker performance on violence than on nudity, difficulty with composite harmful prompts, and the need for better integration between PromptSan-Modify and PromptSan-Suffix. The appendix’s use of separate text classifiers for different NSFW categories also indicates that generalization across unseen category combinations remains challenging (Xie et al., 23 Jun 2025).

These limitations matter because adversarial prompting research repeatedly shows that harmful intent can survive major changes in surface form. SurrogatePrompt exploits semantic displacement in closed-source systems (Ba et al., 2023), TokenProber exploits checker-model discrepancy (Wang et al., 11 May 2025), and IGD shows that even adversarially obfuscated prompts can converge to overlapping unsafe predicted-noise features during diffusion (Yang et al., 5 Aug 2025). A plausible implication is that prompt sanitization alone cannot be assumed to close the entire safety gap, especially when attacks target compositional semantics or the generator’s internal state rather than only surface tokens.

Within the prompt-sanitization literature, however, PromptSan is significant for making NSFW classifiers operational rather than merely evaluative. In PromptSan-Modify, the classifier provides both a safety target and a token-sensitivity signal. In PromptSan-Suffix, text and image classifiers jointly shape a reusable suffix that shifts prompt representations toward safer regions. The resulting framework is therefore neither pure prompt blocking nor model-side concept erasure. It is a classifier-guided prompt-conditioning method that treats prompt safety as an optimization problem over the textual control channel of a diffusion model (Xie et al., 23 Jun 2025).