FreeInpaint: Prompt-Guided Inpainting

• FreeInpaint is a training-free, prompt-guided image inpainting framework that integrates PriNo and DeGu modules to enhance both prompt alignment and visual coherence.
• It employs two inference modules—PriNo to optimize the initial noise and DeGu to decompose guidance signals—eliminating the need for retraining or fine-tuning.
• The framework shows significant improvements in metrics such as ImageReward and CLIPScore, indicating robust semantic fidelity and superior human aesthetic preference.

FreeInpaint is a training-free, prompt-guided image inpainting framework designed to solve the dual challenge of prompt alignment and visual rationality in modern diffusion-based inpainting models. It operates as an inference-time plug-in for pre-trained text-to-image diffusion models (such as Stable Diffusion), requiring no network retraining or parameter fine-tuning. The framework introduces two inference modules—Prior-Guided Noise Optimization (PriNo) and Decomposed Training-Free Guidance (DeGu)—which together steer both the initial noise and each step of denoising to maximize semantic agreement with user text while producing visually coherent compositions (Gong et al., 24 Dec 2025).

1. Problem Setting and Motivation

Text-guided image inpainting involves synthesizing new content in an arbitrarily shaped masked region of an input image, conditional on a user-provided text prompt. Given an input image $I \in \mathbb{R}^{h \times w \times 3}$, a binary mask $M \in \{0,1\}^{h \times w}$, and a descriptive prompt $c$, the goal is to generate an output $I^o$ where the masked region contains content faithful to $c$ and blends naturally with the context. In diffusion frameworks (e.g., Stable Diffusion), the masked input is embedded to a latent $z^m = \mathcal{E}(I \odot (1-M))$, sampling proceeds from initial noise $z_T$ conditioned on $(c, z^m, M)$, and the result is decoded and blended to form the output.

Existing methods exhibit two main limitations:

• Prompt misalignment: models, even with prompt conditioning, often hallucinate content merely fitting the local context, ignoring the prompt semantics.
• Visual rationality loss: strict prompt alignment (e.g., attention-only methods) can cause incoherent seams, textural discontinuities, or color mismatches between the inpainted region and known background.

FreeInpaint addresses these by dynamically optimizing the initial noise to control attention in the earliest denoising steps and combining multiple guidance signals for stepwise correction (Gong et al., 24 Dec 2025).

2. Prior-Guided Noise Optimization (PriNo)

The PriNo module optimizes the initial noise latent $z_T$ prior to reverse diffusion, directly influencing the model's attention maps at the earliest step. The central hypothesis is that different realizations of $z_T$ induce different U-Net cross- and self-attention maps, which modulate where the model “looks” when filling the mask.

Given cross-attention $A^c \in \mathbb{R}^{h' \times w'}$ and self-attention $A^s \in \mathbb{R}^{h' \times w'}$ at the first denoising step (past mask downsampling $M'$), the framework minimizes a composite loss: $$\mathcal{L}_c = \sum_{i,j} \left[(1 - M'_{ij})A^c_{ij} - M'_{ij}A^c_{ij}\right],\quad \mathcal{L}_s = \sum_{i,j} \left[(1 - M'_{ij})A^s_{ij} - M'_{ij}A^s_{ij}\right],$$ with a KL regularizer $\mathcal{L}_{\mathrm{KL}}$ to keep the optimized noise close to standard normal. PriNo optimizes the mean and variance $(\mu, \sigma)$ of $z_T \sim \mathcal{N}(\mu, \sigma^2)$ via SGD for several iterations or rounds and selects the best $z_T$ that most strongly directs early attention inside the mask (Gong et al., 24 Dec 2025).

3. Decomposed Training-Free Guidance (DeGu)

The DeGu module performs conditional correction at every DDPM/DDIM denoising step. Standard classifier-free or reward-based guidance typically combines all objectives into a single term, which can overweight prompt alignment or coherence. DeGu instead decomposes the posterior into three explicit rewards:

• $r_c(z_t, c)$: a local CLIPScore for text-prompt alignment in the mask.
• $r_m(z_t, z^m)$: InpaintReward for masked-region coherence with the context.
• $r_q(z_t)$: ImageReward measuring overall human aesthetic preference.

The composite objective augments the conditional score of the denoiser, correcting the output as: $$\hat\epsilon_t = \epsilon_\theta - \gamma_c \sqrt{\bar\alpha_t} \nabla_{z_t} r_c - \gamma_m \sqrt{\bar\alpha_t} \nabla_{z_t} r_m - \gamma_q \sqrt{\bar\alpha_t} \nabla_{z_t} r_q,$$ where $\epsilon_\theta$ is the uncorrected noise prediction, $\gamma_*$ are guidance coefficients, and $\sqrt{\bar\alpha_t}$ modulates guidance strength across timesteps. This multi-term correction allows balancing semantic fidelity, compositional coherence, and visual quality (Gong et al., 24 Dec 2025).

4. Algorithmic Workflow and Hyperparameters

FreeInpaint’s inference comprises two sequential modules:

1. PriNo: For each of one or more random restarts, a random $\epsilon \sim \mathcal{N}(0, I)$ is sampled, its $(\mu, \sigma)$ parameters optimized to minimize the early attention loss, and the best candidate $z_T$ is selected.
2. DeGu: Starting from $z_T$, for each step $t$, gradients of reward models for prompt alignment, region coherence, and aesthetic quality are taken with respect to the current latent, and the denoiser output is corrected before DDPM update.

Key hyperparameters include PriNo attention loss weights ($\lambda_1 = 1$, $\lambda_2 = 5$, $\lambda_3 \approx 500$), number of PriNo optimization rounds ($\tau_{\rm round}$), DeGu scales $(\gamma_c, \gamma_m, \gamma_q)$, and denoising step count (50 for SD1.5, 20 for SDXL, etc.) (Gong et al., 24 Dec 2025).

5. Experimental Evaluation

Comprehensive evaluations on EditBench (free-form masks, Mask-Rich local prompts) and re-annotated MSCOCO (object layout with detailed LLaVA captions) demonstrate significant quantitative gains. Key comparison metrics include Local/Global CLIPScore (prompt alignment), InpaintReward and LPIPS (visual rationality), and ImageReward/HPSv2 (perceptual quality/human preference). Against SD1.5-Inpainting, PowerPaint, BrushNet, SDXL-Inpainting, SD3I, and the training-free HD-Painter baseline:

• FreeInpaint yielded +15–25% improvements in ImageReward, ~+1 point HPSv2, +2–3 points in Global CLIP, −5–10% in LPIPS, and the largest InpaintReward improvements relative to all backbones.
• In user studies (30 samples, 59 raters), FreeInpaint was preferred 64.5% of the time—outperforming SDI (16.2%) and SDI+HDP (19.3%) (Gong et al., 24 Dec 2025).
• Qualitative examples reveal robust text rendering inside the mask, fine detail preservation, and seamless texture/lighting transitions.

6. Analysis of Approach and Limitations

FreeInpaint’s superior performance derives from strict early attention steering (via PriNo) and instance-level, multi-objective denoising correction (DeGu), decoupling high-level structure from context compatibility and human aesthetic factors. The plug-and-play nature makes FreeInpaint compatible with any pre-trained inpainting U-Net or DiT model, enabling rapid integration into broader workflows.

Limitations include the potential failure on extremely small masks (<5×5 px), where attention steering and reward signals become unreliable, and indirect sensitivity to biases in reward models (CLIP, InpaintReward, ImageReward). Future work is suggested in hierarchical attention for ultra-tiny regions, scenario-adaptive reward structures, and optimizing compute by reducing PriNo optimization rounds or guiding only sparse denoising steps (Gong et al., 24 Dec 2025).

7. Context Within Free-Form Inpainting Research

FreeInpaint operates within a family of recent prompt-aware, training-free diffusion inpainting methods. In contrast to HarmonPaint (Li et al., 22 Jul 2025), which manipulates U-Net self-attention at inference for structure and style harmonization, FreeInpaint introduces explicit reward decomposition and prior-guided noise optimization for more direct prompt alignment. Relative to LanPaint (Zheng et al., 5 Feb 2025), which employs Langevin dynamics and guided scoring for exact conditional inference, FreeInpaint targets direct reward optimization and attention steering for the prompt-driven setting. Collectively, this body of research converges towards robust, high-fidelity, and prompt-faithful inpainting under arbitrary masks by leveraging and refining the capabilities of powerful pre-trained diffusion architectures.