Differentiable Smudge Operator

• Differentiable Smudge Operator is a neural rendering primitive that blends pigments via smooth, gradient-accessible procedures emulating physical smudging.
• It reformulates classical sequential smudge steps into a one-shot, weighted integration along cubic Bézier strokes to optimize stroke geometry efficiently.
• It integrates into stroke reconstruction pipelines to recover realistic color transitions and soft tonal modulations in digital painting.

A differentiable smudge operator is a neural rendering primitive designed to emulate the pigment blending and smooth color transitions produced by physical smudging actions in painting, while ensuring full differentiability for gradient-based fitting or optimization. It achieves this by reformulating classical, sequential smudge procedures into architecturally linear, backpropagation-friendly routines based on differentiable sampling, weighted integration, and compositional updates, thereby enabling optimization of geometric and appearance parameters via gradient descent in reconstruction or synthesis frameworks (Jiang et al., 17 Nov 2025).

1. Formal Definition and Mathematical Structure

Let $I(x)$ denote the current canvas RGB image over domain $x \in \Omega$. A smudge stroke is parameterized as $$\Theta_{\text{smudge}} = \{x_s, x_c, x_e, r_s, r_e\}$$, where $x_s, x_c, x_e \in \mathbb{R}^2$ are the cubic Bézier start, control, and end points, and $r_s, r_e \in \mathbb{R}$ are the stroke radii at start and end, respectively. Fixed blending coefficients $\alpha_c$, $\alpha_s \in [0,1]$ govern the mixing ratios.

The operator applies $N+1$ “stamps” along the Bézier curve, each located at

$$x_k = (1-t_k)^2 x_s + 2(1-t_k)t_k x_c + t_k^2 x_e,\quad r_k = (1-t_k) r_s + t_k r_e,\quad t_k = k/N$$

for $k = 0, \ldots, N$. The $k$-th local stamp covers a patch $S_k(x) = I(x)\cdot 1_{\|x-x_k\| \leq r_k}$, with the indicator implemented via differentiable grid or mask sampling.

The core blending update is originally recursive: $$C_k^k(x) = \alpha_c \cdot B_{k-1}(x) + (1-\alpha_c) \cdot C_k^{k-1}(x) \ B_k(x) = \alpha_s \cdot B_{k-1}(x) + (1-\alpha_s) \cdot C_k^k(x)$$ with $C_0^0 = B_0 = S_0$; the final output is $\tau_s(x) = C_N^N(x)$.

2. Differentiable Reformulation and Gradient Propagation

The sequential recurrence impedes gradient flow to early stamps and uses non-differentiable binary masks. The differentiable smudge operator addresses these by:

• Replacing hard binary masks with smooth, differentiable grid sampling for $S_k(x)$ extraction.
• Implementing a one-shot brush initialization via Beta kernel-weighted summations along arc-length parameterization:

$$t_i = \ell_i / L\,; \quad K_{k,i} = t_i^{a-1}(1-t_i)^{b-1}\ ; \quad \bar K_{k,i} = K_{k,i}/\sum_{m=0}^k K_{k,m}$$

$B_k^0(x) = \sum_{i=0}^k \bar K_{k,i}\, C_i^0(x)$

where $a,b>0$ are kernel shape parameters, $\ell_i$ is the arc length up to $x_i$, and $L = \ell_N$.

• Using only two additional linear blending steps post-initialization:

$$C_k^k(x) = \alpha_c \cdot B_{k-1}^0(x) + (1-\alpha_c)\cdot C_k^{k-1}(x) \ B_k^k(x) = \alpha_s \cdot B_k^0(x) + (1-\alpha_s)\cdot C_k^k(x)$$

This provides smooth and efficient gradient propagation $\partial \tau_s/\partial \Theta_{\text{smudge}}$ via the differentiable grid-sampling and weighting kernels.

3. Operator Parameters and Hyperparameters

All parameters directly controlling the smudge operation conform to:

Parameter	Type/Range	Role
$x_s, x_c, x_e$	$\mathbb{R}^2$	Bézier control (learned)
$r_s, r_e$	$\mathbb{R}$	End radii (learned)
$\alpha_c$	$[0,1]$	Canvas-blend ratio (fixed)
$\alpha_s$	$[0,1]$	Brush-retention ratio (fixed)
$a, b$	$>0$	Kernel shape (fixed)
$N$	$\mathbb{N}$	Sample count (fixed, e.g., 10–100)

No additional learned weights are introduced; only stroke geometry is optimized at the smudge reconstruction stage (Jiang et al., 17 Nov 2025).

4. Algorithmic Procedure and Implementation

The differentiable smudge operator can be summarized by the following pseudocode:

StrokeBasedSmudge(Θ_smudge, I, N, α_c, α_s, a, b): // Sample along cubic Bézier for k in 0..N: t_k = k/N x_k = (1–t_k)² x_s + 2(1–t_k)t_k x_c + t_k² x_e r_k = (1–t_k) r_s + t_k r_e // Extract initial canvas patches via differentiable grid‐sample for k in 0..N: C_k^0(x) = I(x) · 𝟙_{‖x–x_k‖≤r_k} B_k^0(x) = C_k^0(x) // One‐shot brush initialization using Beta kernel compute cumulative arc‐lengths ℓ_k and total L for k in 0..N: for i in 0..k: K_{k,i} = t_i^{a−1} (1–t_i)^{b−1} normalize: \bar K_{k,i} = K_{k,i} / ∑_{m=0}^k K_{k,m} B_k^0(x) = ∑_{i=0}^k \bar K_{k,i} · C_i^0(x) // Canvas & brush update (linear blends) for k in 1..N: C_k^k(x) = α_c · B_{k−1}^0(x) + (1–α_c) · C_k^{k−1}(x) B_k^k(x) = α_s · B_k^0(x) + (1–α_s) · C_k^k(x) return τ_s(x) = C_N^N(x)

The indicator mask $𝟙_{‖x–x_k‖≤r_k}$ is realized as a differentiable mask via grid sampling (spatial transformer) with radius parameter $r_k$.

5. Integration into Stroke-by-Stroke Reconstruction Pipelines

The operator is utilized as the third phase of a staged stroke reconstruction process:

1. Stroke appearance reconstruction: Parallel, stamp-based paint renderer forms single- and dual-color Bézier strokes.
2. Texture stylization: A style generation module (e.g., StyleGAN latent $w$ per stroke) synthesizes painting-style textures conditioning on geometry.
3. Smudge stroke reconstruction: The differentiable smudge operator $\varphi_s(\Theta_{\text{smudge}}, I)$ is applied to regions with high reconstruction error. Each smudge stroke’s output is composited into the current canvas.
4. Optimization: Losses (pixel-wise $\mathcal{L}_1$, perceptual $\mathcal{L}_{\text{perc}}$, gradient-magnitude $\mathcal{L}_{\text{mag}}$, segmentation $\mathcal{L}_{\text{seg}}$, area $\mathcal{L}_{\text{area}}$) are computed between the smudged and target images. Backpropagation through $\varphi_s$ updates $\Theta_{\text{smudge}}$.
5. Resolution refinement: The process repeats on a finer grid, iteratively refining stroke and smudge geometry (Jiang et al., 17 Nov 2025).

6. Qualitative and Algorithmic Impact

Applying the differentiable smudge operator:

• Eliminates “blocking” artifacts and color banding caused by the hard abutment of dual-color strokes.
• Redistributes pigment along the Bézier path: early brush stamps absorb downstream pigment and re-deposit it, achieving gradual color gradients and soft tonal modulation.
• Enables the recovery of human-grade color blending and naturalistic highlight transitions (e.g., glazes in oil painting, watercolor washes, and ink gradations).
• Ensures that, due to full differentiability, geometric stroke parameters ($x_s, x_c, x_e, r_s, r_e$) are optimized for best match to target imagery under complex, painterly objectives.

Figures provided in (Jiang et al., 17 Nov 2025) demonstrate superior edge smoothness, realism in shading, and greater expressivity relative to non-smudge renderers.

7. Broader Significance and Connections

The differentiable smudge operator represents a modular neural primitive for end-to-end optimization of painterly processes, directly supporting the recovery of expressive image structures in an inverse rendering setting. It complements neural paint and stylization modules by bridging discrete stroke synthesis and continuous pigment blending. This approach is distinct from standard convolutional or procedural image blending in its:

• End-to-end differentiable, parameter-driven formulation,
• Emulation of the physical process of brush smudging,
• Plug-in compatibility with neural inverse rendering, and
• Direct optimization of geometric (Bézier) primitives.

A plausible implication is the extension of this operator to other differentiable media blending processes (e.g., charcoal, pastels) and broader usage in reconstruction, editing, and generative painting systems where interpretability and continuous parameter refinement are critical (Jiang et al., 17 Nov 2025).