Adaptive Anisotropic Textured Gaussians

• The paper introduces A²TG, a method that adaptively allocates anisotropic textures to 3D Gaussian splats based on local scene detail and photometric gradients.
• It integrates gradient-guided texture upscaling and anisotropy alignment within a 2D Gaussian Splatting pipeline, delivering improved PSNR and reduced LPIPS under fixed memory budgets.
• The approach efficiently pairs scene geometry with textured splats to optimize GPU memory usage and render high-fidelity details even in high-frequency regions.

Adaptive Anisotropic Textured Gaussians (A²TG) constitute a data-driven, spatially adaptive texture parameterization framework for Gaussian Splatting in 3D scene representation. The approach equips each Gaussian primitive with a learnable, anisotropic texture whose size, aspect ratio, and sampling density adapt to both the geometry and local appearance complexity. This enables efficient allocation of texture resources, aligning with splat orientation and scene detail, and facilitates high-fidelity rendering under tight memory constraints (Hsu et al., 14 Jan 2026, Wei et al., 16 Dec 2025, Xie et al., 28 Nov 2025).

1. Mathematical Model: Geometry and Texture Parameterization

The foundational primitive is a 2D Gaussian "splat" parameterized by spatial mean $\mu \in \mathbb{R}^2$, anisotropic covariance $\Sigma \in \mathbb{R}^{2 \times 2}$, and appearance attributes (RGBA texture):

$$G_i(u) = \exp\left(-\frac{1}{2}(u-\mu_i)^\top \Sigma_i^{-1}(u-\mu_i)\right)$$

Anisotropy is encoded via eigen-decomposition: $$\Sigma_i = R_i \operatorname{diag}(S_{u_i}^2, S_{v_i}^2) R_i^\top, \quad a_i = S_{u_i}/S_{v_i}$$ where $R_i \in SO(2)$ is the rotation, $S_{u_i}, S_{v_i}$ are scale factors along principal axes.

Each Gaussian stores an RGBA texture of resolution $(T_{u_i}, T_{v_i})$. Crucially, $T_{u_i}, T_{v_i}$ are determined adaptively and anisotropically, generally satisfying $T_{u_i}/T_{v_i} \approx a_i$ to align texture shape with geometric anisotropy.

Texture lookup is performed in the local (canonical) frame after mapping screen-space coordinates, rescaling: $$u' = \frac{1}{2}(u+1) T_{u_i}, \quad v' = \frac{1}{2}(v+1) T_{v_i}$$ and sampling via bilinear interpolation.

2. Adaptive Gradient-Guided Texture Growth

A²TG leverages a gradient-guided allocation strategy to determine texture resolution per primitive. This mechanism proceeds as follows:

1. Gradient Accumulation: During training, for each covered pixel $x_j$ and splat $i$, compute photometric loss gradient magnitude $g_{ij} = \|\partial L/\partial u_i(x_j)\|_2$.
2. Selection Threshold: Aggregate $G_i = \sum_j g_{ij}$; if $G_i > K_G$ (a hyperparameter), splat $i$ is marked for texture upscaling.
3. Anisotropic Upscaling Rules: Based on geometric anisotropy ($a_i$) and axis lengths ($S_{u_i}, S_{v_i}$):

• If $a_i > k_A$ and $S_{v_i} < k_s$: $T_{u_i}\leftarrow 2 T_{u_i}$ (width only)
• If $1/a_i > k_A$ and $S_{u_i} < k_s$: $T_{v_i}\leftarrow 2 T_{v_i}$ (height only)
• Otherwise: $(T_{u_i}, T_{v_i}) \leftarrow 2(T_{u_i}, T_{v_i})$

Newly allocated texels are initialized by nearest-neighbor from the old texture and immediately enter joint optimization with all other parameters. This procedure focuses resolution increases along the axes and in regions indicated by persistent rendering error, ensuring memory is concentrated on high-detail or high-visibility parts of the scene (Hsu et al., 14 Jan 2026, Wei et al., 16 Dec 2025).

3. Integration with the 2DGS Rendering Pipeline

A²TG integrates seamlessly with 2D Gaussian Splatting (2DGS) architectures:

• Forward Pass: For each pixel, candidate Gaussians are rasterized and sorted by depth. The canonical ($u,v$) coordinates are computed and mapped to texture coordinates for sampling. Adaptive sampling schemes can further warp ($u,v$) by Gaussian CDF or frequency-aware mappings to concentrate samples in high-density or high-complexity regions (Wei et al., 16 Dec 2025, Xie et al., 28 Nov 2025).
• Alpha Compositing: Colors and opacities from all Gaussians are composited in front-to-back order:

$$C(x) = \sum_i o_i G_i(\varphi_i(x)) \prod_{j<i} (1 - o_j G_j(\varphi_j(x)))$$

where $\varphi_i$ maps camera intersection to Gaussian local frame.

• Texture Management: Textures are bundled in atlases, and zero/one-pixel textures are used for "inactive" or low-contribution splats, minimizing overhead.

4. Advanced Sampling and Frequency Alignment

A complementary perspective found in FACT-GS and related works is the explicit modulation of texel density to match local signal complexity: $$\rho^\star(u,v) \propto (\|\nabla C(u,v)\| + \varepsilon)^\alpha$$ where $\|\nabla C(u,v)\|$ proxies local frequency; $\alpha\ge1$, $\varepsilon>0$. Anisotropic texture sampling is motivated by structure tensor analysis, exploiting eigen-decomposition of $\mathbf{S}(u,v)$ to fit sampling ellipsoids along dominant orientations (Xie et al., 28 Nov 2025). Textures are reparameterized via learnable deformation fields $f_i(u,v) = u + D_i(u)$, with the Jacobian $\frac{\partial f_i}{\partial u}$ modulating sampling density.

Loss terms regularize the warp toward frequency alignment, low anisotropy (to avoid degenerate stretching), and smoothness, e.g.:

• Frequency alignment: $$\mathcal{L}_{\mathrm{freq}} = \int (\det J_i(u) - \rho^\star(u))^2 du$$
• Anisotropy penalty: $$\mathcal{L}_{\mathrm{ani}} = \int (\frac{\sigma_1(u)}{\sigma_2(u)} - 1)^2 du$$
• Total Variation: $$\mathcal{L}_{\mathrm{tv}} = \int \|\nabla D_i(u)\|_1 du$$

This modulation enables sharper depiction of high-frequency, oriented structures and mitigates blur in under-sampled regions (Xie et al., 28 Nov 2025).

5. Historical Perspective: Gaussian Fields with Prescribed Orientation

Earlier work on texture synthesis by locally anisotropic Gaussian fields (LA-FBF) formalized the generation of random fields with prescribed local orientation $\theta(x)$ and fractal regularity $H$ via harmonizable integrals: $$X(x) = \int_{\mathbb{R}^2} (e^{ix\cdot\xi}-1) f^{1/2}(x, \xi) d\widehat{W}(\xi)$$ with spectral density $$f^{1/2}(x, \xi) = \mathbf{1}_{|\arg\xi-\theta(x)|\leq\alpha} \|\xi\|^{-H-1}$$ (Polisano et al., 2014, Polisano et al., 2015). Efficient simulation mechanisms including turning-band algorithms enable band-wise generation of oriented textures, rapidly synthesizing large grids via sums of fractional Brownian motions along prescribed orientations. These models laid theoretical foundations for local adaptation, variogram analysis, and simulation techniques now leveraged in learned texture architectures.

6. Quantitative Evaluation, Efficiency, and Trade-offs

A²TG demonstrates favorable memory-fidelity trade-offs across multiple benchmarks. Under fixed memory budgets, A²TG achieves higher PSNR and lower LPIPS than prior textured Gaussian approaches, owing to its per-splat, detail-driven texture allocation (see Table below, excerpted from (Hsu et al., 14 Jan 2026)). Ablations indicate that both adaptive upscaling and anisotropy materially contribute to efficiency. Many splats remain near 1×1 textures, concentrating capacity only where justified by scene complexity.

Benchmark	Memory (MB)	Method	PSNR	SSIM	LPIPS	# Splats
Mip-NeRF360	200	Textured Gaussians*	28.37	0.832	0.188	410k
Mip-NeRF360	200	A²TG	28.51	0.838	0.174	700k
Tanks & Temples	200	Textured Gaussians*	23.41	0.824	0.164	410k
Tanks & Temples	200	A²TG	23.56	0.828	0.153	690k
DeepBlending	200	Textured Gaussians*	29.51	0.897	0.198	410k
DeepBlending	200	A²TG	29.86	0.900	0.187	700k

Ablation:

• No upscaling: PSNR=25.72, Mem=23.2 MB
• No anisotropy: PSNR=26.01, Mem=31.1 MB
• Full A²TG: PSNR=26.01, Mem=29.7 MB

This suggests anisotropic upscaling saves ≈1.4 MB over isotropic square upscaling at equivalent visual quality. A²TG also scales parameter count efficiently, since many splats are not upscaled unless error thresholds motivate texture growth.

7. Limitations and Future Directions

Current A²TG frameworks focus on adaptive upscaling; automatic downscaling or texture retraction is not implemented. Selection thresholds ($K_G$, $k_A$, $k_s$) require careful tuning for optimal allocation; misconfigured thresholds can lead to under- or over-texturing. Extreme anisotropy may produce highly elongated textures, which can be suboptimal for GPU hardware. Runtime overheads (training and inference) are low, maintaining real-time rendering, but texture lookups and atlas management introduce modest additional GPU memory traffic (Hsu et al., 14 Jan 2026, Wei et al., 16 Dec 2025).

Extensions include dynamic scene adaptation, multifractional texture fields (spatially varying $H(x)$), and integration of local orientation estimates from external detectors. Theoretical and algorithmic lineage stems from LA-FBF models, with simulation algorithms demonstrating robust generation of locally oriented Gaussian textures (Polisano et al., 2014, Polisano et al., 2015). A plausible implication is that combining stochastic field theory and learnable geometric adaptation will continue to drive advances in spatially adaptive appearance modeling.