LatentLens: Unified Latent-Space Imaging & Interpretability

• LatentLens is a framework that unifies latent-space imaging, reconstruction, and interpretability by integrating optics, machine learning, and vision–language models.
• It employs methods like generative low-light reconstruction, latent space compression, and neural interpretability to enhance imaging performance in scenarios such as aberration correction and privacy-preserving identification.
• Practical implementations demonstrate significant improvements in metrics like PSNR and compression ratios, while also revealing challenges in dataset scaling and calibration for robust real-world usage.

LatentLens encompasses a set of imaging and interpretability methodologies that unify latent-space representations from optics, machine learning, and vision–LLM interpretability. The term "LatentLens" has been independently applied to (1) generative lensless image reconstruction frameworks leveraging physical and neural priors, (2) camera architectures that encode directly into the latent space of generative models for high-efficiency sensing, (3) interpretability tools for mapping visual tokens in VLMs to textual descriptions at every layer, (4) latent representations for blind lens aberration correction, and (5) privacy-preserving identification using learnable lensless masks. Across these domains, the defining theme is the use of latent variable structures and learned embeddings as the primary lens for image formation, inversion, or semantic analysis.

1. Generative Approaches for Lensless Imaging Under Low Light

LatentLens in the context of low-light lensless imaging denotes a two-stage image reconstruction pipeline that integrates model-driven and data-driven mechanisms to address photon-starved, noisy conditions (Liu et al., 7 Jan 2025). The forward acquisition is governed by

$b = Hx + n$

where $b$ denotes the measurement vector, $H$ is the lensless imaging operator (convolutional with calibrated PSF), $x$ is the unknown scene, and $n$ encompasses Poisson, Gaussian, and quantization noise.

Model-Driven Initialization

A learnable Wiener filtering module in the Fourier domain produces an initial estimate:

$$x_{\text{init}} = F^{-1}\{ F(b) \odot [ \overline{F(h)} / ( |F(h)|^2 + \lambda ) ] \}$$

where $F$ denotes DFT, $h$ is the learnable PSF, and $\lambda$ a trainable regularizer. This step extracts the range space component of the measurement, robustly attenuating high-frequency noise while recovering low-frequency scene content.

Latent-Driven Conditional Diffusion

Subsequent refinement decomposes $x_{\text{init}}$ into wavelet sub-bands via two-level 2D Haar DWT, separating low-frequency global structure (LL) from high-frequency details (LH, HL, HH). For the LL band, a conditional generative diffusion process (U-Net architecture, ResNet blocks, cross-attention) denoises in latent space, conditioned on $x_{\text{init}}$'s LL coefficient. High-frequency subbands are processed by a lightweight, depth-separable convolutional network. Bidirectional diffusion training ensures stability and semantic fidelity.

The overall loss combines diffusion noise prediction ($L_1$), reverse-process image fidelity ($L_2$), and HF reconstruction ($L_3$), weighted with perceptual and structural terms. Simulation and experiments demonstrate substantial improvements over ADMM, FlatNet, MWDN, and DeepLIR, especially in low-light, high-noise regimes (e.g., MSE reductions from 0.1371 to 0.0071 and PSNR gains from 8.76 dB to 22.02 dB on real datasets).

2. Latent Space Imaging: Compressing Sensing into Generative Model Latents

LatentLens as an imaging system denotes a paradigm in which image formation is engineered to yield measurements that are directly mapped—via learned optics and an MLP encoder—into the latent space of a pre-trained generative model (Souza et al., 2024).

Imaging Pipeline and Mathematical Structure

A programmable SLM (e.g., DMD) forms binary masks $o_j \in \{0,1\}^{m \times n}$, amplitude-modulating the scene. A single-pixel sensor integrates the masked scene, yielding

$y_j = \langle o_j, I \rangle$

across $d$ masks, resulting in a measurement vector $y \in \mathbb{R}^d$. The measurement operator $M$ is strictly linear and binary.

A digital encoder $D:\mathbb{R}^d \to \mathbb{R}^{512 \times 18}$ (for StyleGAN2 $W^+$ latent) is trained to minimize

$$\mathcal{L}_{\text{total}} = \lambda_{\text{lat}} \| D(Mx) - E(x) \|_1 + \lambda_{\text{id}} [1 - \langle \phi(\hat{x}), \phi(x) \rangle ] + \lambda_{2} \| \hat{x} - x \|_2^2 + \lambda_{\text{pips}} \text{LPIPS}(\hat{x}, x) + \lambda_{\text{energy}} \mathcal{L}_{\text{energy}}$$

where $E$ is a GAN inversion encoder, $\phi$ is a face embedding, and $\mathcal{L}_{\text{energy}}$ regularizes mask energy.

Compression and Efficiency

This setup achieves compression ratios up to 1:1000 for reconstructive applications ($d=64$ vs. $mn=65536$), and as high as 1:16384 for downstream classification. Reconstruction quality (e.g., PSNR ∼ 20–25 dB, SSIM ∼ 0.6 for $d=256$) is strong given the extreme compression, with identity similarity preserved (curricularface cosine ∼ 0.22–0.32). The system also admits high-speed extensions due to mask switching rates, with $d=64$ measurements acquired in <2 ms (enabling 500 Hz video rates).

3. LatentLens for Visual Token Interpretability in VLMs

LatentLens further refers to a method for probing the interpretability of visual tokens within LLM-based VLMs (Krojer et al., 31 Jan 2026). It provides full-sentence semantic descriptions of visual patch representations at all layers of a VLM.

Workflow

• Vision encoder (e.g., CLIP ViT) produces patch embeddings $v_i \in \mathbb{R}^{d_v}$, which are mapped to the LLM embedding space $\mathbb{R}^d$ by a shallow MLP connector:

$h_i^{(0)} = W_2\, \sigma(W_1 v_i + b_1) + b_2$

• The projected visual tokens are prepended to textual tokens and processed through the frozen LLM, yielding activations $h_i^{(\ell)}$.
• For interpretability, a large pool of contextualized text token representations is extracted from the same LLM applied to ∼3 M Visual Genome captions. For each visual activation, top-$k$ nearest neighbors are found among these contextual embeddings by cosine similarity. The matching sentence contexts provide a natural language description of the visual token.

Empirical Findings

• Compared to LogitLens and EmbeddingLens, which operate on the unembedding matrix and raw input embeddings respectively, LatentLens finds 72% of visual tokens interpretable (vs. 23% for LogitLens) across 9 VLMs × 9 layers × 100 patches.
• Nearest-neighbor retrieval in LatentLens consistently yields semantically precise, context-rich phrases (e.g., “cushions on the couch are deep and plush” for a couch patch). The method illuminates the high alignment between projected visual tokens and mid-layer contextualized linguistic representations, evidenced by the “mid-layer leap” phenomenon: even layer-0 visual tokens align to layer 8–16 contextual text in the LLM.

4. Latent Point Spread Function Representations for Aberration Correction

OmniLens++ introduces a latent Point Spread Function Representation (LPR) for scalable and generalizable blind lens aberration correction (Jiang et al., 21 Nov 2025). Here, LatentLens denotes the VQ-VAE-based latent codebook learned over a large LensLib of spatially varying PSFs.

Optical Degradation and LPR

Given an imaging model $$y(u,v) = \sum_{i,j} h_{u,v}[i,j] x[u-i, v-j] + n(u,v)$$, the ISP F map of PSFs is encoded by a vector-quantized VAE, learning codebook atoms $e_k \in \mathbb{R}^{n_z}$. The quantized latent PSF map $z_q$ is then fused with image features in a U-Net backbone for generative correction.

Dataset Scaling and Generalization

AODLibpro, a richly sampled synthetic lens aberration dataset (3600 training, 54 test lenses crossing severity and spatial-variation classes), enables the LPR to cover real-world optical degradation statistics. Ablation studies show performance gains with LPR increase with dataset scale, and FoundCAC (with LPR) achieves best-in-class PSNR/SSIM/LPIPS on both synthetic and real-world aberration data.

5. Privacy-Preserving Identification via Learnable Lensless Masks

A separate instantiation of LatentLens employs learned lensless masks for visual privacy in face identification systems (Canh et al., 2023). The system uses a binary planar mask $H$ (learned via proxy $W$), convolving with the image to acquire a blurred measurement $y = H * x + \eta$. Instead of reconstructing an image, the measurement is directly input to a CNN classifier (ResNet-18).

Losses and Privacy Metrics

To ensure human-imperceptibility while maintaining recognition accuracy, the total loss includes (i) cross-entropy for recognition, (ii) similarity to a maximally blurred reference, (iii) total variation (promoting broad support), (iv) invertibility penalty (enlarged aperture hampers deblurring), and (v) restricted isometry property (RIP) to minimize invertibility.

Performance is evaluated via machine accuracy, AUC-RIP, and subjective human verification tests. Machine identification accuracy remains within 2–5% of the pinhole upper bound, but human subject accuracy on real prototypes drops to chance (35–45%), confirming strong visual privacy.

6. Practical Implementation and Comparative Evaluation

Across diverse instantiations, LatentLens frameworks employ task-specific optimization strategies, training protocols, and hardware architectures.

Training Regimes and Hardware

• Generative low-light imaging employs Adam optimizer, learning rates decayed every 100 epochs, and batch sizes of 22 on 2×RTX 3090 GPUs (Liu et al., 7 Jan 2025).
• Latent space imaging involves mask and MLP co-training via joint minimization over perceptual, identity, and energy losses (Souza et al., 2024).
• LPR-based aberration correction leverages a two-stage freezing and fine-tuning procedure, with codebooks of size 1024 sufficient for optimal performance (Jiang et al., 21 Nov 2025).
• Hardware prototypes have included DMD–photodiode single-pixel systems, amplitude-mask lensless cameras, and CMOS sensor arrays with photolithographically patterned binary masks.

Quantitative Benchmarks

System/Modality	PSNR (dB)	SSIM	LPIPS	Application
LatentLens (2501)	18.83+	0.57+	0.16-	Low-light lensless imaging
LatentLens (2407)	20–25	0.6	—	Sensing-to-latent compression
FoundCAC (2511)	25.85–28.67	0.85+	0.12–0.18	Aberration correction
LwC-RIP (2302)	91–95%	—	—	Privacy-preserving identification

Values representative; see referenced figures and tables for full metrics.

Comparative Advantages

• LatentLens generative approaches outperform classical ADMM and recent neural architectures under severe low-light and noise.
• Latent space imaging attains extreme compression with direct semantic reconstruction, departing from pixel-based measurement paradigms.
• Latent interpretability probes reveal a higher degree of semantic structure in VLMs than suggested by prior unembedding-based methods.

7. Limitations and Prospective Directions

While LatentLens advances a unified latent-space-centric perspective, open challenges remain:

• Generative low-light pipelines require controlled training datasets and hardware recalibration to generalize to uncontrolled, outdoor settings.
• Latent space imaging currently focuses on faces and StyleGAN; broadening to generic scenes and other generative families would require latent structure adaptation.
• Blind aberration correction via LPR necessitates large-scale PSF libraries; performance degrades at small data scales.
• Privacy-preserving lensless imaging is presently evaluated in simple identification settings; scaling to broader biometric or cross-domain tasks is untested.
• Visual token interpretability metrics hinge on the representativeness of the reference corpus and the quality of contextual embeddings; extensions to multi-modal or temporally coherent contexts are potential directions.

Continued development of co-designed optics, scalable training datasets (e.g., AODLibpro), and task-conditioned latent mappings is suggested to further the reach and robustness of LatentLens-based imaging and interpretability systems (Liu et al., 7 Jan 2025, Souza et al., 2024, Krojer et al., 31 Jan 2026, Jiang et al., 21 Nov 2025, Canh et al., 2023).