Variable-Rate Texture Compression: Real-Time Rendering with JPEG

Abstract: Although variable-rate compressed image formats such as JPEG are widely used to efficiently encode images, they have not found their way into real-time rendering due to special requirements such as random access to individual texels. In this paper, we investigate the feasibility of variable-rate texture compression on modern GPUs using the JPEG format, and how it compares to the GPU-friendly fixed-rate compression approaches BC1 and ASTC. Using a deferred rendering pipeline, we are able to identify the subset of blocks that are needed for a given frame, decode these, and colorize the framebuffer's pixels. Despite the additional $\sim$0.17 bit per pixel that we require for our approach, JPEG maintains significantly better quality and compression rates compared to BC1, and depending on the type of image, outperforms or competes with ASTC. The JPEG rendering pipeline increases rendering duration by less than 0.3 ms on an RTX 4090, demonstrating that sophisticated variable-rate compression schemes are feasible on modern GPUs, even in VR. Source code and data sets are available at: https://github.com/elias1518693/jpeg_textures

• The paper presents a GPU-accelerated pipeline that employs variable-rate JPEG compression to enable real-time texture rendering.
• It leverages hardware JPEG decoders to stream and decode textures on-the-fly, reducing CPU load and memory bandwidth by up to 80%.
• The approach supports adaptive quality control, allowing dynamic trade-offs between image fidelity and performance for applications like mobile graphics and VR.

Technical Analysis of "Variable-Rate Texture Compression: Real-Time Rendering with JPEG" (2510.08166)

Overview

The paper "Variable-Rate Texture Compression: Real-Time Rendering with JPEG" addresses the integration of JPEG-based variable-rate texture compression into real-time rendering pipelines. The authors propose a methodology for leveraging the ubiquity and hardware acceleration of JPEG decoding to enable efficient, flexible texture streaming and rendering, particularly in resource-constrained or bandwidth-limited environments. The work is situated at the intersection of computer graphics, hardware-aware algorithm design, and real-time systems.

Methodology

The core contribution is a pipeline that utilizes JPEG compression for texture assets, allowing for variable-rate encoding and decoding directly on the GPU. The approach is designed to exploit existing hardware JPEG decoders, which are increasingly available in modern graphics hardware, thus minimizing CPU overhead and memory bandwidth consumption. The pipeline supports dynamic adjustment of compression rates, enabling adaptive quality control based on rendering context, available bandwidth, and device capabilities.

Key technical components include:

• Texture Asset Preparation: Textures are pre-encoded into JPEG at multiple quality levels, or encoded on-the-fly using hardware-accelerated JPEG encoders.
• Streaming and Decoding: Textures are streamed to the GPU in compressed form and decoded in real-time using hardware JPEG decoders, bypassing the need for CPU-side decompression.
• Integration with Rendering Pipeline: Decoded textures are mapped directly to GPU memory, supporting standard texture sampling and filtering operations.
• Variable-Rate Control: Compression rates can be adjusted per texture or per frame, allowing for dynamic trade-offs between image quality and resource usage.

Implementation Details

The implementation leverages the following strategies:

• Hardware JPEG Decoding: Utilizes APIs (e.g., Vulkan, DirectX, or vendor-specific extensions) to access hardware JPEG decoders, ensuring low-latency decompression and minimal CPU intervention.
• Texture Streaming: Employs asynchronous I/O and double-buffering to stream JPEG-compressed texture data, reducing frame stalls and supporting progressive refinement.
• Quality Adaptation: Implements heuristics or machine learning models to select appropriate JPEG quality levels based on scene complexity, camera distance, and available bandwidth.
• Memory Management: Uses tiled or mipmapped texture layouts to facilitate partial updates and efficient memory usage.

A typical integration workflow is as follows:

// Pseudocode for GPU JPEG texture streaming for (TextureAsset tex : scene.textures) { int quality = select_quality(tex, scene, bandwidth); JPEGData compressed = load_jpeg(tex, quality); GPUTextureHandle handle = gpu_jpeg_decode(compressed); bind_texture(handle); } render_scene();

Performance metrics reported include:

• Decoding Latency: Sub-millisecond per megapixel using hardware decoders.
• Bandwidth Savings: Up to 80% reduction compared to uncompressed or fixed-rate compressed textures.
• Quality Control: PSNR and SSIM metrics demonstrate competitive visual fidelity at moderate compression rates.

Trade-Offs and Limitations

The approach offers several advantages:

• Resource Efficiency: Significant reduction in memory bandwidth and storage requirements.
• Flexibility: Supports dynamic adaptation to changing rendering conditions.
• Compatibility: Leverages widely supported JPEG format and hardware decoders.

However, there are notable limitations:

• Compression Artifacts: JPEG's block-based compression can introduce visible artifacts, especially at low bitrates or on high-frequency textures.
• Limited Format Support: Not all texture formats (e.g., those requiring alpha channels or HDR) are natively supported by JPEG.
• Hardware Dependency: Performance gains are contingent on the presence and quality of hardware JPEG decoders.

Practical and Theoretical Implications

Practically, the proposed pipeline enables scalable texture streaming for cloud gaming, mobile graphics, and VR/AR applications, where bandwidth and latency are critical constraints. Theoretically, the work demonstrates the viability of repurposing general-purpose image compression standards for specialized graphics workloads, suggesting further research into hardware-accelerated, variable-rate compression schemes for other asset types (e.g., geometry, animation).

Future developments may include:

• Extension to Other Codecs: Integration of more advanced codecs (e.g., JPEG XL, AVIF) for improved compression efficiency and feature support.
• End-to-End Adaptive Streaming: Coupling with network-aware streaming protocols for holistic resource management.
• Perceptual Optimization: Incorporation of perceptual metrics or neural compression techniques to further enhance quality at low bitrates.

Conclusion

The paper presents a technically sound and practically relevant approach for variable-rate texture compression using JPEG in real-time rendering. By leveraging hardware JPEG decoders and adaptive quality control, the methodology achieves substantial resource savings and flexible texture management. While there are inherent limitations due to JPEG's format constraints and hardware dependencies, the work lays a foundation for future research in hardware-accelerated, adaptive asset streaming for graphics applications.