Reducing Storage in Large-Scale Photo Sharing Services using Recompression

Abstract: The popularity of photo sharing services has increased dramatically in recent years. Increases in users, quantity of photos, and quality/resolution of photos combined with the user expectation that photos are reliably stored indefinitely creates a growing burden on the storage backend of these services. We identify a new opportunity for storage savings with application-specific compression for photo sharing services: photo recompression. We explore new photo storage management techniques that are fast so they do not adversely affect photo download latency, are complementary to existing distributed erasure coding techniques, can efficiently be converted to the standard JPEG user devices expect, and significantly increase compression. We implement our photo recompression techniques in two novel codecs, ROMP and L-ROMP. ROMP is a lossless JPEG recompression codec that compresses typical photos 15% over standard JPEG. L-ROMP is a lossy JPEG recompression codec that distorts photos in a perceptually un-noticeable way and typically achieves 28% compression over standard JPEG. We estimate the benefits of our approach on Facebook's photo stack and find that our approaches can reduce the photo storage by 0.3-0.9x the logical size of the stored photos, and offer additional, collateral benefits to the photo caching stack, including 5-11% fewer requests to the backend storage, 15-31% reduction in wide-area bandwidth, and 16% reduction in external bandwidth.