VA-CDH: A Variance-Aware Method to Optimize Latency for Caching with Delayed Hits

Abstract: Caches are fundamental to latency-sensitive systems like Content Delivery Networks (CDNs) and Mobile Edge Computing (MEC). However, the delayed hit phenomenon where multiple requests for an object occur during its fetch from the remote server after a miss significantly inflates user-perceived latency. While recent algorithms acknowledge delayed hits by estimating the resulting aggregate delay, they predominantly focus on its mean value. We identify and demonstrate that such approaches are insufficient, as the real aggregate delay frequently exhibits substantial variance in the true production system, leading to suboptimal latency performance when ignored. Thus, we propose VA-CDH, a variance-aware method to optimize latency for caching with delayed hits. It employs a novel ranking function that explicitly incorporates both the empirically estimated mean and standard deviation of aggregate delay, allowing caching decisions to account for its variation. We derive the analytical distribution of aggregate delay under Poisson arrivals as a theoretical contribution, offering more statistical insight beyond the mean value. Through the simulations conducted on synthetic and real-world datasets, we show that VA-CDH reduces the total latency by 1%-6% approximately compared to state-of-the-art algorithms.