Cardinalities estimation under sliding time window by sharing HyperLogLog Counter

Abstract: Cardinalities estimation is an important research topic in network management and security. How to solve this problem under sliding time window is a hot topic. HyperLogLog is a memory efficient algorithm work under a fixed time window. A sliding version of HyperLogLog can work under sliding time window by replacing every counter of HyperLogLog with a list of feature possible maxim (LFPM). But LFPM is a dynamic structure whose size is variable at running time. This paper proposes a novel counter for HyperLogLog which consumes smaller size of memory than that of LFPM. Our counter is called bit distance recorder BDR, because it maintains the distance of every left most "1" bit position. The size of BDR is fixed. Based on BDR, we design a multi hosts' cardinalities estimation algorithm under sliding time window, virtual bit distance recorder VBDR. VBDR allocate a virtual vector of BDR for every host and every physical BDR is shared by several hosts to improve the memory usage. After a small modifcation, we propose another two parallel versions of VBDR which can run on GPU to handle high speed traffic. One of these parallel VBDR is fast in IP pair scanning and the other one is memory efficient. BDR is also suitable for other cardinality estimation algorithms such as PCSA, LogLog.