Efficient disk spilling for fully concurrent (non-partitioned) hash aggregation

Develop an efficient method to spill fully concurrent (non-partitioned) hash-based GROUP BY aggregation—implemented using a shared global hash table and a vector of partial aggregates—to disk so that such operators can handle datasets that exceed main-memory capacity without resorting to partitioning.

Background

The paper studies fully concurrent GROUP BY aggregation where all threads share a global hash table for ticketing and then update partial aggregates. One update strategy entails thread-local partial aggregate vectors that are merged at the end. While this avoids contention, it incurs memory overhead that grows with both the number of threads and the number of distinct keys.

Unlike partitioned aggregation, which has well-established external-memory strategies, the authors note that for non-partitioned (fully concurrent) hash aggregation there is no established approach to spill intermediate state to disk. This limitation is highlighted as a practical obstacle when datasets or key cardinalities exceed available memory.