Ascertain correctness of PRNG implementations across machine learning frameworks

Ascertain whether the pseudorandom number generator implementations used in Python, NumPy (Generator with MT, Philox, and PCG), TensorFlow (Philox-based Generator), and PyTorch correctly implement their claimed algorithms by developing a reliable verification procedure that remains valid even when identical seeds do not yield identical sequences across technologies due to differing seed-to-state initialization functions.

Background

The study finds that identical seeds fed to implementations of the same pseudorandom number generator (Mersenne Twister, PCG, Philox) in different technologies (C, Python, NumPy, TensorFlow, PyTorch) can produce different sequences, likely due to differences in seed-to-state initialization functions. This loss of numerical reproducibility complicates determining whether a given implementation faithfully conforms to the intended algorithm.

The authors note that, beyond Mersenne Twister which provides reference outputs, it is difficult to verify correctness without consistent numerical reproducibility across platforms. They suggest that original authors supplying reference output samples would help, highlighting a need for rigorous, portable conformance testing methods for PRNGs used in machine learning frameworks.

References

From what has been discovered comes more questions: if the loss of reproducibility does not come from the seeding functions, this leads us to a critical inquiry: how can we ascertain that the algorithm in use is a correct implementation of the generator? For us, this is an open question.

— Reproducibility, energy efficiency and performance of pseudorandom number generators in machine learning: a comparative study of python, numpy, tensorflow, and pytorch implementations  (2401.17345 - Antunes et al., 2024) in Section 5 DISCUSSIONS