Machine Learning for Error Correction with Natural Redundancy

Abstract: The persistent storage of big data requires advanced error correction schemes. The classical approach is to use error correcting codes (ECCs). This work studies an alternative approach, which uses the redundancy inherent in data itself for error correction. This type of redundancy, called Natural Redundancy (NR), is abundant in many types of uncompressed or even compressed files. The complex structures of Natural Redundancy, however, require machine learning techniques. In this paper, we study two fundamental approaches to use Natural Redundancy for error correction. The first approach, called Representation-Oblivious, requires no prior knowledge on how data are represented or compressed in files. It uses deep learning to detect file types accurately, and then mine Natural Redundancy for soft decoding. The second approach, called Representation-Aware, assumes that such knowledge is known and uses it for error correction. Furthermore, both approaches combine the decoding based on NR and ECCs. Both experimental results and analysis show that such an integrated scheme can substantially improve the error correction performance.