2000 character limit reached
Interpreting Deepcode, a learned feedback code
Published 26 Apr 2024 in cs.IT, eess.SP, and math.IT | (2404.17519v2)
Abstract: Deep learning methods have recently been used to construct non-linear codes for the additive white Gaussian noise (AWGN) channel with feedback. However, there is limited understanding of how these black-box-like codes with many learned parameters use feedback. This study aims to uncover the fundamental principles underlying the first deep-learned feedback code, known as Deepcode, which is based on an RNN architecture. Our interpretable model based on Deepcode is built by analyzing the influence length of inputs and approximating the non-linear dynamics of the original black-box RNN encoder. Numerical experiments demonstrate that our interpretable model -- which includes both an encoder and a decoder -- achieves comparable performance to Deepcode while offering an interpretation of how it employs feedback for error correction.
- H. Kim, Y. Jiang, S. Kannan, S. Oh, and P. Viswanath, “Deepcode: Feedback codes via deep learning,” Advances in neural information processing systems, vol. 31, 2018.
- A. R. Safavi, A. G. Perotti, B. M. Popovic, M. B. Mashhadi, and D. Gunduz, “Deep extended feedback codes,” arXiv preprint arXiv:2105.01365, 2021.
- M. B. Mashhadi, D. Gunduz, A. Perotti, and B. Popovic, “Drf codes: Deep snr-robust feedback codes,” arXiv preprint arXiv:2112.11789, 2021.
- Y. Shao, E. Ozfatura, A. Perotti, B. Popovic, and D. Gündüz, “Attentioncode: Ultra-reliable feedback codes for short-packet communications,” IEEE Transactions on Communications, 2023.
- E. Ozfatura, Y. Shao, A. G. Perotti, B. M. Popović, and D. Gündüz, “All you need is feedback: Communication with block attention feedback codes,” IEEE Journal on Selected Areas in Information Theory, vol. 3, no. 3, pp. 587–602, 2022.
- H. Kim, Y. Jiang, S. Kannan, S. Oh, and P. Viswanath, “Deepcode and modulo-sk are designed for different settings,” arXiv preprint arXiv:2008.07997, 2020.
- A. Ben-Yishai and O. Shayevitz, “Simple modulo can significantly outperform deep learning-based deepcode,” arXiv preprint arXiv:2008.01686, 2020.
- J. Kim, T. Kim, D. Love, and C. Brinton, “Robust non-linear feedback coding via power-constrained deep learning,” arXiv preprint arXiv:2304.13178, 2023.
- J. Schalkwijk and T. Kailath, “A coding scheme for additive noise channels with feedback–i: No bandwidth constraint,” IEEE Transactions on Information Theory, vol. 12, no. 2, pp. 172–182, 1966.
- O. Shayevitz and M. Feder, “Optimal feedback communication via posterior matching,” IEEE Transactions on Information Theory, vol. 57, no. 3, pp. 1186–1222, 2011.
- A. Ben-Yishai and O. Shayevitz, “Interactive schemes for the awgn channel with noisy feedback,” IEEE Transactions on Information Theory, vol. 63, no. 4, pp. 2409–2427, 2017.
- Z. Chance and D. J. Love, “Concatenated coding for the awgn channel with noisy feedback,” IEEE Transactions on Information Theory, vol. 57, no. 10, pp. 6633–6649, 2011.
- R. Mishra, D. Vasal, and H. Kim, “Linear coding for awgn channels with noisy output feedback via dynamic programming,” IEEE Transactions on Information Theory, 2023.
- M. Lian, C. Häger, and H. D. Pfister, “What can machine learning teach us about communications?” in 2018 IEEE Information Theory Workshop (ITW). IEEE, 2018, pp. 1–5.
- D. Sussillo and O. Barak, “Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks,” Neural computation, vol. 25, no. 3, pp. 626–649, 2013.
- A. Karpathy, J. Johnson, and L. Fei-Fei, “Visualizing and understanding recurrent networks,” arXiv preprint arXiv:1506.02078, 2015.
- Y. J. Choe, J. Shin, and N. Spencer, “Probabilistic interpretations of recurrent neural networks,” Probabilistic Graphical Models, 2017.
- S. Jain and B. C. Wallace, “Attention is not explanation,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Papers), 2019, pp. 3543–3556.
- S. Wiegreffe and Y. Pinter, “Attention is not not explanation,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, 2019, pp. 11–20.
- N. Devroye, N. Mohammadi, A. Mulgund, H. Naik, R. Shekhar, G. Turán, Y. Wei, and M. Žefran, “Interpreting deep-learned error-correcting codes,” in 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022, pp. 2457–2462.
- A. Mulgund, R. Shekhar, N. Devroye, G. Turán, and M. Žefran, “Evaluating interpretations of deep-learned error-correcting codes,” in 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 2022, pp. 1–8.
- N. Devroye, A. Mulgund, R. Shekhar, G. Turán, M. Žefran, and Y. Zhou, “Interpreting training aspects of deep-learned error-correcting codes,” in 2023 IEEE International Symposium on Information Theory (ISIT). IEEE, 2023, pp. 2374–2379.
- A. Mulgund, N. Devroye, G. Turán, and M. Žefran, “Decomposing the training of deep learned turbo codes via a feasible map decoder,” in 2023 12th International Symposium on Topics in Coding (ISTC). IEEE, 2023, pp. 1–5.
- Y. Jiang, H. Kim, H. Asnani, S. Kannan, S. Oh, and P. Viswanath, “Turbo autoencoder: Deep learning based channel codes for point-to-point communication channels,” Advances in neural information processing systems, vol. 32, 2019.
- S. Lall, J. E. Marsden, and S. Glavaški, “A subspace approach to balanced truncation for model reduction of nonlinear control systems,” International Journal of Robust and Nonlinear Control: IFAC-Affiliated Journal, vol. 12, no. 6, pp. 519–535, 2002.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.