Papers
Topics
Authors
Recent
Search
2000 character limit reached

Parallel Algorithms Align with Neural Execution

Published 8 Jul 2023 in cs.LG | (2307.04049v2)

Abstract: Neural algorithmic reasoners are parallel processors. Teaching them sequential algorithms contradicts this nature, rendering a significant share of their computations redundant. Parallel algorithms however may exploit their full computational power, therefore requiring fewer layers to be executed. This drastically reduces training times, as we observe when comparing parallel implementations of searching, sorting and finding strongly connected components to their sequential counterparts on the CLRS framework. Additionally, parallel versions achieve (often strongly) superior predictive performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Neural Algorithmic Reasoning. Patterns, 2(7):100273, July 2021. ISSN 26663899. doi: 10.1016/j.patter.2021.100273. URL http://arxiv.org/abs/2105.02761. arXiv:2105.02761 [cs, math, stat].
  2. Neural Execution Engines: Learning to Execute Subroutines. Technical Report arXiv:2006.08084, arXiv, October 2020. URL http://arxiv.org/abs/2006.08084. arXiv:2006.08084 [cs, stat] type: article.
  3. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, pages 22084–22102. PMLR, 2022a.
  4. A step towards neural genome assembly. arXiv preprint arXiv:2011.05013, 2020.
  5. Reasoning-modulated representations. In Learning on Graphs Conference, pages 50–1. PMLR, 2022b.
  6. Narti: Neural algorithmic reasoning for trajectory inference.
  7. A generalist neural algorithmic learner. In Learning on Graphs Conference, pages 2–1. PMLR, 2022.
  8. Neural Algorithmic Reasoning with Causal Regularisation, February 2023. URL http://arxiv.org/abs/2302.10258. arXiv:2302.10258 [cs, stat].
  9. Recent advances in convolutional neural network acceleration. Neurocomputing, 323:37–51, 2019.
  10. An evaluation of edge tpu accelerators for convolutional neural networks. arXiv e-prints, pages arXiv–2102, 2021.
  11. Andreas Loukas. What graph neural networks cannot learn: depth vs width, January 2020. URL http://arxiv.org/abs/1907.03199. arXiv:1907.03199 [cs, stat].
  12. Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228, 2015.
  13. Differentiable sorting networks for scalable sorting and ranking supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8546–8555. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/petersen21a.html.
  14. Neural shuffle-exchange networks - sequence processing in o(n log n) time. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/9001ca429212011f4a4fda6c778cc318-Paper.pdf.
  15. What Can Neural Networks Reason About? Technical Report arXiv:1905.13211, arXiv, February 2020. URL http://arxiv.org/abs/1905.13211. arXiv:1905.13211 [cs, stat] type: article.
  16. Neural Execution of Graph Algorithms, January 2020. URL http://arxiv.org/abs/1910.10593. arXiv:1910.10593 [cs, stat].
  17. Efficient parallel algorithms. Cambridge Univ. Press, Cambridge, reprinted edition, 1990. ISBN 978-0-521-38841-2 978-0-521-34585-9.
  18. Limits to parallel computation: P-completeness theory. Oxford University Press, New York, 1995. ISBN 978-0-19-508591-4.
  19. Behrooz Parhami. Introduction to parallel processing: algorithms and architectures. Springer Science & Business Media, 2006.
  20. Pointer graph networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2232–2244. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/176bf6219855a6eb1f3a30903e34b6fb-Paper.pdf.
  21. Algorithmic Concept-Based Explainable Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6):6685–6693, June 2022. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v36i6.20623. URL https://ojs.aaai.org/index.php/AAAI/article/view/20623.
  22. Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Tasks, March 2023. URL http://arxiv.org/abs/2211.00692. arXiv:2211.00692 [cs].
  23. A Nico Habermann. Parallel neighbor-sort (or the glory of the induction principle). 1972.
  24. On Identifying Strongly Connected Components in Parallel. In José Rolim, editor, Parallel and Distributed Processing, volume 1800, pages 505–511. Springer Berlin Heidelberg, Berlin, Heidelberg, 2000. ISBN 978-3-540-67442-9 978-3-540-45591-2. doi: 10.1007/3-540-45591-4_68. URL http://link.springer.com/10.1007/3-540-45591-4_68. Series Title: Lecture Notes in Computer Science.
  25. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. Advances in Neural Information Processing Systems, 35:20232–20242, 2022.
  26. Why is everyone training very deep neural network with skip connections? IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2022. doi: 10.1109/TNNLS.2021.3131813.
  27. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263–1272. PMLR, 2017.
  28. Deep sets. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/f22e4747da1aa27e363d86d40ff442fe-Paper.pdf.
  29. Graph Attention Networks, February 2018. URL http://arxiv.org/abs/1710.10903. arXiv:1710.10903 [cs, stat].
  30. Graph Neural Networks are Dynamic Programmers. Technical Report arXiv:2203.15544, arXiv, June 2022. URL http://arxiv.org/abs/2203.15544. arXiv:2203.15544 [cs, math, stat] type: article.
  31. John H. Reif. Depth-first search is inherently sequential. Information Processing Letters, 20(5):229–234, June 1985. ISSN 00200190. doi: 10.1016/0020-0190(85)90024-9. URL https://linkinghub.elsevier.com/retrieve/pii/0020019085900249.
Citations (7)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up