Papers
Topics
Authors
Recent
Search
2000 character limit reached

The Power of Linear Recurrent Neural Networks

Published 9 Feb 2018 in cs.LG and cs.NE | (1802.03308v9)

Abstract: Recurrent neural networks are a powerful means to cope with time series. We show how autoregressive linear, i.e., linearly activated recurrent neural networks (LRNNs) can approximate any time-dependent function f(t). The approximation can effectively be learned by simply solving a linear equation system; no backpropagation or similar methods are needed. Furthermore, and this is the main contribution of this article, the size of an LRNN can be reduced significantly in one step after inspecting the spectrum of the network transition matrix, i.e., its eigenvalues, by taking only the most relevant components. Therefore, in contrast to other approaches, we do not only learn network weights but also the network architecture. LRNNs have interesting properties: They end up in ellipse trajectories in the long run and allow the prediction of further values and compact representations of functions. We demonstrate this by several experiments, among them multiple superimposed oscillators (MSO), robotic soccer (RoboCup), and stock price prediction. LRNNs outperform the previous state-of-the-art for the MSO task with a minimal number of units.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. Fitting autoregressive models for prediction. Annals of the Institute of Statistical Mathematics 21, 243–247. URL: http://link.springer.com/content/pdf/10.1007/BF02532251.pdf.
  2. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5, 157–166. URL: http://doi.org/10.1109/72.279181.
  3. Training a 3-node neural network is NP-complete. Neural Networks 5, 117–127. URL: http://doi.org/10.1016/S0893-6080(05)80010-3.
  4. Language models are few-shot learners, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (Eds.), Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pp. 1877–1901. URL: http://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  5. Encoding-based memory for recurrent neural networks. Neurocomputing 456, 407–420. URL: http://doi.org/10.1016/j.neucom.2021.04.051.
  6. Users Manual: RoboCup Soccer Server – for Soccer Server Version 7.07 and Later. The RoboCup Federation. URL: http://helios.hampshire.edu/jdavila/cs278/virtual_worlds/robocup_manual-20030211.pdf.
  7. Dynamical Systems and Linear Algebra. volume 158 of Graduate Studies in Mathematics. American Mathematical Society, Providence, Rhode Island. URL: http://doi.org/10.1090/gsm/158.
  8. The asymptotic performance of linear echo state neural networks. Journal of Machine Learning Research 17, 1–35. URL: http://jmlr.org/papers/v17/16-076.html.
  9. Fast linear algebra is stable. Numerische Mathematik 108, 59–91. URL: http://doi.org/10.1007/s00211-007-0114-x.
  10. Deep learning: Methods and applications. Foundations and Trends in Signal Processing 7, 198–387. URL: http://research.microsoft.com/pubs/209355/DeepLearning-NowPublishing-Vol7-SIG-039.pdf.
  11. GNU Octave – A High-Level Interactive Language for Numerical Computations. URL: http://www.octave.org/. edition 4 for Octave version 4.2.1.
  12. Finding structure in time. Cognitive Science 14, 179–211. URL: http://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1402_1.
  13. Progress in RoboCup revisited: The state of soccer simulation 2D, in: Behnke, S., Sheh, R., Sariel, S., Lee, D.D. (Eds.), RoboCup 2016: Robot Soccer World Cup XX. RoboCup International Symposium, Springer Nature Switzerland, Leipzig. pp. 144–156. URL: http://doi.org/10.1007/978-3-319-68792-6_12.
  14. Solving number series with simple recurrent networks, in: Ferrández de Vicente, J.M., Álvarez Sánchez, J.R., de la Paz López, F., Toledo-Moreo, F.J. (Eds.), Natural and Artificial Models in Computation and Biology – 5th International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2013. Proceedings, Part I, Springer. pp. 412–420. URL: http://doi.org/10.1007/978-3-642-38637-4_43.
  15. Deep Learning. Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, London. URL: http://www.deeplearningbook.org.
  16. Minimum spanning trees and single linkage cluster analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics) 18, 54–64. URL: http://doi.org/10.2307/2346439.
  17. On the approximation capability of recurrent neural networks. Neurocomputing 31, 107–123. URL: http://doi.org/10.1016/S0925-2312(99)00174-5.
  18. Darts: User-friendly modern machine learning for time series. Journal of Machine Learning Research 23, 1–6. URL: http://jmlr.org/papers/v23/21-1177.html.
  19. MatLab Guide. 3rd ed., Siam, Philadelphia, PA. URL: http://bookstore.siam.org/ot150/.
  20. Long short-term memory. Neural Computation 9, 1735–1780. URL: http://doi.org/10.1162/neco.1997.9.8.1735.
  21. Matrix Analysis. 2nd ed., Cambridge University Press, New York, NY. URL: http://www.cse.zju.edu.cn/eclass/attachments/2015-10/01-1446086008-145421.pdf.
  22. Approximation capabilities of multilayer feedforward networks. Neural Networks 4, 251–257. URL: http://doi.org/10.1016/0893-6080(91)90009-T.
  23. State-frequency memory recurrent neural networks, in: Precup, D., Teh, Y.W. (Eds.), Proceedings of the 34th International Conference on Machine Learning, PMLR, Sydney, Australia. pp. 1568–1577. URL: http://proceedings.mlr.press/v70/hu17c.html.
  24. Forecasting: principles and practices. OTexts, Melbourne, Australia. URL: http://otexts.com/fpp2/.
  25. Automatic time series forecasting: The forecast package for R. Journal of Statistical Software 27. URL: http://doi.org/10.18637/jss.v027.i03.
  26. Neuromorphic silicon neuron circuits. Frontiers in Neuroscience 5, 73. URL: http://www.frontiersin.org/article/10.3389/fnins.2011.00073.
  27. Echo state network. Scholarpedia 2, 2330. URL: http://doi.org/10.4249/scholarpedia.2330. revision #151757.
  28. Controlling Recurrent Neural Networks by Conceptors. CoRR – Computing Research Repository abs/1403.3369. Cornell University Library. URL: http://arxiv.org/abs/1403.3369.
  29. Using conceptors to manage neural long-term memories for temporal patterns. Journal of Machine Learning Research 18, 387–429. URL: http://dl.acm.org/doi/abs/10.5555/3122009.3122022.
  30. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 2, 78–80. URL: http://doi.org/10.1126/science.1091277.
  31. Principal component analysis, in: Lovric, M. (Ed.), International Encyclopedia of Statistical Science. Springer, Berlin, Heidelberg, pp. 1094–1096. URL: http://doi.org/10.1007/978-3-642-04898-2_455.
  32. RoboCup: A challenge problem for AI. AI Magazine 18, 73–85. URL: http://www.aaai.org/ojs/index.php/aimagazine/article/view/1276/1177.
  33. Balanced echo state networks. Neural Networks 36, 35–45. URL: http://doi.org/10.1016/j.neunet.2012.08.008.
  34. Fast classification learning with neural networks and conceptors for speech recognition and car driving maneuvers, in: Chomphuwiset, P., Kim, J., Pawara, P. (Eds.), Proceedings of the 14th Multi-Disciplinary International Conference on Artificial Intelligence (MIWAI), Springer Nature Switzerland. pp. 45–57. URL: http://link.springer.com/chapter/10.1007/978-3-030-80253-0_5, doi:10.1007/978-3-030-80253-0_5.
  35. Computational Intelligence. A Methodological Introduction. 2nd ed., Springer, London. URL: http://link.springer.com/book/10.1007/978-1-4471-7296-3.
  36. SNIP: Single-shot network pruning based on connection sensitivity, in: International Conference on Learning Representations. URL: http://arxiv.org/abs/1810.02340.
  37. Reservoir computing trend on software and hardware implementation. Global Journal of Researches in Engineering (F) 17. URL: http://engineeringresearch.org/index.php/GJRE/article/download/1654/1585.
  38. A Critical Review of Recurrent Neural Networks for Sequence Learning. CoRR – Computing Research Repository abs/1506.00019. Cornell University Library. URL: http://arxiv.org/abs/1506.00019.
  39. Predicting Stock Prices Using Recurrent Neural Networks. WAIT – Wernigeröder Automatisierungs- und Informatiktexte 01/2020. Automation and Computer Sciences Department. Harz University of Applied Sciences. URL: http://dx.doi.org/10.25673/35875. in German.
  40. Transformer-based capsule network for stock movement prediction, in: Proceedings of the First Workshop on Financial Technology and Natural Language Processing, Macao, China. pp. 66–73. URL: http://www.aclweb.org/anthology/W19-5511.
  41. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation 14, 2531–2560. URL: http://doi.org/10.1162/089976602760407955.
  42. Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks. Neural Computation 25, 671–696. URL: http://doi.org/10.1162/NECO_a_00411. pMID: 23272918.
  43. Learning recurrent neural networks with Hessian-free optimization, in: Proceedings of the 28th International Conference on Machine Learning, pp. 1033–1040. URL: http://dl.acm.org/doi/10.5555/3104482.3104612.
  44. Difference between memory and prediction in linear recurrent networks. Physical Review E 96, 032308 [1–7]. URL: http://doi.org/10.1103/PhysRevE.96.032308.
  45. Neuromorphic electronic systems. Proceedings of the IEEE 78, 1629–1636. URL: http://ieeexplore.ieee.org/document/58356.
  46. Analysing soccer games with clustering and conceptors, in: Akyama, H., Obst, O., Sammut, C., Tonidandel, F. (Eds.), RoboCup 2017: Robot Soccer World Cup XXI. RoboCup International Symposium, Springer Nature Switzerland, Nagoya, Japan. pp. 120–131. URL: http://doi.org/10.1007/978-3-030-00308-1_10.
  47. RoboCupSimData: Software and data for machine learning from RoboCup simulation league, in: Holz, D., Genter, K., Saad, M., von Stryk, O. (Eds.), RoboCup 2018: Robot Soccer World Cup XXII. RoboCup International Symposium, Springer Nature Switzerland, Montréal, Canada. pp. 230–237. URL: http://doi.org/10.1007/978-3-030-27544-0_19.
  48. Importance estimation for neural network pruning, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11256–11264. URL: http://doi.org/10.1109/CVPR.2019.01152.
  49. Stock market’s price movement prediction with lstm neural networks, in: International joint conference on neural networks (IJCNN), IEEE. URL: http://doi.org/10.1109/IJCNN.2017.7966019.
  50. Training recurrent networks online without backtracking. CoRR – Computing Research Repository abs/1507.07680. Cornell University Library. URL: http://arxiv.org/abs/1507.07680.
  51. Learning Input and Recurrent Weight Matrices in Echo State Networks. CoRR – Computing Research Repository abs/1311.2987. Cornell University Library. URL: http://arxiv.org/abs/1311.2987.
  52. Pre-training of recurrent neural networks via linear autoencoders, in: Advances in Neural Information Processing Systems 27 (NIPS 2014), pp. 3572–3580. URL: http://papers.nips.cc/paper/5271-pre-training-of-recurrent-neural-networks-via-linear-autoencoders.
  53. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning 28, 1310–1318. URL: http://proceedings.mlr.press/v28/pascanu13.pdf.
  54. Learning long-term dependencies with deep memory states, in: Lifelong Learning: A Reinforcement Learning Approach Workshop, International Conference on Machine Learning. URL: http://pdfs.semanticscholar.org/2e09/9bf26976e2334a9c4a2ad1aefc28cf83299b.pdf.
  55. Predicting numbers: An AI approach to solving number series, in: Bach, J., Edelkamp, S. (Eds.), KI 2011: Advances in Artificial Intelligence – Proceedings of the 34th Annual German Conference on Artificial Intelligence, Springer, Berlin. pp. 255–259. URL: http://doi.org/10.1007/978-3-642-24455-1_24.
  56. Pruning algorithms – a survey. IEEE Transactions on Neural Networks 4, 740–747. URL: http://ieeexplore.ieee.org/document/248452, doi:10.1109/72.248452.
  57. Predicting stock prices using LSTM. International Journal of Science and Research (IJSR) 6, 1754–1756. URL: http://www.ijsr.net/archive/v6i4/ART20172755.pdf.
  58. Training recurrent networks by Evolino. Neural Computation 19, 757–779. URL: http://doi.org/10.1162/neco.2007.19.3.757.
  59. Stock market analysis: A review and taxonomy of prediction techniques. International Journal of Financial Studies 7. URL: http://www.mdpi.com/2227-7072/7/2/26, doi:10.3390/ijfs7020026.
  60. Exact solutions for recursive principal components analysis of sequences and trees, in: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (Eds.), Artificial Neural Networks – ICANN, Springer, Berlin, Heidelberg. pp. 349–356. URL: http://link.springer.com/chapter/10.1007/11840817_37.
  61. Periodicity detection by neural transformation, in: Van Dyck, E. (Ed.), ESCOM 2017 – 25th Anniversary Conference of the European Society for the Cognitive Sciences of Music, IPEM, Ghent University, Ghent, Belgium. pp. 159–162. URL: http://artint.hs-harz.de/fstolzenburg/papers/Sto17b.pdf.
  62. The power of linear recurrent neural networks, in: Brunner, D., Jaeger, H., Parkin, S., Pipa, G. (Eds.), Cognitive Computing – Merging Concepts with Hardware, Hannover. URL: http://www.ai.rug.nl/minds/cogcompconf.html. received Prize for Most Technologically Feasible Poster Contribution.
  63. Nonlinear Dynamics and Chaos. With Applications to Physics, Biology, Chemistry, and Engneering. 2nd ed., CRC Press, Boca Raton, FL. URL: http://doi.org/10.1201/9780429492563.
  64. Reinforcement Learning: An Introduction. 2nd ed., MIT Press. URL: http://incompleteideas.net/book/the-book.html.
  65. Random matrices: Universality of ESDs and the circular law. The Annals of Probability 38, 2023–2065. URL: http://projecteuclid.org/euclid.aop/1282053780.
  66. Asymptotic Fisher memory of randomized linear symmetric echo state networks. Neurocomputing 298, 4–8. URL: http://doi.org/10.1016/j.neucom.2017.11.076.
  67. ReservoirPy: An efficient and user-friendly library to design echo state networks, in: Artificial Neural Networks and Machine Learning – ICANN 2020. Springer, pp. 494–505. URL: https://doi.org/10.1007/978-3-030-61616-8_40.
  68. Legendre memory units: Continuous-time representation in recurrent neural networks, in: Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, Canada. URL: http://proceedings.neurips.cc/paper/2019/file/952285b9b7e7a1be5aa7849f32ffff05-Paper.pdf.
  69. Short-term memory in orthogonal neural networks. Physical Review Letters 92, 148102. URL: http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.92.148102.
  70. Decoupled echo state networks with lateral inhibition. Neural Networks 20, 365–376. URL: http://doi.org/10.1016/j.neunet.2007.04.014.
Citations (5)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

Sign Up for Free