Papers
Topics
Authors
Recent
Search
2000 character limit reached

Studying Hopfield models via fully lifted random duality theory

Published 29 Nov 2023 in math.PR, cond-mat.stat-mech, cs.IT, math-ph, math.IT, math.MP, math.ST, and stat.TH | (2312.00071v1)

Abstract: Relying on a recent progress made in studying bilinearly indexed (bli) random processes in \cite{Stojnicnflgscompyx23,Stojnicsflgscompyx23}, the main foundational principles of fully lifted random duality theory (fl RDT) were established in \cite{Stojnicflrdt23}. We here study famous Hopfield models and show that their statistical behavior can be characterized via the fl RDT. Due to a nestedly lifted nature, the resulting characterizations and, therefore, the whole analytical machinery that produces them, become fully operational only if one can successfully conduct underlying numerical evaluations. After conducting such evaluations for both positive and negative Hopfield models, we observe a remarkably fast convergence of the fl RDT mechanism. Namely, for the so-called square case, the fourth decimal precision is achieved already on the third (second non-trivial) level of lifting (3-sfl RDT) for the positive and on the fourth (third non-trivial) level of lifting (4-sfl RDT) for the corresponding negative model. In particular, we obtain the scaled ground state free energy $\approx 1.7788$ for the positive and $\approx 0.3279$ for the negative model.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (99)
  1. D. Achlioptas and Y. Peres. The threshold for random k-SAT is 2⁢k⁢l⁢n⁢k−O⁢(k)2𝑘𝑙𝑛𝑘𝑂𝑘2klnk-{O}(k)2 italic_k italic_l italic_n italic_k - italic_O ( italic_k ). Journal of the AMS, 17:947–973, 2004.
  2. A. E. Alaoui and M. Sellke. Algorithmic pure states for the negative spherical perceptron. 2020. available online at http://arxiv.org/abs/2010.15811.
  3. D. J. Aldous. Asymptotics in the random assignment problem. Probab Theory Related Fields, 93:507–534, 1992.
  4. D. J. Aldous. The zeta(2) limit in the random assignment problem. Random Structures Algorithms, 18:381–418, 2001.
  5. Exact expectations and distributions for the random assignment problem. Combinat. Probab. Comput., 11(3):217–248, 2002.
  6. Phys. Rev. A, 32:1007, 1985.
  7. Storing infinite number of patterns in a spin glass model of neural networks. Phys. Rev. Letters, 55:1530, 1987.
  8. P. Baldi and S. Venkatesh. Number od stable points for spin-glasses and neural networks of higher orders. Phys. Rev. Letters, 58(9):913–916, Mar. 1987.
  9. The replica symmetric approximation of the analogical neural network. J. Stat. Physics, July 2010.
  10. Equlibrium statistical mechanics of bipartite spin systems. Journal of Physics A: Mathematical and Theoeretical, 44(245002), 2011.
  11. How glassy are neural networks. J. Stat. Mechanics: Thery and Experiment, July 2012.
  12. M. Bayati and A. Montanari. The lasso risk of gaussian matrices. 2010. available online at http://arxiv.org/abs/1008.2581.
  13. M. Bayati and A. Montanari. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Transactions on Information Theory, 57(2), 2011.
  14. A. Bovier and V. Gayrard. Hopfield models as generalized random mean field models. In mathematical aspects of spin glasses and neural networks, Progr. Prob., 41:3–89, 1998.
  15. Asymmetric Little spin glas model. Physical Review B, 46(9), September 1992.
  16. Eigenstates and limit cycles in the SK model. Journal of Physics A: Mathematical and General, 21:4201, 1988.
  17. S. H. Cameron. Tech-report 60-600. Proceedings of the bionics symposium, pages 197–212, 1960. Wright air development division, Dayton, Ohio.
  18. S. Chatterjee. A generalization of the Lindenberg principle. The Annals of Probability, 34(6):2061–2076.
  19. A. Coja-Oghlan. The asymptotic k-SAT threshold. Proceedings of the forty-fifth annual ACM symposium on theory of computing (STOC), pages 804–813, 2014.
  20. D. Coppersmith and G. Sorkin. Constructive bounds and exact expectations for the random assignment problem. Random Structures Algorithms, 15:113–144, 1999.
  21. D. Coppersmith and G. Sorkin. On the expected incremental cost of a minimum assignment. Contemporary combinatories, Bolyai Society Mathematical Studies, 10(8), 2002. B. Bollobas (Editor), Springer, New York.
  22. T. Cover. Geomretrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, (EC-14):326–334, 1965.
  23. Saturation level of the Hopfield model for neural network. Europhys. Lett., (2):337, 1986.
  24. D. Dean and F. Ritort. Squared interaction matrix Sherrington-Kirkpatrick model for a spin glass. Phys. Rev. B, 65:224209, 2002.
  25. Satisfiability Threshold for Random Regular nae-sat. Communications in Mathematical Physics, 341(2):435–489, 2015.
  26. Satisfiability threshold for random regular NAE-SAT. Proceedings of the forty-sixth annual ACM symposium on theory of computing (STOC), pages 814–822, 2015.
  27. D. Donoho. High-dimensional centrally symmetric polytopes with neighborlines proportional to dimension. Disc. Comput. Geometry, 35(4):617–652, 2006.
  28. D. Donoho and J. Tanner. Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Preprint, 2009. available at arXiv:0906.2530.
  29. Jamming in multilayer supervised learning models. Phys. Rev. Lett., 123(16):160602, 2019.
  30. S. Franz and G. Parisi. The simplest model of jamming. Journal of Physics A: Mathematical and Theoretical, 49(14):145001, 2016.
  31. Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems. SciPost Physics, 2:019, 2017.
  32. Critical jammed phase of the linear perceptron. Phys. Rev. Lett., 123(11):115702, 2019.
  33. Surfing on minima of isostatic landscapes: avalanches and unjamming transition. SciPost Physics, 9:12, 2020.
  34. A. Frieze and N. Wormald. Random k-Sat: a tight threshold for moderately growing k. Combinatorica, 28:297–305, 2005.
  35. E. Gardner. The space of interactions in neural networks models. J. Phys. A: Math. Gen., 21:257–270, 1988.
  36. E. Gardner and B. Derrida. Optimal storage properties of neural networks models. J. Phys. A: Math. Gen., 21:271–284, 1988.
  37. F. Guerra. Broken replica symmetry bounds in the mean field spin glass model. Comm. Math. Physics, 233:1–12, 2003.
  38. H. Gutfreund and Y. Stein. Capacity of neural networks with discrete synaptic couplings. J. Physics A: Math. Gen, 23:2613, 1990.
  39. D. O. Hebb. Organization of behavior. New York: Wiley, 1949.
  40. J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proc. Nat. Acad. Science, 79:2554, 1982.
  41. R. D. Joseph. The number of orthants in n𝑛nitalic_n-space instersected by an s𝑠sitalic_s-dimensional subspace. Tech. memo 8, project PARA, 1960. Cornel aeronautical lab., Buffalo, N.Y.
  42. Covering cubes by random half cubes with applications to biniary neural networks. Journal of Computer and System Sciences, 56:223–252, 1998.
  43. W. Krauth and M. Mezard. Storage capacity of memory networks with binary couplings. J. Phys. France, 50:3057–3066, 1989.
  44. J. W. Lindeberg. Eine neue herleitung des exponentialgesetzes in der wahrscheinlichkeitsrechnung. Math. Z., 15:211–225, 1922.
  45. S. Linusson and J. Wastlund. A proof of Parisi’s conjecture on the random assignment problem. Probabil Theory Related Fields, 128(3):419–440, 2004.
  46. W. A. Little. The existence of persistent states in the brain. Math. Biosci., 19(1-2):101–120, 1974.
  47. Threshold values of random K-SAT from the cavity method. Random Struct. Alg., 28:340–373, 2006.
  48. M. Mezard and G. Parisi. On the solution of the random link matching problem. J Physique, 48:1451–1459, 1987.
  49. Analytic and algorithmic solution of random satisfiability problems. Science, 297:812–815, 2002.
  50. M. Molloy. The freezing threshold for k-colourings of a random graph. Proceedings of the forty-third annual ACM symposium on theory of computing (STOC), pages 921–930, 2012.
  51. Proofs of the Parisi and Coppersmith-Sorkin random assignment conjectures. Random Structures and Algorithms, 27(4):413–444, 2005.
  52. R. O.Winder. Threshold logic. Ph. D. dissertation, Princetoin University, 1962.
  53. D. Panchenko. A connection between the Ghirlanda-Guerra identities and ultrametricity. The Annals of Probability, 38(1):327–347, 2010.
  54. D. Panchenko. The Ghirlanda-Guerra identities for mixed p-spin model. Comptes Rendus Mathematique, 348(3-4):189–192, 2010.
  55. D. Panchenko. The Parisi ultrametricity conjecture. Ann. Math., 77(1):383–393, 2013.
  56. D. Panchenko. The Sherrington-Kirkpatrick model. Springer Science & Business Media, 2013.
  57. D. Panchenko. On the replica symmetric solution of the l-sat model. Electronic Journal of Probability, 19, 2014.
  58. G. Parisi. Infnite number of order parameters for spin-glasses. Phys. Rev. Lett., 43:1754–1756, 1979.
  59. G. Parisi. Breaking the symmetry in SK model. J. Physics, A13:1101, 1980.
  60. G. Parisi. A sequence of approximated solutions to the SK model for spin glasses. Journal of Physics A: Mathematical and General, 13(4):L115, 1980.
  61. G. Parisi. Order parameter for spin glasses. Phys. Rev. Lett., 50:1946, 1983.
  62. L. Pastur and A. Figotin. On the theory of disordered spin systems. Theory Math. Phys., 35(403-414), 1978.
  63. The replica-symmetric solution without the replica trick for the Hopfield model. Journal of Statistical Physics, 74(5/6), 1994.
  64. L. Schlafli. Gesammelte Mathematische AbhandLungen I. Basel, Switzerland: Verlag Birkhauser, 1950.
  65. M. Shcherbina and B. Tirozzi. The free energy of a class of Hopfield models. Journal of Statistical Physics, 72(1/2), 1993.
  66. M. Shcherbina and B. Tirozzi. On the volume of the intrersection of a sphere with random half spaces. C. R. Acad. Sci. Paris. Ser I, (334):803–806, 2002.
  67. M. Shcherbina and B. Tirozzi. Rigorous solution of the Gardner problem. Comm. on Math. Physics, (234):383–422, 2003.
  68. D. Sherrington and S. Kirkpatrick. Solvable model of a spin glass. Phys. Rev. Letters, 35:1792–1796, 1972.
  69. H. Steffan and R. Kuhn. Replica symmetry breaking in attractor neural network models. Z. Phys. B, 95:249–260, 1994.
  70. M. Stojnic. A framework for perfromance characterization of LASSO algortihms. available online at http://arxiv.org/abs/1303.7291.
  71. M. Stojnic. Upper-bounding ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization weak thresholds. available online at http://arxiv.org/abs/1303.7289.
  72. M. Stojnic. Various thresholds for ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. available online at http://arxiv.org/abs/0907.3666.
  73. M. Stojnic. Another look at the Gardner problem. 2013. available online at http://arxiv.org/abs/1306.3979.
  74. M. Stojnic. Asymmetric Little model and its ground state energies. 2013. available online at http://arxiv.org/abs/1306.3978.
  75. M. Stojnic. Bounding ground state energy of Hopfield models. 2013. available online at http://arxiv.org/abs/1306.3764.
  76. M. Stojnic. Discrete perceptrons. 2013. available online at http://arxiv.org/abs/1303.4375.
  77. M. Stojnic. Lifting ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization strong and sectional thresholds. 2013. available online at http://arxiv.org/abs/1306.3770.
  78. M. Stojnic. Lifting/lowering Hopfield models ground state energies. 2013. available online at http://arxiv.org/abs/1306.3975.
  79. M. Stojnic. Negative spherical perceptron. 2013. available online at http://arxiv.org/abs/1306.3980.
  80. M. Stojnic. Regularly random duality. 2013. available online at http://arxiv.org/abs/1303.7295.
  81. M. Stojnic. Spherical perceptron as a storage memory with limited errors. 2013. available online at http://arxiv.org/abs/1306.3809.
  82. M. Stojnic. Fully bilinear generic and lifted random processes comparisons. 2016. available online at http://arxiv.org/abs/1612.08516.
  83. M. Stojnic. Generic and lifted probabilistic comparisons – max replaces minmax. 2016. available online at http://arxiv.org/abs/1612.08506.
  84. M. Stojnic. Bilinearly indexed random processes – stationarization of fully lifted interpolation. 2023. available online at arxiv.
  85. M. Stojnic. Fully lifted interpolating comparisons of bilinearly indexed random processes. 2023. available online at arxiv.
  86. M. Stojnic. Fully lifted random duality theory. 2023. available online at arxiv.
  87. M. Talagrand. Rigorous results for the Hopfield model with many patterns. Prob. theory and related fields, 110:177–276, 1998.
  88. M. Talagrand. An assignment problem at high temperature. Ann. Probab., 31(2):818–848, 2003.
  89. M. Talagrand. The Generic Chaining. Springer-Verlag, 2005.
  90. M. Talagrand. The Parisi formula. Annals of mathematics, 163(2):221–263, 2006.
  91. M. Talagrand. Mean field models and spin glasse: Volume II. A series of modern surveys in mathematics 55, Springer-Verlag, Berlin Heidelberg, 2011.
  92. M. Talagrand. Mean field models and spin glasses: Volume I. A series of modern surveys in mathematics 54, Springer-Verlag, Berlin Heidelberg, 2011.
  93. S. Venkatesh. Epsilon capacity of neural networks. Proc. Conf. on Neural Networks for Computing, Snowbird, UT, 1986.
  94. J. Wastlund. A proof of a conjecture of Buck, Chan, and Robbins on the expected value of the minimum assignment. Random Structures Algorithms, 26(1-2):237–251, 2004.
  95. J. Wastlund. An easy proof of the ζ⁢(2)𝜁2\zeta(2)italic_ζ ( 2 ) limit in the random assignment problem. Electronic Communications in Probability, 14:1475, 2009.
  96. J. Wastlund. Replica symmetry of the minimum matching. Annals of Mathematics, 175(3):1061–1091, 2012.
  97. J. G. Wendel. A problem in geometric probablity. Mathematics Scandinavia, 11:109–111, 1962.
  98. R. O. Winder. Single stage threshold logic. Switching circuit theory and logical design, pages 321–332, Sep. 1961. AIEE Special publications S-134.
  99. J. Y. Zhao. The Hopfield model with superlinearly many patterns. 2011. available online at http://arxiv.org/abs/1108.4771.
Citations (5)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Authors (1)

  1. Mihailo Stojnic 

Collections

Sign up for free to add this paper to one or more collections.

Sign Up