Papers
Topics
Authors
Recent
Search
2000 character limit reached

Continual Learning by Three-Phase Consolidation

Published 12 Mar 2024 in cs.LG and cs.CV | (2403.14679v1)

Abstract: TPC (Three-Phase Consolidation) is here introduced as a simple but effective approach to continually learn new classes (and/or instances of known classes) while controlling forgetting of previous knowledge. Each experience (a.k.a. task) is learned in three phases characterized by different rules and learning dynamics, aimed at removing the class-bias problem (due to class unbalancing) and limiting gradient-based corrections to prevent forgetting of underrepresented classes. Several experiments on complex datasets demonstrate its accuracy and efficiency advantages over competitive existing approaches. The algorithm and all the results presented in this paper are fully reproducible thanks to its publication on the Avalanche open framework for continual learning.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” Psychology of Learning and Motivation, vol. 24, pp. 109–165, 1989. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0079742108605368
  2. S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “iCaRL: Incremental Classifier and Representation Learning,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 2001–2010.
  3. Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 12, pp. 2935–2947, Dec 2018.
  4. P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: a strong, simple baseline,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS’20.   Red Hook, NY, USA: Curran Associates Inc., 2020.
  5. Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, “Large scale incremental learning,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).   Los Alamitos, CA, USA: IEEE Computer Society, jun 2019, pp. 374–382. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR.2019.00046
  6. M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer, “Class-incremental learning: Survey and performance evaluation on image classification,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 45, no. 05, pp. 5513–5533, may 2023.
  7. D. Maltoni and V. Lomonaco, “Continuous learning in single-incremental-task scenarios,” Neural Networks, vol. 116, pp. 56–73, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608019300838
  8. J. G. Klinzing, N. Niethard, and J. Born, “Mechanisms of systems memory consolidation during sleep,” Nature Neuroscience, vol. 22, no. 10, pp. 1598–1610, Oct 2019. [Online]. Available: https://doi.org/10.1038/s41593-019-0467-3
  9. O. C. González, Y. Sokolov, G. P. Krishnan, J. E. Delanois, and M. Bazhenov, “Can sleep protect memories from catastrophic forgetting?” eLife, vol. 9, p. e51005, aug 2020. [Online]. Available: https://doi.org/10.7554/eLife.51005
  10. T. L. Hayes, G. P. Krishnan, M. Bazhenov, H. T. Siegelmann, T. J. Sejnowski, and C. Kanan, “Replay in Deep Learning: Current Approaches and Missing Biological Elements,” Neural Computation, vol. 33, no. 11, pp. 2908–2950, 10 2021. [Online]. Available: https://doi.org/10.1162/neco\_a\_01433
  11. G. Graffieti, D. Maltoni, L. Pellegrini, and V. Lomonaco, “Generative negative replay for continual learning,” Neural Networks, vol. 162, pp. 369–383, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608023001235
  12. H. Hemati, A. Cossu, A. Carta, J. Hurtado, L. Pellegrini, D. Bacciu, V. Lomonaco, and D. Borth, “Class-incremental learning with repetition,” in Proceedings of The 2nd Conference on Lifelong Learning Agents, ser. Proceedings of Machine Learning Research, S. Chandar, R. Pascanu, H. Sedghi, and D. Precup, Eds., vol. 232.   PMLR, 22–25 Aug 2023, pp. 437–455. [Online]. Available: https://proceedings.mlr.press/v232/hemati23b.html
  13. J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521–3526, 2017. [Online]. Available: https://www.pnas.org/content/114/13/3521
  14. G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural Networks, vol. 113, pp. 54–71, may 2019.
  15. Z. Mai, R. Li, J. Jeong, D. Quispe, H. Kim, and S. Sanner, “Online continual learning in image classification: An empirical survey,” Neurocomputing, vol. 469, pp. 28–51, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231221014995
  16. T. Lesort, T. George, and I. Rish, “Continual learning in deep networks: an analysis of the last layer,” 2022.
  17. B. Zhao, X. Xiao, G. Gan, B. Zhang, and S.-T. Xia, “Maintaining discrimination and fairness in class incremental learning,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020, pp. 13 205–13 214.
  18. S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019, pp. 831–839.
  19. L. Caccia, R. Aljundi, N. Asadi, T. Tuytelaars, J. Pineau, and E. Belilovsky, “New insights on reducing abrupt representation change in online continual learning,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=N8MaByOzUfb
  20. V. Lomonaco, L. Pellegrini, G. Graffieti, and D. Maltoni, “Architect, Regularize and Replay (ARR): a Flexible Hybrid Approach for Continual Learning,” in Towards Human Brain Inspired Lifelong Learning, X. Li, R. Savitha, A. Ambikapathi, S. Sundaram, and H. M. Fayek, Eds.   Singapore: World Scientific Publishing, 2024, ch. 6. [Online]. Available: {https://www.worldscientific.com/worldscibooks/10.1142/13689}
  21. V. Lomonaco, D. Maltoni, and L. Pellegrini, “Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches,” in CVPR Workshop on Continual Learning for Computer Vision, 2020, pp. 246–247. [Online]. Available: https://openaccess.thecvf.com/content_CVPRW_2020/html/w15/Lomonaco_Rehearsal-Free_Continual_Learning_Over_Small_Non-I.I.D._Batches_CVPRW_2020_paper.html
  22. D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2014. [Online]. Available: http://arxiv.org/abs/1312.6114
  23. C. T. Fan, M. E. Muller, and I. Rezucha, “Development of sampling plans by using sequential (item by item) selection techniques and digital computers,” Journal of the American Statistical Association, vol. 57, no. 298, pp. 387–402, 1962. [Online]. Available: http://www.jstor.org/stable/2281647
  24. D. Isele and A. Cosgun, “Selective experience replay for lifelong learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/11595
  25. L. Pellegrini, G. Graffieti, V. Lomonaco, and D. Maltoni, “Latent replay for real-time continual learning,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 10 203–10 209.
  26. F. Zenke, B. Poole, and S. Ganguli, “Continual Learning Through Synaptic Intelligence,” in Proceedings of the 34th International Conference on Machine Learning (ICML), vol. 70, 2017, pp. 3987–3995.
  27. V. Lomonaco and D. Maltoni, “Core50: a new dataset and benchmark for continuous object recognition,” in Proceedings of the 1st Annual Conference on Robot Learning, ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldberg, Eds., vol. 78.   PMLR, Nov 2017, pp. 17–26. [Online]. Available: https://proceedings.mlr.press/v78/lomonaco17a.html
  28. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 248–255.
  29. A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” Tech. Rep., 2009.
  30. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” 2017.
  31. O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in Proceedings of the 30th International Conference on Neural Information Processing Systems, ser. NIPS’16.   Red Hook, NY, USA: Curran Associates Inc., 2016, p. 3637–3645.
  32. A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. S. Torr, and M. Ranzato, “On tiny episodic memories in continual learning,” 2019.
  33. Stanford. (CS231N), “Tiny imagenet challenge, cs231n course.” [Online]. Available: https://tiny-imagenet.herokuapp.com/
  34. M. Masana, T. Tuytelaars, and J. van de Weijer, “Ternary feature masks: Zero-forgetting for task-incremental learning,” in IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021.   Computer Vision Foundation / IEEE, 2021, pp. 3570–3579.
  35. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 770–778.
  36. A. López-Cifuentes, M. Escudero-Viñolo, J. Bescós, and Álvaro García-Martín, “Semantic-aware scene recognition,” Pattern Recognition, vol. 102, p. 107256, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0031320320300613
  37. V. Lomonaco, L. Pellegrini, A. Cossu, A. Carta, G. Graffieti, T. L. Hayes, M. De Lange, M. Masana, J. Pomponi, G. M. van de Ven, M. Mundt, Q. She, K. Cooper, J. Forest, E. Belouadah, S. Calderara, G. I. Parisi, F. Cuzzolin, A. S. Tolias, S. Scardapane, L. Antiga, S. Ahmad, A. Popescu, C. Kanan, J. van de Weijer, T. Tuytelaars, D. Bacciu, and D. Maltoni, “Avalanche: an end-to-end library for continual learning,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2021, pp. 3595–3605.
  38. A. Carta, L. Pellegrini, A. Cossu, H. Hemati, and V. Lomonaco, “Avalanche: A pytorch library for deep continual learning,” Journal of Machine Learning Research, vol. 24, no. 363, pp. 1–6, 2023. [Online]. Available: http://jmlr.org/papers/v24/23-0130.html
  39. D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17, 2017, pp. 6470–6479.
  40. D. Madaan, J. Yoon, Y. Li, Y. Liu, and S. J. Hwang, “Representational continuity for unsupervised continual learning,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=9Hrka5PA7LW
  41. J. Smith, C. Taylor, S. Baer, and C. Dovrolis, “Unsupervised progressive learning and the stam architecture,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Z.-H. Zhou, Ed.   International Joint Conferences on Artificial Intelligence Organization, 8 2021, pp. 2979–2987, main Track. [Online]. Available: https://doi.org/10.24963/ijcai.2021/410
  42. J. Soch, “Proof: Kullback-Leibler divergence for the normal distribution,” in The Book of Statistical Proofs, J. Soch, Ed., 2020. [Online]. Available: https://statproofbook.github.io/P/norm-kl.html

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up