Papers
Topics
Authors
Recent
Search
2000 character limit reached

CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation

Published 10 Oct 2023 in cs.CV | (2310.06368v1)

Abstract: Class incremental semantic segmentation aims to strike a balance between the model's stability and plasticity by maintaining old knowledge while adapting to new concepts. However, most state-of-the-art methods use the freeze strategy for stability, which compromises the model's plasticity.In contrast, releasing parameter training for plasticity could lead to the best performance for all categories, but this requires discriminative feature representation.Therefore, we prioritize the model's plasticity and propose the Contrast inter- and intra-class representations for Incremental Segmentation (CoinSeg), which pursues discriminative representations for flexible parameter tuning. Inspired by the Gaussian mixture model that samples from a mixture of Gaussian distributions, CoinSeg emphasizes intra-class diversity with multiple contrastive representation centroids. Specifically, we use mask proposals to identify regions with strong objectness that are likely to be diverse instances/centroids of a category. These mask proposals are then used for contrastive representations to reinforce intra-class diversity. Meanwhile, to avoid bias from intra-class diversity, we also apply category-level pseudo-labels to enhance category-level consistency and inter-category diversity. Additionally, CoinSeg ensures the model's stability and alleviates forgetting through a specific flexible tuning strategy. We validate CoinSeg on Pascal VOC 2012 and ADE20K datasets with multiple incremental scenarios and achieve superior results compared to previous state-of-the-art methods, especially in more challenging and realistic long-term scenarios. Code is available at https://github.com/zkzhang98/CoinSeg.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Memory aware synapses: Learning what (not) to forget. In Proc. European Conference on Computer Vision, pages 139–154, 2018.
  2. Expert gate: Lifelong learning with a network of experts. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 3366–3375, 2017.
  3. Incremental and decremental support vector machine learning. Advances in neural information processing systems, 13, 2000.
  4. Modeling the background for incremental learning in semantic segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 9233–9242, 2020.
  5. Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. Advances in neural information processing systems, 34, 2021.
  6. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Recognition and Machine Intelligence, 40(4):834–848, 2017.
  7. Masked-attention mask transformer for universal image segmentation. arXiv preprint arXiv:2112.01527, 2021.
  8. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34, 2021.
  9. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882, 2021.
  10. Continual prototype evolution: Learning online from non-stationary data streams. In Proc. IEEE International Conference on Computer Vision, pages 8250–8259, 2021.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  12. Plop: Learning without forgetting for continual semantic segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 4040–4050, 2021.
  13. The pascal visual object classes (voc) challenge. International Journal on Computer Vision, 88(2):303–338, 2010.
  14. Global knowledge calibration for fast open-vocabulary segmentation, 2023.
  15. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.
  16. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122, 2016.
  17. Deep learning. nature, 521(7553):436–444, 2015.
  18. Learning without forgetting. IEEE Transactions on Pattern Recognition and Machine Intelligence, 40(12):2935–2947, 2017.
  19. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  20. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
  21. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  22. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  23. Recall: Replay-based continual learning in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7026–7035, 2021.
  24. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  25. Incremental learning techniques for semantic segmentation. In Proc. IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  26. Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 1114–1124, 2021.
  27. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11321–11329, 2019.
  28. A study of gaussian mixture models of color and texture features for image classification and segmentation. Pattern recognition, 39(4):695–706, 2006.
  29. Genetic-based em algorithm for learning gaussian mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1344–1348, 2005.
  30. Learn++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 31(4):497–508, 2001.
  31. Encoder based lifelong learning. In Proc. IEEE International Conference on Computer Vision, pages 1320–1328, 2017.
  32. icarl: Incremental classifier and representation learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  33. Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
  34. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
  35. Incremental learning through deep adaptation. IEEE Transactions on Pattern Recognition and Machine Intelligence, 42(3):651–663, 2018.
  36. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
  37. Overcoming catastrophic forgetting with hard attention to the task. In Proc. International Conference on Machine Learning, pages 4548–4557, 2018.
  38. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
  39. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  40. Memory replay gans: Learning to generate new categories without forgetting. Advances in Neural Information Processing Systems, 31, 2018.
  41. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077–12090, 2021.
  42. Prototype mixture models for few-shot semantic segmentation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16, pages 763–778. Springer, 2020.
  43. Uncertainty-aware contrastive distillation for incremental semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  44. Representation compensation networks for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7053–7064, 2022.
  45. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  46. Class-incremental learning via deep model consolidation. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1131–1140, 2020.
  47. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics, 50(9):3855–3865, 2020.
  48. Mining unseen classes via regional objectness: A simple baseline for incremental segmentation. Advances in neural information processing systems, 35, 2022.
  49. Scene parsing through ade20k dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 633–641, 2017.
  50. Ctp: Towards vision-language continual pretraining via compatible momentum contrast and topology preservation, 2023.
Citations (17)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up