Papers
Topics
Authors
Recent
Search
2000 character limit reached

Pseudo-label Learning with Calibrated Confidence Using an Energy-based Model

Published 15 Apr 2024 in cs.CV | (2404.09585v1)

Abstract: In pseudo-labeling (PL), which is a type of semi-supervised learning, pseudo-labels are assigned based on the confidence scores provided by the classifier; therefore, accurate confidence is important for successful PL. In this study, we propose a PL algorithm based on an energy-based model (EBM), which is referred to as the energy-based PL (EBPL). In EBPL, a neural network-based classifier and an EBM are jointly trained by sharing their feature extraction parts. This approach enables the model to learn both the class decision boundary and input data distribution, enhancing confidence calibration during network training. The experimental results demonstrate that EBPL outperforms the existing PL method in semi-supervised image classification tasks, with superior confidence calibration error and recognition accuracy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Zhu Xiaojin, “Semi-supervised learning literature survey,” Tech. Rep. 1530, Computer Sciences, University of Wisconsin-Madison, 2005.
  2. “A survey on transfer learning,” IEEE Transactions on Knowledge and Data Engineering, pp. 1345–1359, 2010.
  3. Settles Burr, “Active learning literature survey,” Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009.
  4. Dong-Hyun Lee, “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in Proceedings of the International Conference on Machine Learning, 2013, p. 896.
  5. “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” in Proceedings of the Annual Conference on Neural Information Processing Systems, 2020, pp. 596–608.
  6. “On calibration of modern neural networks,” in Proceedings of the International Conference on Machine Learning, 2017, pp. 1321–1330.
  7. “Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers,” in Proceedings of the International Conference on Machine Learning, 2001, pp. 609–616.
  8. “In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning,” in Proceedings of the International Conference on Learning Representations, 2021.
  9. “Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning,” Proceedings of the Annual AAAI Conference on Artificial Intelligence, vol. 35, no. 8, pp. 6912–6920, 2021.
  10. “Debiased learning from naturally imbalanced pseudo-labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14647–14657.
  11. “Class balanced adaptive pseudo labeling for federated semi-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 16292–16301.
  12. “Confidence-aware pseudo-label learning for weakly supervised visual grounding,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2828–2838.
  13. “Contactless pulse estimation leveraging pseudo labels and self-supervision,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 20588–20597.
  14. “Self-training with noisy student improves imagenet classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  15. “Meta pseudo labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11557–11568.
  16. “Boostmis: Boosting medical image semi-supervised learning with adaptive pseudo labeling and informative active annotation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20666–20676.
  17. “Bayesian learning via stochastic gradient langevin dynamics,” in Proceedings of the International Conference on Machine Learning, 2011, pp. 681–688.
  18. “Divergence triangle for joint training of generator model, energy-based model, and inferential model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8670–8679.
  19. “Implicit generation and modeling with energy based models,” in Proceedings of the Annual Conference on Neural Information Processing Systems, 2019, vol. 32.
  20. “Cooperative learning of energy-based model and latent variable model via MCMC teaching,” in Proceedings of the Annual AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  21. “Residual energy-based models for text generation,” in Proceedings of the International Conference on Learning Representations, 2020.
  22. “Residual energy-based models for text,” Journal of Machine Learning Research, vol. 22, no. 40, pp. 1–41, 2021.
  23. “Your classifier is secretly an energy based model and you should treat it like one,” in Proceedings of the International Conference on Learning Representations, 2020.
  24. “Improved contrastive divergence training of energy based models,” in Proceedings of the International Conference on Machine Learning, 2021.
  25. “Edropout: Energy-based dropout and pruning of deep neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 10, pp. 5279–5292, 2022.
  26. “Energy-based continuous inverse optimal control,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2022.
  27. “Calibrating deep neural networks using explicit regularisation and dynamic data pruning,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 1541–1549.
  28. “Bridging precision and confidence: A train-time loss for calibrating object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11474–11483.
  29. “Obtaining well calibrated probabilities using bayesian binning,” in Proceedings of the Annual AAAI Conference on Artificial Intelligence, 2015, vol. 29.
  30. John Platt et al., “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” Advances in Large Margin Classifiers, vol. 10, no. 3, pp. 61–74, 1999.
  31. “Transforming classifier scores into accurate multiclass probability estimates,” in Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2002, pp. 694–699.
  32. “Learning for single-shot confidence calibration in deep neural networks through stochastic inferences,” in Proc. Conf. Comput. Vis. and Pattern Recog., 2019, pp. 9030–9038.
  33. “Be confident! towards trustworthy graph neural networks via confidence calibration,” Proceedings of the Annual Conference on Neural Information Processing Systems, vol. 34, 2021.
  34. “Dynamically weighted balanced loss: Class imbalanced learning and confidence calibration of deep neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 7, pp. 2940–2951, 2022.
  35. “mixup: Beyond empirical risk minimization,” in Proceedings of the International Conference on Learning Representations, 2018.
  36. “On mixup training: Improved calibration and predictive uncertainty for deep neural networks,” Proceedings of the Annual Conference on Neural Information Processing Systems, vol. 32, 2019.
  37. Hideaki Hayashi, “A hybrid of generative and discriminative models based on the Gaussian-coupled softmax layer,” IEEE Transactions on Neural Networks and Learning Systems (Early Access), 2024.
  38. “Learning multiple layers of features from tiny images,” 2009.
  39. “Reading digits in natural images with unsupervised feature learning,” in NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  40. “Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms,” arXiv preprint arXiv:1708.07747, 2017.
  41. “Medmnist v2 - a large-scale lightweight benchmark for 2d and 3d biomedical image classification,” Scientific Data, vol. 10, no. 41, 2023.
  42. “Wide residual networks,” in Proceedings of the British Machine Vision Conference, 2016, pp. 87.1–87.12.
  43. “Pseudo-labeling and confirmation bias in deep semi-supervised learning,” in Proceedings of the International Joint Conference on Neural Networks, 2020, pp. 1–8.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up