Papers
Topics
Authors
Recent
Search
2000 character limit reached

Neural Random Forest Imitation

Published 25 Nov 2019 in cs.LG and stat.ML | (1911.10829v2)

Abstract: We present Neural Random Forest Imitation - a novel approach for transforming random forests into neural networks. Existing methods propose a direct mapping and produce very inefficient architectures. In this work, we introduce an imitation learning approach by generating training data from a random forest and learning a neural network that imitates its behavior. This implicit transformation creates very efficient neural networks that learn the decision boundaries of a random forest. The generated model is differentiable, can be used as a warm start for fine-tuning, and enables end-to-end optimization. Experiments on several real-world benchmark datasets demonstrate superior performance, especially when training with very few training examples. Compared to state-of-the-art methods, we significantly reduce the number of network parameters while achieving the same or even improved accuracy due to better generalization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Deep learning on small datasets without pre-training using cosine loss. In IEEE Winter Conference on Applications of Computer Vision (WACV), pp.  1360–1369, 2020.
  2. Neural Random Forests. Sankhya A, 81:347–386, December 2019.
  3. Small data, big decisions: Model selection in the small-data regime. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp.  1035–1044. PMLR, Jul 2020.
  4. Breiman, L. Random forests. Machine learning, 45(1):5–32, 2001.
  5. Classification and Regression Trees. Wadsworth and Brooks, Monterey, CA, 1984.
  6. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol., 2(3):1–27, 2011.
  7. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493–2537, 2011. ISSN 1532-4435.
  8. A decision heuristic for Monte Carlo tree search doppelkopf agents. In 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp.  1–8, November 2017. ISBN 978-1-5386-2726-6.
  9. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
  10. Rethinking importance weighting for deep learning under distribution shift. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  11. Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res., 15(1):3133–3181, 2014. ISSN 1532-4435.
  12. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp.  6645–6649, 2013.
  13. Why we need a small data paradigm. BMC Medicine, 17, 2019.
  14. Deep neural network initialization with decision trees. IEEE Transactions on Neural Networks and Learning Systems, 30(5):1286–1295, 2019.
  15. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 25, 2012.
  16. Random forests versus neural networks – what’s best for camera localization? In IEEE International Conference on Robotics and Automation (ICRA), pp.  5118–5125, 2017.
  17. On oblique random forests. In Machine Learning and Knowledge Discovery in Databases, pp. 453–469, 2011.
  18. Human-level control through deep reinforcement learning. Nature, 518:529–533, 2 2015. ISSN 0028-0836.
  19. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp.  807–814, 2010. ISBN 978-1-60558-907-7.
  20. Modern neural networks generalize on small data sets. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
  21. A survey of the usages of deep learning for natural language processing. IEEE Transactions on Neural Networks and Learning Systems, 32:604–624, 2021.
  22. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proc. Interspeech 2019, pp.  2613–2617, 2019.
  23. Self-training for few-shot transfer across extreme task differences. In Proceedings of the International Conference on Learning Representations, 2021.
  24. Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised methods. IEEE transactions on pattern analysis and machine intelligence, 2020.
  25. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  26. Quinlan, J. R. C4.5: programs for machine learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993. ISBN 1-55860-238-0.
  27. Object recognition from very few training examples for enhancing bicycle maps. In IEEE Intelligent Vehicles Symposium (IV), June 2018.
  28. Learning convolutional neural networks for object detection with very little training data. Multimodal Scene Understanding, 2019.
  29. ChimeraMix: Image classification on small datasets via masked feature mixing. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI, pp.  1298–1305, 7 2022.
  30. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
  31. Mapping auto-context decision forests to deep convnets for semantic segmentation. In Proceedings of the British Machine Vision Conference (BMVC), September 2016.
  32. Fully convolutional cross-scale-flows for image-based defect detection. In Winter Conference on Applications of Computer Vision (WACV), January 2022.
  33. Learning Representations by Back-Propagating Errors, pp. 696–699. MIT Press, Cambridge, MA, USA, 1988. ISBN 0262010976.
  34. Sethi, I. K. Entropy nets: from decision trees to neural networks. Proceedings of the IEEE, 78(10):1605–1613, Oct 1990. ISSN 0018-9219.
  35. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
  36. Fsce: Few-shot object detection via contrastive proposal encoding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  37. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 27, 2014.
  38. Comparative analysis of image classification algorithms based on traditional machine learning and deep learning. Pattern Recognition Letters, 141:61–67, 2021. ISSN 0167-8655.
  39. Welbl, J. Casting random forests as artificial neural networks (and profiting from it). In German Conference on Pattern Recognition, 2014.
  40. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations (ICLR), 2017.
  41. Pyramid scene parsing network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  6230–6239, 2017.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 1 like about this paper.

Sign Up for Free