ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention
Abstract: Sketch semantic segmentation is a well-explored and pivotal problem in computer vision involving the assignment of pre-defined part labels to individual strokes. This paper presents ContextSeg - a simple yet highly effective approach to tackling this problem with two stages. In the first stage, to better encode the shape and positional information of strokes, we propose to predict an extra dense distance field in an autoencoder network to reinforce structural information learning. In the second stage, we treat an entire stroke as a single entity and label a group of strokes within the same semantic part using an auto-regressive Transformer with the default attention mechanism. By group-based labeling, our method can fully leverage the context information when making decisions for the remaining groups of strokes. Our method achieves the best segmentation accuracy compared with state-of-the-art approaches on two representative datasets and has been extensively evaluated demonstrating its superior performance. Additionally, we offer insights into solving part imbalance in training data and the preliminary experiment on cross-category training, which can inspire future research in this field.
- Doodleformer: Creative sketch drawing with transformers. In European Conference on Computer Vision, pages 338–355. Springer, 2022.
- Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. Computer Vision and Image Understanding, 164:27–37, 2017.
- Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression. Computers & Graphics, 71:77–87, 2018.
- End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020.
- Adaptively-realistic image generation from stroke and sketch with diffusion model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4054–4062, 2023.
- A flexible framework for online document segmentation by pairwise stroke distance learning. Pattern Recognition, 48(4):1197–1210, 2015.
- An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
- Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6824–6835, 2021.
- Creative sketch generation. arXiv preprint arXiv:2011.10039, 2020.
- A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477, 2017.
- Data-driven segmentation and labeling of freehand sketches. ACM Transactions on Graphics (TOG), 33(6):1–10, 2014.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Robust flow-guided neural prediction for sketch-based freeform surface modeling. ACM Transactions on Graphics (TOG), 37(6):1–12, 2018a.
- Free2cad: Parsing freehand drawings into cad commands. ACM Transactions on Graphics (TOG), 41(4):1–16, 2022.
- Toward deep universal sketch perceptual grouper. IEEE Transactions on Image Processing, 28(7):3219–3231, 2019.
- Fast sketch segmentation and labeling with deep learning. IEEE computer graphics and applications, 39(2):38–51, 2018b.
- Sketchgan: Joint sketch completion and recognition with generative adversarial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5830–5839, 2019.
- An intriguing failing of convolutional neural networks and the coordconv solution. Advances in neural information processing systems, 31, 2018.
- Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
- Scheduled sampling for transformers. arXiv preprint arXiv:1906.07651, 2019.
- Generalising fine-grained sketch-based image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 677–686, 2019.
- Sketchsegnet+: An end-to-end learning of rnn for multi-class sketch semantic segmentation. Ieee Access, 7:102717–102726, 2019.
- Sketchformer: Transformer-based representation for sketched structure. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14153–14162, 2020.
- Stylemeup: Towards style-agnostic sketch-based image retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8504–8513, 2021.
- Example-based sketch segmentation and labeling using crfs. ACM Transactions on Graphics (TOG), 35(5):1–9, 2016.
- Free hand-drawn sketch segmentation. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part I 12, pages 626–639. Springer, 2012.
- Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Pointer networks. Advances in neural information processing systems, 28, 2015.
- Multi-column point-cnn for sketch segmentation. Neurocomputing, 392:50–59, 2020.
- Sketchsegnet: A rnn model for labeling sketch strokes. In 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2018.
- Sketchgnn: Semantic sketch segmentation with graph neural networks. ACM Transactions on Graphics (TOG), 40(3):1–13, 2021.
- Sketchnet: Sketch classification with web images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1105–1113, 2016.
- Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
- Sketch-segformer: Transformer-based segmentation for figurative and creative sketches. IEEE Transactions on Image Processing, 2023.
- Part-level sketch segmentation and labeling using dual-cnn. In Neural Information Processing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, December 13-16, 2018, Proceedings, Part I 25, pages 374–384. Springer, 2018.
- 2d freehand sketch labeling using cnn and crf. Multimedia Tools and Applications, 79(1-2):1585–1602, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.