Hierarchical NeuroSymbolic Approach for Comprehensive and Explainable Action Quality Assessment
Abstract: Action quality assessment (AQA) applies computer vision to quantitatively assess the performance or execution of a human action. Current AQA approaches are end-to-end neural models, which lack transparency and tend to be biased because they are trained on subjective human judgements as ground-truth. To address these issues, we introduce a neuro-symbolic paradigm for AQA, which uses neural networks to abstract interpretable symbols from video data and makes quality assessments by applying rules to those symbols. We take diving as the case study. We found that domain experts prefer our system and find it more informative than purely neural approaches to AQA in diving. Our system also achieves state-of-the-art action recognition and temporal segmentation, and automatically generates a detailed report that breaks the dive down into its elements and provides objective scoring with visual evidence. As verified by a group of domain experts, this report may be used to assist judges in scoring, help train judges, and provide feedback to divers. Annotated training data and code: https://github.com/laurenok24/NSAQA.
- The kimore dataset: Kinematic assessment of movement and clinical scores for remote monitoring of physical rehabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(7):1436–1448, 2019.
- Neural-symbolic learning systems - foundations and applications. In Perspectives in Neural Computing, 2012.
- Who’s better, who’s best: Skill determination in video using deep ranking.
- Who’s better? who’s best? pairwise deep ranking for skill determination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6057–6066, 2018.
- The pros and cons: Rank-aware temporal attention for skill determination in long videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7862–7871, 2019.
- Action quality assessment using siamese network-based deep metric learning. IEEE Transactions on Circuits and Systems for Video Technology, 31(6):2260–2273, 2020.
- Amy Kwan. Usa diving elearning center.
- A survey of vision-based human action evaluation methods. Sensors, 19(19):4129, 2019.
- End-to-end learning for action quality assessment. In Pacific Rim Conference on Multimedia, pages 125–134. Springer, 2018a.
- Scoringnet: Learning key fragment for action quality assessment with ranking loss in skilled sports. In Asian Conference on Computer Vision, pages 149–164. Springer, 2018b.
- Manipulation-skill assessment from videos with spatial attention network. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
- Towards unified surgical skill assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9522–9531, 2021.
- Extraction and classification of diving clips from continuous video footage. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 38–48, 2017.
- Action assessment by joint relation graphs. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6331–6340, 2019.
- Action quality assessment across multiple actions. In 2019 IEEE winter conference on applications of computer vision (WACV), pages 1468–1476. IEEE, 2019.
- Measuring the quality of exercises. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 2241–2244. IEEE, 2016.
- Learning to score olympic events. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 20–28, 2017.
- What and how well you performed? a multitask learning approach to action quality assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 304–313, 2019.
- Vi-net—view-invariant quality of human movement assessment. Sensors, 20(18):5258, 2020.
- Neurosymbolic programming for science. arXiv preprint arXiv:2210.05050, 2022.
- Uncertainty-aware score distribution learning for action quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
- Python tutorial. Centrum voor Wiskunde en Informatica Amsterdam, The Netherlands, 1995.
- Learning to score figure skating sport videos. IEEE transactions on circuits and systems for video technology, 30(12):4578–4590, 2019.
- Finediving: A fine-grained dataset for procedure-aware action quality assessment, 2022.
- Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
- Group-aware contrastive regression for action quality assessment. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7919–7928, 2021.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.