2000 character limit reached
E-TSL: A Continuous Educational Turkish Sign Language Dataset with Baseline Methods
Published 5 May 2024 in cs.CL, cs.CV, and cs.LG | (2405.02984v2)
Abstract: This study introduces the continuous Educational Turkish Sign Language (E-TSL) dataset, collected from online Turkish language lessons for 5th, 6th, and 8th grades. The dataset comprises 1,410 videos totaling nearly 24 hours and includes performances from 11 signers. Turkish, an agglutinative language, poses unique challenges for sign language translation, particularly with a vocabulary where 64% are singleton words and 85% are rare words, appearing less than five times. We developed two baseline models to address these challenges: the Pose to Text Transformer (P2T-T) and the Graph Neural Network based Transformer (GNN-T) models. The GNN-T model achieved 19.13% BLEU-1 score and 3.28% BLEU-4 score, presenting a significant challenge compared to existing benchmarks. The P2T-T model, while demonstrating slightly lower performance in BLEU scores, achieved a higher ROUGE-L score of 22.09%. Additionally, we benchmarked our model using the well-known PHOENIX-Weather 2014T dataset to validate our approach.
- Sign language recognition, generation, and translation: An interdisciplinary perspective. pages 16–31, 10 2019.
- Aligning subtitles in sign language videos, 2021.
- Neural sign language translation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7784–7793, 2018.
- Neural machine translation for sign language: Effect of data augmentation on bleu and rouge scores. In In: Proceedings of The 16th International Conference on Machine Vision Applications (MVA’19), pages 1–6, 2019.
- Sign language transformers: Joint end-to-end sign language recognition and translation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10023–10033, 2020.
- Content4all open research sign language translation datasets. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1–5, Los Alamitos, CA, USA, 2021. IEEE Computer Society.
- A simple multi-modality transfer learning baseline for sign language translation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5110–5120, 2022.
- Two-stream network for sign language recognition and translation. In Advances in Neural Information Processing Systems, 2022.
- Fully Convolutional Networks for Continuous Sign Language Recognition, pages 697–714. 11 2020.
- Addressing resource scarcity across sign languages with multilingual pretraining and unified-vocabulary datasets. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
- Self-mutual distillation learning for continuous sign language recognition. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11283–11292, 2021.
- Hand pose guided 3d pooling for word-level sign language recognition. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 3428–3438, 2021.
- Evaluation of american sign language generation by native asl signers. ACM Trans. Access. Comput., 1(1), may 2008.
- Sign language translation with hierarchical spatio-temporal graph neural network. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2131–2140, Los Alamitos, CA, USA, 2022. IEEE Computer Society.
- D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
- Assessing the deaf user perspective on sign language avatars. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’11, page 107–114, New York, NY, USA, 2011. Association for Computing Machinery.
- Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA, 2002. Association for Computational Linguistics.
- C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. pages 74–81, 2004.
- Mediapipe: A framework for building perception pipelines, 06 2019.
- S. Morrissey and W. Andy. An example-based approach to translating sign language. In Machine Translation Summit, 2005.
- B.S. Parton. Sign Language Recognition and Translation: A Multidisciplined Approach From the Field of Artificial Intelligence. The Journal of Deaf Studies and Deaf Education, 11(1):94–101, 09 2005.
- Progressive transformers for end-to-end sign language production. In Computer Vision – ECCV 2020, pages 687–705, Cham, 2020. Springer International Publishing.
- Autsl: A large scale multi-modal turkish sign language dataset and baseline methods. IEEE Access, 8:181340–181355, 2020.
- Using motion history images with 3d convolutional networks in isolated sign language recognition. IEEE Access, 10:18608–18618, 2022.
- Isolated sign recognition with a siamese neural network of rgb and depth streams. In IEEE EUROCON 2019 -18th International Conference on Smart Technologies, pages 1–6, 2019.
- Evaluation of hidden markov models using deep cnn features in isolated sign recognition. Multimedia Tools and Applications, 80:19137 – 19155, 2021.
- Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
- A new dataset for end-to-end sign language translation: The greek elementary school dataset. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 1958–1967, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.
- World Health Organization. Deafness and hearing loss. Accessed: 04-13-2024.
- Bosphorussign22k sign language recognition dataset. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 181–188, Marseille, France, 2020. European Language Resources Association (ELRA).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.