Global Academy of Technology
Rajarajeshwari Nagar, Bengaluru-560098
Department of Electronics and Communication
Engineering
Project Phase I Presentation
(21ECE76)
On
“Real-Time Sign Language Translation using CNN-LSTM
Hybrid Architecture for Spatial-Temporal Gesture
Synthesis
and
Grammar
Refinement.
”
PRESENTED BY:
Ms C Saanvi
Ms Varsha P V
Mr Chidaksh Babu
1GA21EC028
1GA21EC165
1GA21EC032
Under the guidance of : Shubha GN
1
CONTENTS
1.
Introduction
2.
Literature survey
3.
Problem definition
4.
Objectives
5.
Methodology (Option)
6.
Expected Results
7.
References
2
INTRODUCTION
1. This project aims to bridge the communication gap for individuals relying on sign language by translating gestures into text
in real time using advanced deep learning techniques.
2. This project addresses critical challenges such as limited sign language proficiency among the general population, high costs,
and limited availability of interpreters, as well as the inability of existing tools to recognize dynamic gestures.
3.LSTM (Long Short-Term Memory) is a type of recurrent neural network (RNN) designed to handle sequential data
CNN (Convolutional Neural Network) is a deep learning model that specializes in extracting spatial features from images
4.By combining Convolutional Neural Networks (CNNs) for spatial feature extraction and Long Short-Term Memory (LSTM)
for temporal sequence modeling, the system captures the complexity of hand gestures.
5.Attention mechanisms optimize the recognition of critical frames within gesture sequences, while the MediaPipe Holistic
pipeline ensures accurate hand tracking by capturing 126 hand landmarks in three dimensions.
6.Users can train the system on custom datasets, allowing adaptability to various sign languages and dialects. Real-time
processing enables seamless translation of gestures to text, with grammar correction performed using the language_tool_python
library to ensure clarity and coherence.
7.The project’s applications span healthcare, education, customer service, workplace accessibility, and social integration,
promoting inclusivity and accessibility for individuals with hearing and speech impairments.
3
LITERATURE SURVEY
Sl
Paper/Journal
No.’;
name
Author Year of
name publication
Overview
1.
Sign Language Production With Pan Xie, Taiying 2024
Latent Motion Transformer - Peng, Yao Du,
Applications of Computer Vision Qipeng Zhang
(WACV)
This research introduces a novel method for Sign Language Production (SLP) that
generates high-quality sign videos directly from sign gloss without relying on human
poses. It uses a two-stage model: (i) an improved 3D VQ-GAN for learning latent
representations and (ii) sequence-to-sequence attention for better feature ordering. To
enhance semantic coherence, the model incorporates perceptual and reconstruction
losses. Experiments on WLASL and NMFs-CSL datasets show superior performance
over previous approaches.
2.
Sign Language Translation with
Sentence Embedding Supervision German Research Center for
Artificial Intelligence
2024
Yasser
Hamidullah Josef,
van
Genabith
Cristina EspañaBonet
This research introduces a novel gloss-free Sign Language Translation (SLT) method
using sentence embeddings instead of gloss annotations, enabling multilingual learning
without manual labeling. Evaluated on German (PHOENIX-2014T) and American
(How2Sign) sign languages, it significantly outperforms other gloss-free approaches,
narrowing the gap with gloss-based systems.
3.
Towards
Privacy-Aware
Sign Phillip Rust, Bowen 2024
Shi,
Skyler
Language Translation at Scale Wang, Necati Cihan
ACL-anthology
This research proposes SSVP-SLT, a privacy-aware, two-stage SLT framework using
self-supervised pretraining on anonymized videos and supervised fine-tuning. It
achieves state-of-the-art performance on How2Sign, addressing both data scarcity and
privacy concerns.
4.
iSign: A Benchmark for Indian Abhinav,Sudeep
Sign Language Processing Modi,Monali
ACL-anthology
Barbate
This research presents iSign, a benchmark for Indian Sign Language (ISL) processing,
featuring the largest ISL-English dataset. It introduces multiple NLP tasks like
SignVideo2Text and Text2Pose, benchmarks them with baseline models, and provides
linguistic insights, streamlining ISL research and enhancing accessibility for the NLP
and sign language communities.
5.
Unsupervised
Sign
Language Zhengs
Xhui, 2024
Translation and Generation
Xing Wang
Camgöz
2024
USLNet is an unsupervised sign language translation and generation model that learns
from non-parallel text and video data. It employs single-modality reconstruction and
cross-modality back-translation, tackling text-video discrepancies with a sliding window
method. Experiments on BBC-Oxford and Open-Domain ASL datasets show its 4
effectiveness, rivaling supervised models.
LITERATURE SURVEY
Sl
No.
Paper/Journal
name
Author Year of
name publication
6.
End-to-End Sign Language Video Yu
Duan
Generation with Latent Diffusion
Changsheng Xu
-CVVSR
Huaiwen Zhang
7.
LLMs are Good
Translators-CVVSR
Sign
Overview
· 2024
·
This research pioneers end-to-end text-to-sign video translation, capturing full-body and
facial expressions for enhanced expressiveness. It improves diffusion models with
multimodal conditions and a motion-aware framework for better text-visual alignment.
Experiments on multiple datasets demonstrate superior semantic accuracy, naturalness,
and expressiveness compared to state-of-the-art methods.
Language Jia Gong, Lin Geng 2024
Foo, Yixuan He
SignLLM leverages large language models (LLMs) for Sign Language Translation
(SLT) by transforming sign videos into a language-like representation. It introduces
vector-quantized visual sign tokens and an alignment module to enhance semantic
compatibility. Experiments show state-of-the-art gloss-free performance on SLT
benchmarks, improving translation readability and accuracy.
8.
YouTube-ASL: A Large-Scale, OpenDomain American Sign LanguageEnglish Parallel Corpus
-Arrve
David
Uthus,
Tanzer,
Georg
2024
YouTube-ASL is a massive ASL dataset with ~1000 hours of videos and >2500 unique
signers, significantly larger than previous datasets. It enables state-of-the-art fine-tuned
ASL-to-English translation and achieves the first nontrivial zero-shot results. This
resource advances machine learning research for ASL by addressing data scarcity and
supporting diverse signers.
9.
An Isolated American Sign Language Thad Starner, Sean 2024
Dataset Collected via Smartphones
Forbes,
Matthew
-Oxgh
So, David Martin
PopSign is a smartphone-based game that helps hearing parents of deaf infants learn
sign language by integrating sign recognition into gameplay. The PopSign ASL v1.0
dataset, collected from 47 Deaf adult signers using Pixel 4A cameras, features 250 ASL
signs with over 175,000 verified examples. A baseline LSTM model achieves 84.2%
accuracy, demonstrating its potential for educational use.
10.
ASL Citizen: A Community-Sourced Aakasha
Desai, 2024
Dataset for Advancing Isolated Sign Lauren Berger
Language Recognition
-Oxgh
ASL Citizen is the first crowdsourced dataset for Isolated Sign Language Recognition
(ISLR), featuring 83,399 videos of 2,731 ASL signs from 52 signers in diverse settings.
Designed for ASL dictionary retrieval, it enables users to match their demonstrated
signs via webcam. Supervised models trained on this dataset achieve 63% accuracy and
91% recall-at-10.
Garrett
Manfred
5
LITERATURE SURVEY
Sl
No.
Paper/Journal
name
Author Year of
name publication
Overview
11.
Leveraging Large Language Models Ryan
Wong
for
Gloss-Free
Sign
Language Richard Brodi
Translation-Argv
, 2024
Sign2GPT is a framework for gloss-free sign language translation that integrates
pretrained vision and language models using lightweight adapters, addressing data
scarcity and computational constraints. It introduces a novel pretraining strategy using
pseudo-glosses to enhance sign representation learning. Tested on RWTH-PHOENIXWeather 2014T and CSL-Daily, it achieves state-of-the-art performance.
12.
An Explicit Multi-modal Fusion Cong Hu, Biao Fu, 2024
Method for Sign Language Translation Pei Yu
-Argv
Sign Language Production (SLP) faces significant multimodal semantic gaps due to
limited cross-modal alignment labeling, unlike sign language translation. Generating
complex visual semantics from text remains challenging. Similar issues are addressed in
cross-modal retrieval, vision-language understanding, and multimodal sentiment
analysis by aligning and comparing cross-modal semantics in rich multimodal data.
13.
Unsupervised
Sign
Language Bian Wu, Chenyong 2024
Translation and Generation-WordXl
Guan
USLNet is the first unsupervised model for sign language translation and generation,
learning from single-modality data without parallel datasets. It combines singlemodality reconstruction and cross-modality back-translation, addressing text-video
alignment issues using a sliding window method. Tested on BOBSL and OpenASL, it
demonstrates competitive performance against supervised models.
14.
Uncertainty-aware sign language video Ronglai
retrieval with probability distribution Fangyun We
modeling-WordXl
UPRet introduces an uncertainty-aware probability distribution framework for sign
language video retrieval, addressing the complexity and uncertainty in sign language.
By enabling flexible mappings between video and text, it overcomes limitations of finegrained annotation scarcity. UPRet achieves state-of-the-art performance on How2Sign
(59.1%), PHOENIX-2014T (72.0%), and CSL-Daily (78.4%).
15.
Signer
Diversity-driven
Data Honcho Fu, Guan 2024
Augmentation for Signer-Independent Che
Sign Language-Agvrv
Zuo, 2024
The Signer Diversity-driven Data Augmentation (SDDA) method enhances signerindependent SLT by enriching signer diversity without using costly signer identity
labels. Combining adversarial training and diffusion-based augmentation, SDDA
generates diverse signer examples. Consistency and discrimination losses further
improve signer-independent feature learning, achieving state-of-the-art SLT
6
performance in signer-independent settings.
PROBLEM STATEMENT
The aim of this project is to develop a system using machine learning techniques like a hybrid CNN-LSTM model which
uses hand tracking to enable real-time sign language recognition and translation into text, improving communication for
individuals with hearing impairments and enhancing social inclusion. Additionally, it incorporates grammar correction
to ensure the accuracy of the translated sentences.
This Project combines improves upon the literature survey as given below:
CNN-LSTM model with attention mechanisms providing:
1. Better Temporal and Spatial Feature Extraction: CNNs handle spatial features, LSTMs manage temporal sequences,
and attention. It selectively prioritizes critical frames
2. Enhanced Long-Range Dependency Handling: Attention focuses on key gestures without the need for exhaustive
sequence processing like TCNs
3. Real-Time Efficiency: The combination reduces computational complexity, making it more suitable for real-time
translation tasks compared to the heavier TCN approach. In summary, by integrating LSTMs and attention, this project
effectively addresses the vanishing gradient issue while maintaining computational efficiency and accuracy,
outperforming TCN-based models in both performance and speed.
4. Enhanced Frame Prioritization: The attention mechanism improves recognition of critical frames, something bidirectional LSTMs lack by default.
7
8
Advantages & Disadvantages
 Advantages
 Disadvantages
1. Customizable Data Collection: Users can create
personalized sign language datasets through a data
collection process, enhancing the model's adaptability to
various sign languages.
2. Real-Time Predictions: The system offers real-time
hand gesture recognition using hand landmarks,
facilitating immediate translation of sign language into
text.
3. Accurate Hand Tracking: The use of MediaPipe's
Holistic pipeline ensures precise hand tracking, which is
crucial for effective sign language recognition.
4. Grammar Correction: Integration with the language
tool python library allows for grammar correction of
predicted sentences, improving the readability and
accuracy of translations
1. Data Collection Requirement: Users need to collect
and label their own datasets, which can be timeconsuming and may require substantial effort to achieve
a comprehensive dataset.
2. Potential Performance Variability: The accuracy of
the model is highly dependent on the quality and
diversity of the training data, insufficient data can lead to
inefficient performance.
3. Technical Proficiency Needed: Setting up and training
the model may require a certain level of technical
expertise in machine learning and programming, which
could be a barrier for some users.
9
OBJECTIVES
A. Spatial-Temporal Gesture Analysis:
Building a deep learning framework using CNN for capturing spatial features of hand shapes and LSTM for tracking
temporal sequences of gestures to accurately map sign language movements.
B. Gesture Recognition:
Developing a system that performs accurately even with variations like different lighting conditions, hand positions,
or gesture speeds by enhancing the adaptability of the neural network architecture.
C. Grammar-Aware Text Generation:
Integrating Natural Language Processing (NLP) tools to ensure the translated text maintains proper sentence structure
and grammar, making the output easily understandable.
D. Real-Time Hand Tracking:
Using MediaPipe Holistic for precise hand landmark detection and motion tracking to ensure quick and accurate
gesture input processing.
E. Data Enhancement Techniques:
Applying techniques such as flipping, scaling, and frame shuffling to generate diverse training data, improving the
model's ability to recognize different types of gestures.
F.
10
Methodology
The block diagram below depicts the methodology for sign language translation:
Fig 1:Block diagram depicting the methodology
11
1. Create a Dataset
• This step involves collecting and annotating gesture sequences to build a labeled dataset.
• Tools like MediaPipe Holistic are used to extract hand landmarks and store the data in a structured format.
• This data serves as the foundation for training and testing the model.
2. Spatial Feature Extraction Using CNN
• The CNN layers process image data to extract spatial features such as hand shapes, finger orientations, and movement patterns.
• These features are critical for identifying the static aspects of gestures.
3. Temporal Dependencies Are Captured Using LSTM
• LSTM layers are used to analyze the temporal relationships between frames in a gesture sequence.
• This step captures how gestures evolve over time, allowing for interpretation of dynamic gestures.
4. Capturing Hand Gesture Sequences
• This step involves recording a series of gesture frames over time to form complete sequences.
• These sequences are essential for understanding the context and meaning of the gestures.
5. Feature Extraction
• The extracted features (both spatial from CNN and temporal from LSTM) are combined and refined.
• This integrated feature set is used to classify gestures accurately.
6. Sentence Generation
• Recognized gestures are mapped to corresponding words, which are then structured into sentences.
• This involves grouping predictions into meaningful outputs.
7. Grammar Correction
• The generated sentences are passed through a grammar correction module (e.g., using language_tool_python).
• This step ensures the final output is grammatically correct, enhancing readability and usability.
12
APPLICATIONS
1. Helps individuals who use sign language communicate with people who do not understand it by translating sign
language into text or speech in real time
2. Can be used in classrooms or workshops to aid in communication between teachers and hearing-impaired students.
3. Students learning sign language can use the system to validate their signs and improve accuracy.
4. Enhances communication between healthcare providers and patients with hearing impairments.
5. Collects data to analyze different sign languages and their variations, contributing to linguistic and AI research.
6. When combined with speech-to-text or text-to-speech tools, this system can facilitate communication across different
spoken languages, breaking barriers for sign language users.
13
EXPECTED RESULTS
Gesture Recognition:The system will effectively classify various hand gestures with high precision, supporting dynamic and complex sign
language recognition.
Feature Extraction:The CNN will capture hand shapes and positions, while LSTM models will learn gesture sequences over time, enhanced
by attention mechanisms focusing on key frames.
Real-Time Processing:Integration with MediaPipe Holistic allows real-time hand tracking and low-latency processing for smooth user
interaction.
Text Translation: Recognized gestures will be translated into grammatically correct text using automated refinement tools.
14
REFERENCES
1 Kumar, P., Verma, A., & Raj, S. (2023). Gesture Recognition in Sign Language Using CNN and Bi-directional LSTM. IEEE Transactions on
Pattern Analysis, 45(5), 1568-1580.
2 Yang, T., Zhang, H., & Li, Q. (2023). Gesture Translation in Sign Language with Deep Reinforcement Learning. AI & Robotics, 29(4), 220-234.
3 Zhang, W., Liu, B., & Wang, T. (2023). Hand Gesture Recognition for Sign Language Translation Using CNN and Attention Mechanism. Journal
of Vision and Graphics, 27(2), 60-74.
4 Zhou, Y., & Chen, H. (2023). Efficient Hand Gesture Recognition for Sign Language Translation Using Transformer Networks. Computer Vision
and Image Understanding, 214, 1-12.
5 Li, M., & Zhao, X. (2023). Sign Language Recognition Using 3D Convolutional Neural Networks. IEEE Transactions on Image Processing,
32(5), 1356-1367.
6 Wang, J., & Hu, L. (2023). Cross-Domain Sign Language Translation Using Generative Adversarial Networks. IEEE Transactions on
Cybernetics, 53(7), 4873-4884.
7 Zhang, S., & Liu, Z. (2023). A Comprehensive Survey on Sign Language Recognition and Translation: Methods, Applications, and Future
Directions. IEEE Access, 11, 24560-24577.
8 Singh, A., & Gupta, N. (2022). Deep Learning-Based Gesture Recognition for Sign Language Translation. Journal of Machine Learning
Research, 23(4), 123-135.
9 Kumar, R., & Sharma, P. (2022). Multi-Modal Sign Language Translation Using Deep Learning. IEEE Access, 10, 12578-12589.
10 Kumar, A., & Sharma, R. (2022). Vision-based and Wearable-based Sign Language Recognition: A Comparative Study. Pattern Recognition,
132, 107830.
11 Soni, R., Jain, P., & Gupta, S. (2022). Real-Time Sign Language Translation with Machine Learning. Journal of Real-Time Systems, 58(3),
98-112.
12 Li, F., & Liu, P. (2021). Hybrid CNN-LSTM Models for Gesture Recognition in Sign Language. International Journal of Machine Learning,
43(3), 71-82.
13 Lee, S., Choi, J., & Kim, D. (2021). End-to-End Sign Language Recognition Using Deep Learning. IEEE Transactions on Neural Networks,
32(10), 2053-2067.
14 Zhang, H., Li, Y., & Wei, Z. (2021). Sign Language Recognition with Attention-Based CNN-LSTM Model. Pattern Recognition Letters, 145,
43-52.
15 Liu, J., & Wang, X. (2021). Sign Language Recognition Using Temporal Convolutional Networks. IEEE Transactions on Signal Processing, 69,
15
2642-2653.
THANK YOU
16