AI-Powered Multi-Modal Sign Language Detection and Sentence Completion
A real-time AI accessibility system that detects sign language gestures and converts them into natural language sentences using computer vision and GPT.
Communication can be difficult for people who use sign language when others around them do not understand the gestures. This project solves that problem by building a real-time sign language detection system that can recognize hand gestures and convert them into meaningful natural language.
The system uses YOLOv8 with a custom annotated dataset prepared through Roboflow to detect sign gestures in real time. After detecting the gesture, the output is processed with the OpenAI GPT API to form more complete and natural sentences instead of only showing isolated words.
The project is designed with future accessibility features in mind, including voice synthesis support. This makes it useful for building assistive communication tools that can help bridge the gap between sign language users and non-sign language users.
Key benefits
- Detects sign language gestures in real time
- Uses YOLOv8 with a custom annotated dataset
- Converts detected gestures into natural language sentences
- Supports future voice synthesis and accessibility features
- Helps improve communication for sign language users
What's included
- Real-time sign language detection system
- Custom dataset annotation using Roboflow
- YOLOv8-based gesture recognition model
- GPT-powered sentence completion workflow
- Flask/Streamlit-based application interface
Use Cases
- Sign language translation support
- Accessibility communication tools
- Real-time gesture recognition
- Assistive AI applications
- Future voice-enabled sign language systems
Key deliverables
- YOLOv8 sign detection model
- Custom annotated dataset workflow
- OpenAI GPT-based sentence generation
- Real-time computer vision pipeline
- Flask/Streamlit application setup
- Accessibility-focused system architecture