Mobile Sign Language Translation System for Deaf Community
Mehrez Boulares, Mohamed Jemni · 2012 · Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2207016.2207049
Summary
This paper from the University of Tunis presents a mobile web service for translating written text into sign language using a 3D virtual signing agent on Android devices. The system addresses two interrelated accessibility challenges: deaf and hard of hearing people with low written language literacy need sign language translations of text content (such as SMS messages and emails), while hearing people need tools to learn sign language for communication with deaf individuals. The architecture has three components: a Sign Modelling Language (SML) database that stores sign descriptions as XML-based notation following the H-Anim specification for humanoid body joint articulation; a 3D rendering engine that converts SML descriptions into animated avatar output; and a video conversion module that streams the result to mobile devices. The system uses a web tool that allows different sign language communities to collaboratively create and store their own sign dictionaries, addressing the diversity of sign languages across cultures. Users enter text on the Android app (either typed or from received SMS), which sends an HTTP request to the web service; the service looks up SML animations word by word, renders them into 3D animation, and returns a streaming video URL.
Key findings
The system offers several advantages over video-based sign language approaches. Avatar-based animation using SML notation produces fluid transitions between signs — a problem that video concatenation cannot solve, since merging separate video clips loses the natural flow between signs and can make meaning unclear. The collaborative multi-community approach allows different deaf communities to build their own sign dictionaries, reflecting the reality that sign languages vary significantly by region and culture. Performance testing showed an average of 5 seconds per sign for new animations (on an i5 processor with dedicated graphics), with previously rendered animations served instantly via cached URLs. The system supports two input modes: manual text entry with auto-completion (suggesting existing signs from the database) and direct translation of received SMS messages. The web-based sign creation tool uses a 3D virtual agent interface with sliders to control joint rotations and a timeline for sequencing movements, making sign creation accessible to non-technical users.
Relevance
This paper represents early work on mobile sign language translation that anticipated the growing importance of avatar-based signing technology. The collaborative, community-driven approach to sign dictionary creation is particularly valuable given that there are over 300 distinct sign languages worldwide with no universal standard. For accessibility practitioners, the system highlights that text-based content is not inherently accessible to all deaf users — many deaf individuals have limited written language literacy and rely primarily on sign language, making text-to-sign translation an important accessibility service alongside traditional captioning. The mobile-first approach also recognises that smartphones are increasingly the primary computing device for many users, including in developing regions where this research originated. The technical approach of using lightweight XML-based sign notation rather than video storage addresses bandwidth and scalability concerns that remain relevant for mobile accessibility solutions.
Tags: sign language · deaf accessibility · mobile accessibility · machine translation · signing avatar · assistive technology · web services · 3D animation
Standards referenced: H-Anim · X3D