The Implementation of a Vocabulary and Grammar for an Open-Source Speech-Recognition Programming Platform
Jean K. Rodriguez-Cartagena, Andrea C. Claudio-Palacios, Natalia Pacheco-Tallaj, Valerie Santiago González, Patricia Ordonez-Franco · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2811346
Summary
This paper presents the development of a standardized vocabulary and grammar for voice-based programming, designed to make coding accessible to people with limited hand mobility or visual impairments. The work is part of the larger Kavita Project, which aims to create an open-source spoken programming language platform. A key challenge in voice-based programming is ambiguity—every programmer has their own way of verbally describing code, creating inconsistency that speech recognition software cannot reliably interpret. The researchers addressed this by developing a generic spoken language for C-based programming languages with clear, unambiguous commands. They chose gerunds (verb forms ending in "-ing") as keywords because they are more phonetically distinguishable by speech recognition systems, reducing recognition errors. The system is built on Simon, an open-source voice recognition platform, making it freely available rather than dependent on commercial software like Dragon Naturally Speaking. The vocabulary covers essential programming constructs: defining variables, printing strings and variables, creating expressions with operators, storing user input, and control flow structures like for loops and if/else statements. For example, saying "defining integer variable" followed by "total" produces the code "int total;".
Key findings
The researchers validated their vocabulary and grammar through a survey of 33 respondents from the programming community, including senior computer science students and faculty members from two Puerto Rican universities. The survey presented proposed voice commands alongside their intended C++ output, gathering feedback on clarity and naturalness. The methodology involved analyzing a Perl script containing all elements of a generic C-based language, then developing voice commands that map to these constructs. Ten programming structures were covered in the survey, including variable definition, string and variable printing, equation results, user input, expressions with various operators, relational and logical expressions, for loops, and if/else structures. The project builds on prior work including Spoken Java (Andrew Begel's research showing programmers had "little trouble" learning a verbal Java system) and Tavis Rudd's Python voice coding demonstration at PyCon 2013. However, those systems used short sounds or cryptic commands optimized for experienced programmers—the Kavita Project aims for a more learnable, universally designed approach accessible to newcomers.
Relevance
This research addresses an important but often overlooked accessibility barrier: programming careers are effectively closed to people who cannot type extensively due to motor impairments. Carpal tunnel syndrome, repetitive strain injuries, and conditions affecting hand mobility can force programmers out of the profession or prevent entry entirely. Voice-based programming could open this career path. The open-source foundation (Simon platform) is significant—commercial voice recognition software is expensive and may not be accessible in all contexts. The focus on learnability through natural language commands (using gerunds and descriptive phrases) prioritizes newcomers and people learning to program, not just experienced developers seeking efficiency gains. For accessibility practitioners, this demonstrates the importance of designing assistive technology that works for beginners, not just power users. The survey-based validation approach also models how to involve the target community in refining assistive tools. While the paper represents early-stage work with a working Java implementation, it points toward a future where coding bootcamps and computer science education could be fully accessible to students with motor impairments.
Tags: speech recognition · voice input · programming · motor impairment · repetitive strain injury · assistive technology · open source · alternative input · carpal tunnel syndrome