Design and Psychometric Evaluation of American Sign Language Translations of Usability Questionnaires
Larwan Berke, Matt Huenerfauth, Kasmira Patel · 2019 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/3314205
Summary
This paper presents the first formally translated and psychometrically validated American Sign Language (ASL) versions of two widely used usability questionnaires: the System Usability Scale (SUS) and the Net Promoter Score (NPS). Many deaf adults in the United States have lower levels of English reading literacy because English is not their first language — ASL is a distinct language with its own grammar and syntax. Yet no standardized usability instruments existed in ASL, effectively excluding these users from rigorous HCI research and commercial usability testing. The translation process followed established cross-linguistic instrument translation best practices across five stages: expert team translation (including native ASL signers from the Deaf community), high-quality video recording of a native signer performing the instruments, multiple rounds of back-translation evaluation where ASL interpreters independently translated the ASL videos back into English, summative user studies with DHH participants evaluating real software artifacts, and public dissemination of the final instruments. The back-translation process was particularly meticulous, revealing nuanced linguistic challenges — for example, the ASL sign SUGGEST could not substitute for "recommend" in the NPS because its connotation differed, and the sign MUST carried a deontic (obligation) meaning rather than the epistemic (likelihood) meaning intended in the original English. The team conducted four user studies across two phases involving a total of 72 DHH participants evaluating websites, apps, and physical products.
Key findings
The ASL-SUS demonstrated excellent psychometric properties comparable to the original English SUS. Internal reliability was strong with a Cronbach's Alpha of 0.898 (95% CI [0.88, 0.92]), closely matching the English SUS Alpha of 0.911. Criterion validity was confirmed through significant correlation with the ASL-Adjective Scale (Pearson's r = 0.759, p < 0.001), with no significant difference from the English SUS-to-Adjective correlation. Construct validity was established through factor analysis confirming the same two-factor structure (Usable and Learnable) as the English SUS, with Tucker's Phi congruence scores of 0.94-0.97 compared to prior published loadings. Mean scores for ASL-SUS (56.65) and English SUS (52.11) were not significantly different. The ASL-NPS also showed strong psychometric properties: internal reliability of 0.697 (at the "reasonable" threshold), and criterion validity confirmed through correlation with ASL-SUS (r = 0.712, p < 0.001). None of the 60 DHH participants in the ASL studies reported difficulty understanding or responding to the ASL-SUS questions. The research also demonstrated that cultural and linguistic considerations extend beyond simple word substitution — the team had to carefully navigate concepts like the distinction between evaluating a product versus recommending it to a friend, which carry different cultural weight in Deaf community discourse.
Relevance
This work removes a significant methodological barrier to including DHH users in HCI research and commercial usability evaluation. For researchers and UX practitioners, the freely available ASL-SUS and ASL-NPS instruments (videos, answer sheets, and response datasets published at RIT's lab website and ACM Digital Library) provide ready-to-use tools for conducting rigorous usability studies with deaf participants. The rigorous translation and validation methodology serves as a model for translating other standardized instruments into ASL or other signed languages — the authors explicitly note that different signed languages worldwide would each require their own translation efforts. For organizations committed to inclusive user research, these instruments address a practical gap: without validated tools in a participant's primary language, response data may reflect language comprehension difficulties rather than actual usability assessments. The work also highlights the broader principle that inclusive research requires inclusive instruments, not just inclusive recruitment.
Tags: American Sign Language · deaf and hard of hearing · usability questionnaires · psychometrics · SUS · NPS · back-translation · inclusive research methods · instrument translation