Volunteer-Based Online Studies With Older Adults and People with Disabilities
Qisheng Li, Krzysztof Z. Gajos, Katharina Reinecke · 2018 · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '18) · doi:10.1145/3234695.3236360
Summary
This paper validates volunteer-based online experiments as a methodology for conducting large-scale accessibility research with people with disabilities and older adults — populations that are notoriously difficult to recruit for laboratory studies. The researchers replicated four established laboratory studies on LabintheWild, a platform where volunteers participate in experiments in exchange for personalized performance feedback rather than monetary compensation. The four studies covered: (1) a Weather Prediction Task testing probabilistic learning in people with dyslexia (3,786 participants); (2) Sternberg's Memory Task examining age-related cognitive decline (18,026 participants); (3) Baron-Cohen's Social Intelligence "Reading the Mind in the Eyes" test related to autism (123,928 participants); and (4) a Fitts' Law pointing task studying age-related motor decline (209,916 participants). Across these studies, 355,656 total participants took part, with 1.35% self-reporting disabilities and 2.25% being over age 65. The researchers additionally analyzed participant comments and forum discussions about LabintheWild experiments to understand what motivates people with disabilities to participate.
Key findings
All four LabintheWild studies successfully replicated the key findings of their original laboratory counterparts, with much larger and more diverse samples. The Weather Prediction Study confirmed that people with dyslexia achieve lower forecasting accuracy (55% vs. 57% for controls), though it did not replicate the interaction effect found in the smaller original study. The Memory Study confirmed the age-related complexity effect across ages 22-99 as a continuous variable rather than binned groups. The Social Intelligence Study confirmed that people with ASD scored significantly lower (M=22.92 vs. M=26.29, Cohen's d=.73). The Fitts' Law Study replicated all four established findings about age-related motor decline. Beyond replication, the qualitative analysis of comments and forums revealed four powerful motivations for participation by people with disabilities: self-diagnosis (using tests to confirm suspected disabilities), comparison (posting and comparing scores with others who share their condition), self-experimentation (testing themselves before and after medications or interventions), and storytelling (sharing personal experiences and providing context for their results). Participants frequently used experiments not designed for specific disabilities as diagnostic tools — for example, autism forums shared the Social Intelligence Test even though the LabintheWild version never mentioned autism.
Relevance
This research has significant methodological implications for the accessibility field. The persistent challenge of recruiting participants with specific disabilities limits most accessibility studies to small samples, weakening generalizability. This paper demonstrates that volunteer-based platforms can attract thousands of participants with disabilities at no recruitment cost, producing data quality comparable to laboratory studies. For researchers, the key design insight is that personalized feedback — showing participants how they compare to others — is the primary motivator, replacing monetary compensation. The qualitative findings about participant motivations are equally important: people with disabilities are hungry for tools that help them understand their own abilities, compare with peers who share their condition, and track changes over time. This suggests that accessibility research platforms should provide disability-specific comparison groups, support longitudinal self-tracking, and create safe spaces for community discussion. The finding that participants appropriate experiments for self-diagnosis also raises ethical considerations — platforms should provide validated assessments and clear disclaimers to prevent misdiagnosis.
Tags: research methodology · online experimentation · accessibility research · aging · participant recruitment · crowdsourcing · dyslexia · autism · cognitive accessibility
Standards referenced: ISO 9241-9