Challenging and Improving Current Evaluation Methods for Colour Identification Aids
Connor Geddes, David R. Flatla · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 22) · doi:10.1145/3517428.3544818
Summary
This paper critically examines how Colour Vision Deficiency (CVD) aids are evaluated, exposing fundamental flaws in common evaluation practices. CVD affects approximately 5% of the population and impairs the ability to accurately identify and discriminate between colours — a challenge with real-world consequences from misreading transit maps to missing colour-coded medical symptoms. While various CVD aids exist (haptic, visual recolouring, and sound-based), the methods used to evaluate their effectiveness are problematic. Most evaluations rely on diagnostic tests like the Ishihara plates, which are too easy to pass with any aid, or on "indirect" evaluations where non-CVD participants view the world through CVD simulations rather than involving actual CVD participants. The authors propose two new evaluation tasks to address gaps in existing methods. The Category Task assesses connotative colour judgement — matching a colour to a meaning or category — which no previous evaluation task addressed. The Distinction Task measures how many distinct colour names an aid allows a CVD user to accurately identify simultaneously, providing a practical upper-bound metric for aid effectiveness. Both new tasks, along with three previously developed tasks (Selection, Transition, and Sorting), were tested in an online study with 284 participants, including 26 with confirmed CVD. The study used the Machado simulation method at three severity levels and three levels of contextual information provided to simulated participants.
Key findings
The results demonstrate conclusively that CVD simulations do not produce results equivalent to those from actual CVD participants, regardless of how much contextual information is provided to simulated participants. Across five colour tasks, statistically significant differences were found between CVD participants and all simulation conditions (p < .0001 in most tasks). CVD participants were consistently more accurate at naming colours than non-CVD participants under simulation, because people with CVD develop deep knowledge over a lifetime of navigating colour in a world not designed for them. The specific colours that CVD and simulated participants confused differed wildly — for example, CVD participants who incorrectly sorted "orange" colours typically named them "red" or "green," while simulated participants confused "orange" with "yellow" and "brown." Providing more information about CVD to simulated participants made no meaningful difference in their performance or behaviour. The two new evaluation tasks proved effective: the Category Task showed simulation accuracy close to CVD participants (the only task to do so), while the Distinction Task revealed clear differences in colour-naming strategies between groups.
Relevance
This research has significant implications for anyone developing or evaluating colour accessibility tools. The central finding — that CVD simulations should NOT be used to evaluate CVD aids — challenges a widespread practice in accessibility research where the majority of studies use simulations as their primary or sole evaluation method. For web accessibility practitioners, this reinforces that simulation tools built into browser developer tools (Chrome and Firefox both include CVD simulation) are useful for awareness but cannot substitute for testing with actual users who have CVD. The paper also highlights four types of colour challenges people with CVD face (comparative, denotative, connotative, and aesthetic), providing a framework for more comprehensive colour accessibility evaluation. Organizations developing colour-accessible interfaces should involve CVD participants directly rather than relying on simulation-based validation.
Tags: colour vision deficiency · colour blindness · CVD simulation · evaluation methods · colour identification aids · visual accessibility