A user evaluation of the SADIe transcoder
Darren Lunn, Sean Bechhofer, Simon Harper · 2008 · Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '08) · doi:10.1145/1414471.1414498
Summary
This paper presents a user evaluation of SADIe (Structural-Semantics for Accessibility and Device Independence), a system that uses Semantic Web technologies to make web pages more accessible to screen reader users. The core problem SADIe addresses is that web pages convey structural meaning visually through CSS rendering — for example, a navigation menu is visually distinct from main content through its positioning and styling — but this implicit visual structure is lost when a screen reader produces a sequential audio rendering. SADIe works by annotating CSS classes found in a website's stylesheet with an ontology that classifies page elements into roles such as "Removable" (banners, ads, unnecessary clutter) and "Priority" (main content). The ontology has two layers: an upper ontology defining abstract roles common across all websites, and a lower ontology specific to each site that maps its CSS classes to those roles. SADIe then offers three transcoding modes: "Defluff" which removes elements classified as removable, "Reorder" which promotes important content to the top of the page, and "Menu" which moves navigation to the bottom where it is accessible but does not block main content. Four visually impaired participants (three profoundly blind, one partially sighted, all daily JAWS users aged 26-45) evaluated the system across 20 web pages chosen from the top 100 most-visited websites, performing fact-based question tasks on both original and SADIe-transcoded versions.
Key findings
On average, participants completed tasks significantly faster using SADIe-transcoded pages (mean = 54.19 seconds, SE = 10.48) compared to original pages (mean = 163.25 seconds, SE = 49.87). A t-test confirmed statistical significance (t(7,6) = -2.14, p = 0.033, p < 0.05). Importantly, there were no instances where users failed to complete a task on the SADIe version, while four task failures occurred on original pages. However, the evaluation also revealed nuanced findings: on two pages (Page 2 and Page 8), SADIe actually increased task completion time. For Page 2, the transcoding removed familiar structural cues — the user reached the main story within three seconds but rapidly tabbed past it because the page was not in the expected format. This highlighted that users develop spatial mental models of familiar page layouts, and restructuring can disrupt these models. Qualitative feedback revealed the critical importance of headings as navigational landmarks — participants relied heavily on JAWS' heading navigation feature, and pages with poor heading structure were frustrating regardless of transcoding. Participants also described "Coping Strategies" including searching for cached keywords, counting links, and using heading jumps to orient themselves — strategies the researchers compared to findings by Yesilada et al. on how visually impaired users navigate using structural cues.
Relevance
This study provides important evidence that semantic transcoding — transforming web pages based on the roles of their visual components — can meaningfully improve the screen reader experience. The statistically significant reduction in task completion time (from nearly 3 minutes to under 1 minute on average) demonstrates the real cost of visual clutter for non-visual browsing. For accessibility practitioners, the evaluation reveals several crucial insights: first, that removing irrelevant content ("defluffing") may be even more impactful than adding accessibility features; second, that users develop spatial models of websites and disrupting familiar layouts can be counterproductive; and third, that proper heading structure is the single most important navigational aid for screen reader users. The ontology-based approach of annotating CSS classes rather than individual page elements offers scalability, since a single site annotation applies across all pages sharing the same stylesheet. While the small sample size (4 participants, 20 pages) limits generalizability, the mixed-methods approach combining timed tasks with qualitative observation provides rich insights into real screen reader usage patterns.
Tags: web accessibility · screen readers · semantic web · transcoding · visual impairment · ontology · web content transformation · CSS · assistive technology
Standards referenced: WCAG