Capture: A Desktop Display-Centric Text Recorder
Oren Laadan, Andrew Shu, Jason Nieh · 2012 · Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2012) · doi:10.1145/2384916.2384919
Summary
This paper presents Capture, a display-centric text recording system that continuously tracks all onscreen text and metadata across both foreground and background windows with low overhead and without requiring modifications to applications, window systems, or operating system kernels. The fundamental problem is that while computer displays are designed for human visual consumption, the ability of computers to programmatically process display content lags far behind the rate at which information is generated. Screen readers, the primary assistive technology for accessing display content, have critical limitations: they are generally limited to the foreground window, they often cannot keep up with application update rates, and they require users to actively navigate or define "hot spots" to monitor for changes. Capture addresses these limitations through an intelligent caching architecture that integrates with the standard accessibility framework (GNOME AT-SPI on Linux). At its core is a "mirror accessibility tree" — a cached copy of the entire desktop's accessibility tree that is continuously updated via an event handler. The event handler employs five optimization mechanisms: event queuing, grouping (coalescing multiple events from the same source), deferring (batching bursty GUI transactions), reordering (prioritizing high-value events), and filtering/aging (discarding stale events).
Key findings
Evaluation against the Orca screen reader on eight real desktop applications (Adobe Reader, Firefox, GEdit, GNOME Terminal, OpenOffice Impress, OpenOffice Writer, Pidgin messaging, Thunderbird email) showed dramatic differences. Capture achieved 100% text coverage across all application workloads, while Orca achieved 100% for only two workloads and delivered under 60% coverage for half of them. For applications like Adobe Reader and OpenOffice Impress, Orca captured 0% of onscreen text. In multi-application scenarios (foreground + background windows), Capture maintained 53-95% coverage while Orca dropped to 0-33%. Runtime overhead was comparable between the two systems — both under 3% for all workloads — meaning Capture's dramatically better coverage came at no additional performance cost. The caching architecture enables novel capabilities beyond traditional screen reading: virtual hot spots that notify users of text changes anywhere on screen without specifying physical locations; desktop-wide text search that includes transient content (pop-ups, status messages, dynamically loaded text) that would otherwise be lost; and complete content logging for auditing and review purposes.
Relevance
This paper identifies and addresses a fundamental limitation of screen reader architecture that remains relevant today: the gap between what is visually displayed and what is programmatically accessible to assistive technology. The finding that a widely-used screen reader (Orca) captured 0% of content from some common applications is stark evidence that technical accessibility (having an accessibility API) does not guarantee functional accessibility. For accessibility practitioners and developers, the mirror tree caching approach offers an architectural model for how assistive technologies can more completely access display content. The concept of tracking background window changes is particularly important as modern computing increasingly involves multi-window, multi-application workflows. The virtual hot spot feature — notifying users when specific text or patterns appear anywhere on screen — addresses a persistent pain point for screen reader users who may miss dynamically updated content. While the prototype was Linux-specific, the architectural principles apply to any platform with an accessibility framework.
Tags: screen reader · accessibility API · accessibility tree · text recording · desktop accessibility · GNOME · Linux · AT-SPI · caching architecture · assistive technology