Mining Web Interactions to Automatically Create Mash-Ups
Jeffrey P. Bigham, Ryan S. Kaminsky, Jeffrey Nichols · 2009 · Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST 2009) · doi:10.1145/1622176.1622215
Summary
This paper introduces TX2, a Firefox browser extension that automatically creates meta-search mash-ups by mining the web interactions of multiple users to discover relationships between query forms on different websites. The deep web — information accessible only through web forms — contains an order of magnitude more information than the surface web, but users are typically restricted to searching one site at a time. TX2 addresses this by recording user interactions with web forms (via CoScripter's ActionShot technology), using a connector service to find related forms across different sites based on shared input patterns, and an integrator that simultaneously queries matched sites and presents combined results on a single page. For example, when a user searches a library catalog, TX2 automatically searches other library catalogs and presents all results together. The system uses feature generators to extract semantic types from form inputs (dates, airport codes, zip codes) and matches forms across sites using a script model abstraction that groups similar interactions into meta-steps. Results from external sites are integrated into the original page using DOM structure analysis, with options to preserve either the source site's visual styling or match the host page's styling. Users can reorder, interleave, or filter results by source.
Key findings
TX2 successfully matched and combined results from 6 of 10 tested site pairs across libraries, travel, and search engine categories. The system correctly discovered connections between related sites in each category from a corpus of 13,159 interactive steps across 5,459 web pages recorded from 6 users over three months. The matching algorithm also found additional valid connections beyond the manually-recorded test pairs, such as southwest.com matching travel sites and Santa Clara Library matching other library sites. Failures occurred when sites used non-standard form submission methods (JavaScript-based submissions on eBay and Myspace) or when TX2 could not detect repeating result elements (video sharing sites). The paper explicitly notes an accessibility application: TX2 can be used to create custom accessible versions of websites by authoring a new accessible interface and connecting it to existing deep web resources, allowing blind or cognitively impaired users to access information through interfaces optimized for their needs.
Relevance
While primarily a web engineering contribution, this paper has significant accessibility implications that the authors explicitly identify. The ability to automatically create custom interfaces to existing web resources is directly relevant to accessibility: users who find one site's interface more accessible than another can access information from less accessible sites through the preferred interface. This concept anticipates modern approaches to web accessibility overlays and alternative interfaces. The idea that "people want great interfaces AND great information" and that users should be able to independently choose the interface and the information source is a powerful accessibility principle. TX2 demonstrates that mining collective user behavior can automatically surface connections between resources that would otherwise require programming expertise to create — democratizing access to the deep web for non-technical users. The work connects to Bigham's broader accessibility research agenda by exploring how technology can reduce barriers to information access, and the browser extension approach (modifying web content client-side) is the same pattern used in his accessibility tools like WebInSight and the Social Accessibility Project.
Tags: web accessibility · web development · information retrieval · end-user programming · personalization · user interface design