Reviews

Search results

• UIClip: A Data-driven Model for Assessing User Interface Design

  Jason Wu, Yi-Hao Peng, Xin Yue Amanda Li, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols · 2024 · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24)

  This paper introduces UIClip, a computational model that automatically assesses UI design quality and visual relevance from a screenshot and natural language description. Built on OpenAI's CLIP B/32 architecture (151 million parameters), UIClip is fine-tuned on a novel…

  UI design · machine learning · design quality assessment · computer vision · CLIP

• Towards Automated Accessibility Report Generation for Mobile Apps

  Amanda Swearngin, Jason Wu, Xiaoyi Zhang, Esteban Gomez, Jen Coughenour, Rachel Stukenborg, Bhavya Garg, Greg Hughes, Adriana Hilliard, Jeffrey P. Bigham, Jeffrey Nichols · 2024 · ACM Transactions on Computer-Human Interaction

  This paper presents a system for automatically generating whole-app accessibility reports for mobile apps, addressing key limitations of existing accessibility scanning tools. The work begins with formative interviews with eight accessibility QA professionals at a large…

  automated testing · accessibility testing · mobile accessibility · app crawling · machine learning

• AutoChemplete - Making Chemical Structural Formulas Accessible

  Merlin Knaeble, Gabriel Sailer, Zihan Chen, Thorsten Schwarz, Kailun Yang, Mario Nadj, Rainer Stiefelhagen, Alexander Maedche · 2023 · Proceedings of the 20th International Web for All Conference (W4A)

  This paper presents AutoChemplete, an interactive labeling tool that makes chemical structural formulas accessible to blind and low-vision (BLV) students. Chemical structural formulas — visual diagrams showing atoms, bonds, and molecular geometry — are fundamental to studying…

  STEM accessibility · blind and low vision · chemistry · machine learning · interactive labeling

• AutoChemplete - Making Chemical Structural Formulas Accessible (Extended Abstract)

  Merlin Knaeble, Gabriel Sailer, Zihan Chen, Thorsten Schwarz, Kailun Yang, Mario Nadj, Rainer Stiefelhagen, Alexander Maedche · 2023 · Proceedings of the 20th International Web for All Conference (W4A)

  This extended abstract presents AutoChemplete, an interactive labeling tool designed to make chemical structural formulas accessible to blind and low vision (BLV) students. The paper highlights a stark gap in STEM education: while 69% of US BLV students express interest in STEM…

  STEM accessibility · chemistry accessibility · blind and low vision · interactive labeling · machine learning

• OPTIMAL-EM: Optimised Population Sourcing for Web Accessibility Evaluation

  Alexander Hambley, Yeliz Yesilada, Markel Vigo, Simon Harper · 2023 · Proceedings of the 20th International Web for All Conference (W4A '23)

  This extended abstract presents OPTIMAL-EM, a prototypical tool and framework designed to optimise the web accessibility evaluation process by systematically selecting representative pages from a website for auditing. Currently, accessibility evaluations supported by…

  automated accessibility testing · web accessibility · accessibility evaluation · web crawling · machine learning

• Towards Optimised Population Sourcing for Web Accessibility Evaluation

  Alexander Hambley · 2023 · Proceedings of the 20th International Web for All Conference (W4A '23)

  This doctoral consortium extended abstract presents a PhD project proposing a novel framework and prototypical tool for optimising web accessibility evaluation through statistically representative page sampling. The work is the single-author companion to the multi-author…

  automated accessibility testing · web accessibility · accessibility evaluation · web crawling · machine learning

• The Sem-Lex Benchmark: Modeling ASL Signs and their Phonemes

  Lee Kezar, Jesse Thomason, Naomi Caselli, Zed Sehyr, Elana Pontecorvo · 2023 · ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

  This paper introduces the Sem-Lex Benchmark, the largest curated dataset of its kind for American Sign Language (ASL) isolated sign recognition, containing over 84,000 videos of isolated sign productions from 41 deaf ASL signers. The dataset addresses two critical barriers in…

  sign language recognition · American Sign Language · machine learning · phonology · dataset

• AdaptiveSound: An Interactive Feedback-Loop System to Improve Sound Recognition for Deaf and Hard of Hearing Users

  Hang Do, Quan Dang, Jeremy Zhengqi Huang, Dhruv Jain · 2023 · ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

  This paper introduces AdaptiveSound, the first feedback-loop sound recognition system designed for deaf and hard of hearing (DHH) users. Sound recognition tools help DHH people become aware of environmental sounds ranging from safety-critical alerts like fire alarms and sirens…

  deaf and hard of hearing · sound recognition · human-in-the-loop · machine learning · incremental learning

• Reimagining Machine Learning's Role in Assistive Technology by Co-Designing Exergames with Children Using a Participatory Machine Learning Design Probe

  Jared Duval, Laia Turmo Vidal, Elena Márquez Segura, Yinchu Li, Annika Waern · 2023 · Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2023)

  This paper fundamentally reframes the role of machine learning in assistive technology, arguing that ML models do not need to be accurate to be valuable — they can serve as sources of play and motivation rather than diagnostic tools. The researchers developed Cirkus, a…

  machine learning · participatory design · exergames · children · sensory processing

• "Not There Yet": Feasibility and Challenges of Mobile Sound Recognition to Support Deaf and Hard-of-Hearing People

  Jeremy Zhengqi Huang, Hriday Chhabria, Dhruv Jain · 2023 · Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS)

  This paper presents the first longitudinal field study of a mobile sound recognition system used by Deaf and hard of hearing (DHH) people in their daily lives. The researchers deployed SoundWatch, a smartwatch-based app that uses a deep learning model (Google YAMNet…

  deaf and hard of hearing · sound recognition · wearable technology · smartwatch · assistive technology

• Machine Generation of Audio Description for Blind and Visually Impaired People

  Virgínia P. Campos, Tiago M. U. de Araújo, Guido L. de Souza Filho, Luiz M. G. Gonçalves · 2023 · ACM Transactions on Accessible Computing

  This paper presents an extension to CineAD, a system for automatically generating audio descriptions (AD) for videos. The authors address a critical accessibility gap: most videos, films, and cultural programming lack audio descriptions, leaving blind and visually impaired (BVI)…

  audio description · blind and visually impaired · computer vision · machine learning · video accessibility

• Never-ending Learning of User Interfaces

  Jason Wu, Rebecca Krosnick, Eldon Schoop, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols · 2023 · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST)

  This paper introduces the Never-ending UI Learner, an automated system that continuously crawls real mobile applications to learn semantic properties of user interfaces. The system addresses a fundamental limitation of current approaches to UI understanding: most machine…

  machine learning · mobile accessibility · UI semantics · automated crawling · tappability

• WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

  Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, Jeffrey P. Bigham · 2023 · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

  This paper introduces WebUI, a large-scale dataset of approximately 400,000 web pages automatically crawled and paired with visual, semantic, and stylistic metadata extracted from the browser engine. The dataset addresses a critical bottleneck in UI understanding research:…

  machine learning · computer vision · UI modeling · web semantics · transfer learning

• WearSkill: Personalized and Interchangeable Input with Wearables for Users with Motor Impairments

  Ovidiu-Andrei Schipor, Laura-Bianca Bilius, Radu-Daniel Vatavu · 2022 · Proceedings of the 19th International Web for All Conference (W4A)

  This paper introduces WearSkill, a web-based application designed to provide personalized and interchangeable input for wearable computing, with a specific focus on users with motor impairments. The system addresses a fundamental challenge in wearable accessibility: most…

  wearable technology · motor impairments · personalization · input modalities · gesture input

• Machine Learning for Accessible Web Navigation

  Tlamelo Makati · 2022 · Proceedings of the 19th International Web for All Conference (W4A)

  This extended abstract explores how machine learning techniques, particularly reinforcement learning, can be adapted to improve web navigation accessibility in alignment with WCAG Guideline 2.4 (Navigable). The author observes that ML techniques have already been applied to…

  machine learning · web accessibility · reinforcement learning · web navigation · query optimization

• On the Identification of Accessibility Bug Reports in Open Source Systems

  Wajdi Aljedaani, Mohamed Wiem Mkaouer, Stephanie Ludi, Ali Ouni, Ilyes Jenhani · 2022 · Proceedings of the 19th International Web for All Conference (W4A)

  This paper addresses a significant gap in software engineering research: the automated identification of accessibility-related bug reports in open-source projects. While bug tracking systems like Bugzilla and Monorail accumulate vast numbers of reports, manually sifting through…

  accessibility bug reports · machine learning · open source · bug classification · automated testing

• The story behind Dytective: how we brought research results on dyslexia and accessibility to Spanish public schools

  Luz Rello · 2022 · Proceedings of the 19th International Web for All Conference (W4A)

  This invited talk paper presents the decade-long research and entrepreneurial journey behind Dytective, a platform that combines machine learning and gamified exercises to detect risk of dyslexia and provide personalized interventions. Dyslexia affects up to 10% of the global…

  dyslexia · machine learning · screening · serious games · social entrepreneurship

• Evaluating Haptic Technology in Accessibility of Digital Audio Workstations for Visual Impaired Creatives

  Christina Karpodini · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '22)

  This doctoral consortium paper proposes a two-stage research programme to make Digital Audio Workstations (DAWs) more accessible for musicians with visual impairments through haptic feedback. The author argues that modern music production has become an increasingly visual…

  haptic technology · visual impairment · music accessibility · digital audio workstation · vibrotactile feedback

• Blind Users Accessing Their Training Images in Teachable Object Recognizers

  Jonggi Hong, Jaina Gandhi, Ernest Essuah Mensah, Farnaz Zamiri Azar, Kyungjun Lee, Hernisa Kacorri · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '22)

  This paper introduces MYCam, an open-source iOS testbed application designed to help blind users build and improve personalized object recognizers. While teachable object recognizers allow users to train custom models by taking photos of objects they want to recognize, blind…

  teachable AI · object recognition · blind users · machine learning · camera interaction

• A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers

  Sanjana Shivani Chintalapati, Jonathan Bragg, Lucy Lu Wang · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '22)

  This paper examines the quality and availability of alt text for data visualizations in scientific publications, focusing on papers from the HCI and accessibility communities (CHI and ASSETS conferences, 2010-2020). The authors processed over 25,000 paper PDFs and found that…

  alt text · data visualization · scientific documents · blind and low vision · document accessibility

• Data Representativeness in Accessibility Datasets: A Meta-Analysis

  Rie Kamikubo, Lining Wang, Crystal Marte, Amnah Mahmood, Hernisa Kacorri · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '22)

  This paper conducts a systematic meta-analysis of demographic representativeness in 190 accessibility datasets — datasets sourced from people with disabilities and older adults — spanning from 1984 to 2021. The authors examine how age, gender, and race and ethnicity are…

  AI fairness · datasets · representation · diversity · inclusion

• AIDE: An Automatic Image Description Engine for Review Imagery

  Rachana Sreedhar, Nicole Tan, Jingyue Zhang, Kim Jin, Spencer Gregson, Eli Moreta-Feliz, Niveditha Samudrala, Shrenik Sadalgi · 2022 · Proceedings of the 19th International Web for All Conference (W4A)

  This paper from Wayfair presents AIDE, a multimodal machine learning system that automatically generates contextual alt-text for user-submitted review images in e-commerce — a category of imagery that is particularly inaccessible because it is user-generated, unpredictable in…

  alt text · computer vision · blindness · visual impairment · screen readers

• Automatically Generating and Improving Voice Command Interface from Operation Sequences on Smartphones

  Lihang Pan, Chun Yu, JiaHui Li, Tian Huang, Xiaojun Bi, Yuanchun Shi · 2022 · Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

  This paper presents AutoVCI, a system that automatically generates voice command interfaces (VCIs) for smartphone tasks from recorded touch operation sequences, enabling hands-free and eyes-free interaction without requiring programming expertise, corpora, or hand-written rules.…

  voice command interface · mobile accessibility · speech recognition · natural language understanding · programming by demonstration

• "Every Website Is a Puzzle!": Facilitating Access to Common Website Features for People with Visual Impairments

  Natã M. Barbosa, Jordan Hayes, Smirity Kaushik, Yang Wang · 2022 · ACM Transactions on Accessible Computing

  This paper addresses a fundamental challenge for screen reader users: while sighted users can rely on visual conventions to find common website features (log in at top right, contact at bottom), these shortcuts are inaccessible to users with visual impairments who must linearly…

  visual impairment · screen readers · web accessibility · machine learning · crowdsourcing

• Technical Perspective: Computation Where the (Inter)Action Is

  Jeffrey P. Bigham · 2022 · Communications of the ACM

  This one-page technical perspective accompanies the SoundWatch paper in Communications of the ACM. Bigham uses SoundWatch — a smartwatch prototype that detects audio events and displays descriptions for deaf and hard-of-hearing people — as a lens to explore broader questions…

  wearable technology · deaf and hard of hearing · sound recognition · smartwatch · assistive technology