UI Agent

An AI-powered software system that can autonomously interact with graphical user interfaces on behalf of a user, performing tasks by interpreting natural language commands and translating them into interface actions such as clicking buttons, entering text, and navigating between screens. In accessibility contexts, UI agents promise to make inaccessible or complex interfaces easier to use for people with disabilities, particularly blind and low-vision users who face barriers with conventional screen reader navigation. However, current agents often complete tasks end-to-end without involving users in critical decisions, raising concerns about user agency. Effective accessible UI agents should proactively pause at decision points and provide feedback through accessible modalities.