UI Detection

The use of computer vision and machine learning to automatically identify and classify user interface elements (buttons, text fields, icons, toggles, etc.) from screenshots or screen pixels. In accessibility contexts, UI detection enables systems to generate accessibility metadata for applications that lack proper developer-provided semantic information, allowing screen readers and other assistive technologies to interact with otherwise inaccessible interfaces. Modern UI detection typically uses object detection models trained on large annotated datasets of application screenshots.