Designing and Evaluating Head-based Pointing on Smartphones for People with Motor Impairments
Muratcan Cicek, Ankit Dave, Wenxin Feng, Michael Xuelin Huang, Julia Katherine Haines, Jeffry Nichols · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '20) · doi:10.1145/3373625.3416994
Summary
This paper presents a calibration-free head-based pointing (HBP) system for Android smartphones that uses the standard front-facing camera to enable people with motor impairments to control an on-screen cursor through head movements. The system was designed by a team including the lead author who has lived with significant motor impairments and over ten years of personal experience using head-based pointing technologies. The design addresses five core requirements: hardware-free (uses only the built-in camera), customizable (supports dwell, blink, and smile selection methods), calibration-free (maps relative head movement rather than absolute position), precise (pixel-level accuracy), and available as open source. The system uses ML Kit for Firebase to track the nose tip position from the front-facing camera, mapping relative head movements to cursor position changes via a carefully designed pipeline including input smoothing, gain factor scaling, motion thresholding, boundary clipping, and output smoothing. A key innovation is the edge-clipping mechanism that enables intuitive self-calibration — when the cursor reaches a screen edge and the user overshoots, the system discards further movement in that direction, allowing users to naturally re-center the head-to-cursor mapping without explicit calibration. The researchers conducted two Fitts's Law studies: a comparison with Eva Facial Mouse (42 participants without motor impairments) and what they believe is the first Fitts's Law study of mobile head tracking with participants with motor impairments (16 participants).
Key findings
In the comparison study with 42 able-bodied participants, the proposed HBP achieved a grand mean throughput of 0.90 bits per second (bps) versus 0.83 bps for Eva Facial Mouse, with HBP outperforming at lower indices of difficulty (ID < 4 bits) — the range most common in real mobile tasks. HBP maintained linear, predictable performance scaling across difficulty levels while EFM showed non-linear degradation. In the study with 16 participants with motor impairments, the grand mean throughput was 0.61 bps, with significant variation by impairment severity: mild impairments averaged 0.96 bps, moderate averaged 0.20 bps, and 5 participants with severe impairments could not complete any task blocks. Notably, HBP achieved 0.95 bps throughput at the highest difficulty level (ID 5.20 bits), involving targets half the size of Android's recommended touch target after near-full-screen movement, demonstrating fine precision capability. The system performed consistently across different devices and environments, with three remote participants completing tasks on their own phones. Involuntary limb movements causing chair instability posed a significant challenge for some participants during dwell selection, and one of the highest performers was a participant who was physically restrained in his chair, suggesting stability is a key factor.
Relevance
This research addresses a critical accessibility gap — smartphone interaction for people who cannot use touchscreens due to motor impairments. The calibration-free, hardware-free approach is particularly important because it eliminates barriers that often prevent adoption of assistive technologies: cost, setup complexity, and dependence on sighted assistance for configuration. The finding that many participants at a rehabilitation facility were unaware that free head-based pointing systems existed highlights a broader awareness gap in assistive technology distribution. The design guidelines that emerged — including the need for different interfaces for different ability levels, filtering involuntary movements, and potentially modifying the UI itself based on user capabilities — are valuable for any accessibility practitioner designing alternative input methods. The paper also demonstrates the value of lived experience in assistive technology design, as the lead author's decade of personal HBP use directly informed the design requirements and informed decisions that formal user research alone might have missed.
Tags: motor accessibility · head tracking · alternative input · mobile accessibility · Fitts law · computer vision · assistive technology
Standards referenced: ISO 9241-9