Reimagining Wearable AR Gesture Design: Physical Therapy Reasoning in Everyday Contexts
Wei Wu, Binyan Xu, Soonhyeon Kweon, Yujie Wang, Leanne Chukoskie, Casper Harteveld · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26) · doi:10.1145/3772318.3790719
Summary
This CHI 2026 paper reimagines gesture vocabularies for lightweight, everyday augmented reality (AR) glasses — the emerging category of optical see-through wearables (e.g., Snap Spectacles, Meta Orion, Ray-Ban Meta) that are entering consumer and workplace life. The authors argue that current AR gesture sets are inherited from VR headsets and enterprise goggles, optimizing recognizer accuracy while neglecting long-session fatigue, joint safety, and social legibility. The work is grounded in a three-stage, PT-informed elicitation pipeline. Stage 1 reviewed 104 publicly listed Snap Spectacles applications to distill 15 recurring "gesture intents" spanning object manipulations (Move, Rotate, Enlarge, Shrink, Connect, Separate), system commands (Select, Delete, Switch, Open, Close, Draw), and body/spatial actions (Throw, Dodge). Stage 2 built an on-device prototype in Lens Studio 5 that displayed neutral 3D cube animations cueing each intent without biasing toward any existing recognizer. Stage 3 ran a 90–120-minute in-person session with ten licensed U.S. physical therapists (M = 14.8 years of practice; specialties across orthopedics, sports medicine, neurorehabilitation, pediatrics, and ICU care) in three rounds: (1) unaided intuitive execution, (2) PT-guided clinical substitution, and (3) stage-aware card sorting across dramaturgical contexts (front-/back-/off-stage × individual/public). Analysis combined two-pass coding in NVivo (Cohen's κ = 0.82), open coding of think-aloud data, and abductive theory-building. The paper positions PTs as "movement-safety experts" whose clinical reasoning translates intuitive but fragile mid-air gestures into biomechanically stable, socially coherent forms.
Key findings
Four recurring biomechanical principles emerged: (1) Joint–Rotation Substitution — redirect rotation away from spine and wrist to forearm supination/pronation and stance micro-adjustments; (2) Shoulder-Line Workspace Constraint — constrain interaction to a chest-level, elbow-led ±30° envelope, avoiding "gorilla-arm" posture, overhead reach, and posterior reach; (3) Proximal–Distal Stability — shift effort toward proximal joints, treating the shoulder as a stabilizer rather than an actuator; (4) Fine–Gross Motor Integration — establish a gross-motor base before precision, using multi-finger grips and bilateral alternation to distribute load. These principles crystallize into the Everyday-AR Golden Ergonomic Canvas, a Vitruvian-Man-inspired chest-level, elbow-anchored workspace that bounds sustainable interaction. A parallel analysis using Goffman's dramaturgical framing produced six everyday contexts (Amusement, Transit, Utility, Workplace, Social, Presentation) where gesture amplitude, subtlety, and expressivity systematically shift: fine motions dominate off-stage/crowded settings, fine–gross hybrids suit back-stage contexts, and gross expressive gestures are reserved for front-stage performance. The paper challenges "naturalness" as a design goal, showing that gestures users initially perform (overhand throws, wrist flicks, shoulder-led pointing) are often biomechanically fragile once repeated. Eight design implications are distilled, including prioritizing proximal stability, treating fine-motor actions as cognitive load, centering the elbow as the primary input site, enabling scenario-aware gesture modes, and accommodating cultural and generational variation.
Relevance
For accessibility practitioners, this paper offers a rare biomechanically grounded critique of dominant AR interaction paradigms, with direct implications for motor accessibility, repetitive strain prevention, and inclusive design. The Everyday-AR Golden Ergonomic Canvas provides a concrete, recognizer-agnostic reference frame that designers can use to evaluate whether gesture sets respect joint safety, fatigue resistance, and social legibility — all factors that disproportionately affect users with limited range of motion, chronic pain, or atypical movement patterns. The stage-aware framework also extends accessibility thinking beyond physical capability to social discretion, recognizing that disabled users often navigate heightened visibility and stigma in public. Limitations include a U.S.-only PT sample, absence of actual end-user evaluation (no fatigue or recognizer benchmarking), and single-platform focus on Snap Spectacles. Nonetheless, by treating gesture design as clinical-reasoning-driven redesign rather than user elicitation alone, the paper provides durable scaffolding for AR accessibility work as these devices move from novelty to everyday infrastructure.
Tags: augmented reality · gesture interaction · wearable technology · physical therapy · ergonomics · motor skills · interaction design · social acceptability · user elicitation