Camera-Based Closed-Loop Fingertip Deflection Guidance: Pilot Demonstrations in Target Acquisition and Object Retrieval
Tomasz P. Trzpit, Gregory Reardon, Elizabeth M. Gerber, Pedro Lopes, Michael A. Peshkin, J. Edward Colgate · 2026 · Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA ’26) · doi:10.1145/3772363.3799007
Summary
This CHI 2026 Extended Abstracts poster from the Northwestern/University of Chicago team behind the NURing project presents a camera-enabled evolution of their fingertip-deflection guidance wearable. The authors argue that most eyes-free guidance systems for blind and vision-impaired (BVI) users — spoken directions, beeps, vibrotactile alerts — rely on symbolic cues that must be decoded, compete with hearing, or encumber the fingertip itself, which explains why the white cane (continuous, direct, physically grounded) has remained the dominant mobility tool. The NURing instead uses a tendon-driven actuation ring worn above the proximal-interphalangeal (PIP) joint of the index finger: three independently actuated tendons gently deflect the finger in abduction, adduction, flexion, or extension to bias the arm toward a target during free reach — a cue inspired by a child leading an adult by the finger. The novelty of this poster is the integration of a 120°, 60 Hz global-shutter camera onto the ring itself, making the system closed-loop and object-referenced rather than dependent on external instrumented testbeds. ArUco fiducial markers are detected with OpenCV, the 3D target displacement in the camera frame becomes an error vector, and a proportional controller maps that vector to coordinated tendon tensions. The device weighs ~300 g (smartphone-class) and is framed around four design goals drawn from BVI need-finding: preserve hand function, provide continuous embodied direction, maintain user agency (easy to override), and support two-stage interaction (approach then manipulate). The paper reports two pilot demonstrations and sets up a forthcoming controlled BVI study.
Key findings
Two feasibility demonstrations, both conducted blindfolded on sighted pilot users rather than with BVI participants. (1) A Fitts-style target-acquisition task: the user reached ~600 mm toward a 20-mm ArUco target on a touchscreen; deflection cues shaped the trajectory to land within ~10 mm of the target center in under 3 seconds, with 3D trajectory plots showing smooth convergence along X, Y, and Z error axes. (2) An object-retrieval task: five 25-mm physical cubes on a target board, to be grasped in random order and dropped into a lap tray. A novice (<2 hours of device use) retrieved all five in ~40 seconds, including two tracking losses in which a pull-and-release “re-acquire” cue was rendered on all three tendons simultaneously, prompting the user to pull the hand back and resume. The authors show that deflection at the PIP joint leaves the fingerpad free for touch and grasp, satisfying the “preserve hand function” goal while still providing continuous kinesthetic guidance. Although ArUco markers were used for the pilots, the authors argue the guidance pipeline is agnostic to the localization source — object detection, depth sensing, or mixed-reality coordinates could substitute — positioning fingertip deflection as a modality independent of the hard computer-vision problem of robust markerless everyday-object tracking.
Relevance
For accessibility practitioners and assistive-technology designers, this paper is a useful datapoint in the long argument over how to replace or complement the white cane for fine, peripersonal tasks — dropped pills, kitchen utensils, unfamiliar touchscreen controls — that canes and smartphone cameras handle poorly. The “continuous, embodied, naturally interpretable” framing (versus symbolic beeps and buzzes) mirrors long-standing critiques of vibrotactile guidance wearables and is worth citing in design discussions of non-visual UI. Important caveats: all reported data here are from blindfolded sighted users, not BVI participants; the system depends on ArUco fiducials being physically placed on targets, which is not a deployable condition; tracking loss already required a recovery cue in the 5-object pilot; and the device still occupies the dominant hand. This is a feasibility poster, not a validation. The forthcoming BVI study (NASA-TLX, perceived agency, task performance) is where the real accessibility claims will be tested.
Tags: haptics · wearable technology · assistive technology · blindness and low vision · eyes-free interaction · guidance · kinesthetic feedback · computer vision · fingertip deflection · object retrieval