EmojiFan: Designing A Social Interface Supporting Facial Expression Interaction for Blind and Low Vision People in Party Settings
Jinlin Miao, Shan Luo, Yue Chen, Hongyue Wang, Zhejun Zhang, Rina R. Wehbe · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26) · doi:10.1145/3772318.3790944
Summary
This CHI 2026 paper presents EmojiFan, an AI-assisted wearable prototype designed to help blind and low-vision (BLV) adults participate in facial-expression interactions in party settings — a social context the authors identify as particularly hostile to BLV inclusion because of overlapping speech, rapid conversational turn-taking, ambient noise, and heavy reliance on fleeting non-verbal cues. The system pairs a chest-worn ESP32-CAM (feeding video to a fine-tuned DeepFace affect-recognition pipeline) with a hand-held electronic fan whose handle contains an Arduino MEGA2560, three coin vibration motors for left/centre/right directional haptic cues, a gyroscope, and a GIWOX Holo-42W transparent display that renders dynamic emoji animations generated via a LoRA-fine-tuned GPT-4o mapped to user-defined emotional responses. The fan form was chosen deliberately as a culturally familiar, non-stigmatising 'social object' that can be raised or lowered to signal openness to conversation. The work proceeds in two stages. First, a formative study with 10 young Chinese BLV adults (interviews + design-researcher synthesis) surfaces the specific barriers BLV people face after perceiving non-verbal cues, yielding three design considerations: unobtrusive haptic hints, AI-personalised expressions, and a socially acceptable wearable form. Second, a multi-day in-the-field party simulation with 6 BLV participants and 8 sighted partners evaluates EmojiFan using thematic analysis (832 data units, 50 codes, 4 themes). The study's contribution is not a performance benchmark but a design-case demonstration that reframes BLV social participation around autonomy, personalised expression, and AI proxying of facial affect.
Key findings
The formative study identified three intertwined barriers: BLV users cannot reliably access others' non-verbal cues in noisy party environments (P1: 'I have no idea whether they included me'); cannot respond with timely, natural facial expressions of their own (P5 worries 'my smile looks stiff and unattractive'); and avoid existing visual-recognition assistive tools because pointing a phone at a stranger's face is socially unacceptable. Among 10 participants, 63.6% rated their social frequency 4/5, and both loneliness (28.7% moderate, 19.7% severe in the general BLV population cited) and active withdrawal from social events emerged as consequences. The user study produced four themes. (1) EmojiFan facilitated ice-breaking: dynamic emoji displays drew sighted partners into conversations they would otherwise have avoided. (2) BLV users took initiative — raising the fan to invite talk, lowering it to disengage, picking it up again to redirect conversation — shifting from passive recipients to active conversational participants. (3) Sighted partners' empathy increased; they asked clarifying questions when emoji changed, made more eye contact and nodding, and reported reduced awkwardness. (4) The fan form was especially well-matched to party contexts but felt excessive for quieter settings, where participants wanted smaller form-factors (e.g., necklace pendants). Limitations surfaced by participants: users cannot themselves verify which emoji is being displayed in real-time, raising concerns about misrepresentation; DeepFace's six-category Ekman-based recognition has known cultural and non-typical-expression biases; and the simulated-party environment did not replicate the full complexity of real social settings.
Relevance
For accessibility designers, the paper is most useful as a worked example of three design moves that generalise well beyond parties: (a) treat socially acceptable form factors as first-class design constraints equal to technical function — a fan is a culturally-loaded 'social prop' that a phone pointed at a face is not; (b) AI-proxied expression should be personalised by the user in advance (EmojiFan uses a LoRA-fine-tuned mapping from user-described preferences to emoji outputs), preserving agency in a space where traditional AT would either under-serve individuality or stigmatise the user; (c) embrace productive ambiguity in AI output rather than chasing deterministic accuracy, because ambiguous emoji can invite clarifying conversation rather than shutting it down. The paper also contributes an explicit ethics and privacy frame (on-device/local-network processing only, abstract-signal feedback, visible-signage consent) that accessibility teams building camera-based assistive tools can adapt. Limitations to note: the study is small (6 BLV users, 8 sighted partners), geographically and culturally concentrated (Chinese university context; fan symbolism may not transfer), BLV users cannot see their own emoji output to self-correct, and there is no long-term deployment. The paper is therefore best read as a design proposition and ethics case rather than an efficacy trial.
Tags: blind and low vision · visual impairment · facial expression · social accessibility · wearable technology · haptic feedback · AI-assisted assistive technology · emotion recognition · nonverbal communication · emoji · bodystorming · thematic analysis