Designing an AI that knows what it doesn't know
GesturePro reads sign language through a laptop camera and turns it into text. This case study is about the harder half of the work: deciding how a signer and an imperfect model should share control of their words.
On this page
TL;DR
We set out to build a sign-language translator that needs nothing but a webcam. Desk research changed the question. Deaf people have watched hearing teams build “translators” for years, and their main worries are accuracy, being spoken for, and tools that let hearing people skip learning sign. So the design goal moved from translate automatically to help a signer be understood, with the signer in charge of every word that leaves the screen.
- Question
- How might a camera-based model help a Deaf signer communicate a short message to a hearing person, without speaking for them?
- My role
- Led research and interaction design; built the landmark pipeline and the classifier
- What shipped
- A browser prototype: webcam in, 21 hand landmarks per frame, sign classified to on-screen text, with sign-in and history
- Design stance
- A communication aid for short, low-stakes moments. Never a replacement for an interpreter.
An AI product where being wrong has a cost to someone's voice
Most AI features fail quietly: a bad recommendation is ignored. A sign-recognition model fails out loud. If it outputs the wrong word, a hearing person reads words the signer never said. That makes GesturePro a useful study in UX for AI: the model is the easy part to demo and the hard part to trust.
This page walks through the project in design-thinking stages. Each stage ends with the human-AI interaction decision it led to. Where the evidence is thin, it says so.
- hand landmarks read per frame
- 21
- extra hardware: a standard webcam
- 0
- independently deployed services
- 3
- usability-test participants
- 6
The brief arrived as a solution
The capstone brief was “a real-time sign-language translator with no extra hardware.” That is a solution. It assumes three things nobody had checked: that Deaf people want a translator, that translation is the right job, and that a hand-only model can carry a language that also lives in the face and body.
Before
Can sign language be translated in real time without extra hardware?
A feasibility question. It measures the model and says nothing about the person signing.
After
How might a signer use a camera model to be understood in short exchanges, while staying the author of every word?
A desirability question. It puts the signer's control at the centre and limits the scope to moments where an interpreter isn't there.
What the Deaf community has already said about sign-language tech
Before building screens, I read what Deaf people and their organisations have published about tools like this one. Three sources shaped the project most.
Global survey · 2025
“Nothing about us without us”
42 respondents across five continents: 21 Deaf, 12 hard of hearing, 9 hearing. Most supported the tech, but Deaf respondents often gave conditional answers, and Deaf-led development was a repeated condition.
Atwell, Imai, Bragg, Alikhani, arXiv 2025
Press critique · 2016
The sign-language gloves
A student glove that turned signs into speech drew pushback. Critics said the “bridge” ran one way, toward hearing people, and that it missed grammar carried by the face and body. A head shake can negate a sign.
KQED, May 2016
WFD and WASLI · 2018
Statement on signing avatars
Sign languages are full languages, so word-for-word conversion doesn't work. Machine translation can't yet match a qualified interpreter, live or culturally.
World Federation of the Deaf, World Association of Sign Language Interpreters
What the survey numbers say
The survey is small, so these are counts, not population rates.
| Finding | Deaf (21) | Hard of hearing (12) | Hearing (9) |
|---|---|---|---|
| Values translation as a use | about 12 of 21 | 6 of 12 | 6 of 9 |
| Names system accuracy as a concern | about 11 of 21 | 3 of 12 | not reported |
Counts derived from the paper's reported percentages and group sizes. The two most-named concerns overall, each chosen by 28 of 42, were hearing people profiting from the tech and few Deaf people leading it.
What this changed
The people this tool is “for” value translation less than the people building it.
Translation was valued by fewer Deaf and hard-of-hearing respondents than hearing ones. That meant we couldn't assume demand. We had to design for a narrow job, be honest about limits, and keep the signer in control.
Three ways to frame the problem
I wrote three point-of-view statements and tagged each as a symptom or a root cause. The third became the frame for design.
- Symptom
Hearing people need to understand sign language because they can't follow a signer.
This is the framing the gloves used. It centres the hearing person and treats the signer as the problem. Leaves out: the Deaf person's agency entirely.
- Symptom
A Deaf signer needs a cheaper translator because interpreters and special hardware cost too much.
Cost is real, but this framing quietly positions an AI as a cheaper interpreter, which the WFD statement warns against. Leaves out: contexts where an interpreter is the right answer.
- Root cause · chosen
A Deaf signer in a short, unplanned exchange (a counter, a front desk, a classroom aside) needs a quick way to be understood without handing over their voice, because no interpreter is there and writing back and forth is slow.
Scopes the tool to low-stakes moments, keeps the signer as author, and makes accuracy a shared problem rather than a hidden one. Leaves out: people who sign mainly with facial grammar the model can't read, and long conversations.
How might we
HMW let a signer see what the model saw before anyone else reads it?
HMW make fixing a wrong guess faster than signing it again?
HMW be clear up front about what the model can't read, like facial grammar?
Four human-AI interaction principles we designed against
Rather than invent rules, I mapped ideas to published human-AI interaction guidance. Every design decision on this page traces back to one of these.
P1 · Shneiderman, HCAI
High automation and high human control
Automation and control aren't a trade-off on one dial. The model can do the heavy lifting while the person keeps the final say.
P2 · Microsoft HAX, G1 and G2
Make clear what the system can do, and how well
Set expectations before first use, not after the first failure.
P3 · Microsoft HAX, G9 and G10
Support efficient correction; scope when in doubt
When confidence is low, offer options instead of committing to one answer.
P4 · Google PAIR
Errors and graceful failure; explainability and trust
Show enough of the model's reasoning for people to calibrate trust, and give every error a way forward.
Eight decisions, and what each one costs
Each decision is tagged by status. In prototype means it runs in the build today. Designed means it was specified from research and testing but not built in the capstone timeframe.
| Decision | Why | Principle | Trade-off | Status |
|---|---|---|---|---|
| 1. Camera only, in the browser | Removes cost and install barriers. Avoids wearables, which drew the strongest community pushback. | Access | Webcam quality and lighting limit accuracy | In prototype |
| 2. Show the hand skeleton live | The 21-landmark overlay tells the signer what the model actually sees, so they can fix framing before blaming themselves. | P4 | Visual noise over the video | In prototype |
| 3. Show top 3 guesses with confidence, not one answer | A single answer hides uncertainty. Options let the signer pick, and teach them when the model is unsure. | P2, P3 | One extra glance per sign | Designed |
| 4. Signer confirms before text is shown to the other person | Nothing leaves the signer's side unapproved. This is the core control decision. | P1 | Slower than fully automatic output | Designed |
| 5. Scope statement before first use | Says plainly: fingerspelling and isolated signs only; no facial grammar; not for medical, legal or emergency use. | P2 | Less impressive first impression | Designed |
| 6. Fix by tap, re-sign or type | Three ways out of every error. Typing is always available, so the model is never a dead end. | P3, P4 | More controls on a small screen | Designed |
| 7. History the user owns | Sign-in enables history; it should be private by default and deletable by the user. | Trust | Account step before use | History in prototype |
| 8. Two-way by default: reply as text | Answers the “one-way bridge” critique: the hearing person types back, so both sides do work. | Equity | Needs a shared screen or second device | Designed |
Status tags reflect the state of the prototype at time of submission. Decisions marked “Designed” were specified but scoped out of the build due to time constraints.
Who decides at each step
The interaction loop separates what the model does from what the signer decides. Purple steps are the model. Dark steps are the person.
Person
1. Sign
In view of the camera
Model
2. See
21 landmarks drawn live
Model
3. Guess
Top 3 with confidence
Person
4. Confirm or fix
Tap, re-sign, or type
Person
5. Share
Hearing partner reads, replies in text
The model never writes directly to the other person. Step 4 is the gate.
Hypothesis behind the gate
We believe signers will trust the output enough to use it in a real exchange if they confirm each word before it is shown, even though it is slower.
We'll know when signers in testing choose to keep confirmation on when offered an automatic mode, and rate their confidence that the shown text matches what they signed higher than without it.
We're wrong if most signers switch confirmation off, or the extra step makes them abandon the message for typing.
What usability testing showed
We ran moderated usability tests on the working prototype. Participants signed a set of short messages while thinking aloud, then we asked how much they trusted the text on screen.
Participants
6 people: hearing ASL learners and team members familiar with the vocabulary set. Labelled P1 to P6.
Tasks
Sign each of the 28 trained words; attempt a short sequence; recover if the model returned a wrong guess.
Measures
Task success, errors noticed vs missed, time to recover, self-rated trust via free observation and think-aloud.
| Finding | Count | Led to |
|---|---|---|
| ASL grammar differed from what the model expected; sentence formation was hard to match to the trained vocabulary | All 6 of 6 | Decision 3, 4 |
| Real-time server response had noticeable delay; participants paused mid-sign waiting for confirmation | 5 of 6 | Decision 2 |
| Vocabulary felt too narrow; participants wanted to sign words not in the 28-word set and had no recovery path | 4 of 6 | Decision 6 |
Small sample: counts only, no percentages, no claims beyond the 6 people we tested with. Model: YOLOv8n trained for 50 epochs on a custom 28-class ASL dataset (img_size 640). Training outputs were not saved to the repo; evaluation metrics were not recovered. YOLOv8n on datasets of this scale typically reaches mAP50 in the 0.70–0.85 range — treat that as a rough benchmark, not a measured result from this training run.
Who benefits, and who could be harmed
A human-centred AI project has to account for both. These are potential benefits, not measured outcomes.
Could help
- Short, unplanned exchanges. Moments where booking an interpreter isn't possible.
- No cost, no gear. A webcam and a browser, nothing to buy.
- Open source. Deaf developers and researchers can inspect, fork and lead it.
Could harm
- Replacing interpreters. An organisation might point to the tool instead of booking one. The scope statement and copy argue against this.
- Misquoting the signer. Wrong output read as their words. The confirmation gate is the guard.
- Hearing-led design. Our team was not Deaf-led. That is the biggest gap in this project.
An accessibility tool has to be accessible itself
WCAG 2.2 AA requirements we set for the interface around the camera.
| Requirement | Why it matters here |
|---|---|
| Text output at 4.5:1 contrast, resizable to 200% | Output is read at a glance across a counter, often by someone older. |
| Full keyboard path for confirm, fix and type | Not every user can tap precisely while signing. |
| No sound-only signals | Every status (hand lost, low confidence) is visual and in text. |
| Plain-language camera permission | Says what is captured: MediaPipe extracts landmarks locally and draws the skeleton overlay in-browser; only a cropped image frame goes to the server — raw video never leaves the device. Permission copy explains this and links to revoke. |
| Reduced motion respected | The skeleton overlay can be switched to a static outline. |
What we can claim, and what we can't
Supported
- The prototype classifies signs from a standard webcam in real time, with no extra hardware.
- It runs as three containerised services with sign-in and history.
- Usability findings for the 6 people we tested with.
Not yet supported
- That it helps in real conversations. We tested set tasks, not live exchanges.
- That Deaf signers broadly want it. Our sample can't show that.
- Accuracy across signers, skin tones, lighting and signing styles. Not measured.
What I would do with another semester
Deaf co-design partners, paid
Move from testing on Deaf people to designing with them, starting with the scope statement.
Test the confirmation gate
A/B the gate against automatic output in live, low-stakes exchanges. Only a controlled test can show it causes higher trust.
Measure fairness of accuracy
Report accuracy per sign and per signer group, with the denominator, before adding vocabulary.
What the project taught me
The model is not the product. The product is the moment when a person decides whether to trust what the model said. Designing that moment took more thought than training the classifier.
Read the community before writing the brief. Ten years of public Deaf critique would have saved our first framing.
Sources
- Atwell, Imai, Bragg and Alikhani, “Nothing about us without us,” arXiv, 2025 (opens in a new tab)
- KQED, “Invention translates sign language into speech; some in Deaf community unimpressed,” 2016 (opens in a new tab)
- WFD and WASLI, statement on signing avatars, 2018 (opens in a new tab)
- Amershi et al., Guidelines for Human-AI Interaction, Microsoft Research (opens in a new tab)
- Google PAIR, People + AI Guidebook: Errors and Graceful Failure (opens in a new tab)
- Google PAIR, People + AI Guidebook: Explainability and Trust (opens in a new tab)
- Shneiderman, Human-Centered Artificial Intelligence: Reliable, Safe and Trustworthy, 2020 (opens in a new tab)