Work

Vision-Based Finger Snap Detection

First-person finger snap detection, meant for things like smart glasses, so you could pull off cool moves like snapping to turn on the lights.

  • Taoyuan Vocational HS (NTUT-affiliated)
  • HS Year 1
  • 2021
  • Solo project
A computer screen showing hand keypoints detected by the camera

Background

This started as gesture control for a pair of smart glasses I wanted to build. I had use cases in mind like snapping your fingers to turn on the lights, but we ran out of money and the project stalled. I even wanted to add more gestures like double taps and pinches. If I’d actually finished it, it might have been, like, 5% of an Apple Vision Pro 😅

How It Works

  1. Grab a frame from the camera with OpenCV
  2. Run MediaPipe’s hand landmark model to get the keypoint coordinates
  3. Check whether the tips of the index finger, middle finger and thumb are close together, and whether the ring finger is far from the index finger
  4. If so, check whether the middle finger then moves far away from the index finger, and if both motions happen within 0.3 seconds, count it as one complete snap

The Smart Glasses Plan

I figured that putting a screen directly in front of the lenses would be too close to the eye, so you’d have to strain to focus on it and it wouldn’t be comfortable to look at. I’d also thought about pulling the backlight off an LCD to make a see-through display, but whether that would even work, or how transparent it’d be, was anyone’s guess. Because of those bottlenecks, the final idea was to mount the screen at the back end of the glasses’ arm and use a piece of glass at the front to reflect the screen’s light onto the lens.

Even with the display sorted out, there was still the battery and the processor to deal with. Our plan was to run a cable down to a belt carrying the battery and a Raspberry Pi, but sadly a Raspberry Pi was a bit over our budget back then 😅 Funny enough, the Apple Vision Pro ended up using the same approach for its battery, and watching the keynote made me feel a little smug 😂

Diagram of the smart glasses display
Diagram of the smart glasses display

Reflections

This gesture recognition project was basically my practice project when I was first learning Python. Looking back there’s a lot I’d cringe at, but I’ll never forget how excited I was the first time I snapped my fingers and the terminal printed “snap”. It was also my first time using MediaPipe, Google’s easy-to-use and accurate ML toolkit, and I’ve used it a lot since to quickly build fun stuff.

Even though I never fully built the smart glasses, I learned a lot, like where the technical bottlenecks for smart glasses are, and how hard it is to pack serious computing power into something as small as a pair of glasses.

If I were doing this project today, I’d probably put an ESP32-CAM on the glasses and stream the video back to an iOS app I’d write in Swift. That way I could use the powerful A-series chip and even control other apps on the phone, which beats the old Raspberry Pi plan by a mile.