Context
A fun personal project for experimentation and learning, built almost entirely with Claude Code. I wanted to explore what browsing could feel like without a mouse or keyboard. Built two versions: a Chromium-based Electron app using MediaPipe for hand tracking and Deepgram for voice commands, and a WebKit-based native Swift app using Apple Vision and Speech frameworks.
Could I use AI-assisted development to build something genuinely useful while learning about gesture recognition and native app development?
Process
- Defined the interaction model and UX patterns I wanted to explore
- Worked with Claude Code to implement hand tracking using MediaPipe and Apple Vision
- Iterated on gesture sensitivity, debouncing, and state management through conversation
- Built two versions to compare web technologies (Electron) vs native (Swift/SwiftUI)
- Learned about the underlying tech by reviewing and understanding the generated code
Solution
A gesture-controlled browser where open palm activates gesture mode, hand movement scrolls, and pinch clicks. Voice commands handle navigation. The project taught me how AI-assisted development can accelerate learning and prototyping.
Outcomes
- Working prototype that demonstrates gesture and voice browsing
- Deeper understanding of hand tracking, state machines, and native frameworks
- Firsthand experience of AI-assisted development workflow
Learnings
- AI-assisted development is powerful for exploring unfamiliar technical domains
- Defining clear interaction patterns upfront helps guide the implementation
- Building the same thing twice (Electron vs Swift) reveals tradeoffs you'd miss otherwise