It unsettled me a lot about just how much work was put into making the JavaScript version of this work instead of a purely Python version, due to how OpenCV works. I wonder how universal the laggy OpenCV thing is, because my friend faced it too when working on an OpenCV application. Is it so unavoidable that the only option is to not use Python? I really hope that there is another way of going about this.

Anyways, I am very glad that you put in all that effort to make the JavaScript version work well. Working under limitations is sometimes cool. I remember having to figure out how PyTorch evaluated neural networks, and having to convert the PyTorch neural network into Java code that could evaluate the model without any external libraries (it was very inefficient) for a Java code competition. Although there may have been a better way, what I did was good enough.

Mediapipe is a lot of fun to play with and I'm surprised how little it seems to be used.

You might also be interested in Project Gameface, open source Windows and Android software for face input: https://github.com/google/project-gameface

Also https://github.com/takeyamayuki/NonMouse

This is cool, but a moving average filter is pretty bad at removing noise - it tends to be longer than it needs to be because its passband is so bad. Try using a IIR filter instead. You don't need to deal with calculating the coefficients correctly because they'll just be empirically determined.

out = last_out * x + input * (1-x)

Where x is between zero and one. Closer to one, the more filtering you'll do. You can cascade these too, to make a higher order filter, which will work even better.

Amazing work! I have been working on robotifying operation task for my company - a robot hand and a vision that can complete a task on the monitor just like humans do. Have been toying with openAI vision model to get the mouse coordinates but it’s slow and does not return the correct coordinates always (probably due to LLM not understanding geometry)

Anyhow , looking forward to try your approach with mediapipe. Thanks for the write up and demo, inspirational.

I did a very similar project a few months back. My goal was to help alleviate some of the RSI issues I have, and give myself a different input device.

The precision was always tricky, and while fun, i eventually abandoned the project and switched to face tracking and blinking so i didn't have to hold up my hand.

For some reason the idea of pointing my webcam down, didn't dawn on me ever. I then discovered Project Gameface and just started using that.

Happy programming thank you for the excellent write up and read!