SIGN LANGUAGE PREDICTION FOR CHALLENGEEU HACKATHON
IOT • MACHINE LEARNING • EMBEDDED SYSTEMS
THE CHALLENGE
There are over 70 million mute people worldwide. Imagine knowing exactly what to say, but having no way of saying it. This was the aim of our project built during the ChallengeEU Hackathon at SEEU. We aimed to give people a voice, people that often times find it hard to integrate in social circles due to their disabilities. The objective was simple, gather sensor data like acceleration and rotation and capture a few basic movements, so we could train a simple neural network to detect simple gestures such as Hello, Thank you, and Goodbye. We would only be focusing on ASL as it's easier to differentiate between these basic phrases. However, not everything panned out as we had planned.
THE HACKATHON & HARDWARE ARCHITECTURE
Now, being a hackathon, we only had so many sensors we could find in stock and use. The original build was a force sensor, a hear rate sensor, an accelerometer and a gyroscope, plugged into an esp32 lite, which was powered by 7.4V 2S 18650 Lion batteries, stepped down to 5V. Our budget was limited, and A LOT, and I truly mean A LOT went wrong. We had a working prototype, until our esp32-lite gave out. So, we had to borrow an arduino nano 33 sense from the research institute at SEEU (special thanks to them). We built our second prototype, using a raspberry pi, arduino nano, a force sensor and using the accelerometer and gyroscope from the arduino at this point, since it made more sense that way. The raspberry pi worked... until it didn't. Then our battery gave out. Then our raspberry pi stopped working altogether. So, we had to resort to reading data WIRED, straight from the arduino and the force sensor connected to it. (The reasoning behind having an RPi and the Arduino was that we could use the arduino as a gateway of sorts to read the force sensors input, being an analog sensor we had to have some kind of converter since the RPi has no analog pins. And since the arduino had no WiFi, we would be using the raspberry pi to transmit the sensor data through UDP to a laptop, where we could record it and save it to a csv file for furthering processing. So we ended up recording data wired straight to the arduino into a laptop, as you can see in the video below. It wasn't the prettiest solution by any stretch of the imagination, but we had ran out of options at that point in time. We had only 9 hours to gather data and train a bare-bones model. Which meant, we had to make some sacrifices. We cut a lot of the words, and focused on "Hello", and "Thank you". They are distinct enough that we would get a relatively high accuracy rate from our very limited data (limited number of sensors), and time.
CONCLUSION
This was an amazing learning opportunity, getting to gather data from mentors and participants, talking to them and trying to implement their ideas. I would like to thank my team and mentors for the support, and the organizing team for throwing a great hackathon. Maybe for a hackathon of this nature our implementation was "too technical" in a lot of areas, so it could have been a better fit for a more applied hackathon. Nonetheless, all of us learned something new during the two days we were there, and would have done it all over.
