Does nature create wheels? After intense thinking, I couldn’t find any. In nature, there are bug swarms which can be simulated by having a group of little robots that move by vibrating—the topic of my research internship at University of Waterloo with Professor Gennaro Notomista. This then raises the question of control systems. While there are existing computer vision algorithms to move swarm robots, no one at the time has invented a haptic hand system that mapped onto swarms. Our research involved independently learning advanced linear algebra, multivariable calculus; Figuring out from scratch and software engineering stack to create such a control system for swarms. I learned that you can get less mistakes by using Linux, picked up on using ROS for robotics; Then we used PyTorch to build a neural network to translate what I’ve long understood theoretically before into reality. The data came from a haptic hand that we had which familiarized me with hardware. I realized the barrier of entry to robotics was mainly equipment. Throughout the entire pipeline was troubleshooting and more troubleshooting without knowing the practical possibility. I gained technical skills and it was a seed for my interest in swarm intelligence.

Research and personal significance:

Work in haptic gesture recognition could allow a more intuitive interaction in robotics, surgery, VR and gaming. Haptics could go hand-in-hand with Brain-Computer Interfaces to allow individuals with physical disabilities to regain independence and perform tasks with ease. Currently, there is very little literature on recognizing the gesture from the data (e.g. position, velocity of the joints) of a haptic glove. Most existing work is in computer vision recognizing hand gestures, or using a non-machine learning algorithm for haptic gesture.

Personal significance: This research experience sparked a lot of curiosity in bio-inspired robotics and I envision building an ecosystem of bio-inspired robots. Robotics is hard to enter due to equipment. Now that I have equipment, I gained more confidence in my software and hardware skills. Before, I had a basic proficiency in python and a conceptual understanding of machine learning algorithms.

During the project, I read online tutorials of how to use PyTorch library in python, discussed multivariable calculus and quaternions’ use in code, learned tutorials to use ROS, the linux subsystem on windows and ubuntu. I also learned a few tricks:

1. How to use GitHub collaboratively
2. Basic Ubuntu commands
3. The process of troubleshooting hardware

We faced obstacles on a daily basis, mostly with various software errors. We have to not just search for the error but also logically eliminate possible sources of error and fixing it.

Likewise I learned what I could improve on:

1. improving efficiency during coding e.g. keyboard shortcuts during programming
2. reply more frequently to emails
3. readily discussing my current understanding, willingness to be wrong and learning

Steps:

1. Code a neural network to classify gestures of the haptic hand. This was done by installing packages, understanding advanced mathematics behind many tutorials with a Python library (which allows us to use existing code written by other programmers to be efficient) PyTorch, training the neural network, evaluating how the network performed and adjusting the structure of the network.
2. Install the software, ROS, to communicate with the haptic hand live. First, we tested which operating systems were compatible with the software, from Windows to Ubuntu 22.04 to Ubuntu 20.04; where we could access these OSes—through the Windows Subsystem for Linux or dual booting; familiarize ourselves with the new pipeline; install the software while solving many issues and understand the tutorials of the software
3. Install haptic hand’s proprietary software. First, we installed the communication software and used trial-and-error against many issues since there wasn’t much documentation available on the internet. Next, we installed the API, also used trial-and-error against issues and modified the example script.