Wearable Gesture Recognition with Heterogeneous Cameras

Abstract

The purpose of this research was to create a wearable system that recognizes gestures of the user, allowing interaction through hand gestures. The user wears a hat mounted with a regular optical camera and a thermal camera. The combination of these two heterogeneous video streams was used to recognize the user’s gestures in many conditions and environments. First, corners were detected from contrast stretched images using the Shi-Tomasi method. The movement of these corners was then tracked using Lucas-Kanade optical flow analysis. Groups of corners that moved together were defined using hierarchical cluster linkage analysis. To determine how these groups moved with time, a connected components analysis was employed. The motion path was reduced into its cardinal and semi cardinal vector components to encode the motion vector. Subsequently, this data was used to train hidden Markov models for each gesture and each camera. After the evaluation of gesture priority over all hidden Markov models, principal components analysis was performed on this gesture prioritized set to train a one vs one Multiclass recognizer. Finally, a confusion matrix was generated indicating a recognition success rate of 87%. An analysis was performed on the robustness of the algorithm under various luminance, heat and image variance conditions. The contribution of combining optical and thermal video streams vs utilizing either as a single video stream input and found to be a great advantage. Additionally, a video database of gestures was created and will be released so that other researchers can compare algorithms and benchmarks using the same data-set.