Vision-based human posture and gesture recognition for SemuBot

Abstract

This bachelor’s thesis outlines the design of SemuBot’s real-time human posture and gesture recognition system. SemuBot is the first of its kind social assistive robot from Estonia. Social robots in both healthcare and elderly care environments need strong non-verbal communication skills and opportunistic safety monitoring and socializing features. To provide those features without incorporating wearable sensors, the system utilizes the YOLOv8-pose estimation model in combination with an Intel RealSense D435i depth camera. Heuristic algorithms were created to analyze the spatio-temporal relations of the 17 body keypoints. The recognition system also analyzes the relative heights and wrist position to the width of the shoulders in order to determine waving/assistive gestures. The system also includes an opportunistic mechanism to help classify and monitor temporal velocity and the posture of the user, which identifies and classifies the user as standing, sitting, or laying. Evaluated under the constraints of edge computing, the system is able to address the needs with high accuracy and low latency.