Micro Activities Recognition and Macro Worksteps Classification for Industrial IoT Processes
Sprache des Titels:
IoT '21: 11th International Conference on the Internet of Things
Automated understanding of worksteps in industrial assembly work is important for IoT-based assistive guidance technologies in employee-machine collaboration and for industrial IoT (IIoT) environments. Our aim is to identify macro worksteps using depth images and micro activities of employees during the assembly of ATM machines for auxiliary purposes in their daily complex tasks using hand-operated tools. Due to the advance of inertial measurement unit (IMU) technologies and pattern recognition systems , IMU based sensing together with machine learning have gained momentum on workstep recognition and were selected for this study in combination with a depth camera sensor which is mounted on a ceiling with a top-down angle. In this work the focus is at a seamless embedding of non-impeding body-worn IMUs or their integration into smart tools and IoT devices, and the depth sensor ensuring the privacy of the operators, allowing for unobstructed monitoring of tools? usage pattern and thus assembly workstep recognition. The results for this study are evidenced with empirical observations of assembly workstep executions by (i) hand screwing, (ii) screw driver screwing, (iii) machine screwing, (iv) wrench screwing, with the null class being disproportionally dominant in the data set. Deep Learning models including LSTM, Temporal Convolutional Networks (TCN) and CNN architectures are proposed for the detection of micro activities and macro worksteps and the identification of the current workstep which will be beneficial also for the recognition of the transition between each two consecutive macro worksteps. A sophisticated counting mechanism of the classified activities is recognized as the next research challenge, with the extraction of features from each IMU sensor and the temporal information from the depth sensor, integrated in an IoT system.