Dynamic Gesture Recognition Method for Machining Operations Based on Multi-modalData Fusion

NAVIGATE

Table of Contents

STATISTICS

Viewed217

Downloads166

Dynamic Gesture Recognition Method for Machining Operations Based on Multi-modalData Fusion

[HTML] PDF下载 (166)

[1]ZHANG Fuqiang,ZENG Xia,BAI Junyan,et al.Dynamic Gesture Recognition Method for Machining Operations Based on Multi-modalData Fusion[J].Journal of Zhengzhou University (Engineering Science),2024,45(05):30-36.[doi:10.13705/j.issn.1671-6833.2024.02.007]

Copy

Journal of Zhengzhou University (Engineering Science)[ISSN 1671-6833/CN 41-1339/T] Volume: 45 Number of periods: 2024 05 Page number: 30-36 Column: Public date: 2024-08-08

Title:: Dynamic Gesture Recognition Method for Machining Operations Based on Multi-modalData Fusion

Author(s):: ZHANG Fuqiang¹; 2; ZENG Xia¹; 2; BAI Junyan¹; 2; DING Kai¹; 2; 1. Key Laboratory of Road Construction Technology and Equipment of MOE, Changan University, Xian 710064, China; 2. Instituteof Smart Manufacturing Systems, Changan University, Xian 710064, China

Keywords:: multi-modal data fusion; machining operation; dynamic gesture recognition; C3D; Mish activationfunction; human-computer interaction

CLC:: TH166

DOI:: 10.13705/j.issn.1671-6833.2024.02.007

Abstract:: In order to solve the problem of difficulty in improving the recognition accuray and the low robustness ofthe model caused by the lack of feature information provided by single mode data, a dynamic gesture recognitionstrategy based on multi-modal data fusion of machining operations for human-computer interaction was proposed.Firstly, the C3D network model was used to extract features from the depth image and color image modal data basedon the spatial and temporal dimensions of videos. Secondly, the recognition results of the two modal data were fusedaccording to the maximum principle at the decision-making level. Meanwhile, the Relu activation function used inthe original model was replaced by Mish activation function to optimize the gradient update effect. Finally, throughthree sets of comparative experiments, it was found that the average recognition accuracy of six dynamic gesturesreached 96. 8%. The results showed that the proposed method achieved the goal of high accuracy and high robustness of dynamic gesture recognition in machining operation, which would play a role in promoting the application ofhuman-computer interaction technology in actual production scenes.

References:: [1] 李浩, 刘根, 文笑雨, 等. 面向人机交互的数字孪生系统工业安全控制体系与关键技术[ J] . 计算机集成制造系统, 2021, 27(2) : 374-389.
LI H, LIU G, WEN X Y, et al. Industrial safety controlsystem and key technologies of digital twin system oriented to human-machine interaction[ J] . Computer Integrated Manufacturing Systems, 2021, 27(2) : 374-389.
[2] LIU N J, LOVELL B C, KOOTSOOKOS P J, et al.Model structure selection & training algorithms for anHMM gesture recognition system[C]∥Ninth InternationalWorkshop on Frontiers in Handwriting Recognition. Piscataway:IEEE, 2004: 100-105.
[3] HARTMANN B, LINK N. Gesture recognition with inertial sensors and optimized DTW prototypes [ C ] ∥2010IEEE International Conference on Systems, Man and Cybernetics. Piscataway: IEEE, 2010: 2102-2109.
[4] 李浩, 杨森林, 张晓丽. 基于机器视觉的火车驾驶员动态手势识别方法[ J] . 传感器与微系统, 2021, 40(2) : 34-37, 43.
LI H, YANG S L, ZHANG X L. Dynamic gesture recognition method of train driver based on machine vision[ J] .Transducer and Microsystem Technologies, 2021, 40(2) : 34-37, 43.
[5] JIA L S, ZHOU X Z, XUE C Q. Non-trajectory-basedgesture recognition in human-computer interaction basedon hand skeleton data[ J] . Multimedia Tools and Applications, 2022, 81(15) : 20509-20539.
[6] 彭金柱, 董梦超, 杨扬. 基于视觉和肌电信息融合的手势识别方法 [ J] . 郑州大学学报 ( 工学版) , 2021,42(2) : 67-73.
PENG J Z, DONG M C, YANG Y. Human gesture recognition method based on vision and EMG signal information [ J] . Journal of Zhengzhou University ( EngineeringScience) , 2021, 42(2) : 67-73.
[7] CHAKRAVARTHI S S, RAO B N K, CHALLA A P, etal. Gesture recognition for enhancing human computer interaction[ J] . Journal of Scientific & Industrial Research,2023, 82(4) : 438-443.
[8] CHEOK M J, OMAR Z, JAWARD M H. A review ofhand gesture and sign language recognition techniques[ J] . International Journal of Machine Learning and Cybernetics, 2019, 10(1) : 131-153.
[9] MUJAHID A, AWAN M J, YASIN A, et al. Real-timehand gesture recognition based on deep learning YOLOv3model[ J] . Applied Sciences, 2021, 11(9) : 4164.
[10] ADITHYA A, RAJESH R. A deep convolutional neuralnetwork approach for static hand gesture recognition[ J] .Procedia Computer Science, 2020, 171: 2353-2361.
[11] SHARMA S, SINGH S. Vision-based hand gesture recognition using deep learning for the interpretation of signlanguage[ J] . Expert Systems with Applications, 2021,182: 115657.
[12] 刘杰, 王月, 田明. 多尺度时空特征融合的动态手势识别网络[ J] . 电子与信息学报, 2023, 45(7) : 2614-2622.
LIU J, WANG Y, TIAN M. Dynamic gesture recognitionnetwork based on multiscale spatiotemporal feature fusion[ J] . Journal of Electronics & Information Technology,2023, 45(7) : 2614-2622.
[13] SINGH D K. 3D-CNN based dynamic gesture recognitionfor Indian sign language modeling[ J] . Procedia Computer Science, 2021, 189: 76-83.
[14] ZHANG W J, WANG J C. Dynamic hand gesture recognition based on 3D convolutional neural network models[C]∥2019 IEEE 16th International Conference on Networking, Sensing and Control ( ICNSC ) . Piscataway:IEEE, 2019: 224-229.
[15] YANG L C, CHEN J, ZHU W H. Dynamic hand gesturerecognition based on a leap motion controller and two-layer bidirectional recurrent neural network [ J ] . Sensors,2020, 20(7) : 2106.
[16] KHODABANDELOU G, JUNG P G, AMIRAT Y, et al.Attention-based gated recurrent unit for gesture recognition[ J] . IEEE Transactions on Automation Science andEngineering, 2021, 18(2) : 495-507.
[17] LI C K, LI S, GAO Y B, et al. A two-stream neural network for pose-based hand gesture recognition [ J] . IEEETransactions on Cognitive and Developmental Systems,2022, 14(4) : 1594-1603.
[18] PALANISAMY G, THANGASWAMY S S. An efficienthand gesture recognition based on optimal deep embeddedhybrid convolutional neural network-long short term memory network model [ J] . Concurrency and Computation:Practice and Experience, 2022, 34(21) : 1-15.
[19] 谷学静, 周自朋, 郭宇承, 等. 基于 CNN-LSTM 混合模型的动态手势识别方法[ J] . 计算机应用与软件,2021, 38(11) : 205-209.
GU X J, ZHOU Z P, GUO Y C, et al. Dynamic gesturerecognition method based on CNN-LSTM hybrid model[ J ] . Computer Applications and Software, 2021, 38(11) : 205-209.
[20] REHMAN M, AHMED F, KHAN M, et al. Dynamichand gesture recognition using 3D-CNN and LSTM networks[ J] . Computers Materials & Continua, 2021, 70(3) : 4675-4690.
[21] MO D H, TIEN C L, YEH Y L, et al. Design of digitaltwin human-machine interface sensor with intelligent fingergesture recognition[J]. Sensors, 2023, 23(7): 3509.
[22] JI S W, XU W, YANG M, et al. 3D convolutional neuralnetworks for human action recognition [ J] . IEEE Transactions on Pattern Analysis and Machine Intelligence,2013, 35(1) : 221-231.
[23] TRAN D, BOURDEV L, FERGUS R, et al. Learningspatiotemporal features with 3D convolutional networks[C]∥2015 IEEE International Conference on ComputerVision ( ICCV) . Piscataway:IEEE, 2015: 4489-4497.
[24] MERCIONI M A, HOLBAN S. Soft Clipping Mish—a novel activation function for deep learning[C]∥2021 4th International Conference on Information and Computer Technologies (ICICT). Piscataway: IEEE, 2021: 13-17.
[25] 杨婷. 基于深度学习的多模态情感识别[ D] . 南昌:南昌大学, 2021.
YANG T. Multimodal emotion recognition based on deeplearning[D] . Nanchang: Nanchang University, 2021.

Similar References:

Memo

Last Update: 2024-09-02