Open Access Open Access  Restricted Access Subscription Access

A Survey on Deep Learning based Detection of Abnormal Human Behaviour using Computer Vision Human Activity Recognition System

Partha Ghosh, Sayantan Roy, Sombit Bose, Avisek Mondal

Abstract


Abnormal Human activity recognition (Abnormal HAR) systems are very popular among researchers nowadays, they attempt to identify and analyze human activities using acquired information from sensors. Several papers have already been published in the abnormal HAR topics, the technologies in this field have multidisciplinary nature. They need constant updates. Our literature survey divided the approaches into three categories, the first one is about wearable sensor-based approach, the second one is about smartphone sensor-based approach and the third one is about computer vision-based approach. The first collects data from sensors using sensing devices, whereas the smartphone sensor takes input from smartphone sensors such as the gyroscope and accelerometer, and the last classifies activities using pose estimation, which necessitates the estimation of body key points using a neural network. Our literature survey attempts to review the Abnormal HAR systems from the deep learningbased computer vision perspective. We have further categorized the vision-based models into three basic types, the first one is the generative model, the second one is the discriminative model and the third one is about the hybrid ones. We have listed all recent models on various new datasets like UCF-Crime. Abnormal HAR has too many applications such as security, video surveillance, and home monitoring are highly related to abnormal HAR tasks. This establishes a new trend in abnormal HAR systems. Our survey aims to provide the reader an analysis of Vision-based HAR., it also checks the challenges and future scope of our survey.


Keywords


Human Activity Recognition, Deep Learning, Autoencoder, RNN, LSTM, Generative Adversarial Network

Full Text:

PDF

References


Beddiar, D.R., Nini, B., Sabokrou, M., Hadid, A. (2020). Vision-based human activity recognition: a survey. Multimedia Tools and Applications, 79 (41), 30509–30555.

Abuhamad, M., Abusnaina, A., Nyang, D., Mohaisen, D. (2020). Sensor-based cotinuous authentication of smartphones’ users using be-havioral biometrics: A contemporary survey. IEEE Internet of Things Journal, 8 (1), 65–84.

Xiu, Y., Li, J., Wang, H., Fang, Y., Lu, C.: Pose flow: Efficient online pose tracking. arXiv preprint arXiv:1802.00977 (2018)

Srivastava N, Mansimov E, Salakhudinov R (2015) Unsupervised learning of video representations using lstms. In: International conference on machine learning, pp 843–852

Xing L, Qin-kun X (2018) Human action recognition using auto-encode and pnn neural network. Software Guide (1): 4 1608.01529

Ravanbakhsh M, Nabi M, Sangineto E, Marcenaro L, Regazzoni C, Sebe N (2017) Abnormal event detection in videos using generative adversarial nets. In: 2017 IEEE International Conference on Image Processing (ICIP) IEEE, pp 1577–1581

Sabokrou M, Khalooei M, Fathy M, Adeli E (2018) Adversarially learned one-class classifier for novelty detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3379–3388

K.K. Hati, P.K. Sa, and B. Majhi. Lobs: Local background subtracter for video surveillance. In 2012 Asia Pacific Conference on Postgraduate Research in Microelectronics and Electronics, pages 29–34, 2012.

Zadeh, M.Z., Babu, A.R., Jaiswal, A., Makedon, F. (2020). Self-Supervised Human Activity Recognition by Augmenting Generative Adversarial Networks. arXiv preprint arXiv:2008.11755.

Sultani, W., Chen, C., Shah, M. (2018). Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6479-6488).

Mart ´ınez-Mascorro, G. A., Abreu-Pederzini, J.R., Ortiz-Bayliss, J.C., Terashima-Mar´ın, H. (2020). Suspicious behavior detection on shoplift-ing cases for crime prevention by using 3D convolutional neural net-works. arXiv preprint arXiv:2005.02142.

Ferrari, A., Micucci, D., Mobilio, M., Napoletano, P. (2020). On the Wan, B., Jiang, W., Fang, Y., Luo, Z., Ding, G. (2021). Anomaly detection in video sequences: A benchmark and computational model. arXiv preprint arXiv:2106.08570. personalization of classification models for human activity recognition. IEEE Access, 8, 32066–32079.

D. Berjon, C. Cuevas, F. Moran, and N. Garcia. GPU-based implemen-tation of an optimized nonparametric background modeling for real-time moving object detection. IEEE Transactions on Consumer Electronics, 59 (2): 361–369, 2013.

Joshila Grace. L.K and K. Reshmi. Face recognition in surveillance system. In 2015 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), pages 1–5, March 2015.

Ade Nurhopipah and Agus Harjoko. Motion detection and face recog-nition for cctv surveillance system. IJCCS (Indonesian Journal of Computing and Cybernetics Systems), 12:107, 07 2018.

Tang Sze Ling, Liang Kim Meng, Mei Lim, Kadim Zulaikha, Ahmed A. Baha’a, and Ahmed Al-Obaidi. Colour-based object tracking in surveil-lance application. In Proceedings of the International Multi Conference of Engineers and Computer Scientists 2009, IMECS 2009, volume 1, page No pages, 03 2009

J.S. Kim, D.H. Yeom, and Y.H. Joo. Fast and robust algorithm of tracking multiple moving objects for intelligent video surveillance systems. IEEE Transactions on Consumer Electronics, 57 (3): 1165–1170, 2011.

Hou, W. Wan, K. Han, R. Muhammad, and M. Yang. Human detection and tracking over camera networks: Areview. In 2016 International Conference on Audio, Language and Image Processing (ICALIP), pages 574–580, 2016.

JoohyungKang and Sooyeong Kwak. Loitering detection solution for cctv security system. Journal of Korea Multimedia Society, 17, 01 2014.

Jing-Ying Chang, Huei-Hung Liao, and Liang-Gee Chen. Localized detection of abandoned luggage. EURASIP Journal on Advances in Signal Processing, 2010 (1):675784, 2010.

S. Wu, H. Wong, and Z. Yu. A bayesian model for crowd escape behavior detection. IEEE Transactions on Circuits and Systems for Video Technology, 24 (1): 85–98, Jan 2014.

Manuel Alvar, Andrea Torsello, Alvaro Sanchez-Miralles, and Jos e´ Marıa Armingol. Abnormal behavior detection using dominant sets. Machine Vision and Applications, 25 (5): 1351–1368, 2014.

Tian Wang, Meina Qiao, Yingjun Deng, Yi Zhou, Huan Wang, Qi Lyu, and Hichem Snoussi. Abnormal event detection based on analysis of movement information of video sequence. Optik-International Journal for Light and Electron Optics, 152: 50–60, 2018.

Ullah, W., Ullah, A., Haq, I.U., Muhammad, K., Sajjad, M., Baik, S.W. (2021). CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks. Multimedia Tools and Applications, 80 (11), 16979–16995.

Begampure, S., Jadhav, P. (2021). Intelligent video analytics for human action detection: a deep learning approach with transfer learning. International Journal of Computing and Digital System.

AKrizhevsky, I Sutskever, GE Hinton, “Imagenet classification with deep convolutional neural networks”, Communications of the ACM 60 (6), 84–90

Karen Simonyan, Andrew Zisserman Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv preprint arXiv:1409.1556, 2014.,[cs.CV] 10 Dec 2015

28. Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun ”Deep Residual Learning for Image Recognition”, Microsoft Research, arXiv:1512.03385v1,[cs.CV] 10 Dec 2015.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierr Sermanet,Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Ra-binovich, “Going Deeper with Convolutions”, CVPR 2015.

UCSD Anomaly Detection Dataset [Online]. Available from http://www.svcl.ucsd.edu/projects/ anomaly/dataset.html

Liu, W., Luo, W., Lian, D., Gao, S. (2018). Future frame prediction for anomaly detection–a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6536-6545).

Lv, H., Zhou, C., Cui, Z., Xu, C., Li, Y., Yang, J. (2021). Localizing anomalies from weakly-labeled videos. IEEE transactions on image processing, 30, 4505–4515.

The spruceeats. What Is Caviar? [online]. Available from https://www.thespruceeats.com/what-is-caviar-1807004

CAVIAR: Context Aware Vision using Image-based Active Recognition [Online]. Available from https://homepages.inf.ed.ac.uk/rbf/CAVIAR/

Wu, P., Liu, J., Shi, Y., Sun, Y., Shao, F., Wu, Z., Yang, Z. (2020, August). Not only look, but also listen: Learning multimodal violence detection under weak supervision. In European Conference on Computer Vision (pp. 322–339). Springer, Cham.

Aqeel, M., Khan, K.B., Azam, M.A., Ghouri, M.H., Jaskani, F. H. (2020, November). Detection of Anomaly in Videos Using Con-volutional Autoencoder and Generative Adversarial Network model. In 2020 IEEE 23rd International Multitopic Conference (INMIC) (pp. 1–6). IEEE.

M. Sabokrou, M. Fayyaz, M. Fathy, and R. Klette, ”Deep-cascade: Cascading 3d deep neural networks for fast anomaly detection and local-ization in crowded scenes,” IEEE Transactions on Image Processing,vol. 26, pp. 1992-2004, 2017.

Zhu, Y., Newsam, S. (2019). Motion-aware feature for improved video anomaly detection. arXiv preprint arXiv:1907.10211.

Luo, W., Liu, W., Gao, S. (2017). A revisit of sparse coding based anomaly detection in stacked rnn framework. In Proceedings of the IEEE International Conference on Computer Vision (pp. 341–349).

W. Luo, W. Liu, and S. Gao, ”Remembering history with convolutional lstm for anomaly detection,” in 2017 IEEE International Conference on Multimedia and Expo (ICME), 2017, pp. 439–444.

Chong, Y.S., Tay, Y.H. (2017, June). Abnormal event detection in videos using spatiotemporal autoencoder. In International symposium on neural networks (pp. 189–196). Springer, Cham.

Dubey, S., Boragule, A., Jeon, M. (2019, October). 3D ResNet with Ranking Loss Function forAbnormal Activity Detection in Videos. In 2019 International Conference on Control, Automation and Information Sciences (ICCAIS) (pp. 1–6). IEEE.

Zaheer, M.Z., Mahmood, A., Shin, H., Lee, S.I. (2020).Aself-reasoning framework for anomaldetection using video-level labels. IEEE Signal Processing Letters, 27, 1705-1709.

Tian, Y., Pang, G., Chen, Y., Singh, R., Verjans, J.W., Carneiro, G. (2021). Weakly-supervised Video Anomaly Detection with Robust Tem-poral Feature Magnitude Learning. arXiv preprint arXiv:2101.10030.

O¨ ztu¨rk, H.˙I., Can, A.B. (2021). ADNet: Temporal Anomaly Detection in Surveillance Videos. arXiv preprint arXiv:2104.06653. using video-level labels. IEEE Signal Processing Letters, 27, 1705–1709.

Boekhoudt, K., Matei, A., Aghaei, M., Talavera, E. (2021). HR-Crime: Human-Related Anomaly Detection in Surveillance Videos. arXiv preprintarXiv:2108.00246.

Wu, J., Zhang, W., Li, G., Wu, W., Tan, X., Li, Y., ... Lin, L. (2021). Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video. arXiv preprint arXiv:2108.03825.

Zhou S, Shen W, Zeng D, Fang M, Wei Y, Zhang Z. Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Proc Image Commun. 2016; 47: 358–68.

Nasaruddin, N., Muchtar, K., Afdhal, A., Dwiyantoro, A.P.J. (2020). Deep anomaly detection through visual attention in surveillance videos. Journal of Big Data, 7 (1), 1–17.

Liu, J., Shahroudy, A., Xu, D., Wang, G. (2016, October). Spatio temporal lstm with trust gates for 3d human action recognition. In European conference on computer vision (pp. 816–833). Springer, Cham.

Wan, B., Jiang, W., Fang, Y., Luo, Z., Ding, G. (2021). Anomaly detection in video sequences: A benchmark and computational model. arXiv preprint arXiv:2106.08570.

Noori, F.M., Wallace, B., Uddin, M.Z., Torresen, J. (2019, June). A robust human activity recognition approach using open pose, motion features, and deep recurrent neural network. In Scandinavian conference on image analysis (pp. 299–310). Springer, Cham.

Kale, G.V. (2019). Human activity recognition on real time and offline dataset. International Journal of Intelligent Systems and Applications in Engineering, 7 (1), 60–65.

O¨ ztu¨rk, H.˙I., Can, A.B. (2021). ADNet: Temporal Anomaly Detection in Surveillance Videos. ArXiv preprint arXiv:2104.06653.

Wu, J., Zhang, W., Li, G., Wu, W., Tan, X., Li, Y., ... Lin, L. (2021). Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video.arXiv preprint arXiv:2108.03825.

Zhou S, Shen W, Zeng D, Fang M, Wei Y, Zhang Z. Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Proc Image Commun. 2016; 47: 358–68.

Mekruksavanich, S., Jitpattanakul, A. (2021). Biometric user identification based on human activity recognition using wearable sensors: An experiment using deep learning models. Electronics, 10 (3), 308.

Avenue Dataset for Abnormal Event Detection [Online]. Available from http://www.cse.cuhk. edu.hk/leojia/projects/detectabnormal/dataset.html




DOI: https://doi.org/10.37591/jocta.v12i3.869

Refbacks

  • There are currently no refbacks.


Copyright (c) 2021 Journal of Computer Technology & Applications