Novel Emotion Recognition Framework from Facial Expressions Using Spiking Neural Networks on Wearable Edge Devices
Keywords:
Spiking Neural Networks (SNNs), Open Neural Network Exchange (ONNX), Long Short-Term Memory (LSTM), Message Queuing Telemetry Transport (MQTT).Abstract
Real-time emotion recognition from facial expressions holds significant promise for enhancing human-computer interaction and personalizing user experiences. Harnessing the potential of advanced technologies, the research presents a novel approach to real-time emotion recognition from facial expressions using Spiking Neural Networks (SNN) on wearable edge devices. The methodology integrates key technologies including Open Neural Network Exchange (ONNX), Message Queuing Telemetry Transport (MQTT), and Long Short-Term Memory (LSTM) networks to enhance the efficiency and accuracy of emotion recognition systems in practical scenarios. By leveraging ONNX, seamless model interchangeability and deployment across diverse hardware platforms are achieved, ensuring scalability and flexibility in model deployment. Through optimized model conversion and deployment on wearable edge devices, interoperability and efficiency in real-time emotion recognition tasks are ensured. MQTT serves as a lightweight and reliable communication protocol for seamless data exchange between wearable edge devices and external systems, facilitating real-time transmission of facial expression data and inference results. This enables collaborative processing and decision-making across distributed networks, enhancing system responsiveness and scalability. The integration of LSTM networks captures temporal dependencies in facial expressions, improving the accuracy and robustness of emotion recognition systems. LSTM networks excel in modeling sequential data and long-term dependencies, making them suitable for analyzing temporal patterns in facial expressions over time.Top of Form
Downloads
References
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001). Emotion recognition in human-computer interaction. IEEE Signal processing magazine, 18(1), 32-80.
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022). Deep learning in human activity recognition with wearable sensors: A review on advances. Sensors, 22(4), 1476.
Shi, Q., Dong, B., He, T., Sun, Z., Zhu, J., Zhang, Z., & Lee, C. (2020). Progress in wearable electronics/photonics—Moving toward the era of artificial intelligence and internet of things. InfoMat, 2(6), 1131-1162.
Seng, K. P., Lee, P. J., & Ang, L. M. (2021). Embedded intelligence on FPGA: Survey, applications and challenges. Electronics, 10(8), 895.
Mazumder, A. N., Meng, J., Rashid, H. A., Kallakuri, U., Zhang, X., Seo, J. S., & Mohsenin, T. (2021). A survey on the optimization of neural network accelerators for micro-ai on-device inference. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 11(4), 532-547.
Azghadi, M. R., Lammie, C., Eshraghian, J. K., Payvand, M., Donati, E., Linares-Barranco, B., & Indiveri, G. (2020). Hardware implementation of deep network accelerators towards healthcare and biomedical applications. IEEE Transactions on Biomedical Circuits and Systems, 14(6), 1138-1159.
Ferguson, M., Jeong, S., Law, K. H., Levitan, S., Narayanan, A., Burkhardt, R., ... & Lee, Y. T. T. (2019, August). A standardized representation of convolutional neural networks for reliable deployment of machine learning models in the manufacturing industry. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (Vol. 59179, p. V001T02A005). American Society of Mechanical Engineers.
Tosetti, F. (2021). Deep Learning on the Edge: a comparative analysis on Computer Vision for space applications (Doctoral dissertation, Politecnico di Torino).
Dizdarević, J., Carpio, F., Jukan, A., & Masip-Bruin, X. (2019). A survey of communication protocols for internet of things and related challenges of fog and cloud computing integration. ACM Computing Surveys (CSUR), 51(6), 1-29.
Koziolek, H., Grüner, S., & Rückert, J. (2020). A comparison of MQTT brokers for distributed IoT edge computing. In Software Architecture: 14th European Conference, ECSA 2020, L'Aquila, Italy, September 14–18, 2020, Proceedings 14 (pp. 352-368). Springer International Publishing.
Teixeira, T., Granger, É., & Lameiras Koerich, A. (2021). Continuous emotion recognition with spatiotemporal convolutional neural networks. Applied Sciences, 11(24), 11738.
Zheng, Y., & Blasch, E. (2023). Facial Micro-Expression Recognition Enhanced by Score Fusion and a Hybrid Model from Convolutional LSTM and Vision Transformer. Sensors, 23(12), 5650.
Geetha, A. V., Mala, T., Priyanka, D., & Uma, E. (2024). Multimodal Emotion Recognition with deep learning: advancements, challenges, and future directions. Information Fusion, 105, 102218.
Rathi, N., Agrawal, A., Lee, C., Kosta, A. K., & Roy, K. (2021, February). Exploring spike-based learning for neuromorphic computing: Prospects and perspectives. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE) (pp. 902-907). IEEE.
Kaushik, H., Kumar, T., & Bhalla, K. (2022). iSecureHome: A deep fusion framework for surveillance of smart homes using real-time emotion recognition. Applied Soft Computing, 122, 108788.
Datta, G. (2023). Towards Efficient Edge Intelligence with In-Sensor and Neuromorphic Computing: Algorithm-Hardware Co-Design (Doctoral dissertation, University of Southern California).
Pu, G., & Chen, J. (2024). Facial Expression Recognition Based on Convolutional Spiking Neural Network and STDP Fine-Tune.
Aouedi, O. (2024). Towards a Scalable and Energy-Efficient Framework for Industrial Cloud-Edge-IoT Continuum. IEEE Internet of Things Magazine.
Tan, C., Ceballos, G., Kasabov, N., & Puthanmadam Subramaniyam, N. (2020). Fusionsense: Emotion classification using feature fusion of multimodal data and deep learning in a brain-inspired spiking neural network. Sensors, 20(18), 5328.
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2022). Efficient acceleration of deep learning inference on resource-constrained edge devices: A review. Proceedings of the IEEE, 111(1), 42-91.
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., ... & Hussain, A. (2024). Interpreting black-box models: a review on explainable artificial intelligence. Cognitive Computation, 16(1), 45-74.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
