Affective Large Language Model-Driven Virtual Companions for Real-Time Emotion-Adaptive Dialogue Generation in Immersive VR-Based Empathy Training
Keywords:
Affect-aware computing; Large Language Models; Emotion Recognition; Physiological Signals; Virtual Companion; Adaptive Dialogue Generation; Empathy Training; Synthetic Dataset; Human-AI Interaction.Abstract
This research suggests an affect-conscious LLM-based virtual companion model that combines synthetic physiological emotion identification, emotion intensity predictor with adaptive dialogue forms to generate empathetic dialogues. Synthetic dataset of 5000 physiological samples of the four conditions, Calm, Neutral, Distressed and Engaged, was generated and tested on random forest, SVM and ANN classifiers. The models were able to attain a good performance of about 0.98 Macro-F1, which illustrates good performance in emotion classification. The dialogue evaluation based on affect conditioning adaptation was better than the scripted and generic conversation style. The framework shows that it is computationally feasible to implement emotion-directed virtual companions but verification of them in the real world must involve human experiments as well as actual physiological data.
Downloads
References
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W. and Zhang, W., 2022. A systematic review on affective computing: Emotion models, databases, and recent advances. Information Fusion, 83, pp.19-52.
Comas, J., Aspandi, D. and Binefa, X., 2020, November. End-to-end facial and physiological model for affective computing and applications. In 2020 15th IEEE international conference on Automatic Face and Gesture Recognition (FG 2020) (pp. 93-100). IEEE.
Saganowski, S., Perz, B., Polak, A.G. and Kazienko, P., 2022. Emotion recognition for everyday life using physiological signals from wearables: A systematic literature review. IEEE Transactions on Affective Computing, 14(3), pp.1876-1897.
Smith, R., Parr, T. and Friston, K.J., 2019. Simulating emotions: An active inference model of emotional state inference and emotion concept learning. Frontiers in psychology, 10, p.2844.
Bulagang, A.F., Weng, N.G., Mountstephens, J. and Teo, J., 2020. A review of recent approaches for emotion classification using electrocardiography and electrodermography signals. Informatics in Medicine Unlocked, 20, p.100363.
Alexander, R.G., Macknik, S.L. and Martinez-Conde, S., 2020. Microsaccades in applied environments: Real-world applications of fixational eye movement measurements. Journal of Eye Movement Research, 12(6), p.50.
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E. and Mihalcea, R., 2019, July. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 527-536).
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M. and Davison, J., 2020, October. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations (pp. 38-45).
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and Agarwal, S., 2020. Language models are few-shot learners. Advances in neural information processing systems, 33, pp.1877-1901.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.T., Rocktäschel, T. and Riedel, S., 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33, pp.9459-9474.
Slater, M. and Sanchez-Vives, M.V., 2016. Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3, p.74.
Makransky, G. and Petersen, G.B., 2019. Investigating the process of learning with desktop virtual reality: A structural equation modeling approach. Computers & Education, 134, pp.15-30.
Liu, Y., 2020, June. The application of virtual reality in empathy establishment: Foresee the future. In 2020 5th International Conference on Computational Intelligence and Applications (ICCIA) (pp. 188-193). IEEE.
Bertrand, P., Guegan, J., Robieux, L., McCall, C.A. and Zenasni, F., 2018. Learning empathy through virtual reality: multiple strategies for training empathy-related abilities using body ownership illusions in embodied virtual reality. Frontiers in Robotics and AI, 5, p.26.
Kusal, S., Patil, S., Choudrie, J., Kotecha, K., Mishra, S. and Abraham, A., 2022. AI-based conversational agents: a scoping review from technologies to future directions. IEEE access, 10, pp.92337-92356.
Følstad, A. and Brandtzaeg, P.B., 2020. Users' experiences with chatbots: findings from a questionnaire study. Quality and User Experience, 5(1), p.3.
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W. and Taylor, J.G., 2001. Emotion recognition in human-computer interaction. IEEE Signal processing magazine, 18(1), pp.32-80.
Abdu, F.J., Zhang, Y., Fu, M., Li, Y. and Deng, Z., 2021. Application of deep learning on millimeter-wave radar signals: A review. Sensors, 21(6), p.1951.
Strohmann, T., Siemon, D., Khosrawi-Rad, B. and Robra-Bissantz, S., 2023. Toward a design theory for virtual companionship. Human–Computer Interaction, 38(3-4), pp.194-234.
Virvou, M., 2023. Artificial Intelligence and User Experience in reciprocity: Contributions and state of the art. Intelligent Decision Technologies, 17(1), pp.73-125.
Hassija, V., Chakrabarti, A., Singh, A., Chamola, V. and Sikdar, B., 2023. Unleashing the potential of conversational AI: Amplifying chat-GPT’s capabilities and tackling technical hurdles. Ieee Access, 11, pp.143657-143682.
Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X. and Yang, X., 2018. A review of emotion recognition using physiological signals. Sensors, 18(7), p.2074.
Tran, Q.D.L. and Le, A.C., 2023. Exploring bi-directional context for improved chatbot response generation using deep reinforcement learning. Applied Sciences, 13(8), p.5041.
Zhou, Y., Zhi, C., Xu, F., Cui, W., Wang, H., Qin, A., Chen, X., Wang, Y. and Huang, X., 2023. Keyword-Aware Transformers Network for Chinese Open-Domain Conversation Generation. Electronics, 12(5), p.1228.
Caldarini, G., Jaf, S. and McGarry, K., 2022. A literature survey of recent advances in chatbots. Information, 13(1), p.41.
Gerry, L.J., Billinghurst, M. and Broadbent, E., 2022, March. Empathic skills training in virtual reality: a scoping review. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 227-232). IEEE.
Moon, J., 2018. Reviews of social embodiment for design of non-player characters in virtual reality-based social skill training for autistic children. Multimodal Technologies and Interaction, 2(3), p.53.
Zhang, M., Ding, H., Naumceska, M. and Zhang, Y., 2022. Virtual reality technology as an educational and intervention tool for children with autism spectrum disorder: current perspectives and future directions. Behavioral Sciences, 12(5), p.138.
Angulo, C., Chacón, A. and Ponsa, P., 2023. Towards a cognitive assistant supporting human operators in the Artificial Intelligence of Things. Internet of Things, 21, p.100673.
Theis, S., Jentzsch, S., Deligiannaki, F., Berro, C., Raulf, A.P. and Bruder, C., 2023, July. Requirements for explainability and acceptance of artificial intelligence in collaborative work. In International conference on human-computer interaction (pp. 355-380). Cham: Springer Nature Switzerland.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
