TY - JOUR
T1 - Modeling Emotion in Complex Stories
T2 - The Stanford Emotional Narratives Dataset
AU - Ong, Desmond C.
AU - Wu, Zhengxuan
AU - Zhi-Xuan, Tan
AU - Reddan, Marianne
AU - Kahhale, Isabella
AU - Mattek, Alison
AU - Zaki, Jamil
N1 - Publisher Copyright:
© 2010-2012 IEEE.
PY - 2021/7/1
Y1 - 2021/7/1
N2 - Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. We demonstrate several baseline and state-of-the-art modeling approaches on the SEND, including a Long Short-Term Memory model and a multimodal Variational Recurrent Neural Network, which perform comparably to the human-benchmark. We end by discussing the implications for future research in time-series affective computing.
AB - Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. We demonstrate several baseline and state-of-the-art modeling approaches on the SEND, including a Long Short-Term Memory model and a multimodal Variational Recurrent Neural Network, which perform comparably to the human-benchmark. We end by discussing the implications for future research in time-series affective computing.
KW - Affective computing
KW - affect sensing and analysis
KW - emotional corpora
KW - multi-modal recognition
UR - http://www.scopus.com/inward/record.url?scp=85114555331&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85114555331&partnerID=8YFLogxK
U2 - 10.1109/TAFFC.2019.2955949
DO - 10.1109/TAFFC.2019.2955949
M3 - Article
AN - SCOPUS:85114555331
SN - 1949-3045
VL - 12
SP - 579
EP - 594
JO - IEEE Transactions on Affective Computing
JF - IEEE Transactions on Affective Computing
IS - 3
M1 - 8913483
ER -