CS Forum: "Movies and meaning: from low-level features to mind reading" Sergio Benini, University of Brescia

CS department's public guest lecture, open to everyone free-of-charge.

Prof. Sergio Benini
University of Brescia

Host: prof. Tapio Takala
Time: 11:15 (coffee at 11:00)
Venue: T6, CS building

Movies and meaning: from low-level features to mind reading

Abstract:

When dealing with movies, closing the tremendous discontinuity between low-level features and the richness of semantics in the viewers’ cognitive processes, requires a variety of approaches and perspectives at the crossroad between video content analysis, neuroscience, and psychology.

When attempting to relate movie content to users’ affective responses, previous work in content analysis suggests that a direct mapping of audio-visual properties into elicited emotions is difficult, due to the high variability of individual reactions. To reduce the gap between the objective level of features and the subjective sphere of emotions, we exploit the intermediate representation of the connotative properties of movies: the set of shooting and editing conventions that help in transmitting meaning to the audience [1] [2].

One of these stylistic feature, the shot scale, i.e. the distance of the camera from the subject, effectively regulates theory of mind, indicating that increasing spatial proximity to the character triggers higher occurrence of mental state references in viewers’ story descriptions [3] [4] [7]. When considered together with shot duration, meant as length of camera takes, shot scale does not appear as random patterns in movies from the same director, thus it may be also employed for automatic attribution of movie authorship [5].

Movies are also becoming an important stimuli employed in neural decoding, an ambitious line of research within contemporary neuroscience aiming at “mind-reading”. We address the challenge of producing generalizable decoding models, which allow the reconstruction of perceived audiovisual features from human magnetic resonance imaging (fMRI) data without prior training of the algorithm on the decoded content [6] [7]. In this field we also aim at combining fMRI data and deep features in a hybrid model able to predict specific video object classes [8].

References
[1] S. Benini, L. Canini, and R. Leonardi, “A connotative space for supporting movie affective recommendation,” IEEE Transactions on Multimedia, vol. 13, no. 6, pp. 1356–1370, 2011.
[2] L. Canini, S. Benini, and R. Leonardi, “Affective recommendation of movies based on selected connotative features,” Circuits and Systems for Video Technology, IEEE Transactions on, vol. 23, no. 4, pp. 636–647, 2013.
[3] S. Benini, M. Svanera, N. Adami, R. Leonardi, and A. B. Kovács, “Shot scale distribution in art films,” Multimedia Tools and Applications Multimedia Tools and Applications, 1-29, 2016.
[4] K. Bálint, M. Svanera, S. Benini, B. Rooney, “Formal Features Predict Narrative Engagement: Can Low-level Formal Features Predict Narrative Engagement and Enjoyment in Film Viewers? Computational analyses of Tarantino’s Violent Film Scenes”, submitted to Media Psychology, 2017.
[5] M. Svanera, M. Savardi, A. Signoroni, A. B. Kovács, S. Benini, “Who is the director of this movie? Automatic style recognition based on shot features ” submitted to IEEE Transactions on Image Processing, 2017.
[6] G. Raz, M. Svanera; N. Singer, G. Gilam, M. Bleich Cohen, T. Lin, R. Admon, T. Gonen, A. Thaler, R. Y. Granot, R. Goebel, S. Benini, G. Valente, "Robust inter-subject audiovisual decoding in functional magnetic resonance imaging using high-dimensional regression", in Neuroimage, vol. 163, pp. 244-263, December 2017
[7] G. Raz, G. Valente, M. Svanera, S. Benini, and A. B. Kovács, “Shot-scales in the Brain: A Robust Neural Fingerprint of Perceiving Apparent Distance in Movies”, submitted to Projections, 2017.
[8] M. Svanera, S. Benini, G. Raz, T. Hendler, R. Goebel, and G. Valente, "Deep driven fMRI decoding of visual categories", in NIPS Workshop on Representation Learning in Artificial and Biological Neural Networks (MLINI 2016), Barcelona, Spain, December 9, 2016.

Bio

Sergio Benini received his MSc degree in Electronic Engineering (2000, cum laude) and his PhD in Information Engineering (2006) from the University of Brescia, Italy. Between 2001 and 2003 he was with Siemens Mobile Communications R&D. During his Ph.D. he spent almost one year in British Telecom Research in UK. Since 2005 he is Assistant Professor at the University of Brescia. In 2012 he co-founded Yonder, a spin-off company specialized in NLP, Machine Learning, and Cognitive Computing.