The relationship between visual and verbal information and emotion processing: reports from EEG and self-assessment

Abstract

People receive emotion-related information through different mediums. In vision, emotional content could be presented in either a visual or verbal manner (pictorially or as text). In the current thesis, we investigated whether perceived emotional material is processed homogeneously and whether the processing manner (visual/verbal) affects how people process and express emotional content. For that, 108 healthy adults were shown pictures depicting facial expressions (Ekman database) and words presenting six basic emotions (anger, happiness, sadness, surprise, fear, disgust) and one depiction of neutral. The participants had to either mirror the facial expression or express the written emotion while their EEG was recorded, as well as give a rating of the task difficulty. The results for self-reported task difficulty indicated that expressing an emotion was consistently assessed easier in the verbal condition, congruous with our hypothesis that verbal labelling helps reduce the uncertainty related to facial portrayals of expression. We chose three time-intervals for the EEG analyses (representative of the N1, N170 and LPP ERP-components). The results comparing the visual and verbal condition showed that surprise and fear were processed in a significantly different manner in the N1 and N170 time-interval, and all emotions except fear were dissimilar in the LPP time-interval. This is indicative of a preferential treatment for fear stimuli. We concluded that processing emotional content is highly dependent on how it is presented in the task, and the perceived emotional ambiguity emerging from stimuli being presented in a visual or verbal manner could therefore play an important role in how emotions are processed, expressed and perceived by others.