Residential Collegefalse
Status已發表Published
Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities
Jiandian Zeng1; Jiantao Zhou1; Tianyi Liu2
2023-12
Source PublicationIEEE TRANSACTIONS ON MULTIMEDIA
ISSN1520-9210
Volume25Pages:6301-6314
Abstract

Multimodal sentiment analysis aims to extract emotions with multiple data sources, usually under the assumption that all modalities are available. In practice, such a strong assumption does not always hold, and most of multimodal sentiment analysis methods may fail when partial modalities are missing. Some existing works have started to address the missing modality problem; but only considered the single modality missing case, while ignoring the practically more general cases of multiple modalities missing. To this end, in this paper, we propose a Tag-Assisted Transformer Encoder (TATE) network to handle the problem of missing uncertain modalities. Specifically, we design a tag encoding module to cover both the single modality and multiple modalities missing cases, so as to guide the network’s attention to those missing modalities. Besides, a new space projection pattern is adopted to align common vectors, taking into account the different importance of each modality. Afterwards, a Transformer encoder-decoder network is utilized to learn the missing modality features, and the outputs of the Transformer encoder are extracted for the final sentiment classification. Extensive experiments and analyses are conducted on CMU-MOSI, IEMOCAP, and MELD datasets, which show that the proposed method can achieve significant improvements compared with several baselines.

KeywordMultimodal Sentiment Analysis Missing Modality Joint Representation
DOI10.1109/TMM.2022.3207572
URLView the original
Indexed BySCIE
WOS Research AreaComputer Science ; Telecommunications
WOS SubjectComputer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS IDWOS:001098831500047
PublisherIEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 445 HOES LANE, PISCATAWAY, NJ 08855-4141
Scopus ID2-s2.0-85139405659
Fulltext Access
Citation statistics
Document TypeJournal article
CollectionTHE STATE KEY LABORATORY OF INTERNET OF THINGS FOR SMART CITY (UNIVERSITY OF MACAU)
Faculty of Science and Technology
DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE
Corresponding AuthorJiantao Zhou
Affiliation1.State Key Laboratory of Internet of Things for Smart City, and Department of Computer and Information Science, University of Macau, Macau, China
2.Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
First Author AffilicationUniversity of Macau
Corresponding Author AffilicationUniversity of Macau
Recommended Citation
GB/T 7714
Jiandian Zeng,Jiantao Zhou,Tianyi Liu. Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities[J]. IEEE TRANSACTIONS ON MULTIMEDIA, 2023, 25, 6301-6314.
APA Jiandian Zeng., Jiantao Zhou., & Tianyi Liu (2023). Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities. IEEE TRANSACTIONS ON MULTIMEDIA, 25, 6301-6314.
MLA Jiandian Zeng,et al."Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities".IEEE TRANSACTIONS ON MULTIMEDIA 25(2023):6301-6314.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Jiandian Zeng]'s Articles
[Jiantao Zhou]'s Articles
[Tianyi Liu]'s Articles
Baidu academic
Similar articles in Baidu academic
[Jiandian Zeng]'s Articles
[Jiantao Zhou]'s Articles
[Tianyi Liu]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Jiandian Zeng]'s Articles
[Jiantao Zhou]'s Articles
[Tianyi Liu]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.