Residential College | false |
Status | 已發表Published |
Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities | |
Jiandian Zeng1; Jiantao Zhou1; Tianyi Liu2 | |
2023-12 | |
Source Publication | IEEE TRANSACTIONS ON MULTIMEDIA |
ISSN | 1520-9210 |
Volume | 25Pages:6301-6314 |
Abstract | Multimodal sentiment analysis aims to extract emotions with multiple data sources, usually under the assumption that all modalities are available. In practice, such a strong assumption does not always hold, and most of multimodal sentiment analysis methods may fail when partial modalities are missing. Some existing works have started to address the missing modality problem; but only considered the single modality missing case, while ignoring the practically more general cases of multiple modalities missing. To this end, in this paper, we propose a Tag-Assisted Transformer Encoder (TATE) network to handle the problem of missing uncertain modalities. Specifically, we design a tag encoding module to cover both the single modality and multiple modalities missing cases, so as to guide the network’s attention to those missing modalities. Besides, a new space projection pattern is adopted to align common vectors, taking into account the different importance of each modality. Afterwards, a Transformer encoder-decoder network is utilized to learn the missing modality features, and the outputs of the Transformer encoder are extracted for the final sentiment classification. Extensive experiments and analyses are conducted on CMU-MOSI, IEMOCAP, and MELD datasets, which show that the proposed method can achieve significant improvements compared with several baselines. |
Keyword | Multimodal Sentiment Analysis Missing Modality Joint Representation |
DOI | 10.1109/TMM.2022.3207572 |
URL | View the original |
Indexed By | SCIE |
WOS Research Area | Computer Science ; Telecommunications |
WOS Subject | Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications |
WOS ID | WOS:001098831500047 |
Publisher | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 445 HOES LANE, PISCATAWAY, NJ 08855-4141 |
Scopus ID | 2-s2.0-85139405659 |
Fulltext Access | |
Citation statistics | |
Document Type | Journal article |
Collection | THE STATE KEY LABORATORY OF INTERNET OF THINGS FOR SMART CITY (UNIVERSITY OF MACAU) Faculty of Science and Technology DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE |
Corresponding Author | Jiantao Zhou |
Affiliation | 1.State Key Laboratory of Internet of Things for Smart City, and Department of Computer and Information Science, University of Macau, Macau, China 2.Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China |
First Author Affilication | University of Macau |
Corresponding Author Affilication | University of Macau |
Recommended Citation GB/T 7714 | Jiandian Zeng,Jiantao Zhou,Tianyi Liu. Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities[J]. IEEE TRANSACTIONS ON MULTIMEDIA, 2023, 25, 6301-6314. |
APA | Jiandian Zeng., Jiantao Zhou., & Tianyi Liu (2023). Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities. IEEE TRANSACTIONS ON MULTIMEDIA, 25, 6301-6314. |
MLA | Jiandian Zeng,et al."Robust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities".IEEE TRANSACTIONS ON MULTIMEDIA 25(2023):6301-6314. |
Files in This Item: | There are no files associated with this item. |
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment