Residential College | false |
Status | 已發表Published |
Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities | |
Jiandian Zeng1; Tianyi Liu2; Jiantao Zhou1 | |
2022-07-07 | |
Conference Name | 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) |
Source Publication | PROCEEDINGS OF THE 45TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '22) |
Pages | 1545-1554 |
Conference Date | July 11th to 15th, 2022 |
Conference Place | Madrid, Spain |
Country | Spain |
Author of Source | Enrique Amigo ; Pablo Castells ; Julio Gonzalo |
Publication Place | New York, United States |
Publisher | Association for Computing Machinery |
Abstract | Multimodal sentiment analysis has been studied under the assump- tion that all modalities are available. However, such a strong as- sumption does not always hold in practice, and most of multimodal fusion models may fail when partial modalities are missing. Several works have addressed the missing modality problem; but most of them only considered the single modality missing case, and ignored the practically more general cases of multiple modalities missing. To this end, in this paper, we propose a Tag-Assisted Transformer En- coder (TATE) network to handle the problem of missing uncertain modalities. Specifically, we design a tag encoding module to cover both the single modality and multiple modalities missing cases, so as to guide the network’s attention to those missing modalities. Besides, we adopt a new space projection pattern to align common vectors. Then, a Transformer encoder-decoder network is utilized to learn the missing modality features. At last, the outputs of the Transformer encoder are used for the final sentiment classification. Extensive experiments are conducted on CMU-MOSI and IEMO- CAP datasets, showing that our method can achieve significant improvements compared with several baselines. |
Keyword | Multimodal Sentiment Analysis Missing Modality Joint Representation |
DOI | 10.1145/3477495.3532064 |
URL | View the original |
Indexed By | CPCI-S |
WOS Research Area | Computer Science |
WOS Subject | Computer Science, Information Systems |
WOS ID | WOS:000852715901058 |
Scopus ID | 2-s2.0-85135032569 |
Fulltext Access | |
Citation statistics | |
Document Type | Conference paper |
Collection | THE STATE KEY LABORATORY OF INTERNET OF THINGS FOR SMART CITY (UNIVERSITY OF MACAU) Faculty of Science and Technology DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE |
Corresponding Author | Jiantao Zhou |
Affiliation | 1.State Key Laboratory of IoT for Smart City, University of Macau Macau, China 2.Shanghai Jiao Tong University Shanghai, China |
First Author Affilication | University of Macau |
Corresponding Author Affilication | University of Macau |
Recommended Citation GB/T 7714 | Jiandian Zeng,Tianyi Liu,Jiantao Zhou. Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities[C]. Enrique Amigo, Pablo Castells, Julio Gonzalo, New York, United States:Association for Computing Machinery, 2022, 1545-1554. |
APA | Jiandian Zeng., Tianyi Liu., & Jiantao Zhou (2022). Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities. PROCEEDINGS OF THE 45TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '22), 1545-1554. |
Files in This Item: | Download All | |||||
File Name/Size | Publications | Version | Access | License | ||
Tag-assisted Multimo(2207KB) | 会议论文 | 开放获取 | CC BY-NC-SA | View Download |
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment