Modeling Localness for Self-Attention Networks

UM > Faculty of Science and Technology > DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE

Status	已發表Published
	Modeling Localness for Self-Attention Networks
	Yang, B.; Tu, Z.; Wong, F.; Meng, F.; Chao, L.; Zhang, T.
	2018-10-31
Source Publication	Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Pages	4449-4458
Publication Place	Brussels
Publisher	Association for Computational Linguistics
Abstract	Self-attention networks have proven to be of profound value for its strength of capturing global dependencies. In this work, we propose to model localness for self-attention networks, which enhances the ability of capturing useful local context. We cast localness modeling as a learnable Gaussian bias, which indicates the central and scope of the local region to be paid more attention. The bias is then incorporated into the original attention distribution to form a revised distribution. To maintain the strength of capturing long distance dependencies and enhance the ability of capturing short-range dependencies, we only apply localness modeling to lower layers of self-attention networks. Quantitative and qualitative analyses on Chinese⇒English and English⇒German translation tasks demonstrate the effectiveness and universality of the proposed approach.
Keyword	Neural Machine Translation Transformer Self-Attention Models Local Context
Language	英語English
The Source to Article	PB_Publication
PUB ID	42494
Document Type	Conference paper
Collection	DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE
Corresponding Author	Wong, F.
Recommended Citation GB/T 7714	Yang, B.,Tu, Z.,Wong, F.,et al. Modeling Localness for Self-Attention Networks[C], Brussels:Association for Computational Linguistics, 2018, 4449-4458.
APA	Yang, B.., Tu, Z.., Wong, F.., Meng, F.., Chao, L.., & Zhang, T. (2018). Modeling Localness for Self-Attention Networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4449-4458.

Files in This Item:
There are no files associated with this item.

If you have any objections to this item, please fill out the form below and the administrator will contact you as soon as possible.
Content:
Email：	*
Affiliation No.
Verification Code:	Refresh

Any comments and suggestions are welcomed.
Title:	*
Content:
Email：	*
Verification Code:	Refresh