Understanding and Improving Lexical Choice in Non-Autoregressive Translation

UM > Faculty of Science and Technology > DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE

Status	已發表Published
	Understanding and Improving Lexical Choice in Non-Autoregressive Translation
	Ding, L.; Wang, L.; Liu, X.; Wong, F.; Tao, D.; Tu, Z.
	2021-05-03
Source Publication	Ninth International Conference on Learning Representations
Abstract	Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data. To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach. Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words. Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively.
Keyword	Non-Autoregressive Neural Machine Translation
Language	英語English
The Source to Article	PB_Publication
PUB ID	58018
Document Type	Conference paper
Collection	DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE
Recommended Citation GB/T 7714	Ding, L.,Wang, L.,Liu, X.,et al. Understanding and Improving Lexical Choice in Non-Autoregressive Translation[C], 2021.
APA	Ding, L.., Wang, L.., Liu, X.., Wong, F.., Tao, D.., & Tu, Z. (2021). Understanding and Improving Lexical Choice in Non-Autoregressive Translation. Ninth International Conference on Learning Representations.

Files in This Item:
There are no files associated with this item.

If you have any objections to this item, please fill out the form below and the administrator will contact you as soon as possible.
Content:
Email：	*
Affiliation No.
Verification Code:	Refresh

Any comments and suggestions are welcomed.
Title:	*
Content:
Email：	*
Verification Code:	Refresh