Research paperPDF · 15 pages

Attention Is All You Need

Ashish Vaswani · Noam Shazeer · Niki Parmar · Jakob Uszkoreit · Llion Jones · Aidan N. Gomez · Łukasz Kaiser · Illia Polosukhin

2017

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks. We introduce a simpler architecture based solely on attention mechanisms.

1Introduction

Recurrent neural networks have been firmly established as state-of-the-art approaches in sequence modeling and transduction problems such as language modeling and machine translation.

The Transformer follows an encoder-decoder structure. Instead of recurrence, the model relies entirely on attention to draw global dependencies between input and output.

2Model Architecture

Most competitive neural sequence transduction models have an encoder-decoder structure. The encoder maps an input sequence to a continuous representation, and the decoder generates an output sequence one element at a time.

\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Equation 1

3Results

Model	BLEU	Training cost
GNMT + RL	24.6	2.3 × 10¹⁹
ConvS2S	25.2	9.6 × 10¹⁸
Transformer (base)	27.3	3.3 × 10¹⁸
Transformer (big)	28.4	2.3 × 10¹⁹

Table 1. Translation quality and training cost.

4Conclusion

We presented the Transformer, the first sequence transduction model based entirely on attention. The architecture can be trained significantly faster than recurrent or convolutional alternatives.

References

Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate.
Gehring, J. et al. Convolutional sequence to sequence learning.