Attention Is All You Need
- language
- neural networks
Ashish Vaswani and colleagues introduced the Transformer in the 2017 paper Attention Is All You Need. The architecture used attention mechanisms to relate positions in a sequence, dispensing with recurrence and convolution in the proposed encoder-decoder design for translation. Its more parallelizable training and strong results made it influential beyond the original experiments. Later language-model families adapted Transformer ideas in different forms and at very different scales. The paper is an architectural milestone, not the first use of attention or the release date of every later generative model. [Original publication](https://arxiv.org/abs/1706.03762). *Image: Wikimedia Commons (the Transformer's architecture; diagram by Yuening Jia; CC BY-SA 3.0).* **Timeline anchor:** 12 June 2017 CE. The arXiv date. **References:** [Source 1](https://en.wikipedia.org/wiki/Attention_Is_All_You_Need), [Source 2](https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)).
- Date precision
- The arXiv date.
Sources
Related
On other timelines
- The Transformer on The history of computer science by @athan
- Attention Is All You Need reports GPU-trained Transformer models on GPU and ASIC programming · parallel hardware, standards and applications by @athan