What are Transformer Neural Networks?

This short tutorial covers the basics of the Transformer, a neural network architecture designed for handling sequential data in machine learning.

Timestamps:
0:00 - Intro
1:18 - Motivation for developing the Transformer
2:44 - Input embeddings (start of encoder walk-through)
3:29 - Attention
6:29 - Multi-head attention
7:55 - Positional encodings
9:59 - Add & norm, feedforward, & stacking encoder layers
11:14 - Masked multi-head attention (start of decoder walk-through)
12:35 - Cross-attention
13:38 - Decoder output & prediction probabilities
14:46 - Complexity analysis
16:00 - Transformers as graph neural networks

Original Transformers paper:
Attention is All You Need - https://arxiv.org/abs/1706.03762

Other papers mentioned:
(GPT-3) Language Models are Few-Shot Learners - https://arxiv.org/abs/2005.14165
(DALL-E) Zero-Shot Text-to-Image Generation - https://arxiv.org/abs/2102.12092
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - https://arxiv.org/abs/1810.04805
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity - https://arxiv.org/abs/2101.03961
Finetuning Pretrained Transformers into RNNs - https://arxiv.org/abs/2103.13076
Efficient Transformers: A Survey - https://arxiv.org/abs/2009.06732
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth - https://arxiv.org/abs/2103.03404
Do Transformer Modifications Transfer Across Implementations and Applications? - https://arxiv.org/abs/2102.11972
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies - https://ml.jku.at/publications/older/ch7.pdf
Transformers are Graph Neural Networks (blog post) - https://thegradient.pub/transf....ormers-are-graph-neu

Video style inspired by 3Blue1Brown

Music: Trinkets by Vincent Rubinetti

Links:
YouTube: https://www.youtube.com/ariseffai
Twitter: https://twitter.com/ari_seff
Homepage: https://www.ariseff.com

If you'd like to help support the channel (completely optional), you can donate a cup of coffee via the following:
Venmo: https://venmo.com/ariseff
PayPal: https://www.paypal.me/ariseff