GPT is the first of the papers which proved the effectiveness of unsupervised pre-training for language processing tasks. This video is about GPT-1 which became quite an impactful work in the series of GPT papers that we now have (GPT-2 and GPT-3).


Paper: https://www.cs.ubc.ca/~amuham0....1/LING530/papers/rad
code: https://github.com/openai/finetune-transformer-l
Official OpenAI blog: https://openai.com/blog/language-unsupervised/


Paper Abstract:
Natural language understanding comprises a wide range of diverse tasks suchas textual entailment, question answering, semantic similarity assessment, anddocument classification. Although large unlabeled text corpora are abundant,labeled data for learning these specific tasks is scarce, making it challenging fordiscriminatively trained models to perform adequately. We demonstrate that largegains on these tasks can be realized bygenerative pre-trainingof a language modelon a diverse corpus of unlabeled text, followed bydiscriminative fine-tuningon eachspecific task. In contrast to previous approaches, we make use of task-aware inputtransformations during fine-tuning to achieve effective transfer while requiringminimal changes to the model architecture. We demonstrate the effectiveness ofour approach on a wide range of benchmarks for natural language understanding.Our general task-agnostic model outperforms discriminatively trained models thatuse architectures specifically crafted for each task, significantly improving upon thestate of the art in 9 out of the 12 tasks studied. For instance, we achieve absoluteimprovements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% onquestion answering (RACE), and 1.5% on textual entailment (MultiNLI).


AI Bites
YouTube: https://www.youtube.com/c/AIBites
Twitter: https://twitter.com/ai_bites
Patron: https://www.patreon.com/ai_bites
github: https://github.com/ai-bites

Play, save & stream 4K movie videos free on PC with 5KPlayer:
https://www.5kplayer.com/video....-music-player/index.

-----------------------
I DO NOT OWN THIS CLIP
All rights owned by Universal Pictures
https://www.youtube.com/universalpictures

In this video we go through how to code a simple rnn, gru and lstm example. Focus is on the architecture itself rather than the data etc. and we use the simple MNIST dataset for this example.

People often ask what courses are great for getting into ML/DL and the two I started with is ML and DL specialization both by Andrew Ng. Below you'll find both affiliate and non-affiliate links if you want to check it out. The pricing for you is the same but a small commission goes back to the channel if you buy it through the affiliate link.
ML Course (affiliate): https://bit.ly/3qq20Sx
DL Specialization (affiliate): https://bit.ly/30npNrw
ML Course (no affiliate): https://bit.ly/3t8JqA9
DL Specialization (no affiliate): https://bit.ly/3t8JqA9

GitHub Repository:
https://github.com/aladdinpers....son/Machine-Learning

✅ Equipment I use and recommend:
https://www.amazon.com/shop/aladdinpersson

❤️ Become a Channel Member:
https://www.youtube.com/channe....l/UCkzW5JSFwvKRjXABI

✅ One-Time Donations:
Paypal: https://bit.ly/3buoRYH
Ethereum: 0xc84008f43d2E0bC01d925CC35915CdE92c2e99dc

▶️ You Can Connect with me on:
Twitter - https://twitter.com/aladdinpersson
LinkedIn - https://www.linkedin.com/in/al....addin-persson-a95384
GitHub - https://github.com/aladdinpersson

PyTorch Playlist:
https://www.youtube.com/playli....st?list=PLhhyoLH6Ijf