The Inner Workings of GPT
How are large Transformer models like GPT-3 able to generate new text?
Gain a detailed understanding of the internal architecture of these powerful models, as well as insights into why they work, in this video course on GPT, GPT-2, and GPT-3.
Course Contents
The course covers both the applications and implementation of the GPT models through a series of video lectures.
1. What are Transformers?
- An introduction to Transfer Learning
- What applications are the GPT models suited for?
- What are the differences between the three versions?
2. Input Formatting
- How does GPT actually take in text?
- An introduction to word embeddings.
- A look at GPT-2's vocabulary of tokens.
- Details of the GPT-2 Tokenizer.
3. Architecture Overview
- The key building blocks of the GPT models.
- How GPT models understand word order.
4. Self-Attention
- What Self-Attention is actually doing in a Transformer.
- Full implementation details of Self-Attention and Multi-headed Attention.
5. Feed-Forward Network
- The often-overlooked component that contains 2/3s of the model's weights!
- What is the FFN actually doing?
- Full details of its implementation.
6. Decoding Strategies
- An introduction to the techniques used to control how "interesting" vs. "predictable" the text generated by GPT-2 will be.
Ready to learn a whole new skill?
NLP Base Camp Members have complete access to this tutorial
and all of my NLP content!
Course Completion Certificate
Each major module of the course contains a short (optional) quiz to test your knowledge of the material. Once you've achieved a 100% score on all of the quizzes (you can retake them as many times as needed), you will receive a signed certificate of completion!
Frequently Asked Questions
Completing an introductory course in Machine Learning, that includes neural networks, should be enough!
The course assumes that you are already familiar with how a neural network can learn from training data to perform a complex task.
I will talk through some Python code, but no programming is required on your part.
No, but membership includes access to all of my content--including additional material on GPT and related models!
Yes! You can request a student membership here.
The course includes two Python Notebooks which allow you play with GPT-2.
There are no programming assignments in this course, and most of the concepts are explained through illustrations rather than code.
However, as a member, you will have access to many related, code-based tutorials!
