LLM From Scratch is a hands-on workshop where you write every piece of an AI from nothing

Summary

Free hands-on “LLM From Scratch” course that builds a tiny LLM from nothing to a working model.

It comes in six parts: tokenization, transformer, training loop, generation, scaling experiments, and a poetry competition.

Built for hobbyists – runs on a laptop; see the LLM From Scratch GitHub for details.

Have you ever wondered how an LLM works? Perhaps you know some of the basics, such as how they use tokens to get the job done, and how their text generation is somewhat like the autocomplete feature on your phone; however, how would you go about making one from scratch? Where would you even start?

If you want to learn how to make an LLM from scratch, then you’ll need to check out this DIY course, called, uh…”LLM From Scratch.” As one of the best examples of the phrase “does what it says on the tin,” LLM From Scratch starts you off with nothing and walks you through a miniature course to make your own AI.

Related

6 ways anyone can use LM Studio and a local LLM on their PC

Most people can find a use for a local LLM on their PC, and here’s how I use mine.

LLM From Scratch takes you from nothing to creating the ideal AI poet

It’s broken up into six parts

As spotted by Hackaday, LLM From Scratch is a free course that takes you from absolutely nothing to creating your own AI. You’ll start off by learning about tokenization and how LLMs parse data, and then finish by training a model on poetry until you’re happy with it. You’re not going to be taking on ChatGPT or Gemini with it, but it will run on a laptop just fine, so it’s great for hobbyists.

Here are all the courses you can take:

Part

What You’ll Write

Concepts

Part 1: Tokenization

Character-level tokenizer

Character encoding, vocabulary size, why BPE fails on small data

Part 2: The Transformer

Full GPT model architecture

Embeddings, self-attention, layer norm, MLP blocks

Part 3: The Training Loop

Complete training pipeline

Loss functions, AdamW, gradient clipping, LR scheduling

Part 4: Text Generation

Inference and sampling

Temperature, top-k, autoregressive decoding

Part 5: Putting It All Together

Train on real data, experiment

Loss curves, scaling experiments, next steps

Part 6: Competition

Train the best AI poet

Find datasets, scale up, submit your best poem

If you’d like to learn more, be sure to head over to the LLM From Scratch GitHub page to read more about the inspiration and what to expect when undertaking this workshop.

Related

After a year of self-hosting LLMs, I realized the real bottleneck isn’t the GPU

Hardware is just the entry fee for local intelligence.