Last month I hosted an in-person workshop about building your own large language model without any math or ML prerequisites. It covers everything from machine learning fundamentals, deep neural networks, transformer architecture, and pre/post-training. I’m excited to release recordings and training materials for you to watch!

[Start Watching: Build Your Own LLM ]

[Read the slides] [Do the Exercises]

The workshop’s goal is to grok all parts of modern LLM development. So each workshop section builds intuition sequentially: starting from slides teaching the concepts, followed by excel-by-hand exercises developing intuition for the math, and then coding tutorials. One participant, Emily HK, noted: “The best part of this workshop is that all the content is still available online for me to refresh my memory at any point.”. Well, now you have access to these materials as well!

• 23 Workshop Videos @ https://go.JustinAngel.ai/playlist

• 250-page Slide Deck @ https://go.JustinAngel.ai/deck

• 50 Excel and Code Exercises @ https://go.JustinAngel.ai/drive

Why I Recorded the In-Person Workshop

We had over 200 registrations for the in-person workshop in San Francisco! Unfortunately we only had physical space for ~30 attendees. Driven by guilt for having to turn away ~200 people, I ended up spending a week recording, editing and uploading all the materials to YouTube. If you happen to know of a space in SF that can host more than 30 people for the next workshop, let me know?

Video Links

Our first section focuses on sampling LLM token predictions. We have to sample tokens to avoid waiting until the heat-death of the universe to compose a short ideal paragraph. Sampling techniques like temperature, TopP and TopK are then used as a shortcut.

At this point of the workshop, we don’t really know how to build an LLM. So as the enterprising engineers we are, we’ll reverse engineer a classic LLM (GPT-2 small), list out its parts, and create a roadmap to learn about each of those.

ChatGPT is just 100B+ perceptrons. Or said in reverse: a perceptron is the basic unit of work of a neural net. The math formula is straightforward: weight * input + bias. In this section we luxuriate in that simple formula, seeing it in code, in excel demos, and in physical analog demos.

Perceptrons alone can’t easily detect edges (like the ending of sentences), understand non-linear values (like -5 and 5 being the same magnitude), or build some logical structures. Activation Functions solve that gap by introducing non-linearity into neural nets. Our key takeaway from this section is the ReLU^2 activation function which will be the first component of our LLM.

CPUs have 8 physical cores that can run in parallel. So if all we’re computing is 8 neurons, those are great. But when we have 32M perceptrons to compute in parallel, they just can’t match up with the 15,000 parallel compute core GPUs have. In this section we learn about GPU coding starting from CPU, PyTorch modules, torch.compile() fused kernels, CUDA, and Triton.

A single perceptron can only learn one really single rule. To learn language, we need complexity. Which means we’ll need multiple perceptrons grouped together in Layers. And then finally we’ll sequence those layers into a multi-layered perceptron (MLP). This section gives us another component for our final LLM: a growing-shrinking MLP.

If our goal is to teach LLMs the english language, we need to know how far away from that goal we are with our randomly-initialized LLMs spewing gibberish. Loss functions allow us to quantify the distance from that goal. During this section we learn about the theory of traversing loss landscape to minimize loss (i.e. improve english skills in LLMs). We also learn about how Cross-Entropy loss function works in problems without a definitive answer, like next word prediction.

Before we can train a model on the english language, we’ll learn how to train it on predicting A+B%5. At first, this toy example seems trivial. The same manifolds created inside deep neural networks to divide by five, are how LLMs learn any sequential concept like days, months, colours, and word counting.

Now that we’ve successfully trained our first DNN, it’s time to save it to disk, upload it to Hugging Face and load it remotely.

GPT2-small has 72 consecutive matrix multiplications, but activation functions tend to add zeros and small values. What happens when we multiply a bunch of zeros? we get more zeros. Training collapses. GPU time is wasted. People get fired. Recessions start. To avoid that, we learn that choosing a specific statistical distribution for our randomly-initialized values can prevent gradient collapse. Yay!

But in order to avoid training collapse over 72 consecutive matrix multiplications, we really need another trick up our sleeve: Residuals. Combining outputs from the previous layers with the outputs of this layer, allowing each layer to work independently for a bit. It’s like giving the DNN multiple scratchpads. The formula is straightforward: f(x)+x and we’ll see it play out in excel and in code.

Residuals introduced a new problem since f(x)+x can lead to gradient explosion. Intuitively, we can see that f(x)+x>x increases X at every turn, which is how we get a gradient explosion. So we bring in normalization to make sure the values stay clustered around a central point.

So far our DNN only worked on deterministic data with clear answers (e.g., A*B%C has clear answers), but language doesn’t have definitive answers. We can complete the sentence “The capital of France is” with “Paris”, “full of cheese”, or “has a metro strike” all factually correct. Once we introduce data without a definitive solution, the model will overfit on the training data. i.e., it’ll perform better on train data than it can generalize on unseen test data. That’s a problem. Regularization techniques like dropout, gradient clipping, and weight decay can be used to make sure our DNN doesn’t overfit on training data.

Ultimately, LLMs have to return percentage probability for the next words like saying “Paris” is 100% likely. But DNNs / LLMs operate by doing fun matrix multiplication with real numbers, like 1.234567. SoftMax converts those activation values into percentages.

LLMs operate on fun matrix multiplication with numbers, but language operates on strings. Tokenizers are our way of putting a square peg into a round hole by shaving off the edges. We’ll convert words, subwords components, punctuation marks and emojis into integer tokens. But then we have to decide how to best tokenize? Should each character be a token? should each word be a token? Or is there a compromise there? Ultimately, we settle on the BPE compromise tokenizer.

Once we’ve converted “Hello” to 995, what does the LLM know about 995? Token Embeddings represent learnable meanings in high-dimensional space for each token. We see the example of how in high-dimensional embedding space, woman and queen exist in the same distance & direction as man and king. These embeddings learn what each word/subword/token means. But those embeddings still lack positional information which makes “dog bites man” and “man bites dog” equivalent. So we’ll learn about positional encoding and how to add positional information to each token embedding.

Attention is how LLMs “read” sentences. First they read sentence fragments, understand word positions, then they identify subjects & verbs, read question words and understand transformations. It’s prohibitive to teach LLMs each one of those concepts. But Attention just learns that through the power of math and backpropagation.

Bringing it all together we build a Transformer architecture with MLPs, Attention Heads, Embeddings, Activation Functions, Regularization, Normalization, Residuals and everything else we learned about so far. This section is my favourite because it really shows how transformers work end-to-end.

Moving on from architecture to actual training, we need to first teach our LLMs the english language. We do that by taking textual information from the web, from books, from social media, code and other data sources. But extracting text from the internet isn’t straightforward. Do we prefer more lower quality text? Or less higher quality text? In this section we build our pretraining script and let it run for hours to train our model in the english language.

How do we know if our models are any good? Should we use leaderboards? Should we have another LLM act as a judge? Good at what? In this section we review the full options of how LLMs are evaluated, and on what LLMs can be evaluated on.

After our LLM learns to speak english, it’s still pretty bad at following instructions. We can tell it to answer “What is the capital of France?” and it’ll just answer “The capital of France is France.” because it doesn’t yet understand it’s supposed to be a helpful AI assistant. Using Instruction fine-tuning we teach our LLM to write poems, generate lists, explain concepts and more.

Finally, we want our LLMs to be good at things. We want them to know all the knowledge, to correctly solve all the math, and to write the perfect code. RL is our tool to evaluate entire responses, and backpropagate both negative and positive signals. In this section we implement the simples RL offline preference optimization algorithm from scratch: SimPO. First in excel, and then in code.

We couldn’t cover the entirety of building frontier LLMs, but we can give you trailheads to what we didn’t cover: scaling training time to multiple GPUs, optimizing GPU usage at inference time, and making LLMs safe to use.

[Start Watching: Build Your Own LLM ]

[Read the slides] [Do the Exercises]

Subscribe now

Twenty years ago I dropped out of college to start my own business. In 2025, I enrolled and completed both an undergraduate and master’s degree in computer science as an intellectual pursuit. It took less than five months and cost approximately $10k by speedrunning accumulating credits. I’m pretty established in my career (e.g., L8/principal engineer in FANG), so this wasn’t about career advancement. It was about enjoying shifting my attention towards AI/ML.

And there was lots of enjoyment in this process. The structure provided by the programs with their clear-cut expectations led to satisfaction and up-leveled my skills. I remember happily wandering the green fields of the Capodimonte in Napoli eating a pizza while listening to audio lectures on American history thinking “it doesn’t get better than this”. Completing courses in a single day ticked a lot of my emotional-validation boxes. And these new technical skills I gained in machine learning turned out to be tremendously useful.

This post is a how-to guide to those readers interested in replicating my process. Once reading about specific undergraduate credit sources becomes overwhelming for you, just scroll down to the master’s degree section.

Undergraduate: TESU and online courses

The TESU (Thomas Edison State University in New Jersey) degree programs are ideal for adult-learners who both have extensive prior knowledge and acquired learning-as-a-skill. I chose an undergraduate major in computer science, since that’s my area of expertise. My undergraduate degree is primarily cobbled together from online coursework and exams that were then post hoc recognized and included in the degree. This post will explore the mind-numbing technical minutiae of various credit sources.

Why have multiple credits sources?

My #1 priority was speedrunning this undergrad, as time is my one non-renewable resource. TESU is great for that. TESU degrees can be accelerated by having external companies evaluate existing knowledge for credit equivalency. However, each credit provider has its own time horizon and cost. As such, TESU students mix and match credit sources based on their preferred learning style, cost, and/or speed. Ultimately, I optimized for speed.

TESU accepts credits from multiple sources, such as TESU coursework, credit-by-exam, online courses, and prior learning from other institutions. Courses at TESU take 3 months and cost ~$500 per-credit, whereas a Sophia online course can be completed in a few days for ~$10 per-credit. Big difference. Upon completion, providers send official transcripts to TESU for evaluation and recognition. As a concrete example, TESU requires undergrads to demonstrate college-level writing proficiency. I took an online CLEP “college composition” exam for ~$90 and had 6 credits recognized within a week (~$15 per-credit). Effectively, TESU collaborates with external credit providers to validate their degree requirements are met.

TESU online courses take about 3 months to complete, or you can test out of those immediately if you already have that knowledge. In the interest of expediency, I favored demonstrating knowledge over undertaking performative education. I found it much faster to knock out several course requirements each week based on knowledge I previously possessed. I owe this degree to the flexibility of not having to kabuki-theater my way through the higher-education system.

The TESU bachelor of arts in computer science degree requires 120 credits with a very specific breakdown. You can’t just collect 120 CompSci-related credits and call it a degree. Rather, credits must be equivalent to specific courses or applied to specific niche categories. For example, the degree requires Calculus 1 or an equivalent course, which I completed online with Sophia over three days. For category requirements, 15 credits must be “upper-level”/advanced credits which I predominantly completed with StudyDotCom and Saylor. Only 30 credits are free electives that don’t need to correlate to specific courses or categories. I assigned most of these electives towards math and CompSci, but I could’ve chosen music theory or culinary history.

Credit source #1: Credit-by-exam

Credit-by-exam providers such as CLEP, DSST and TECEP allow you to take remote-proctored exams and earn credits the same day. These are primarily designed for armed forces service members and high school students, but they provided me a straightforward way to complete 21 of the 120 required credits. Exams ranged from nutrition and world religions, to ethics and writing. To me as a well-read adult, these exams required more courage than preparation. For each exam, I bought $20 worth of preparation materials, had ChatGPT/Claude summarize those, and read through the bullet points prior to the tests. These summaries allowed me to brush up on college-level material I was already familiar with.

The possibility of failing an exam and the over-formality of the proctored tests led me to deprioritize this format. It just isn’t fun to sit on camera answering questions as someone periodically demands you empty your pockets. Pursuing credits through online courses was more enjoyable, less emotionally taxing, and much more affordable.

Credit source #2: Online courses

Online courses accredited by ACE & NCCRS accounted for 50 of my 120 credits. I took 29 credits with Sophia.org, 15 with Study.com, and 6 with Saylor.com. Courses on Sophia have non-proctored final exams with mixed assignment formats including essay writing, coding projects, multiple-choice questions, and video recordings. Sophia courses included math (e.g., Calculus 1), programming (e.g., Java), humanities (e.g., public speaking), and science (e.g., nutrition). StudyDotCom online courses focused on upper-level coursework in computer science (databases, operating systems, architecture, etc) through multiple-choice quizzes and essays. Additionally, I took six credits with Saylor online courses and they only required proctored multiple-choice questions.

Online courses were by far the most enjoyable part of the degree. Self-paced learning across myriad fields delivered contentment & dopamine, bridged knowledge gaps, and felt emotionally supportive. Since both Sophia and Study allow retries, I experienced less anxiety and perfectionism and just did stuff. Some tricky assignments actually took multiple attempts to get to a perfect score. For example, developing a functioning circuit design for an ALU (Arithmetic Logic Unit, part of a CPU) took several attempts to complete StudyDotCom’s course requirements to prove it’s doing binary math correctly. What could’ve been a stressful experience, ended up instead as both enjoyable and educational.

Credit source #3: Language Proficiency exam

NYU’s Language Proficiency exam translated my multilingual fluency into 16 credits towards a TESU degree. Pun intended. The exam required everyday C2-level language fluency in Spanish across reading, speaking, listening, and writing. That exam itself was genuinely challenging and tested what a native speaker living in a Spanish-speaking country would do day-to-day. Results took several days with most credits ending up in the elective categories for me.

In a sense, this is a good example of how credit-by-exam serves an adult-learning population. Instead of the theatrics of sitting through Spanish classes, shouldn’t learners be allowed to demonstrate sufficient competency?

Credit source #4: Transfer Credits

Transferred credits from a foreign university accounted for 44.5 credits. As mentioned earlier, I dropped out of college in 2006 with 50 credits. SpanTran evaluated those credits with a 6:5 ratio (unclear why that ratio was applied). Those credits partially satisfied specific course requirements in computer science such as discrete math, linear algebra and data structures. I chose SpanTran to perform this evaluation because they directly contacted the foreign university and got the transcripts without my help.

An important caveat: TESU requires that at least 30 credits be taken at TESU or transferred from another school. My transfer credits from twenty years ago covered that requirement. Though for those considering a similar path without prior credits, a common option is to start a WGU CompSci undergrad and then transfer those credits to TESU. Alex Sheppe covers the intricacies and limitations of this path in this video.

Credit source #5: TESU coursework

TESU coursework accounted for six credits. TESU requires that two courses be completed with them: a capstone thesis-writing course and a cornerstone media literacy course. Both courses involve weekly coursework including readings, graded essays, discussion forum participation, multiple-choice quizzes, and more. Since those courses take 3 months to complete start-to-finish, they’re the slowest part of getting the undergraduate. Under certain conditions it’s possible to parallelize these courses, which I did.

I took my thesis writing course quite seriously, read 200 academic papers and wrote a 112-page undergraduate thesis on AI-driven psychotherapy. Since publishing it I’ve received positive feedback and it’s painstakingly slowly going through academic publishing.

Was this undergrad worth it?

Yeah, but for unexpected reasons beyond being a prerequisite for a graduate-level education or career advancement. I found real joy in having structure that encouraged me to focus on topics I normally wouldn’t, like history, psychology, and civics. I enjoyed the slightly subversive element of cranking out an undergraduate degree in three months instead of four years. And I got satisfaction seeing myself earn 3 credits (or 1/40th of an undergraduate) in a single day or two. Spending a few hours in the evenings doing coursework in quaint Italian coffee shops provided grounding in an otherwise turbulent year.

Master’s degree: WGU and Machine Learning

AI chatbots will fundamentally shift the nature of knowledge work, including software engineering. Any software engineer who doesn’t have an intrinsic grasp of AI risk becoming unemployable. Red flags went up when an LLM architecture intro read like hieroglyphic gibberish to me. Was I headed to career obsolescence without in-depth AI/ML knowledge? That anxiety grew within me when I couldn’t scroll twitter without failing to understand the importance of the latest round of AI/LLM research. If you can’t beat the AI hype cycle, you join it.

For my graduate degree, I specialized in AI/ML (artificial intelligence and machine learning) pursuing an MS in computer science at WGU (Western Governors University in Utah). Progression at WGU is based on demonstrating competency through successfully completing industry-level projects. The deliverables are aligned with my expectations of a senior software engineer in terms of both coding and technical documentation. Since I lacked that knowledge in ML, I first spent a month learning the fundamentals. After that, I spent a month completing projects and essays for the degree.

Step #1: Learn ML, AI and data science engineering.

Based on advice from Marina, I tried DataCamp, which ended up being perfect for my hands-on learning style. DataCamp courses teach new concepts through writing code. I started by learning the basics of python and data engineering (numpy, matplot/SNS, and Pandas). Then I moved into foundational ML concepts around supervised, unsupervised and deep-learning. And finally, I tackled specialized topics like NLP, RL, MLOps, statistics, and FinTech.

I nailed down a process that allowed me to finish a few courses every day by having ChatGPT convert DataCamp courses to educational and interactive projects. A DataCamp course includes approximately 20 videos on interlinked topics followed by coding exercises. Watching videos is a bit too slow for my pace as I learn best from technical documentation. Each DataCamp course also offers downloadable PDFs capturing the concepts, slides and code relevant to that course. To accelerate that process, I feed the course PDFs to ChatGPT, have it generate a Google Colab notebook that teaches me concepts, watch videos whenever I’m fuzzy on concepts, and start coding. These projects really shine for me because they’re real code I can read, debug, and modify, which fits my learning style. For those interested, here are the generated Python notebooks covering DataCamp topics.

DataCamp is an effective teaching modality for applied AI/ML. WGU tuition actually includes complete unlimited access to DataCamp, Udemy and LinkedIn courses. Even better, WGU’s curriculum materials reference specific DataCamp courses, which made it straightforward to figure out the correct sequencing.

Step #2: Complete non AI/ML coursework

Roughly half of WGU’s MS-AI program covers software engineering: algorithms, architecture, scripting, coding, programming languages, and methodologies. My guess is that a 22-year-old graduate student would find these courses challenging and time-consuming. Given my extensive background as a software engineer with twenty years’ experience, that wasn’t especially challenging and I completed this portion in under a week. These were all non-trivial project assignments such as coding pathfinding algorithms, automating unix remote servers, and authoring corporate-level security documentation. These are all things I’ve done professionally during my career.

Step #3: Complete AI/ML coursework

The more challenging half of the degree focused on AI, ML, NLP, RL, and DNN. I spent twelve hours daily coding up projects and writing graduate-level papers analyzing them. It’s a real joy to have someone care about my toy projects as I mastered AI/ML. I was genuinely happy getting positive feedback from WGU graders on my critical analysis of a saliency map for my XGBoost-based facial sentiment analysis final project. This portion of the degree culminated with an AWS certification testing practical ML engineering and operations.

Was this master’s worth it?

Yes, and No. It’s nuanced.

Yes, because having WGU set up the goalposts created urgency that up-leveled my technical skillset. Ultimately, WGU’s graduate degree is focused on improving AI/ML engineering skills and it achieved that for me. It brought into sharp focus which parts of ML’s vast historical knowledge are must-haves. The degree curriculum separated the wheat from the chaff of ramping up on AI/ML fundamentals.

No, because I’m ambivalent about whether this path makes sense for other software engineers. I could’ve just learned all those concepts, written all that code, and analyzed all those results without any structure or guidance. And if the tuition of a single semester at WGU is too much for you, you can probably do that. However, I’d find it depressing to grind in isolation on a self-derived AI syllabus. Trusting an expert-led curriculum felt reassuring.

How much did it all cost?

• Graduate degree: ~$4,225

  • WGU single semester tuition: $4,125

  • DataCamp subscription: $100 (included with WGU tuition)

  • Google Colab Pro: free 1/yr with EDU emails

• Undergraduate degree: ~$5,880.

  • TESU tuition for two courses (6 credits) or a pack for 15 credits: $3,600

  • Graduation fee: $300

  • Sophia online courses (3 months, 29 credits): $99 x 3 = ~$300

  • Study online courses (2 months, 15 credits): $250 x 2 = $500

  • Saylor online courses (6 credits): $5 x 2 = $10

  • NYU credit-by-exam (language proficiency, 16 credits): $700

  • CLEP credit-by-exam (2 exams, 9 credits): $200 (could be free through Modern States)

  • DSST credit-by-exam (3 exams, 9 credits): $300

  • TECEP credit-by-exam (1 exam, 3 credits): $150

  • Note: this doesn’t take into account TESU’s accelerate fee or potential optimizations around it.

What’s next?

If you’re a software engineer considering your game plan in a world where code has no intrinsic value: go deep-dive into AI/ML. If you’re wondering if you need a degree: you probably don’t. For me, I found the structure and gamification of an expert-led curriculum grounding and supportive.

I had the immense privilege of pursuing an education without career pressure. I'm considering another master's from CU Boulder focused on AI/ML research (about 50% complete), or possibly coursework from Stanford or JHU on LLMs and mechanistic interpretability. Maybe I’ll undertake research and pursue publication. In the context of my life, following my passions has historically led to positive outcomes.

Next month I’ll be hosting a free full day build-your-own-LLM workshop in San Francisco. Drop a comment if you’re interested in attending in-person or want to see me post a recording here.

— Justin

Leave a comment