Even your state-of-the-art LLMs are trained with a beefed-up version of this ancient algorithm, generally described through a mountain-climbing analogy. Look around your current position in the loss landscape, find the direction of the steepest descent, then take a step toward it. Yet no one ever tells you what’s really happening behind the scenes.

Here’s the secret math behind gradient descent:

If you are still here, I have a small favor to ask of you. My YouTube channel is still small, and if you enjoy my content, you can support me by

1. watching the video,

2. subscribing to my channel,

3. and leaving a comment on YouTube with your questions and thoughts.

Over the last few months, I’ve been focused on making my deep dives into the mathematics of machine learning better and clearer. You already love my drawings, and moving from still images to animated visualizations is a light-year leap in quality. This is the future of The Palindrome.

It only takes a couple of clicks from you, but it means the world to me. With your support, I can continue being a full-time content creator, delivering bangers like the video above for you.

In any case, thanks for being here!

Cheers,
Tivadar

It’s Tivadar from The Palindrome. The next post is already in the final stages (it’s going to be one of my best ones), but before that, I’d love to ask you a quick favor.

I’m running a short reader survey to better understand who’s reading, what you’re working on, and what you’d like to see more of.

I’ll use the results to improve the newsletter and make sure any future sponsors are actually relevant to you, and future issues will stay relevant to you. Individual responses won’t be shared, and your contact information will only be used if you explicitly opt in.

It only takes a few minutes:

Let's see the survey!

Thanks!

Cheers,
Tivadar

P.S. Here’s a quick preview from my next video. Can you guess what I’m working on?

Your support makes it possible for me to write these high value, high signal posts every week.

Subscribe now

Hey!

It’s Tivadar from The Palindrome. Before we dive into today’s post, I’d like to ask you a favor.

I’m running a short reader survey to better understand who you are, what you’re working on, and what you’d like to see more of.

I’ll use the results to improve the newsletter and help match it with better sponsors. Individual responses won’t be shared, and your contact information will only be used if you explicitly opt in.

If you complete the survey, you’re an absolute legend. More importantly, you’ll be helping me make The Palindrome even better.

Let's see the survey!

Thanks!

Let’s get back on track. Today’s post is a video-essay version of one of my earlier posts. This is a new format I’m experimenting with, turning my non-technical writing into podcast-like experiences that you can listen to while driving or perhaps washing the dishes.

I’ve been focusing on videos lately because I find the medium better suited to my technical content. If you enjoy my posts, don’t forget to subscribe to my YouTube channel!

Cheers,
Tivadar

Video version

Written version

The single most useful programming exercise I ever did was also the one I hated most: writing C code with a pen, on paper.

When I was a young mathematics student, one of my first university courses was called “Introduction to Programming,” where the computer scientists in our faculty threw us, innocent and naive mathematicians, straight into the deep end.

In the Eastern school system, we believe in immediately applying pressure so intense that it weeds out the unworthy faster than they can chug their welcome drinks at the regular semester launch parties. So, fifteen minutes into the first lecture, we were already dereferencing pointers left and right, because everything else was trivial. At least, according to our professor.

And by the way, this was the same in our math courses. The first sentence of our linear algebra class was “let F be a field, then V is a vector space over F if...”. And just like that, our heads went spinning.

I remember having classmates so technologically inept that they could hardly manage to navigate in Windows XP (the most popular OS at the time), let alone compile C code. So, the “Introduction” to Programming class felt like a punishment for most of us.

And the rock bottom of our whole miserable experience were the coding-on-paper exercises, where we had to write functioning C programs with a pen, on paper.

Coding on paper is antithetical to how we code in practice. Whenever I’m working on a project, I never move in a straight line. I refactor. I bugfix. I go back and forth.

These days, large language models write most of the code I ship. (Even the animation you are watching right now was written by Codex, my coding agent of choice.) But when I actually write the code myself, I use Jupyter Notebooks, where I take a shot at implementing the function, test it on a couple of examples, and fix all the dumb mistakes I’ve made, perhaps even prompting an LLM or two meanwhile. (Or Googling when I feel old school.)

None of that is available on paper. No autocomplete, no syntax checks, no code formatter.

With pen and paper, you can’t run the script, you don’t have autocomplete, and once you’ve written the “code,” you can’t really change it.

It’s linear, uninteractive, and tedious. Coding on paper felt like a punishment.

Is there a point?

I used to believe there wasn’t; now I know there is.

The revelation finally came by drawing analogies, my Number One Tool for noticing patterns. Sometimes, patterns are hard to see in the thing we’re looking at directly. But when we look at something similar, suddenly patterns become obvious.

(Sometimes, we are emotionally charged, like I am about coding on paper, which I used to hate. However, taking another perspective helped me see clearly.)

So, let’s talk about a different problem. What do you think about having an LLM write your essay for you?

Even when generative AI is allowed, you cheat yourself by outsourcing your cognition to a language model. The purpose of writing lies not only in the final outcome, ready to transfer your thoughts to other minds, but in forming those thoughts as well!

What is writing? As said by Donald M. Murray in his famous essay “Teach Writing as a Process Not Product”, writing is

“...the process of discovery through language. It is the process of exploration of what we know and what we feel about what we know through language. It is the process of using language to learn about our world, to evaluate what we learn about our world, to communicate what we learn about our world.” - Donald M. Murray

Coding - that is, engineering and building systems - is also a process, not just a product.

Here’s the catch: you can only get better at delivering products if you master the process. There are no shortcuts.

“But Tivadar, how to master the process?”, you might ask.

By practice. Lots of practice. And by far the best way to practice is to restrict your tools. To make it as hard as possible.

Have you ever seen an athlete exercise with a mask that limits airflow? That’s just to make things hard, to help them perform even better when the restriction is lifted.

When I was practicing Kung Fu, our trainer instructed us to practice complex moves with our non-dominant hands. This technique is magic: after stumbling around with a wooden stick in my left hand for a couple of minutes, I could effortlessly perform the flashy movements with my right hand, which is my dominant one.

Another example: I love playing video games; a significant portion of my life skills are from some game, one way or another. My favorite one by far is Elden Ring, which is notoriously hard. Even the most basic enemies can easily whoop you if you are not on guard.

By purposefully denying yourself of the countless tools available in the game, say, you refuse to level up, you are forced to get good at reading attack patterns and dodging at the perfect moments. When any boss can one-shot you, there’s no room for error.

Thinking about all of these life lessons I learned, I suddenly realized that coding on paper fits the above. It took me a decade and a half, but I’m finally there.

Coding on paper forces you to pay attention to details (instead of letting autocomplete do it for you), to be deliberate (instead of brute forcing your way to a solution through trial and error), and to figure things out on your own (instead of outsourcing your cognitive abilities to an LLM).

Whenever you are alone with a pen and a piece of paper, there’s room for your thoughts.

Now, I know that by restricting our tools, we can build the muscles whose work we offload to tools.

Muscles that will carry all the cognitive load for you later.

Thanks for reading!

If you liked this post, you should support me with a paid subscription. It’s $100 a year, or $10 a month, and in return you get well-researched machine learning/mathematics deep dives such as Machine Learning is Not Just Statistics, Matrices and Graphs, or Vectorization in Theory + Vectorization in Practice.

Your support makes it possible for me to write these high value, high signal posts every week.

Subscribe now

Your support makes it possible for me to write these high value, high signal posts every week.

Subscribe now

Hey!

It’s Tivadar from The Palindrome. Back when I wrote the Mathematics of Machine Learning book, I realized

1. what a great language Python is,

2. and that object-oriented Python should be the first thing taught to every machine learning engineer.

So, I’ve been thinking about publishing a series, but my Python skills are not exactly top-of-the-line; I just hack and slash until things work. Fortunately, I found the best person who could do that!

It’s my pleasure to introduce Stephen Gruppetta, author of The Python Coding Stack, and my longtime online friend from back when X was called Twitter.

If you ever wanted to become a power user of Python and take advantage of all the heavy machinery provided by classes, operator overloading, inheritance, composition, etc., this is the article for you.

(It’s the second post in the object-oriented Python miniseries. Check the first one here for the foundations of OOP!)

Dig in!

Cheers,
Tivadar

A class is a template you can use to create several objects that share the same characteristics. All the objects created from a class will have the same structure and can perform the same actions.

However, sometimes you need to create objects that don’t have the same attributes but that are still similar to each other. You may need objects that share some characteristics but are sufficiently different from each other, so you can’t create them using the same class.

You want to avoid writing a brand new class from scratch. Instead, you’d like to reuse some of the code from the original class.

There are several options for handling overlap between classes. Inheritance is a common tool to link classes. However, inheritance is not always the right solution. Composition is an alternative technique to use a class as part of another class. In this article, you’ll learn about inheritance and composition. You’ll learn how to use each technique, and just as importantly, when to use them.

The Velocity Class

You learned about classes in the first article in this series. You wrote a Vector class that you can use to create vector instances.

You’ll make some changes to this class as you progress through this article. However, you’ll primarily work on a new class: the Velocity class.

You’ll consider two different routes to create the Velocity class. Velocity is a vector: it has a magnitude and a direction. Therefore, you can start from the Vector class you defined in the first article and use it as a starting point to build the Velocity class. This is the inheritance route.

However, you can also create a class that has a Vector instance as one of its attributes. You’ll also explore this composition route in this article.

Read more

The primary goal of this post series is to teach you the Representational Similarity Analysis (RSA), which is a machine-learning analysis that is used to compare distributed representations in different systems.

If you haven’t already read Part 1 in this series, please do so! It provides necessary background about how the RSA score is calculated and interpreted.

As a brief reminder, an RSA (representational similarity analysis) works by comparing cosine similarity matrices across different embeddings spaces (layers, blocks, models, etc.). The idea is that different embeddings spaces may have distinct coordinate systems and even different dimensionalities, but if their internal representational structures are similar, the relative similarities should be strongly correlated even if the vectors are distinct.

The additional goals of this second post are (1) to learn more about RSA and category specificity, and (2) to learn how to dissect the “hid…

Read more

It’s Tivadar from The Palindrome. Back when I wrote the Mathematics of Machine Learning book, I realized

1. what a great language Python is,

2. and that object-oriented Python should be the first thing taught to every machine learning engineer.

So, I’ve been thinking about publishing a series, but my Python skills are not exactly top-of-the-line; I just hack and slash until things work. Fortunately, I found the best person who could do that, and even better, he agreed to write a special article for you!

It’s my pleasure to introduce , author of , and my longtime online friend from back when X was called Twitter.

If you ever wanted to become a power user of Python and take advantage of all the heavy machinery provided by classes, operator overloading, inheritance, composition, etc., this is the article for you.

Dig in!

Cheers,
Tivadar

“A computer program stores data and does stuff with the data.”

This is not the most technical definition of a computer program you’ll see. But it’s a valid one. When you learn to code, you learn about data structures to store different types of data. And you also learn about tools needed to manipulate and transform the data. You often define functions containing code to “do stuff” with the data.

Your code will contain data structures and functions, and you pass those data structures to the functions when needed.

Object-oriented programming (OOP) brings these two aspects together into a single unit. This unit is the object, which contains the data and the tools needed to manipulate the data. This may not sound like much, but it enables you to think about the problem you’re trying to solve differently. You can visualize your problem in a way closer to how humans see the world.

Let’s look at some examples, starting with a concrete one. Consider a country. There’s plenty of data relevant to a country: its name, population size, geographical area, capital city, and more. All countries have these attributes. Therefore, you can create a template that includes these attributes that you can use each time you want to represent a different country. Countries also include people, so these can be included in the data for each country, too.

But countries also perform actions. They issue passports, they collect taxes, they create laws, and so on. OOP urges you to think of a single unit to represent a country, which includes all the data and tools needed for the country to perform the actions required. You’d create a class called Country–this is the template you need to create lots of countries. The class doesn’t represent a specific country but the idea of a country. Once you define the class, you can create as many instances of the class as you need. Each instance represents a specific country.

Classes and Instances • The Vector Class

But let’s work on a different example in this article. Let’s consider a vector. One way to view vectors is as entities with both magnitude and direction. In three-dimensional (3D) Euclidean geometry, a vector is represented by three numbers.

Let’s put on our OOP hat. We need a unit in our program to represent a vector. It needs to represent its data and its functionality. Let’s start by creating a class called Vector:

Admittedly, this class doesn’t do much for now. The ellipsis (...) is just a placeholder that’s valid Python syntax. You’ll add more to this class soon. In the previous section, I mentioned that a class is a template for creating many objects modelled from the same blueprint. Let’s create a few instances of this class:

There’s only one Vector class. But now you have two instances of this class. You create an instance of the class when you add parentheses after the class name. The objects referenced by v1 and v2 are separate objects that occupy different areas of your computer’s memory. You can confirm that they’re different objects by showing their identity using Python’s id() function. The two instances of the Vector class have different identities:

You’ll get different values from the ones shown here when you run this code on your computer. But what matters is that the two numbers you get are different from each other. You can also confirm that v1 and v2 represent different objects by using v1 is v2, which returns False.

Note that the terms object and instance are both commonly used to refer to the unit created by a class. They refer to the same thing.

However, these are “blank” objects. They don’t have anything beyond the bare minimum a Python object needs. Let’s add some data.

Read more

returns to The Palindrome! You know I’m a big fan of his work, and if you are into machine learning, you should be too. His posts always strike the perfect balance between educational, practical, and entertaining.

He recently published the book 50 ML Projects to Understand LLMs, and his upcoming two-part series on exploring word representations is taken directly from the book. If you want to understand how Large Language Models work under the hood, don’t miss the post below.

Enjoy!

Cheers,
Tivadar

Read more

“How to get started in machine learning?” is one of the most common questions I get. I have a couple of default answers, but they are based more on my personal experience than on science.

Inspired by this, I’ve mapped out the knowledge graph of machine learning, building a hierarchy of concepts that can guide you from the foundations to the state of the art.

A couple of fascinating patterns have emerged from my journey: the thin spine of mathematics that holds up the entire knowledge graph, the central concepts like gradient descent that enable modern machine learning as we know it, and more.

Here’s the video where I talk about my findings:

Read more

I’m finishing up my video about the knowledge graph of machine learning, which will be released next week; I’ll do a live premiere right here on Substack Live, with a discussion after the viewing. (You can join here.)

This video will mark a milestone for me: instead of relying on hand-crafted slides and a presentation-style exposition, I built a scripting engine on top of the knowledge graph explorer that turns a JSON script like

[{
  "at": 0.0,
  "action": "autoRotate",
  "axis": "y",
  "speed": 0.01,
  "duration": 9.3,
  "windDown": 2.0,
  "easing": "linear"
},
{
  "at": 9.3,
  "action": "selectNode",
  "nodeId": "gpt",
  "showPrerequisites": true,
  "showDependents": false,
  "duration": 2.0
}]

into a beautifully rendered video.

Here’s a sneak peek:

The machine learning knowledge graph project could serve as a template for my future content. From now on, I’ll go full multimodal, meaning that I’ll

• build interactive visualizations (such as the knowledge graph explorer),

• then write posts and record videos, aided by the interactive tool.

Now that the current video-in-progress is about to be finished, what’s next?

Read on.

Read more

This week, it’s my pleasure to introduce , a fellow scholar working at the forefront of physics, mathematics, and computer science.

One of the main reasons behind the success of modern science is mathematical modeling, the process of translating complex real-life observations into a language that allows us to generalize, understand, and predict.

If you have enjoyed this post, subscribe to his newsletter , a space dedicated to translating the complexity of the empirical world, from your cells to entire societies, into language that is as simple as possible, though not necessarily simpler.

Read more

Read more

Let’s try something new. In recent months, I started to feel that the weekly publishing schedule takes its toll on the quality of my posts. I want to take more time per post to give you some high-quality technical content. I’m inspired by amazing writers such as Sebastian Raschka () or Cameron R. Wolfe (), who publish once a month, but each post is a work of art.

On the other hand, I miss you. A monthly schedule feels too long, and I have a lot to share with you. Ever since I got my ChatGPT Max subscription with access to Codex, my creativity is out of bounds.

So, here’s a new format. Each week, I’m sending you my unfiltered stream of consciousness, all the projects that I’m currently working on. Think of it as joining me for a coffee, where we talk about all the exciting/revolutionary/insane ideas we have in our minds.

This week, there are two things on my mind: knowledge graphs and Minecraft.

Let’s start with knowledge graphs.

The Complete Map of Machine Learning

If you are a regular reader, you know that one of the most common questions I get is “which part of mathematics do I need to study machine learning?” My default answer, based on my decade of experience, is: a ton of linear algebra, a decent amount of calculus, and a snippet of probability theory.

To be honest, I’m not completely satisfied with my reply. So, I dug deep with my newly found agentic AI-fueled superpower to find what is scientifically backed.

Without further ado: here’s the full knowledge graph of mathematics and machine learning. 2081 nodes, 5149 edges.

(The images are screenshots from the interactive graph explorer I’m building, which will be open source and publicly available.)

Let’s unravel this.

Read more

Yesterday we concluded our first monthly workshop! As promised, here is the full recording, exclusive to paid subscribers.

You can access the Jupyter Notebook lecture notes here: https://github.com/the-palindrome/mathematics-of-machine-learning-workshop

The next monthly workshop is already in the works; it’s going to be the next iteration of the Neural Networks From Scratch course. The tentative date is April 18th, 15:00–19:00 CET, but stay tuned for the announcement, as this date might change.

Thanks again so much for attending!

Cheers,
Tivadar

Read more

I’m finally working on my upcoming book Machine Learning From Zero, the sequel to Mathematics of Machine Learning.

The first chapter has just been finished, which sets the foundations for the neat stuff, like implementing neural networks from scratch. (Which is the core of the book.)

Without further ado, here’s an exclusive preview.

Enjoy!

P.S. I’m writing this book in Jupyter Notebooks, and I turn them into Substack posts with NotebookPress, a tool I’m building to bring technical writing on Substack to the next level. If you write math-and-code-heavy content, you should check it out.

Machine learning is training predictive models from data.

Sure, we can be academic about it and refine the definition of machine learning by looking at the countless of nuances, but that’s not what we are here to do. We are here to understand the core fundamentals of machine learning — the fundamentals that will take you further than anything else.

I believe the only way to aquire deep knowledge of any technical subject is to take it apart and put it back together again. This is what we are here to do.

The fundamental machine learning setup consists of:

1. a dataset, usually coming in the form of input and target variables,

2. a parametric function that models the relation between the input and the target variables,

3. and a loss function that measures the model’s fit to the data.

Let’s start with the data.

To look behind the curtain of machine learning algorithms, we have to precisely formulate the problems that we deal with. Three important parameters determine a machine learning paradigm: the input, the output, and the training data.

All machine learning tasks boil down to finding a model that provides additional insight into the data, i.e., a function f that transforms the input x into the useful representation y. This can be a prediction, an action to take, a high-level feature representation, and many more. We’ll learn about all of them.

Mathematically speaking, the basic machine learning setup consists of:

1. a dataset 𝒟,

2. a function f that describes the true relation between the input and the output,

3. and a parametric model h — also called a hypothesis — that serves as our estimation of f.

Remark. (Common abuses of machine learning notation.)

Note that although the function f only depends on the input x, the parametric model h also depends on the parameters and the training dataset.

Thus, it is customary to write h(x) as h(x; w, 𝒟), where w represents the parameters, and 𝒟 is our training dataset.

This dependence is often omitted, but keep in mind that it’s always there.

We make no restrictions about how the model f̂ is constructed. It can be a deterministic function like h(x) = −13.2 x² + 0.92 x + 3.0 or a probability distribution h(x) = P(Y = y ∣ X = x). Models have all kinds of families like generative, discriminative, and more. We’ll talk about them in detail; in fact, models will be the focal points of the majority of the chapters.

First, let’s focus on the paradigms themselves. There are four major ones:

• supervised learning,

• unsupervised learning,

• semi-supervised learning,

• and reinforcement learning.

What are these?

Supervised learning

The most common paradigm is supervised learning. There, we have inputs 𝐱ᵢ ∈ ℝᵐ and ground truth labels yᵢ that form our training dataset

Although the labels can be anything like numbers or text, they are all available for us. The goal is to construct a function that models the relationship between the input x and the target variable y.

Let’s saddle up and see a couple of examples.

Read more

They are the perfect medium for math-and-code-heavy technical content: they support LaTeX snippets, code execution, and, to top it all, enable interactive exploration. Every time I’m reading a hands-on tutorial about some fancy new framework, I cannot resist the urge to jump into edit mode and break the code in ways no author can think of.

Unfortunately, if you choose to write in Jupyter Notebooks, you either abandon content distribution by platforms such as Substack, LinkedIn, or X (because they don’t support the format) or manually convert the notebooks to satisfy every possible whim of every possible editor.

So, I built a tool that enables writers to publish Jupyter Notebooks on Substack (and other platforms) with a couple of clicks. It’s called NotebookPress, and it solves four major pain points:

• LaTeX rendering,

• code snippets,

• user interactivity,

• and cross-platform compatibility.

Let me give you a t…

Read more

Please welcome , author of the brilliant Substack. I’ve been following his work for years, and I’m honored to have him here for a guest post.

By the way, he is hosting a workshop on February 28th titled “Machine Learning and Generative AI System Design,” and he has kindly offered a 35% discount for readers of The Palindrome.

The code TIVADAR35 is valid until February 24th.”

Reserve Your Seat

Now, I’ll pass the mic to Sairam.

Enjoy!

Cheers,
Tivadar

Read more

Yes, I know that it is February. It took me a bit longer to write this post.

I also know that your inbox is (was) overloaded with triumphant 2025 reviews and grand plans for 2026. To respect your time, here's a no-bullshit summary of what you'll get from The Palindrome this year, and if you want the details, just read on.

All subscribers:

• I'm finishing my Machine Learning From Zero book this year, where we'll implement all the fundamental algorithms from scratch. This'll be the topic for my technical posts.

• I'll do more explainer-style videos, like this one from the Matrices and Graphs post.

Paid subscribers:

• Monthly live workshops, streamed live right here on Substack.

• First workshop: Mathematics of Machine Learning, March 7th, 15:00 - 20:00 CET.

• Second workshop: Neural Networks from Scratch, date TBD.

Read more

Today’s post is a very special one, written by my friend Miklós, whom I met during our PhD years. (Which was more than ten years ago. I feel old.) He is one of the smartest people I know, and he’s been doing impressive research projects since then.

One of his latest projects made the news recently, because the data collection took place on the International Space Station (ISS). This is interesting in itself, but what you rarely see is the “backend” side of science, the stuff that don’t make the news, but makes or breaks a research project of this scale.

What follows is a deep-dive report on the entire lifecycle of a space-bound project:

• grant proposal (moving from a whiteboard to orbit),

• agile problem solving (like jumping hoops to meet Apple Store regulations),

• stakeholder coordination (managing the logistics between international space agencies),

• project management (handling “no second chance” execution under the pressure of shifting launch windo…

Read more

Because of its recursive nature, computing the Fibonacci sequence via brute force is computationally expensive.

However, the Fibonacci numbers have a simple and beautiful closed-form expression written in terms of the golden ratio (φ) and the conjugate golden ratio (ψ).

Read more