It’s Tivadar from The Palindrome. Back when I wrote the Mathematics of Machine Learning book, I realized

1. what a great language Python is,

2. and that object-oriented Python should be the first thing taught to every machine learning engineer.

So, I’ve been thinking about publishing a series, but my Python skills are not exactly top-of-the-line; I just hack and slash until things work. Fortunately, I found the best person who could do that, and even better, he agreed to write a special article for you!

It’s my pleasure to introduce , author of , and my longtime online friend from back when X was called Twitter.

If you ever wanted to become a power user of Python and take advantage of all the heavy machinery provided by classes, operator overloading, inheritance, composition, etc., this is the article for you.

Dig in!

Cheers,
Tivadar

“A computer program stores data and does stuff with the data.”

This is not the most technical definition of a computer program you’ll see. But it’s a valid one. When you learn to code, you learn about data structures to store different types of data. And you also learn about tools needed to manipulate and transform the data. You often define functions containing code to “do stuff” with the data.

Your code will contain data structures and functions, and you pass those data structures to the functions when needed.

Object-oriented programming (OOP) brings these two aspects together into a single unit. This unit is the object, which contains the data and the tools needed to manipulate the data. This may not sound like much, but it enables you to think about the problem you’re trying to solve differently. You can visualize your problem in a way closer to how humans see the world.

Let’s look at some examples, starting with a concrete one. Consider a country. There’s plenty of data relevant to a country: its name, population size, geographical area, capital city, and more. All countries have these attributes. Therefore, you can create a template that includes these attributes that you can use each time you want to represent a different country. Countries also include people, so these can be included in the data for each country, too.

But countries also perform actions. They issue passports, they collect taxes, they create laws, and so on. OOP urges you to think of a single unit to represent a country, which includes all the data and tools needed for the country to perform the actions required. You’d create a class called Country–this is the template you need to create lots of countries. The class doesn’t represent a specific country but the idea of a country. Once you define the class, you can create as many instances of the class as you need. Each instance represents a specific country.

Classes and Instances • The Vector Class

But let’s work on a different example in this article. Let’s consider a vector. One way to view vectors is as entities with both magnitude and direction. In three-dimensional (3D) Euclidean geometry, a vector is represented by three numbers.

Let’s put on our OOP hat. We need a unit in our program to represent a vector. It needs to represent its data and its functionality. Let’s start by creating a class called Vector:

Admittedly, this class doesn’t do much for now. The ellipsis (...) is just a placeholder that’s valid Python syntax. You’ll add more to this class soon. In the previous section, I mentioned that a class is a template for creating many objects modelled from the same blueprint. Let’s create a few instances of this class:

There’s only one Vector class. But now you have two instances of this class. You create an instance of the class when you add parentheses after the class name. The objects referenced by v1 and v2 are separate objects that occupy different areas of your computer’s memory. You can confirm that they’re different objects by showing their identity using Python’s id() function. The two instances of the Vector class have different identities:

You’ll get different values from the ones shown here when you run this code on your computer. But what matters is that the two numbers you get are different from each other. You can also confirm that v1 and v2 represent different objects by using v1 is v2, which returns False.

Note that the terms object and instance are both commonly used to refer to the unit created by a class. They refer to the same thing.

However, these are “blank” objects. They don’t have anything beyond the bare minimum a Python object needs. Let’s add some data.

Read more

returns to The Palindrome! You know I’m a big fan of his work, and if you are into machine learning, you should be too. His posts always strike the perfect balance between educational, practical, and entertaining.

He recently published the book 50 ML Projects to Understand LLMs, and his upcoming two-part series on exploring word representations is taken directly from the book. If you want to understand how Large Language Models work under the hood, don’t miss the post below.

Enjoy!

Cheers,
Tivadar

What you will learn in this 2-part post series

The primary goal of this post series is to teach you the Representational Similarity Analysis (RSA), which is a machine-learning analysis that compares distributed representations in different systems. It was originally developed as a neuroscience tool to compare how image categories are represented in different parts of the brain, and was then adapted to compare across different brain-imaging methods, different species, and between brains and computer vision models. Here in this post, I’ll teach you how to use RSA to explore whether different language models encode words in similar ways.

The secondary goal of this post series is to use RSA to learn about LLM architecture and representations. You’ll use RSA as a tool to peer into the internal calculations and representations inside LLMs, including embeddings vectors and the famous “attention” algorithm in the transformer blocks.

This post roughly corresponds to Project 12 in my recent book on using machine-learning projects to understand how LLMs work. Don’t worry, you don’t need the book to follow this post.

The main prerequisite for these posts is understanding correlation and cosine similarity. If you don’t know those two analyses — or if you want a refresher — check out this post. If you want to follow the code that accompanies this post, you’ll also need some Python coding skills.

How to use the code with these posts

Here’s the most important learning tip: Don’t just read the post; follow along with the code! The accompanying code will reproduce all the figures in this post — but you can do so much more by thinking of the code as a starting-point for your continued explorations. Try changing parameters, adding new words or categories, using different similarity/distance metrics, different models, etc.

The code is available here on my GitHub. In the video below, I show how to get and run the code using Google Colab. You can also download the notebook file and run it locally, but I recommend using Colab because you won’t need to worry about local installations or library versions.

📌 The Palindrome breaks down advanced math and machine learning with visuals that make everything click.

Join now while it’s 33% off (until the end of April) and lock in your discount for life. Unlock exclusive posts and 150+ deep dives into the heart of machine learning.

Upgrade to Paid (33% Off)

What are “embeddings” in language models like Claude?

Language models do not process text; they process numbers. When you write a prompt to a chatbot, that text is converted into a sequence of high-dimensional vectors called embeddings vectors. You can think of an embeddings vector as a coordinate, just like how the two numbers [1,-3] can be represented as a coordinate in a 2D space. So, each word is encoded as a coordinate, like the following illustration:

But here’s the thing: That diagram is really oversimplified. In practice, the dimensionality of word embeddings in language models isn’t two; it’s several thousand. Even GPT2-small — one of the smallest LLMs — has an embeddings dimensionality of 768. The models we’ll use in this post have an embeddings dimensionality of 1024. Even a 3-dimensional graph can be confusing and difficult to interpret.

And even more confusing is this: the dimensions are not human-crafted, nor do they correspond to human-interpretable traits like size and friendliness. In fact, we cannot interpret the axes at all. And when I write “we” I’m not referring to you and me; humans cannot understand what the axes mean, because the axes don’t mean anything in the sense of corresponding to physical characteristics of nature.

Still, those embeddings vectors are key to unlocking how LLMs represent and calculate information, and understanding those embeddings is key to technical AI safety and efficient fine-tuning of large models, in addition to research into complex systems and emergence.

How can we compare different embeddings?

All LLMs have an embeddings layer, but not all embeddings layers are the same. Different training sets, embeddings dimensionalities, and model training goals produce different embeddings vectors in different models. And because the embeddings matrix is initialized with random numbers, even the exact same LLM architecture with the exact same training data will have different embeddings vectors.

So how can we compare the embeddings vectors from different models? The answer is we cannot directly compare them.

However, it is possible to compare different embeddings by examining whether they have similar internal statistical structures. And that’s what you’re going to learn in this post.

RSA stands for representational similarity analysis and is a technique to compare representations across multiple encodings. The idea is this: Instead of correlating embeddings vectors directly between models, calculate similarities within models across embeddings, and then determine whether the patterns of across-embeddings similarities are similar.

In other words, the embeddings vector of “banana” might be very different between BERT and GPT-2, but the way that “banana” and “apple” relate to each other within each model might be similar.

Extract embeddings from two LLMs

We’ll work with the BERT-large model and the GPT-2-medium model. I’ll explain why I chose those two models in a moment, and then at the end of this post, I’ll explore a different pair of models.

The code below imports the BERT model and its tokenizer (a tokenizer is an algorithm that converts text into a sequence of integers that it is used to pick out the corresponding embeddings vectors).

I used “B” at the end of the variable names to disambiguate from the GPT-2 variables that end with “G”. The code for importing GPT-2-medium looks very similar and is in the online code file.

Let’s check the sizes of the embeddings matrices:

BERT embeddings shape: [30522, 1024]
GPT2 embeddings shape: [50257, 1024]

The size of the matrices corresponds to the number of items in the vocab (how many tokens each model has learned; that’s around 30k for BERT and 50k for GPT-2), and the dimensionality of the embeddings vectors (1024).

The two models have a different vocab size (briefly: BERT has mostly whole words while GPT-2 has subwords for non-Latin languages and code), but — germane to our application — the embeddings dimension (1024) is exactly the same for both models. That means we can directly compare the embeddings vectors between the two models.

Spoiler alert: Directly comparing their embeddings will not be insightful. But that will lead us to discover something amazing when we apply RSA.

Directly comparing model embeddings

For the rest of this project, we will use 34 words in three semantic categories. The idea is to compare — both directly and via RSA — whether BERT and GPT2 encode these words or their semantic categories in similar ways. Here are the categories and words:
Space: galaxy, asteroid, comet, cosmos, space, sun, planet, moon, star, orbit
Furniture: ceiling, sofa, couch, carpet, door, window, lamp, chair, table, rug, bed, floor, wall
Fruit: pear, grape, banana, cherry, peach, apple, seed, jelly, orange, lime, fruit

These words are all encoded as single tokens (when including a preceding space for GPT-2, which is a result of the byte-pair encoding algorithm).

Those tokens are row indices into the embeddings matrices, which we use to extract a submatrix of the embeddings from each model:

The two submatrices have the same size: 18 by 1024, corresponding to the 18 words and 1024 embeddings dimensions.

Now for the question at hand: do BERT and GPT-2 embed those words in the same way? We can answer that question with a correlation analysis. The idea is simple: If the two models use the same embeddings dimensions, then their embeddings vectors for the word “galaxy” should be identical — or at least very strongly correlated.

Let’s see what the data show:

The answer is a clear and resounding NOPE! The embeddings are completely different, with basically zero-valued correlations with small non-zero values attributable to noise in finite samples.

Let’s take a step back. Does it even make sense to compare embeddings directly? Doing so relies on the assumption that the embeddings dimensions are identical, and thus the embeddings values can be lined up and quantitatively compared.

But embeddings don’t work that way. In fact, embeddings matrices start off as random numbers and then are trained based on large datasets (the Internet text). Because of the random initialization, even training the same model on the exact same training set will create different embeddings matrices.

Segue to RSA.

Representational similarity analysis

Embeddings vectors cannot be directly compared across models. But, the relations across vectors within models can be compared, and then those within-model relations can be compared across models.

In practice, you calculate the RSA score in two steps:

Step 1: Calculate the cosine similarity between all pairs of embeddings vectors within each model.

Step 2: Correlate the two sets of cosine similarity values.

Let’s walk through the analysis, and then I’ll provide more discussion about how to interpret the RSA score, why Step 1 used cosine similarity while Step 2 used correlation, and other similarity metrics to calculate RSA.

The figure below shows the two within-model cosine similarity matrices as heatmaps.

There are several results visible in Figure 3, including that the three semantic categories are visible as the block-diagonal structure in the similarity matrices, and that the cosine similarities appear overall stronger in GPT-2 compared to BERT (note the color levels with equal color map limits).

The more interesting result, however, is the RSA scatter plot.

Here’s the interpretation: The token pairs that have higher similarity in GPT-2’s embeddings also have higher similarity in BERT’s embeddings. This indicates that although the embeddings spaces are different, there are statistical patterns inside the embeddings spaces that are similar across these two models.

Here’s a helpful analogy: Imagine representing the number “7” as an image. Then make a copy of that image and rotate it. The information content of the two images is the same, and yet their direct pixel-wise correlation is nearly zero:

To complete the RSA analogy, you’d need to create images of many more numbers, calculate cosine similarity across the number-images within each embeddings, then correlate those similarity values between embeddings “A” and “B”.

That’s just a helpful way to think about the motivation of RSA; don’t push the analogy too far.

In fact, here is an important note: The embeddings spaces of different models — especially models trained for different purposes (GPT-2 for text generation, BERT for text classification) — are not simply rotated versions of each other. They really are distinct embeddings spaces, but they do share some internal semantic structure that can be measured with an RSA.

As for why the RSA score is a correlation although the within-model relations were calculated using similarity: Within each model, the mean offsets of different vectors are relevant and meaningful. Across models, however, cosine similarities might be overall stronger or weaker; the question we ask with RSA is whether the relationships across token pairs are similar in different models. For that analysis, we do not want mean offsets to bias the score. Indeed, the distribution of similarity values is different between GPT-2 and BERT (see x- and y-axes in Figure 4). If you’re unsure of why a mean offset impacts cosine similarity and not correlation, you can check out my post on the topic.

Regarding the similarity metric: Cosine similarity is appropriate for embeddings vectors, but the analysis works the same way for any similarity or distance metric, such as correlation or Euclidean distance. In fact, if you use a distance metric, the analysis is termed “representational dissimilarity analysis.”

By the way, you may have noticed that I didn’t incorporate the three categories into the analysis. Great observation! I’ll use those categories in Part 2 of this 2-part post series. What I’d like you to do now is think about what analyses you might do with those categories.

RSA with different embeddings sizes

A key advantage of RSA is that it is robust to the dimensionality within each model. For example, how can you compare the embeddings between BERT-large (embeddings dimensionality of 1024) and GPT-2-XL (embeddings dimensionality of 1600)?

You can explore that in the code simply by importing gpt2-xl instead of gpt2-medium.

Well, when you try the direct correlation analysis (Figure 2), you get errors and cannot run the analysis or generate the plots. Both a correlation coefficient and a scatter plot are defined only for two vectors of the same dimensionality.

But don’t be concerned! I already discussed why the direct correlation analysis is invalid for comparing embeddings.

The RSA analyses, on the other hand, look great.

Conclusions and what’s coming up in the next post

RSA is a powerful analysis for comparing how different systems — brains or machines — represent information, when those representations are in different spaces with different dimensionalities. RSA was originally developed in neuroscience, then applied in many other machine-learning applications.

RSA is also easy to understand, easy to code, easy to interpret, and easy to visualize — all ideal properties of a machine-learning analysis.

In this post, you learned about how the analysis works and saw an application using token embeddings in language models. The goal of Part 2 in this series will be to apply this analysis to explore how the embeddings change inside the LLMs, as those embeddings vectors traverse the transformer stack. You’ll also learn how to explore category selectivity.

“How to get started in machine learning?” is one of the most common questions I get. I have a couple of default answers, but they are based more on my personal experience than on science.

Inspired by this, I’ve mapped out the knowledge graph of machine learning, building a hierarchy of concepts that can guide you from the foundations to the state of the art.

A couple of fascinating patterns have emerged from my journey: the thin spine of mathematics that holds up the entire knowledge graph, the central concepts like gradient descent that enable modern machine learning as we know it, and more.

Here’s the video where I talk about my findings:

Before you go: if you find value in my work, consider upgrading to a paid subscription. Your support is what makes it possible for me to keep creating high-quality educational content like this.

Right now is the best time to join—there’s a 33% discount until the end of April, and you can lock in that price for good.

Upgrade to Paid (33% off)

With a premium subscription, you’ll get access to

• the monthly live workshops,

• the subscriber chat,

• and the archive of 150+ deep dives into mathematics and machine learning.

In any case, I’m grateful you’re here. Your time and attention mean a lot, and The Palindrome wouldn’t exist without you.

Cheers,
Tivadar

I’m finishing up my video about the knowledge graph of machine learning, which will be released next week; I’ll do a live premiere right here on Substack Live, with a discussion after the viewing. (You can join here.)

This video will mark a milestone for me: instead of relying on hand-crafted slides and a presentation-style exposition, I built a scripting engine on top of the knowledge graph explorer that turns a JSON script like

[{
  "at": 0.0,
  "action": "autoRotate",
  "axis": "y",
  "speed": 0.01,
  "duration": 9.3,
  "windDown": 2.0,
  "easing": "linear"
},
{
  "at": 9.3,
  "action": "selectNode",
  "nodeId": "gpt",
  "showPrerequisites": true,
  "showDependents": false,
  "duration": 2.0
}]

into a beautifully rendered video.

Here’s a sneak peek:

The machine learning knowledge graph project could serve as a template for my future content. From now on, I’ll go full multimodal, meaning that I’ll

• build interactive visualizations (such as the knowledge graph explorer),

• then write posts and record videos, aided by the interactive tool.

Now that the current video-in-progress is about to be finished, what’s next?

Read on.

Read more

This week, it’s my pleasure to introduce , a fellow scholar working at the forefront of physics, mathematics, and computer science.

One of the main reasons behind the success of modern science is mathematical modeling, the process of translating complex real-life observations into a language that allows us to generalize, understand, and predict.

If you have enjoyed this post, subscribe to his newsletter , a space dedicated to translating the complexity of the empirical world, from your cells to entire societies, into language that is as simple as possible, though not necessarily simpler.

Read more

Read more

Let’s try something new. In recent months, I started to feel that the weekly publishing schedule takes its toll on the quality of my posts. I want to take more time per post to give you some high-quality technical content. I’m inspired by amazing writers such as Sebastian Raschka () or Cameron R. Wolfe (), who publish once a month, but each post is a work of art.

On the other hand, I miss you. A monthly schedule feels too long, and I have a lot to share with you. Ever since I got my ChatGPT Max subscription with access to Codex, my creativity is out of bounds.

So, here’s a new format. Each week, I’m sending you my unfiltered stream of consciousness, all the projects that I’m currently working on. Think of it as joining me for a coffee, where we talk about all the exciting/revolutionary/insane ideas we have in our minds.

This week, there are two things on my mind: knowledge graphs and Minecraft.

Let’s start with knowledge graphs.

The Complete Map of Machine Learning

If you are a regular reader, you know that one of the most common questions I get is “which part of mathematics do I need to study machine learning?” My default answer, based on my decade of experience, is: a ton of linear algebra, a decent amount of calculus, and a snippet of probability theory.

To be honest, I’m not completely satisfied with my reply. So, I dug deep with my newly found agentic AI-fueled superpower to find what is scientifically backed.

Without further ado: here’s the full knowledge graph of mathematics and machine learning. 2081 nodes, 5149 edges.

(The images are screenshots from the interactive graph explorer I’m building, which will be open source and publicly available.)

Let’s unravel this.

Read more

Yesterday we concluded our first monthly workshop! As promised, here is the full recording, exclusive to paid subscribers.

You can access the Jupyter Notebook lecture notes here: https://github.com/the-palindrome/mathematics-of-machine-learning-workshop

The next monthly workshop is already in the works; it’s going to be the next iteration of the Neural Networks From Scratch course. The tentative date is April 18th, 15:00–19:00 CET, but stay tuned for the announcement, as this date might change.

Thanks again so much for attending!

Cheers,
Tivadar

Read more

I’m finally working on my upcoming book Machine Learning From Zero, the sequel to Mathematics of Machine Learning.

The first chapter has just been finished, which sets the foundations for the neat stuff, like implementing neural networks from scratch. (Which is the core of the book.)

Without further ado, here’s an exclusive preview.

Enjoy!

P.S. I’m writing this book in Jupyter Notebooks, and I turn them into Substack posts with NotebookPress, a tool I’m building to bring technical writing on Substack to the next level. If you write math-and-code-heavy content, you should check it out.

Machine learning is training predictive models from data.

Sure, we can be academic about it and refine the definition of machine learning by looking at the countless of nuances, but that’s not what we are here to do. We are here to understand the core fundamentals of machine learning — the fundamentals that will take you further than anything else.

I believe the only way to aquire deep knowledge of any technical subject is to take it apart and put it back together again. This is what we are here to do.

The fundamental machine learning setup consists of:

1. a dataset, usually coming in the form of input and target variables,

2. a parametric function that models the relation between the input and the target variables,

3. and a loss function that measures the model’s fit to the data.

Let’s start with the data.

To look behind the curtain of machine learning algorithms, we have to precisely formulate the problems that we deal with. Three important parameters determine a machine learning paradigm: the input, the output, and the training data.

All machine learning tasks boil down to finding a model that provides additional insight into the data, i.e., a function f that transforms the input x into the useful representation y. This can be a prediction, an action to take, a high-level feature representation, and many more. We’ll learn about all of them.

Mathematically speaking, the basic machine learning setup consists of:

1. a dataset 𝒟,

2. a function f that describes the true relation between the input and the output,

3. and a parametric model h — also called a hypothesis — that serves as our estimation of f.

Remark. (Common abuses of machine learning notation.)

Note that although the function f only depends on the input x, the parametric model h also depends on the parameters and the training dataset.

Thus, it is customary to write h(x) as h(x; w, 𝒟), where w represents the parameters, and 𝒟 is our training dataset.

This dependence is often omitted, but keep in mind that it’s always there.

We make no restrictions about how the model f̂ is constructed. It can be a deterministic function like h(x) = −13.2 x² + 0.92 x + 3.0 or a probability distribution h(x) = P(Y = y ∣ X = x). Models have all kinds of families like generative, discriminative, and more. We’ll talk about them in detail; in fact, models will be the focal points of the majority of the chapters.

First, let’s focus on the paradigms themselves. There are four major ones:

• supervised learning,

• unsupervised learning,

• semi-supervised learning,

• and reinforcement learning.

What are these?

Supervised learning

The most common paradigm is supervised learning. There, we have inputs 𝐱ᵢ ∈ ℝᵐ and ground truth labels yᵢ that form our training dataset

Although the labels can be anything like numbers or text, they are all available for us. The goal is to construct a function that models the relationship between the input x and the target variable y.

Let’s saddle up and see a couple of examples.

Read more

They are the perfect medium for math-and-code-heavy technical content: they support LaTeX snippets, code execution, and, to top it all, enable interactive exploration. Every time I’m reading a hands-on tutorial about some fancy new framework, I cannot resist the urge to jump into edit mode and break the code in ways no author can think of.

Unfortunately, if you choose to write in Jupyter Notebooks, you either abandon content distribution by platforms such as Substack, LinkedIn, or X (because they don’t support the format) or manually convert the notebooks to satisfy every possible whim of every possible editor.

So, I built a tool that enables writers to publish Jupyter Notebooks on Substack (and other platforms) with a couple of clicks. It’s called NotebookPress, and it solves four major pain points:

• LaTeX rendering,

• code snippets,

• user interactivity,

• and cross-platform compatibility.

Let me give you a t…

Read more

Please welcome , author of the brilliant Substack. I’ve been following his work for years, and I’m honored to have him here for a guest post.

By the way, he is hosting a workshop on February 28th titled “Machine Learning and Generative AI System Design,” and he has kindly offered a 35% discount for readers of The Palindrome.

The code TIVADAR35 is valid until February 24th.”

Reserve Your Seat

Now, I’ll pass the mic to Sairam.

Enjoy!

Cheers,
Tivadar

Read more

Yes, I know that it is February. It took me a bit longer to write this post.

I also know that your inbox is (was) overloaded with triumphant 2025 reviews and grand plans for 2026. To respect your time, here's a no-bullshit summary of what you'll get from The Palindrome this year, and if you want the details, just read on.

All subscribers:

• I'm finishing my Machine Learning From Zero book this year, where we'll implement all the fundamental algorithms from scratch. This'll be the topic for my technical posts.

• I'll do more explainer-style videos, like this one from the Matrices and Graphs post.

Paid subscribers:

• Monthly live workshops, streamed live right here on Substack.

• First workshop: Mathematics of Machine Learning, March 7th, 15:00 - 20:00 CET.

• Second workshop: Neural Networks from Scratch, date TBD.

Read more

Today’s post is a very special one, written by my friend Miklós, whom I met during our PhD years. (Which was more than ten years ago. I feel old.) He is one of the smartest people I know, and he’s been doing impressive research projects since then.

One of his latest projects made the news recently, because the data collection took place on the International Space Station (ISS). This is interesting in itself, but what you rarely see is the “backend” side of science, the stuff that don’t make the news, but makes or breaks a research project of this scale.

What follows is a deep-dive report on the entire lifecycle of a space-bound project:

• grant proposal (moving from a whiteboard to orbit),

• agile problem solving (like jumping hoops to meet Apple Store regulations),

• stakeholder coordination (managing the logistics between international space agencies),

• project management (handling “no second chance” execution under the pressure of shifting launch windo…

Read more

Because of its recursive nature, computing the Fibonacci sequence via brute force is computationally expensive.

However, the Fibonacci numbers have a simple and beautiful closed-form expression written in terms of the golden ratio (φ) and the conjugate golden ratio (ψ).

Read more

How did they compute it?

Let me show you:

Read more

For the longest time, people have told me that I should start sharing my educational content in video format.

I’ve gotten compliments like the fact that my illustrations resemble those of 3B1B, for which I’m extremely honored.

So, after a few years of second-guessing whether I should dive into video content, I’ve finally decided to give it a try…

…and, in the short time I’ve been doing it, the results have amazed me.

Introducing The Palindrome YouTube channel

I’ve had a YouTube account since 2011, but just very recently I started shifting towards the creator side instead of being just a plain consumer.

My goal for the next couple of weeks is simple: every fan-favorite post from The Palindrome newsletter and from my social media posts will be turned into video.

So far, there are three videos uploaded to the channel. These are sort of the greatest hits of my educational content. Those that, no matter how many times I republish them, they always get a lot of engagement and I keep gett…

Read more

I am dead serious. Deep down, this tiny rule is the cog in many methods. Ones that you use every day.

Here is what it is, how it works, and why it is essential:

Read more

Hope that you enjoyed the Holidays! This time, I have a brand new video post for you. I'm really into video making now, so you'll see many more like this one in the future.

Enjoy!

Read more

Here’s a quick compilation of the most popular posts we published on The Palindrome in 2025.

Take a chance to read some of them, in case you missed them, or revisit your favorites.

Here’s the full list:

The 10 most popular posts of 2025

1. The Roadmap of Mathematics for Machine Learning

2. Machine Learning is not just Statistics

3. Your Machine Learning Library

4. The Competitive Programmer’s Introduction to Graph Theory

5. Coding on Paper

6. Representing Graphs

7. The Camel Principle

8. Introduction to Algorithmic Analysis

9. Comparing Algorithms

10. The Most Important Skill You Were Never Taught

I hope you enjoy reading them as much as I enjoyed making them for you.

And, if you have a favorite that didn’t make the list, reply to this email and let me know! I reply to all messages from you.

Read more

In a single variable, the chain rule says that

holds for any two univariate differentiable functions f and g. It’s simple, compact, and user-friendly.

However, the math quickly becomes complicated when we introduce multiple variables. And we have to, because there are billions of parameters to tune.

In the general case, we are interested in the composition of the vector-vector function g: ℝⁿ → ℝᵐ with the vector-scalar function f: ℝᵐ → ℝ. For instance, if we have

• a machine learning model with n parameters x = (x₁, ..., xₙ),

• a training dataset of m samples d₁, ..., dₘ,

• predictions g(x) = (g₁(x, d₁), ..., gₘ(w, dₘ)),

• and a loss function f,

then fitting the model is equivalent to finding the maximum of the composite function h(x) = f(g(x)). (The dependence on the data in g(x) is omitted for simplicity.)

To compute the gradient, that is,

(where ∂ᵢh denotes the partial derivative of the function h with respect to its i-th variable), the chain rule says that

This post is about what’s behind this formula.

Read more