Your support makes it possible for me to write these high value, high signal posts every week.

Subscribe now

Hey!

It’s Tivadar from The Palindrome. Back when I wrote the Mathematics of Machine Learning book, I realized

1. what a great language Python is,

2. and that object-oriented Python should be the first thing taught to every machine learning engineer.

So, I’ve been thinking about publishing a series, but my Python skills are not exactly top-of-the-line; I just hack and slash until things work. Fortunately, I found the best person who could do that!

It’s my pleasure to introduce Stephen Gruppetta, author of The Python Coding Stack, and my longtime online friend from back when X was called Twitter.

If you ever wanted to become a power user of Python and take advantage of all the heavy machinery provided by classes, operator overloading, inheritance, composition, etc., this is the article for you.

(It’s the second post in the object-oriented Python miniseries. Check the first one here for the foundations of OOP!)

Dig in!

Cheers,
Tivadar

A class is a template you can use to create several objects that share the same characteristics. All the objects created from a class will have the same structure and can perform the same actions.

However, sometimes you need to create objects that don’t have the same attributes but that are still similar to each other. You may need objects that share some characteristics but are sufficiently different from each other, so you can’t create them using the same class.

You want to avoid writing a brand new class from scratch. Instead, you’d like to reuse some of the code from the original class.

There are several options for handling overlap between classes. Inheritance is a common tool to link classes. However, inheritance is not always the right solution. Composition is an alternative technique to use a class as part of another class. In this article, you’ll learn about inheritance and composition. You’ll learn how to use each technique, and just as importantly, when to use them.

The Velocity Class

You learned about classes in the first article in this series. You wrote a Vector class that you can use to create vector instances.

You’ll make some changes to this class as you progress through this article. However, you’ll primarily work on a new class: the Velocity class.

You’ll consider two different routes to create the Velocity class. Velocity is a vector: it has a magnitude and a direction. Therefore, you can start from the Vector class you defined in the first article and use it as a starting point to build the Velocity class. This is the inheritance route.

However, you can also create a class that has a Vector instance as one of its attributes. You’ll also explore this composition route in this article.

Read more

The primary goal of this post series is to teach you the Representational Similarity Analysis (RSA), which is a machine-learning analysis that is used to compare distributed representations in different systems.

If you haven’t already read Part 1 in this series, please do so! It provides necessary background about how the RSA score is calculated and interpreted.

As a brief reminder, an RSA (representational similarity analysis) works by comparing cosine similarity matrices across different embeddings spaces (layers, blocks, models, etc.). The idea is that different embeddings spaces may have distinct coordinate systems and even different dimensionalities, but if their internal representational structures are similar, the relative similarities should be strongly correlated even if the vectors are distinct.

The additional goals of this second post are (1) to learn more about RSA and category specificity, and (2) to learn how to dissect the “hidden layers” of an LLM, and in particular, the Query, Keys, and Values vectors inside the transformer block. Those Q, K, and V vectors are part of the mechanism by which LLMs figure out what information from previous words are relevant for using the current word to make predictions about subsequent words.

You will discover that the adjustment vectors are largely uncorrelated while their RSA scores are quite high. These results show that although individual attention matrices have idiosyncratic internal calculations, they learn meaningful representations and words that allow them to interact in elegant and semantically meaningful ways.

If you want to learn more about the attention algorithm, I can humbly recommend my post on the topic.

This post roughly corresponds to Project 38 in my recent book on using machine-learning projects to understand how LLMs work. Don’t worry, you don’t need the book to follow this post.

How to use the code with these posts

The accompanying code file will reproduce all the figures in this post — but you can do so much more by thinking of the code as a starting-point for your continued explorations. Try changing parameters, adding new words or categories, using different similarity/distance metrics, different models, etc.

The code is available here on my GitHub. In the video below, I show how to get and run the code using Google Colab. You can also download the notebook file and run it locally, but I recommend using Colab because you won’t need to worry about local installations or library versions.

What are the Q, K, and V vectors in the attention algorithm?

You’ve probably heard of the “attention algorithm” at the heart of the LLM transformer block. Attention is an elegant and clever trick that allows language models to determine how much information is contained in each pair of tokens and how that information is relevant for generating predictions about new text.

You can read more about the attention algorithm in this linked post; the paragraphs below provide a very brief summary.

There are three sets of activations for each token position, called the Query (Q), Keys (K), and Values (V) matrices. The idea is this: For each pair of tokens, the dot product between their corresponding Q and K vectors creates a scalar weighting value, with higher dot products indicating that more importance (attention) should be paid to that pair; that weight value then determines how much relevant information in V gets added onto the embeddings vectors.

The result of the attention algorithm is an adjustment to the embeddings vectors as they pass through the transformer stack. In other words, the embeddings vectors you worked with in the previous post are rotated and scaled by each transformer layer, and those adjustments nudge the vectors from pointing towards the tokens in the input (e.g., the text prompt you gave to Claude) towards other tokens to generate an appropriate and context-relevant output.

The attention calculation is separated into “heads,” which are low-dimensional views of the hidden states that capture distinct features of the text and are combined at the end of the attention algorithm. I decided to ignore the attention heads for this post. That’s partly because we’re not working with the QKᵀ dot products, which is where the representations become head-specific, and partly in the interest of focusing on the mechanics and interpretations of RSA. An interesting extension of this project could involve running the analyses separately per-head, although that would create a dimensionality-explosion due to the number of heads and matrices that could be compared with RSA.

Import and inspect GPT-2-XL

I decided to use GPT-2-XL for this post. The code to import the model and its tokenizer from Hugging Face was shown in the previous post. The screenshot below shows an overview of the model architecture.

If you’re new to PyTorch models and LLMs, then this overview might look intimidating. The relevant information for us here is that there are 48 transformer layers (h is for “hidden layer”), and each hidden layer contains an attention block (attn) among other components like layerNorm and MLP. The Q, K, and V vectors are calculated in the c_attn layer. That matrix is 4800×1600. The 1600 corresponds to the embeddings dimensionality, and 4800 corresponds to the Q, K, and V matrices concatenated into one wide matrix (4800 = 3×1600).

Access the internal calculations using hooks

In the previous post, we didn’t need to prompt the model because we could just grab the embeddings vectors for each word we wanted to analyze. However, accessing the internal calculations of an LLM is a little more involved. The reason is that the transformer modifies each embeddings vector according to context (previous words in the text); in other words, the representation of the word “the” depends on all the words that come before it. Thus, we need to prompt the model with some text in order to analyze its internals.

But that creates a new problem: Even small LLMs create huge data matrices during each forward pass, and storing all of those internal calculations for each prompt would require terabytes of space. Therefore, the internal calculations are destroyed as soon as they’re no longer needed.

Fortunately, PyTorch provides a special method (like a function) that we can implant into the model that allows us to grab the internal calculations before they are destroyed. It’s called a “hook function” and looks like this:

There’s a lot going on in that code, and it might look intimidating if you’re new to working with LLMs in Python. But the idea is to implant the “hook” function into the attention block of each transformer layer, make a copy of the Q, K, and V activations, convert them to NumPy, and store them in a dictionary called activations that we can access later. This function gets called each time we prompt the model with some text.

Now we’re ready to prompt the model using the 34 words in 3 categories that you used in the previous post.

But here’s the thing about language models: They’re not trained to process isolated words; they’re trained to extract rich and context-specific meaning from sequences of words like sentences and paragraphs that contain hundreds or thousands of words. Presenting one token at a time to an LLM will elicit unusual and outlier-like activations. Indeed, most interpretability analyses of LLMs specifically exclude the first token in the sequence because of its extreme activation patterns.

So I’ve done something very simple: I’ve presented to the model the sentence “The next word is ” substituting “” for each of 34 words that we want to analyze. This is good experimental design because it means that all words have identical context, and thus any differences and similarities can only be attributed to world-knowledge that the model learned about each word.

The screenshot below shows code that creates the batch of token sequences and prompts GPT-2 with those sequences. When the model runs through its calculations, the hook function is activated, copying and storing the attention activations vectors into the dictionary.

Output:

(dict_keys(['attn_0_q', 'attn_0_k', 'attn_0_v', 'attn_1_q', 'attn_1_k', 'attn_1_v', 'attn_2_q', 'attn_2_k', 'attn_2_v', 'attn_3_q', 'attn_3_k', 'attn_3_v', 'attn_4_q', 'attn_4_k', 'attn_4_v', 'attn_5_q', 'attn_5_k', 'attn_5_v', 'attn_6_q', 'attn_6_k', 'attn_6_v', 'attn_7_q', 'attn_7_k', 'attn_7_v', 'attn_8_q', 'attn_8_k', 'attn_8_v', 'attn_9_q', 'attn_9_k', 'attn_9_v', 'attn_10_q', 'attn_10_k', 'attn_10_v', 'attn_11_q', 'attn_11_k', 'attn_11_v', 'attn_12_q', 'attn_12_k', 'attn_12_v', 'attn_13_q', 'attn_13_k', 'attn_13_v', 'attn_14_q', 'attn_14_k', 'attn_14_v', 'attn_15_q', 'attn_15_k', 'attn_15_v', 'attn_16_q', 'attn_16_k', 'attn_16_v', 'attn_17_q', 'attn_17_k', 'attn_17_v', 'attn_18_q', 'attn_18_k', 'attn_18_v', 'attn_19_q', 'attn_19_k', 'attn_19_v', 'attn_20_q', 'attn_20_k', 'attn_20_v', 'attn_21_q', 'attn_21_k', 'attn_21_v', 'attn_22_q', 'attn_22_k', 'attn_22_v', 'attn_23_q', 'attn_23_k', 'attn_23_v', 'attn_24_q', 'attn_24_k', 'attn_24_v', 'attn_25_q', 'attn_25_k', 'attn_25_v', 'attn_26_q', 'attn_26_k', 'attn_26_v', 'attn_27_q', 'attn_27_k', 'attn_27_v', 'attn_28_q', 'attn_28_k', 'attn_28_v', 'attn_29_q', 'attn_29_k', 'attn_29_v', 'attn_30_q', 'attn_30_k', 'attn_30_v', 'attn_31_q', 'attn_31_k', 'attn_31_v', 'attn_32_q', 'attn_32_k', 'attn_32_v', 'attn_33_q', 'attn_33_k', 'attn_33_v', 'attn_34_q', 'attn_34_k', 'attn_34_v', 'attn_35_q', 'attn_35_k', 'attn_35_v', 'attn_36_q', 'attn_36_k', 'attn_36_v', 'attn_37_q', 'attn_37_k', 'attn_37_v', 'attn_38_q', 'attn_38_k', 'attn_38_v', 'attn_39_q', 'attn_39_k', 'attn_39_v', 'attn_40_q', 'attn_40_k', 'attn_40_v', 'attn_41_q', 'attn_41_k', 'attn_41_v', 'attn_42_q', 'attn_42_k', 'attn_42_v', 'attn_43_q', 'attn_43_k', 'attn_43_v', 'attn_44_q', 'attn_44_k', 'attn_44_v', 'attn_45_q', 'attn_45_k', 'attn_45_v', 'attn_46_q', 'attn_46_k', 'attn_46_v', 'attn_47_q', 'attn_47_k', 'attn_47_v']), (34, 5, 1600))

The size of each data tensor is 34×5×1600. There are 34 target words embedded into a sentence comprising 5 tokens (“The next word is ___”) with an embeddings dimensionality of 1600.

We’re almost ready for the analyses! The last preparatory step is to create two matrix masks that will allow us to identify the word pairs that are within-category (e.g., galaxy-comet and bed-wall) vs. across-category (e.g., star-sofa and window-banana).

To create those masks, I identified the diagonal blocks corresponding to the word pairs of the same category, and subtracted that from an upper-diagonal matrix. The result is the two matrices in panels C and D in Figure 1 below.

The upshot is this: I can apply those mask matrices to cosine similarity matrices to extract the similarity values within- vs. across-categories, and then apply the RSA method you learned about in the previous post.

Correlating Q, K, and V activations

The goal of Part 2 is to correlate the activations between pairs of attention activation matrices. Spoiler alert: The correlations will be close to zero. But you need to see how small these correlations are, in order to appreciate the insights gained from applying the RSA technique.

To begin, I ran the analysis in layer index 6. The scatter plots in Figure 2 show the correlations between all pairs of attention vectors for the second token position.

The correlation between Q and K is weakly negative, while the other two pairs have correlations near zero. The negative correlation between Q and K stems from the shift towards negative values in QKᵀ, which is an important part of the attention algorithm but isn’t relevant here. Overall, it seems that the activations in one matrix are unrelated to the activations in the other matrices — even though the tokens and contexts are identical.

By the way, I asked you to extract the data from the second token instead of the final (target) token to demonstrate that the weak correlations are not trivially due to processing different tokens; these scatter plots reflect identical token sequences at this point. In fact, some of the correlations are even closer to zero when using the final token. You can explore that yourself in the code.

Next I repeated the analysis for each transformer layer in a for-loop. That’s a lot of scatter plots to look at, so instead I visualized the correlation coefficients (Figure 3).

The results are similar throughout the model: Near-zero correlations for the pairs involving the V vectors, and weakly negative correlations between Q and K across most layers (those negative correlations impose sparsity for reasons that I detail in other posts).

What have we learned so far? In the previous post, you saw that although the embeddings spaces in different models are not directly comparable, their internal statistical relational structures are highly consistent, at least in the small sample we examined. How about the different attention matrices here in this post; do they have high RSA scores despite low direct correlations? Perhaps you’ve guessed that the answer is Yes, but let’s not trust our intuition; instead, let’s gather statistical evidence to make data-informed decisions.

We will start with focusing on data from one transformer layer to build visualization, intuition, and code, and then we’ll expand that analysis to all the layers.

Cosine similarities and RSA (one layer)

Remember from the previous post that RSA involves correlating similarity values, not correlating dimension-coordinate values. So to calculate an RSA score, we first need to calculate the within-matrix similarity values across all word pairs.

I’ll start with data from layer 6. In the code, I extract the Q, K, and V activations from the final token from all batch sequences, and calculate the token × token cosine similarity matrices within each attention matrix.

The cosine similarity matrices are interesting to look at: The within-category similarities (block-diagonals) are visually higher than the across-category off-diagonal elements, and the overall similarities are highest for K and weakest for V.

Now we can calculate the pairwise RSA scores by correlating the non-redundant and non-trivial cosine similarity values across the pairs of attention matrices. I’m not separating into within- vs. across-category submatrices yet; the RSA scores here are based on all words.

After having seen the weak correlations of activation values across the attention matrices, the RSA is a remarkable and refreshing change: Even the “weakest” RSA is still around .85.

The interpretation of the results so far is that the Q, K, and V matrices represent token updates in distinct (often orthogonal) ways, but the nature of how those adjustments relate to each other is comparable across the matrices. In other words, the internal representations are similar while the coordinate spaces are distinct.

That is no accident: If the three sets of attention vectors were already so closely correlated, then they would be redundant and the attention algorithm wouldn’t be terribly useful. Instead, each of the three matrices (especially V) is trained into orthogonal spaces so that their unique contributions to the hidden-state adjustments can be information-rich and context-selective.

But before we get too excited about this result, let’s see if these observations are unique to this layer, or whether we see similar results in other layers.

Laminar profile of RSA scores

The online code file shows how to repeat the RSA analysis in a for-loop over all transformer layers. Figure 6 below shows the laminar profiles of the correlation coefficients (RSA scores).

With exception of the first transformer layer, the RSA scores are all roughly equally strong, around .9, and not visually obviously changing with depth into the model.

Category separability in one layer

We still haven’t incorporated the categories into the analyses. That’s the goal of the last two sections of this post. We will quantify the category separability of cosine similarity values within each of the attention matrices using an effect size calculation called Cohen’s d (unrelated 🙂). And then we’ll calculate the RSA scores separately for the similarity values within- vs. across-categories.

Before getting into the details, let’s build some intuition by visualizing distributions of cosine similarity values. I’ll use the within- and across-category mask matrices I created earlier to isolate the within- and across-category cosine similarity values from one transformer layer. See Figure 7.

The distributions are clearly well-separated: For each attention matrix, the similarity values are stronger within- vs. across-categories. This result shows that even deep inside the attention algorithm, LLMs have some relational structure that incorporates semantic world-knowledge into its token adjustment vectors.

The d-values in the titles are Cohen’s d effect size values that I calculated using the compute_effsize function in the pingouin library. Cohen’s d is the difference of the means of the distributions, scaled by their standard deviations. It’s closely related to the t-value, but is scaled to give more interpretable results. Effect sizes of around 2.5 are very large — in the experimental psychology literature, by comparison, researchers are very happy with effect sizes of around .6 to .8.

Laminar profile of category separability and RSA

The final part of this post is to calculate the effect size and RSA score for each layer, in order to determine whether category specificity evolves across the transformer stack.

The coding here is fairly straightforward, and requires only some minor modifications, for example splitting the RSA calculations by category class and storing the results per-layer.

Here are the results:

The results show high category separability and RSA scores throughout the depth of the model, with some relative decrease in separability around the middle layers.

Interestingly, the across-category RSA scores seem to be weaker than the within-category scores. Let’s investigate that observation more.

The scatter plot below shows the direct comparison of within- to across-category RSA scores, with each marker indicating matrix per layer, and the diagonal line showing unity.

Every single marker is above the line of unity, meaning that all within-category RSA scores are larger than all across-category scores. You don’t need a statistical test to see that it’s a real effect!

By the way, this is not simply due to a ceiling effect of RSA scores (they are bound by 1), because correcting for the ceiling by re-running the analysis using the Fisher-z transform doesn’t change the key result. I don’t show that re-analysis here, but you can test it by applying the np.atanh function to the RSA scores.

These results reveal that semantic information is present within each attention matrix, can be quantified by a linear analysis (though this does not prove that the representations are fully linear), and the relational structures are comparable across matrices despite them being nearly orthogonal spaces.

So you wanna learn more?

If you think that leveraging machine-learning techniques to investigate and understand LLM architecture and internal calculations is a useful approach, then please consider checking out my book from which these posts were adapted, and/or my 90-hour video-based course on LLM architecture, training, and interpretability. And of course, you can check out my other Substack posts.

One. Breaking the rules is often the best course of action.

I can’t even count the number of math-breaking ideas that propelled science forward by light-years.

We have set theory because Bertrand Russell broke the notion that “sets are just collections of things.” We have complex numbers because Gerolamo Cardano kept the computations going when encountering √−1, refusing to acknowledge that it doesn’t exist. We have non-Euclidean spaces because János Bolyai did not accept that, given a line and an external point, there’s only one parallel line that intersects the point.

A triangle of three right angles breaks one space, but creates another.

No assumption should be set in stone, and I’m not afraid to challenge them. You shouldn’t be afraid either.

📌 If you find value in my work, consider supporting The Palindrome with a paid subscription! Your support is what makes independent and high-quality education possible.

Subscribe now

Two. You have to deeply understand the rules to successfully break them.

Let’s be honest: breaking rules is a cliché.

All of my examples of successful rule-breakings above were executed by experts in their fields: Russell was a master of logic, Cardano of algebra, and Bolyai of geometry.

Think of doing mathematics science as holding a large marble orb in your hands, with treasure inside. You want to open it, but the marble is so sturdy that you can’t just smash it open.

What you can do, however, is to obsessively study it in detail until you find a tiny crack, giving you a way in.

If you don’t know what you are doing, you are just an elephant in a porcelain store.

This holds true outside of math. Miles Davis, legendary jazz musician, famously stated that

“Once is a mistake, twice is jazz.”

Mistakes are easy. Jazz is hard.

Three. Understanding happens when you take things slow.

At the university, most of my classes were in the classic chalk + blackboard style, and let me tell you, taking notes on formulas, definitions, and theorems by hand from a blackboard is infinitely more beneficial than relying on a PowerPoint presentation.

We were thinking and working together with the lecturers instead of just hanging along for the ride.

Something magical happens when you copy formulas by hand. Your mind automatically wraps itself around the concepts, bending them, studying them.

It’s not just true for math but for any topic.

That’s why typing out a solution from an LLM response (or StackOverflow if you are old-school) is better than copying and pasting it.

Four. The best way to learn is to solve problems.

I just said that “understanding happens when you take things slow.”

Well, the easiest way to take things slow is to be forced to by pushing your skills and knowledge to the limit.

This happens when you are working with tools you don’t know how to use, on problems you don’t know how to solve.

Until I implemented my own neural network library from scratch, my understanding was limited to the user level. I prepared my datasets, ran a couple of epochs, evaluated the model, and repeated the process until I liked what I saw.

Soon, it wasn’t enough. I wanted to dig deeper, so I built a neural network framework from scratch. And another one. And another one that became mlfz, the cornerstone of my next book.

Consuming YouTube tutorials is an extremely inefficient way to improve at anything. If you want to get good, get your hands dirty and start solving problems.

Think.

Build.

Take it apart and put it back together again.

Five. There are no shortcuts to mastery.

Once upon a time, the legendary Greek mathematician Euclid was summoned by King Ptolemy I Soter (not to be confused with the astronomer Ptolemy), who wanted to study geometry.

Judging from the fact that, two thousand years later, we still call classical geometry Euclidean, you have probably guessed that Euclid was pretty good. However, the king soon got impatient and asked if there was a shortcut.

Euclid replied that

“There is no royal road to geometry.”

The man invented the axiomatic approach to mathematics, wrote the second-most printed book of all time, and told a king to f**k off. That’s a pretty impressive resume.

But what does no royal road mean?

That there are no shortcuts. You can’t pay your way to knowledge; you must put in the work for math and anything else that’s worth doing.

Six. Always tackle one issue at a time. Look at special cases, then add complexity step by step.

I’m quite an impatient person. Even though my recklessness has gotten better with age, sometimes I still add two new features to a single commit. I just can’t help myself.

However, if you are a frequent reader of mine, you probably heard me say, “We’ll add complexity one layer at a time,” or something along these lines.

This is not an accident.

Whenever I want to understand a new concept, I know that mentally juggling with more than one ball thought is a recipe for failure, so I avoid it at all costs. Once I’m familiar with the basics, I add more and more details.

Think of the very first time you drove a car. Personally, my head was spinning so hard I didn’t even know whether I was coming or going. Letting off the gas and gently pressing the brake with one leg, smashing the clutch with the other, finding the right gear with my right hand, and holding the steering wheel steady with the other. That’s just slowing down before a turn. (I’m European, so I learned to drive on a manual.)

After twenty years of experience, I can do all this in my sleep, and I learned it one step at a time.

Driving on autopilot is the problem now, but that’s a topic for another day.

Seven. Finding the right perspective is half the success.

The all-time most-read post of The Palindrome is titled Matrices and Graphs, and it is secretly about this principle. By looking at matrices as graphs, we can immediately prove complex and profound results. Like in computer science, each implementation has its pros and cons. The proper choice of data structure can make or break a problem.

Matrices on one hand, graphs on the other.

Algebraic expressions like a + ib on one hand, vectors on the Euclidean plane on the other.

Driving your startup into the ground on one hand, a priceless learning opportunity on the other.

Success and failure are matters of perspective: you win, or you learn.

Eight. Asking questions is a superpower.

“It’s not that there are no stupid questions,” the words of my professor echo in my ears, “it’s that not asking your questions is stupid.”

You probably don’t know this about me, but I’m quite shameless. I never had a problem raising my hand and asking my question during a lecture, no matter what it was.

My mid-(and sometimes post)-lecture discussions with the professors followed a notable trajectory of improvement, from “What does that ∀ symbol on the board mean?” to discussing open problems and eventually solving a couple of them.

The key to that improvement was my blatant disregard for others’ perceptions of me, daring to play some wrong notes to master my instrument.

Nine. Talent is just the icing on the cake. The rest is hard work and perseverance.

When people discover that I have a PhD in mathematics, one of the most common reactions is, “you must be a genius.”

Let me tell you, this could not be further from the truth.

I was never a mathlete or a brilliant researcher. I am a slow thinker. I have problems with mental arithmetic.

However, three redeeming qualities helped me get where I am today. I am

• emotionally resilient,

• curious,

• and a hard worker.

Without these, no amount of brilliance can put you at the top or even in the middle of the pack.

Ten. Don’t give too much credit to advisors and professors. They are people like you, and experience is the only thing they have over you.

I follow quite a few notable people who have successfully realized their vision, solving problems to move our world forward. Their audiences are full of the young and motivated, yearning to carve out their piece of the pie, failing to realize that the only path they can walk is their own.

What worked for Elon Musk or Paul Erdős might not work for you.

Why? Because you have a different personality, a different socio-economic background, and potentially a different zeitgeist.

“No man ever steps in the same river twice, for it’s not the same river and he’s not the same man.”

Thanks for reading! If you have enjoyed the post, consider supporting me with a paid subscription. I’m an independent researcher/content creator, and your support is what makes work like this one possible.

Subscribe now

It’s Tivadar from The Palindrome. Back when I wrote the Mathematics of Machine Learning book, I realized

1. what a great language Python is,

2. and that object-oriented Python should be the first thing taught to every machine learning engineer.

So, I’ve been thinking about publishing a series, but my Python skills are not exactly top-of-the-line; I just hack and slash until things work. Fortunately, I found the best person who could do that, and even better, he agreed to write a special article for you!

It’s my pleasure to introduce , author of , and my longtime online friend from back when X was called Twitter.

If you ever wanted to become a power user of Python and take advantage of all the heavy machinery provided by classes, operator overloading, inheritance, composition, etc., this is the article for you.

Dig in!

Cheers,
Tivadar

“A computer program stores data and does stuff with the data.”

This is not the most technical definition of a computer program you’ll see. But it’s a valid one. When you learn to code, you learn about data structures to store different types of data. And you also learn about tools needed to manipulate and transform the data. You often define functions containing code to “do stuff” with the data.

Your code will contain data structures and functions, and you pass those data structures to the functions when needed.

Object-oriented programming (OOP) brings these two aspects together into a single unit. This unit is the object, which contains the data and the tools needed to manipulate the data. This may not sound like much, but it enables you to think about the problem you’re trying to solve differently. You can visualize your problem in a way closer to how humans see the world.

Let’s look at some examples, starting with a concrete one. Consider a country. There’s plenty of data relevant to a country: its name, population size, geographical area, capital city, and more. All countries have these attributes. Therefore, you can create a template that includes these attributes that you can use each time you want to represent a different country. Countries also include people, so these can be included in the data for each country, too.

But countries also perform actions. They issue passports, they collect taxes, they create laws, and so on. OOP urges you to think of a single unit to represent a country, which includes all the data and tools needed for the country to perform the actions required. You’d create a class called Country–this is the template you need to create lots of countries. The class doesn’t represent a specific country but the idea of a country. Once you define the class, you can create as many instances of the class as you need. Each instance represents a specific country.

Classes and Instances • The Vector Class

But let’s work on a different example in this article. Let’s consider a vector. One way to view vectors is as entities with both magnitude and direction. In three-dimensional (3D) Euclidean geometry, a vector is represented by three numbers.

Let’s put on our OOP hat. We need a unit in our program to represent a vector. It needs to represent its data and its functionality. Let’s start by creating a class called Vector:

Admittedly, this class doesn’t do much for now. The ellipsis (...) is just a placeholder that’s valid Python syntax. You’ll add more to this class soon. In the previous section, I mentioned that a class is a template for creating many objects modelled from the same blueprint. Let’s create a few instances of this class:

There’s only one Vector class. But now you have two instances of this class. You create an instance of the class when you add parentheses after the class name. The objects referenced by v1 and v2 are separate objects that occupy different areas of your computer’s memory. You can confirm that they’re different objects by showing their identity using Python’s id() function. The two instances of the Vector class have different identities:

You’ll get different values from the ones shown here when you run this code on your computer. But what matters is that the two numbers you get are different from each other. You can also confirm that v1 and v2 represent different objects by using v1 is v2, which returns False.

Note that the terms object and instance are both commonly used to refer to the unit created by a class. They refer to the same thing.

However, these are “blank” objects. They don’t have anything beyond the bare minimum a Python object needs. Let’s add some data.

Read more

returns to The Palindrome! You know I’m a big fan of his work, and if you are into machine learning, you should be too. His posts always strike the perfect balance between educational, practical, and entertaining.

He recently published the book 50 ML Projects to Understand LLMs, and his upcoming two-part series on exploring word representations is taken directly from the book. If you want to understand how Large Language Models work under the hood, don’t miss the post below.

Enjoy!

Cheers,
Tivadar

Read more

“How to get started in machine learning?” is one of the most common questions I get. I have a couple of default answers, but they are based more on my personal experience than on science.

Inspired by this, I’ve mapped out the knowledge graph of machine learning, building a hierarchy of concepts that can guide you from the foundations to the state of the art.

A couple of fascinating patterns have emerged from my journey: the thin spine of mathematics that holds up the entire knowledge graph, the central concepts like gradient descent that enable modern machine learning as we know it, and more.

Here’s the video where I talk about my findings:

Read more

I’m finishing up my video about the knowledge graph of machine learning, which will be released next week; I’ll do a live premiere right here on Substack Live, with a discussion after the viewing. (You can join here.)

This video will mark a milestone for me: instead of relying on hand-crafted slides and a presentation-style exposition, I built a scripting engine on top of the knowledge graph explorer that turns a JSON script like

[{
  "at": 0.0,
  "action": "autoRotate",
  "axis": "y",
  "speed": 0.01,
  "duration": 9.3,
  "windDown": 2.0,
  "easing": "linear"
},
{
  "at": 9.3,
  "action": "selectNode",
  "nodeId": "gpt",
  "showPrerequisites": true,
  "showDependents": false,
  "duration": 2.0
}]

into a beautifully rendered video.

Here’s a sneak peek:

The machine learning knowledge graph project could serve as a template for my future content. From now on, I’ll go full multimodal, meaning that I’ll

• build interactive visualizations (such as the knowledge graph explorer),

• then write posts and record videos, aided by the interactive tool.

Now that the current video-in-progress is about to be finished, what’s next?

Read on.

Read more

This week, it’s my pleasure to introduce , a fellow scholar working at the forefront of physics, mathematics, and computer science.

One of the main reasons behind the success of modern science is mathematical modeling, the process of translating complex real-life observations into a language that allows us to generalize, understand, and predict.

If you have enjoyed this post, subscribe to his newsletter , a space dedicated to translating the complexity of the empirical world, from your cells to entire societies, into language that is as simple as possible, though not necessarily simpler.

Read more

Read more

Let’s try something new. In recent months, I started to feel that the weekly publishing schedule takes its toll on the quality of my posts. I want to take more time per post to give you some high-quality technical content. I’m inspired by amazing writers such as Sebastian Raschka () or Cameron R. Wolfe (), who publish once a month, but each post is a work of art.

On the other hand, I miss you. A monthly schedule feels too long, and I have a lot to share with you. Ever since I got my ChatGPT Max subscription with access to Codex, my creativity is out of bounds.

So, here’s a new format. Each week, I’m sending you my unfiltered stream of consciousness, all the projects that I’m currently working on. Think of it as joining me for a coffee, where we talk about all the exciting/revolutionary/insane ideas we have in our minds.

This week, there are two things on my mind: knowledge graphs and Minecraft.

Let’s start with knowledge graphs.

The Complete Map of Machine Learning

If you are a regular reader, you know that one of the most common questions I get is “which part of mathematics do I need to study machine learning?” My default answer, based on my decade of experience, is: a ton of linear algebra, a decent amount of calculus, and a snippet of probability theory.

To be honest, I’m not completely satisfied with my reply. So, I dug deep with my newly found agentic AI-fueled superpower to find what is scientifically backed.

Without further ado: here’s the full knowledge graph of mathematics and machine learning. 2081 nodes, 5149 edges.

(The images are screenshots from the interactive graph explorer I’m building, which will be open source and publicly available.)

Let’s unravel this.

Read more

Yesterday we concluded our first monthly workshop! As promised, here is the full recording, exclusive to paid subscribers.

You can access the Jupyter Notebook lecture notes here: https://github.com/the-palindrome/mathematics-of-machine-learning-workshop

The next monthly workshop is already in the works; it’s going to be the next iteration of the Neural Networks From Scratch course. The tentative date is April 18th, 15:00–19:00 CET, but stay tuned for the announcement, as this date might change.

Thanks again so much for attending!

Cheers,
Tivadar

Read more

I’m finally working on my upcoming book Machine Learning From Zero, the sequel to Mathematics of Machine Learning.

The first chapter has just been finished, which sets the foundations for the neat stuff, like implementing neural networks from scratch. (Which is the core of the book.)

Without further ado, here’s an exclusive preview.

Enjoy!

P.S. I’m writing this book in Jupyter Notebooks, and I turn them into Substack posts with NotebookPress, a tool I’m building to bring technical writing on Substack to the next level. If you write math-and-code-heavy content, you should check it out.

Machine learning is training predictive models from data.

Sure, we can be academic about it and refine the definition of machine learning by looking at the countless of nuances, but that’s not what we are here to do. We are here to understand the core fundamentals of machine learning — the fundamentals that will take you further than anything else.

I believe the only way to aquire deep knowledge of any technical subject is to take it apart and put it back together again. This is what we are here to do.

The fundamental machine learning setup consists of:

1. a dataset, usually coming in the form of input and target variables,

2. a parametric function that models the relation between the input and the target variables,

3. and a loss function that measures the model’s fit to the data.

Let’s start with the data.

To look behind the curtain of machine learning algorithms, we have to precisely formulate the problems that we deal with. Three important parameters determine a machine learning paradigm: the input, the output, and the training data.

All machine learning tasks boil down to finding a model that provides additional insight into the data, i.e., a function f that transforms the input x into the useful representation y. This can be a prediction, an action to take, a high-level feature representation, and many more. We’ll learn about all of them.

Mathematically speaking, the basic machine learning setup consists of:

1. a dataset 𝒟,

2. a function f that describes the true relation between the input and the output,

3. and a parametric model h — also called a hypothesis — that serves as our estimation of f.

Remark. (Common abuses of machine learning notation.)

Note that although the function f only depends on the input x, the parametric model h also depends on the parameters and the training dataset.

Thus, it is customary to write h(x) as h(x; w, 𝒟), where w represents the parameters, and 𝒟 is our training dataset.

This dependence is often omitted, but keep in mind that it’s always there.

We make no restrictions about how the model f̂ is constructed. It can be a deterministic function like h(x) = −13.2 x² + 0.92 x + 3.0 or a probability distribution h(x) = P(Y = y ∣ X = x). Models have all kinds of families like generative, discriminative, and more. We’ll talk about them in detail; in fact, models will be the focal points of the majority of the chapters.

First, let’s focus on the paradigms themselves. There are four major ones:

• supervised learning,

• unsupervised learning,

• semi-supervised learning,

• and reinforcement learning.

What are these?

Supervised learning

The most common paradigm is supervised learning. There, we have inputs 𝐱ᵢ ∈ ℝᵐ and ground truth labels yᵢ that form our training dataset

Although the labels can be anything like numbers or text, they are all available for us. The goal is to construct a function that models the relationship between the input x and the target variable y.

Let’s saddle up and see a couple of examples.

Read more

They are the perfect medium for math-and-code-heavy technical content: they support LaTeX snippets, code execution, and, to top it all, enable interactive exploration. Every time I’m reading a hands-on tutorial about some fancy new framework, I cannot resist the urge to jump into edit mode and break the code in ways no author can think of.

Unfortunately, if you choose to write in Jupyter Notebooks, you either abandon content distribution by platforms such as Substack, LinkedIn, or X (because they don’t support the format) or manually convert the notebooks to satisfy every possible whim of every possible editor.

So, I built a tool that enables writers to publish Jupyter Notebooks on Substack (and other platforms) with a couple of clicks. It’s called NotebookPress, and it solves four major pain points:

• LaTeX rendering,

• code snippets,

• user interactivity,

• and cross-platform compatibility.

Let me give you a t…

Read more

Please welcome , author of the brilliant Substack. I’ve been following his work for years, and I’m honored to have him here for a guest post.

By the way, he is hosting a workshop on February 28th titled “Machine Learning and Generative AI System Design,” and he has kindly offered a 35% discount for readers of The Palindrome.

The code TIVADAR35 is valid until February 24th.”

Reserve Your Seat

Now, I’ll pass the mic to Sairam.

Enjoy!

Cheers,
Tivadar

Read more

Yes, I know that it is February. It took me a bit longer to write this post.

I also know that your inbox is (was) overloaded with triumphant 2025 reviews and grand plans for 2026. To respect your time, here's a no-bullshit summary of what you'll get from The Palindrome this year, and if you want the details, just read on.

All subscribers:

• I'm finishing my Machine Learning From Zero book this year, where we'll implement all the fundamental algorithms from scratch. This'll be the topic for my technical posts.

• I'll do more explainer-style videos, like this one from the Matrices and Graphs post.

Paid subscribers:

• Monthly live workshops, streamed live right here on Substack.

• First workshop: Mathematics of Machine Learning, March 7th, 15:00 - 20:00 CET.

• Second workshop: Neural Networks from Scratch, date TBD.

Read more

Today’s post is a very special one, written by my friend Miklós, whom I met during our PhD years. (Which was more than ten years ago. I feel old.) He is one of the smartest people I know, and he’s been doing impressive research projects since then.

One of his latest projects made the news recently, because the data collection took place on the International Space Station (ISS). This is interesting in itself, but what you rarely see is the “backend” side of science, the stuff that don’t make the news, but makes or breaks a research project of this scale.

What follows is a deep-dive report on the entire lifecycle of a space-bound project:

• grant proposal (moving from a whiteboard to orbit),

• agile problem solving (like jumping hoops to meet Apple Store regulations),

• stakeholder coordination (managing the logistics between international space agencies),

• project management (handling “no second chance” execution under the pressure of shifting launch windo…

Read more

Because of its recursive nature, computing the Fibonacci sequence via brute force is computationally expensive.

However, the Fibonacci numbers have a simple and beautiful closed-form expression written in terms of the golden ratio (φ) and the conjugate golden ratio (ψ).

Read more

How did they compute it?

Let me show you:

Read more

For the longest time, people have told me that I should start sharing my educational content in video format.

I’ve gotten compliments like the fact that my illustrations resemble those of 3B1B, for which I’m extremely honored.

So, after a few years of second-guessing whether I should dive into video content, I’ve finally decided to give it a try…

…and, in the short time I’ve been doing it, the results have amazed me.

Introducing The Palindrome YouTube channel

I’ve had a YouTube account since 2011, but just very recently I started shifting towards the creator side instead of being just a plain consumer.

My goal for the next couple of weeks is simple: every fan-favorite post from The Palindrome newsletter and from my social media posts will be turned into video.

So far, there are three videos uploaded to the channel. These are sort of the greatest hits of my educational content. Those that, no matter how many times I republish them, they always get a lot of engagement and I keep gett…

Read more

I am dead serious. Deep down, this tiny rule is the cog in many methods. Ones that you use every day.

Here is what it is, how it works, and why it is essential:

Read more