The Math You Missed Behind Gradient Descent
Why gradient descent works
The single most important optimization algorithm in AI: gradient descent.
Even your state-of-the-art LLMs are trained with a beefed-up version of this ancient algorithm, generally described through a mountain-climbing analogy. Look around your current position in the loss landscape, find the direction of the steepest descent, then take a step toward it. Yet no one ever tells you what’s really happening behind the scenes.
Here’s the secret math behind gradient descent:
If you are still here, I have a small favor to ask of you. My YouTube channel is still small, and if you enjoy my content, you can support me by
watching the video,
subscribing to my channel,
and leaving a comment on YouTube with your questions and thoughts.
Over the last few months, I’ve been focused on making my deep dives into the mathematics of machine learning better and clearer. You already love my drawings, and moving from still images to animated visualizations is a light-year leap in quality. This is the future of The Palindrome.
It only takes a couple of clicks from you, but it means the world to me. With your support, I can continue being a full-time content creator, delivering bangers like the video above for you.
In any case, thanks for being here!
Cheers,
Tivadar