Prologue: The Map of the Territory

Let us begin our investigation by establishing a precise correspondence between two worlds: the world of classical mechanics, where particles move through physical space under forces, and the world of machine learning, where algorithms navigate through parameter space seeking to minimize loss.

This is not merely an analogy—it is an exact mathematical mapping. Every symbol, every equation in physics has its precise counterpart in optimization theory. Let us make this mapping explicit from the very beginning:

PHYSICAL WORLD		↔	MACHINE LEARNING WORLD
Symbol	Meaning	↔	Symbol	Meaning
$\mathbf{x}$	Position of particle	↔	$\mathbf{w}$	Weights/parameters of neural network
$U(\mathbf{x})$	Potential energy	↔	$L(\mathbf{w})$	Loss function (error function)
$\mathbf{v}$	Velocity of particle	↔	$\mathbf{v}$	Velocity in momentum (same symbol!)
$m$	Mass of particle	↔	$m$	Virtual “mass” of optimizer
$\gamma$	Friction coefficient	↔	$\gamma$	Damping coefficient of optimizer
$-\nabla U(\mathbf{x})$	Force pushing downhill	↔	$-\nabla L(\mathbf{w})$	Gradient pushing toward minimum
$t$	Physical time (continuous)	↔	$t$	Training iteration (discrete)
$\Delta t$	Time step	↔	$\Delta t$	One optimization step

Chapter I: The Fundamental Equations

1.1 The Conservative Force and the Loss Gradient

Let us begin with the most fundamental equation: how does the landscape—whether physical or parametric—exert a “force” that guides motion toward lower regions?

1.2 Newton’s Second Law and Parameter Inertia

But force alone does not determine motion. We must specify how the system responds to this force. Newton’s second law provides the answer in physics, and we can write its exact counterpart for optimization:

Combining with our force/gradient equation:

1.3 Friction: Energy Dissipation

In the real world, motion is always subject to friction. For motion through a viscous medium, the friction force is proportional to velocity. This is crucial—let us write both versions:

1.4 The Complete Equation of Motion

Now we combine all forces—the driving force from the gradient and the resistive force from friction—to obtain the complete equation of motion:

Chapter II: The Characteristic Time $\tau$

2.1 The Fundamental Time Scale

Let us now introduce the most important quantity in our entire analysis: the characteristic time scale $\tau = m/\gamma$. This single number determines the entire behavior of our system.

2.2 Rewriting the Equation in Terms of $\tau$

Let us divide our equation of motion by $\gamma$ to make the role of $\tau$ explicit:

Chapter III: The Limit $\tau \to 0$ and the Emergence of Gradient Descent

3.1 The Overdamped Regime: When $\tau$ Becomes Negligible

Now we arrive at the crucial moment. What happens when $\tau$ becomes very small? This occurs when:

• $m \to 0$ (negligible mass)
• or $\gamma \to \infty$ (infinitely strong friction)
• or both, as long as $m/\gamma \to 0$

In this limit, the term $\tau \frac{d\mathbf{v}}{dt}$ becomes negligible compared to the other terms. Mathematically:

3.2 From Continuous Dynamics to Discretization

In both physics simulations and machine learning, we work with discrete time steps. Let us discretize our overdamped equation:

Now we discretize using $\Delta t$ (time step):

3.3 The Learning Rate: $\eta = \Delta t / \gamma$

We now define the learning rate as the ratio of time step to friction coefficient:

3.4 The Final Gradient Descent Equation

Substituting $\eta = \Delta t / \gamma$, we obtain the canonical form of gradient descent:

Chapter IV: Momentum – When $m \neq 0$

4.1 Restoring the Mass

Let us now explore what happens when we do not take the extreme limit $m \to 0$. Instead, we allow the particle (or the optimizer) to retain some mass, some memory of its previous motion.

We return to our fundamental equation, keeping $\tau = m/\gamma$ finite but still assuming we’re in a reasonably damped regime ($\tau$ not too large):

4.2 Discretization with Memory

Let us discretize this equation carefully, approximating the derivative as:

Substituting into our equation and multiplying both sides by $\Delta t$:

Factoring out $\mathbf{v}_t$ from the first two terms:

4.3 The Momentum Coefficient: $\rho = 1 – \Delta t / \tau$

Now we define the momentum coefficient $\rho$ (rho):

4.4 The Final Equation with Momentum

Defining $\rho = 1 – \Delta t / \tau$ and $\eta = \Delta t / \gamma$, we obtain the classical momentum method:

Chapter V: Visual Comparison and Final Tables

5.1 Complete Comparison Table

Aspect	Pure Gradient Descent	GD with Momentum
	$m=0$, $\gamma \to \infty$, $\tau \to 0$	$m>0$, $\gamma$ finite, $\tau$ finite
Equation (physics)	$\mathbf{x}_{t+1} = \mathbf{x}_t – \eta \nabla U(\mathbf{x}_t)$	$\mathbf{v}_{t+1} = \rho \mathbf{v}_t – \eta \nabla U(\mathbf{x}_t)$ $\mathbf{x}_{t+1} = \mathbf{x}_t + \mathbf{v}_{t+1}$
Equation (ML)	$\mathbf{w}_{t+1} = \mathbf{w}_t – \eta \nabla L(\mathbf{w}_t)$	$\mathbf{v}_{t+1} = \rho \mathbf{v}_t – \eta \nabla L(\mathbf{w}_t)$ $\mathbf{w}_{t+1} = \mathbf{w}_t + \mathbf{v}_{t+1}$
State variables	Only $\mathbf{x}$ (or $\mathbf{w}$)	$\mathbf{x}$ and $\mathbf{v}$ (or $\mathbf{w}$ and $\mathbf{v}$)
Memory of past	None ($\tau=0$)	Yes, through $\mathbf{v}_t$
Physics: particle in…	Very thick honey	Oil or air (depending on $\rho$)
Response to constant $\nabla U$ or $\nabla L$	Constant velocity	Acceleration ($\mathbf{v}$ grows)
In narrow valleys	Can oscillate (zig-zag)	Smoother trajectory
Shallow minima	Gets stuck easily	Can jump them thanks to momentum
Overshooting	Impossible (no inertia)	Possible if $\rho$ too high
Convergence	Slower but stable	Faster but can oscillate
Hyperparameters	Only $\eta$	$\eta$ and $\rho$
Typical values	$\eta \approx 0.01\text{-}0.1$	$\eta \approx 0.01$, $\rho \approx 0.9$

Epilogue: The Beauty of Exact Mapping

What we have discovered in this long journey is not merely an analogy, but an exact mathematical isomorphism—a one-to-one correspondence between the equations of dissipative classical mechanics and the algorithms of gradient-based optimization.

Every symbol in physics has its precise counterpart in machine learning. Every physical parameter—mass, friction, relaxation time—has its corresponding role in the optimizer. The gradient descent algorithm is not inspired by physics; it is physics, applied to the abstract space of neural network parameters rather than physical space.

When we set $m = 0$ and $\gamma \to \infty$ (while keeping $\eta = \Delta t / \gamma$ finite), we obtain pure gradient descent: a system with no memory, moving through parameter space like a particle in infinitely viscous honey, always following the instantaneous gradient.

When we allow $m > 0$ and $\gamma$ finite, we obtain the momentum method: a system that remembers its past motion, accumulates velocity in consistent directions, and can coast through unfavorable regions toward better minima.

The learning rate $\eta$ is not an arbitrary tuning parameter—it is precisely $\Delta t / \gamma$, the ratio of temporal discretization to friction. The momentum coefficient $\rho$ is not a magic number—it is exactly $1 – \Delta t / \tau$, encoding how much velocity persists from one iteration to the next.

Yet we must remember: the loss landscapes of neural networks are stranger than any physical terrain. They exist in spaces of unimaginable dimension, they shift with each mini-batch, and they contain structures—sharp versus flat minima, mode connectivity, loss surface geometry—that we are only beginning to understand.

But the physics gives us a foundation, a language, a set of intuitions that guide us through this strange landscape. And that, perhaps, is the deepest lesson: that the mathematics of the natural world and the mathematics of artificial intelligence are not separate domains, but different manifestations of the same underlying principles.