From Laplace to Neural Networks (Part 1)

Time-series predictions have been always interesting to me (and it’s been very hard for me), but I realized that I had never thought about predicting the time-series of physical system dynamics. So I looked up something from my old physics books and gave it a shot.

Laplace’s view

Laplace embraced the view¹ that if we know the positions and velocities of all the particles in the universe, then we would know the future for all time.

“We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed, if this intellect were also vast enough to submit these data to analysis, it would embrace in a single formula the movements of the greatest bodies of the universe and those of the tiniest atom; for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes.”

– Pierre Simon de Laplace, A Philosophical Essay on Probabilities²

This determinism has been later challenged by a series of development in modern science, such as irreversibility in quantum mechanics or thermodynamics, chaos theory, etc. It can still be a philosophical question³, but one thing is clear; even assuming everything is deterministic, nothing is linear in nature. Most of the real problems are highly nonlinear, or I’d say, chaotically deterministic or deterministically chaotic.

Nonlinear problems are much harder to solve and they are often high-dimensional (e.g., computational structural biology , fluid dynamics, material research, etc.), which requires massive computations. Thus, a traditional approach to study physical systems encounters many challenges. Recently, some (frustrated) physicists have found machine learning as a hopeful alternative.

The key questions that physicists have for “machine” are:

1. Can it learn the law of nature only from data?
2. Can it predict chaotic behaviors or nonlinear dynamics?

To me, this sounds a bit crazy - physicists introduced model (i.e., PDEs) to describe the law of nature, and now they’re trying to do something in opposite ways? But that is actuallly happening now - ML for Physics or Physics for ML. This kinds of research evolution is pretty captivating, which motivated me to write this post. As a soft starter, here I consider classical pendulum problems.

Plane (Ideal) Pendulum

Plane (ideal) pendulum is the most popular example in classical physics and its study is a key to understanding the nonlinear dynamics of many other systems⁴.

Single Pendulum

Consider a particle of mass $m$ constrained by a weightless, extensionless rod to move in a vertical circle of radius $L$ .

The gravitational force acts downward, but the component of this force influencing the motion is perpendicular to the support rod, and it is simply

The plane pendulum is a nonlinear system with a symmetric restoring force. The equation of motion can be obtained by equating the torque ($=LF(\theta)$) about the support axis to the product of the angular acceleration ($\ddot{\theta}$) and the rotational inertia ($I=mL^2$) about the same axis:

or,

with $w_0^2=\cfrac{g}{L}$. Thus, if we define new variable $u = [\theta, \dot\theta]^T$, we can construct a system of two first order differential equations that we can solve numerically:

def equation_motion(self, u, t) -> List[float]:
    """Equations of motion."""
    [theta, theta_dot] = u
    return [theta_dot, -self.g / self.L1 * np.sin(theta)]

We are going to use scipy.intergrate.odeint⁵ to find the numerical solutions.

def solve_ode(self) -> npt.NDArray[Any]:
    """Solve ODE."""
    sol = odeint(
        func=self.equation_motion,
        y0=self.init_cond.u0,
        t=self.time_coord.t_grid,
    )
    return sol

Here is the solution (with $m=1.5, L=1, \theta(0)=\frac{80}{180}\pi, \dot\theta(0)=0$). Note that the total energy is preserved over time.

The time-series of $\theta$, $\dot\theta$, $x$, and $y$ are periodic curves:

We often look at its phase diagram and trajectories:

Note that there is an exact analytical solution for this problem, but it is in the form of the Jacobi elliptic integral, and one can skip the details for now.

Double Pendulum

Double pendulum consists of two masses and two rods. The situation is slightly more complicated, since the two masses interact with each other.

But using the Euler-Lagrangian equation⁶, we can obtain a system of equations of motion and solve it numerically.

The equation of motions are implemented and solved similarly, and the solved results are in the below:

(with $m_1=1.5, m_2=2, L_1=1, L_2=1.5, \theta_1(0)=\frac{60}{180}\pi, \theta_2(0) = \frac{80}{180}\pi, \dot\theta_1(0)=\dot\theta_2(0)=0$)

The time series of ($\theta_1$, $\dot\theta_1$), ($\theta_2$, $\dot\theta_2$), ($x_1$, $y_1$), and ($x_2$, $y_2$) are not periodic anymore:

The phase diagram and trajectories shows the coupling relation between $\theta_1$ and $\theta_2$.

Predicting pendulum time-series - ARIMA?

So far we have seen that the single pendulum exhibits periodic dynamics, while the double pendulum exhibits more chaotic and nonlinear dynamics.

Here we attempt to predict the time-series of pendulum phase states. ARIMA is one of the most commonly used methods to model univariate time-series. Its extension to reflect seasonality is SARIMA.

For simplicity, we focus on predicting $\theta$ of single pendulum and $\theta_1$ of double pendulum. The time-series data is splitted into training and test region, and the predicted values are compared with the actuals in the both regions:

• In-sample: both ARIMA and SARIMA work fine.
• Out-of-sample: ARIMA fails for single/double pendulums. SARIMA works for single pendulum, but not for double pendulum. I further conducted a grid search for parameters, but couldn’t find a SARIMA model that works well for double pendulum predictions.

In short, for our double pendulum, (S)ARIMA seems to be fine for curve-fitting, but not for predictions. It is not surprising, since (S)ARIMA cannot approximate all complex dynamics and generalize. (S)ARIMA was tried just as a warm-up.

We want to try something else.

What’s Next

In Part 2, we’ll adopt neural networks to predict pendulum dynamics. Particularly, we will see if it learns any physics purely from data and how embedding physics helps it learn better.

Codes: https://github.com/uriyeobi/hamiltonian_neural_networks

Notes