Closed-form Continuous-time Neural Models
Abstract: Continuous-time neural processes are performant sequential decision-makers that are built by differential equations (DE). However, their expressive power when they are deployed on computers is bottlenecked by numerical DE solvers. This limitation has significantly slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show it is possible to closely approximate the interaction between neurons and synapses -- the building blocks of natural and artificial neural networks -- constructed by liquid time-constant networks (LTCs) efficiently in closed-form. To this end, we compute a tightly-bounded approximation of the solution of an integral appearing in LTCs' dynamics, that has had no known closed-form solution so far. This closed-form solution substantially impacts the design of continuous-time and continuous-depth neural models; for instance, since time appears explicitly in closed-form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared to differential equation-based counterparts. More importantly, in contrast to ODE-based continuous networks, closed-form networks can scale remarkably well compared to other deep learning instances. Lastly, as these models are derived from liquid networks, they show remarkable performance in time series modeling, compared to advanced recurrent models.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview: What is this paper about?
This paper shows a way to make a certain kind of brain-inspired AI much faster and easier to use. These AIs model how things change over time using math called differential equations (rules for how something changes from moment to moment). Normally, computers have to “step” through these rules little by little, which is slow. The authors find a clever shortcut: a direct formula (an approximate “closed-form” solution) that lets the AI jump straight to the answer without taking all those tiny steps. They then build a new, fast neural network called a Closed-form Continuous-depth network (CfC) based on this idea.
Goals: What questions are they trying to answer?
- Can we avoid slow step-by-step math solvers and still keep the power of continuous-time neural networks?
- Is there a good, direct formula that closely matches how certain brain-like models (called liquid time-constant networks, or LTCs) behave?
- If we build a new model using this formula, will it be accurate on real tasks and also much faster?
Approach: How did they do it (in everyday language)?
Think of an app that predicts how something changes over time—like a car’s steering angle or a patient’s health. Many models use a “step-by-step” simulator (a numerical solver) to update the prediction a tiny bit at a time. That’s like walking across a room by taking hundreds of tiny steps.
The authors instead look for a shortcut: a direct math expression that tells you where you’ll end up—like teleporting to the other side of the room. Here’s the idea in simple terms:
- Start with a well-known brain-inspired model (LTCs) where “neurons” and “synapses” (connections) change continuously in time.
- Use math for linear differential equations to simplify the problem. This reduces the solution to an expression that contains one tricky integral (an “area under a curve”).
- Because the input signal can be anything, that integral usually can’t be written exactly. So they approximate it:
- First, they imagine the input as small “constant” chunks (piecewise constant), which makes the integral easy to handle.
- Then they craft a tight approximation that works well even when the input isn’t actually piecewise constant.
- They prove that the error of this approximation shrinks over time (it decays exponentially), which means it’s reliably close to the original model.
- Using this formula, they design a new neural network (CfC) that:
- Has an explicit time term (time is a direct input), so it naturally handles irregular timing (like missing measurements or uneven time gaps).
- Replaces some “too-strong” exponential terms with smoother “sigmoid gates” (imagine doors that open or close gradually over time). This makes training more stable and reduces vanishing gradients.
- Uses a shared “backbone” network that splits into three heads (f, g, h): one head controls the time gate (how fast things change), and the others compute the actual update. This balances speed, stability, and expressiveness.
Analogy: Numerical solvers are like watching every frame of a movie to see what happens. A closed-form solution is like skipping to the exact scene you want. CfC builds a model that lets you skip ahead accurately, without losing the story.
Main Findings: What did they discover and why does it matter?
Here are the key takeaways from their experiments across very different tasks:
- Big speedups: CfCs run much faster than solver-based models—often 10x to 100x faster, and in some cases even more. This is because they don’t need the slow step-by-step solver.
- Strong accuracy:
- Event-based XOR (irregular timing): CfCs reach around 99–100% accuracy, handling irregularly sampled sequences where many models struggle.
- Medical time series (PhysioNet ICU mortality): CfC matches or slightly beats leading methods (about 0.84 AUC), while training per epoch is roughly 160–220 times faster than some continuous-time baselines.
- Movie reviews (IMDB sentiment): CfC with a mixed-memory wrapper reaches top-tier accuracy (~88.3%), competing with strong recurrent models.
- Physics prediction (Walker2D, irregular timings): CfCs achieve the lowest error among many baselines, even outperforming Transformers here, and they train fast.
- Scales well: Because CfCs avoid heavy solvers, they handle larger models and longer sequences more easily.
- Practical bonus: You can even “compile” an already-trained LTC model into its closed-form version to speed up inference (the part where the model makes predictions).
Why this matters: You get the advantages of continuous-time modeling (great for real-world timing and dynamics) without paying the time cost of numerical solvers.
Implications: What could this change in the real world?
- Faster, more reliable time-based AI: CfCs make it practical to use continuous-time models for things like health monitoring, self-driving cars, finance, and physics simulations—especially when data arrives at irregular times.
- Better on-device AI: The speed and simplicity can make continuous-time models feasible on robots, phones, or other devices with limited compute.
- New research directions: Having a closed-form backbone helps scientists study and scale continuous-time neural nets more easily, opening doors to better brain-inspired models and faster scientific simulation tools.
In short, this paper shows how to turn a slow, step-by-step process into a fast, direct one—keeping the brain-like strengths of continuous-time models while making them practical and scalable.
Collections
Sign up for free to add this paper to one or more collections.