A reference for learning-based control

ADP-MATLAB

From equations to
closed-loop experiments.

Study, reproduce and extend adaptive dynamic programming in MATLAB. Each method connects its mathematical assumptions to explicit code, recorded experiments and numerical checks.

Run your first example GitHub code Download ZIP

ADP-MATLAB 0.5.3 · MIT · Release & license status

Maintained by Junkai Tan·The Hong Kong Polytechnic University

8 runnable entries

Begin with an experiment.

An analytic baseline, a learned dynamics model, and a neural value function.

Linear quadraticIntegral PI

Recorded double-integrator states converge toward zero under the learned policy.

Learn a feedback law.

A two-state problem with an analytic LQR solution makes each policy-evaluation round and closed-loop check inspectable.

Recorded result5 policy-evaluation rounds

Run the baseline

Nonlinear dynamicsKoopman PI

Fifty recorded pendulum trajectories approach the equilibrium under a policy learned with a Koopman generator.

Control through a learned model.

Identify a Koopman generator, improve the policy, then compare it with policy iteration using the true dynamics.

Recorded experimentLearned policy · all 50 initial states
Normalized pendulum

Explore Koopman PI

Neural value functionsHJB PINN

Contours of the recorded trained pendulum value function, evaluated on the common grid at continuation horizon four.

Approximate the value function.

Train a physics-informed network with horizon continuation, then inspect its HJB residual and closed-loop behavior.

Recorded experimentNeural V(x, 0) · final stage T = 4 s
Reduced training

Explore neural HJB

Figures use saved MATLAB experiments. Each example has its own assumptions and numerical settings. Provenance · Editable figures & data

A traceable experiment

Keep the mathematics
close to the code.

Start with a method’s problem and equation map. Run its native implementation, then inspect the saved configuration, trajectory and diagnostics.

The shared entry point selects a method. Each implementation retains the assumptions, dimensions and numerical choices its mathematics requires.

Read the implementation guide

The integral-PI baseline

J(π) = ∫₀^∞ (xᵀQx + uᵀRu) dtu = π(x)

Problem: adp.models.doubleIntegrator
Learning: adp.learn.integralPI
Evaluation: adp.sim.rollout
Record: result.config · result.metrics
result.evaluation · runDir

The analytic LQR solution is an independent comparison; it is not supplied to the learner.

Find the method for your problem.

8 runnable entries, including 7 paper implementations. Expand a row for the command, dependencies and recorded scope.

Start fromLinear systems Learned dynamics Neural HJB Stochastic control

Implemented methods and runnable examples
Method / source	Problem & learning	Example
Integral policy iterationReference baseline	CT linear quadratic · known B	Double integrator
Analytic LQR comparator; two-state example. The saved baseline reaches the analytic LQR gain in five policy-evaluation rounds; the final gain error is 8.37e-14. Products MATLAB Native function `demo_integral_pi` Local checks 46 recorded passing tests Run from the repository root `demo_reproductions( ... 'baseline');` Entry source (.m)Equation notes (.md)Experiment & checks (.json)
Koopman generator + PIL4DC 2025	CT nonlinear · identified generator	Normalized pendulum
Normalized pendulum variant; not every paper table. Fifty recorded closed-loop trajectories have a maximum terminal state norm of 1.23e-3. Average cumulative cost across the 50 initial states is compared with true-model PI: the maximum absolute difference over 0–10 s is 3.15e-5. Products MATLAB Native function `demo_koopman` Local checks 8 recorded passing tests Run from the repository root `demo_reproductions( ... 'koopman_l4dc2025');` Entry source (.m)Equation notes (.md)Experiment & checks (.md)Original paper
Mean-field LQGAutomatica 2025	Stochastic CT · two-gain PI	Finite population LQG
Finite-sample social optimization; low-sample failures retained. The documented 4,000-path run reports gain errors of approximately 0.53% and 0.85%. Low-sample failures and independent training repeats are retained. Products MATLAB Native function `demo_meanfield_lqg2025` Local checks 8 recorded passing tests The default example uses 100 training paths. The saved 0.53% / 0.85% gain errors come from a separate 4,000-path record. Default: 100 training paths `demo_reproductions( ... 'meanfield_lqg2025');` This existing procedure pools batches under a common probe and fits at 100, 400, 1,000 and 4,000 paths. A small-sample fit may stop it before 4,000. Exact replay of the historical numbers still needs the archived configuration and data; see the run guide. Precision procedure: 40 × 100 paths `addpath(fullfile(pwd, ... 'reproductions','meanfield_lqg2025')); runId = char(datetime('now', ... 'Format','yyyyMMdd-HHmmss-SSS')); outputDir = fullfile(pwd,'runs', ... ['meanfield-precision-' runId]); precision = mf_precision_check( ... outputDir,mf_config(),40);` Entry source (.m)Equation notes (.md)Experiment & checks (.md)Run guide (.md)Original paper
Off-policy Q-learningIEEE TAC 2023	DT LQR · matrix Bellman equation	Data-based LQR
Data-based LQR; explicit MIMO initialization variant. The implementation documents a data-based stabilizing initialization, a discrete LQR comparison, and an explicit MIMO initialization variant. The linked record contains local checks. Products MATLAB, Control System Toolbox Native function `demo_qlearning_tac2023` Local checks 7 recorded passing tests Run from the repository root `demo_reproductions( ... 'qlearning_tac2023');` Entry source (.m)Equation notes (.md)Experiment & checks (.json)Original paper
Infinite-horizon HJB PINNIJRNC 2025	Neural HJB · horizon continuation	Scalar LQR & pendulum
Reduced LQR/pendulum training; corrected quartic cost identified. The saved reduced pendulum experiment completes four closed-loop evaluations at each continuation horizon, from 1 through 4. The figure shows the trained value on the common grid at horizon 4. Products MATLAB, Deep Learning Toolbox Native function `run_pinn_reproduction` Local checks 5 recorded passing tests Smoke trains scalar LQR at horizon 1 and pendulum at horizons 1–2 for 10 updates each, plus two 12-update prefix checks. Short workflow: smoke `demo_reproductions( ... 'pinn_infinite_horizon2025', ... 'smoke');` The horizon-4 figure uses reduced training: 2,000 LQR updates and 3,000 per pendulum horizon, 1–4. This is the smaller network experiment, not the full paper schedule. Figure configuration: reduced `[result,runDir] = demo_reproductions( ... 'pinn_infinite_horizon2025', ... 'reduced');` Entry source (.m)Equation notes (.md)Experiment & checks (.json)Run guide (.md)Original paper
Safe epigraph PINNICML 2025	Epigraph HJB · neural value	Constrained boat navigation
Boat example; collision and budget violations remain. Of the same 64 candidates, reduced training executes 64: 19 collisions, 47 budget violations. The author checkpoint executes 63: 1 collision, 9 budget violations. Events may overlap. Products MATLAB, Deep Learning Toolbox Native function `demo_safe_pinn` Local checks 13 recorded passing tests This trains and evaluates a new MATLAB network. The run guide below separately explains author-checkpoint inference using external converted weights and an existing result.mat with the comparison inputs. Reduced training: 5,000 updates `outputRoot = fullfile( ... pwd,'runs','safe_pinn_icml2025'); [result,runDir] = demo_reproductions( ... 'safe_pinn_icml2025', ... outputRoot,5000);` Entry source (.m)Equation notes (.md)Experiment & checks (.md)Run guide (.md)Original paper
Robust Koopman PIPreprint 2026	Lifted bilinear · robust PI	Lifted bilinear control
Held-out error bound fails on 25.55% of points. The fitted error bound fails on 25.55% of held-out points in the preserved evaluation. Products MATLAB Native function `demo_robust_koopman` Local checks 8 recorded passing tests Run from the repository root `demo_reproductions( ... 'robust_koopman2026');` Entry source (.m)Equation notes (.md)Experiment & checks (.md)Original paper
Bias-policy iterationAutomatica 2026	Unknown CT nonlinear · fixed data	Pendulum & two-link arm
Pendulum/arm variants; arm cost is 8.84% above local LQR. The recorded arm variant stabilizes, with cost 8.84% above the local LQR comparison. Products MATLAB; optional: Control System Toolbox Native function `demo_bias_pi` Local checks 12 recorded passing tests Default: pendulum `demo_reproductions( ... 'bias_pi_automatica2026');` The displayed 8.84% cost gap belongs to this local-multistart arm variant over five seconds. Its LQR comparison requires Control System Toolbox. Recorded arm variant `[result,runDir] = demo_reproductions( ... 'bias_pi_automatica2026', ... [],struct('example','arm'));` Entry source (.m)Equation notes (.md)Experiment & checks (.md)Run guide (.md)Original paper

CT: continuous time. DT: discrete time. Algorithmic assumptions and numerical variants are documented with each implementation.

Getting started

Your first run
in MATLAB.

Download the v0.5.3 source or clone the GitHub repository. Open its root folder in MATLAB and run the commands alongside.

The baseline uses base MATLAB with the standard JVM. Expect a learned gain close to [1, √3].

Installation & usage 中文说明

check_environment();
% List available methods
demo_reproductions();

[result, runDir] = ...
    demo_reproductions( ...
        'baseline');
result.learning.K

Expected gain1.0000 1.7321

Inspect result.evaluation for the trajectory and result.metrics for the numerical checks. runDir points to the saved experiment.

Results you can inspect.

Local tests, recorded experiments and paper comparisons answer different questions. Keep the settings with the result.

107 local checks passedMATLAB R2025b Update 6 · macOS Apple silicon

Tests include two single Adam updates. Full training and paper-figure reproduction are recorded separately. Validation record

Integral PI: learning, feedback and valueView the complete baseline figure

Integral policy iteration gain error, recorded phase trajectory and normalized cost–value decomposition.

Double integrator, Q = I, R = 1, x₀ = [1, −1]ᵀ. Contours use the analytic LQR value. Final gain error: 8.37 × 10⁻¹⁴; eight-second cost: 1.4641008. PDF · Editable figure & data

Recorded differences

These results describe specific numerical variants; the settings differ across methods.

Experiment	Observed result
Safe epigraph PINN	Of the same 64 candidates, reduced training executes 64: 19 collisions, 47 budget violations. The author checkpoint executes 63: 1 collision, 9 budget violations. Events may overlap.
Robust Koopman PI	The fitted error bound fails on 25.55% of held-out points.
Bias-PI arm variant	The recorded closed loop stabilizes; cost is 8.84% above the local LQR comparison.

Download test results · Recorded experiment index

Build on the reference

Add the next
reproducible example.

Contribute a mathematical problem, an equation map, an explicit MATLAB implementation and an experiment with an independent comparison. Preserve the configuration and historical results.

The repository includes templates and six optional research skills. Current priorities include FxT-CL-ACI, constrained control and an offline MATLAB/Simulink robot example.

Contribution guide Development roadmap

Citation & release

Credit the method.
Record the version.

Cite the original paper when using an implementation, and include the ADP-MATLAB version or commit.

Software citation metadata

Software version: 0.5.3
Public release: Published
Software license: MIT

Initial references: Frank Lewis’s software, FxT-CL-ACI, and ADP algorithms. Attribution details