Canyon Run: Continuous Reinforcement Learning
A fixed-wing aircraft learns to fly a winding, 600 m canyon. Rumoca simulates the aircraft and its inner rate controller. A PPO actor and critic learn four simultaneous, continuous commands: forward thrust and body roll, pitch and yaw rates. Start with CartPole for a smaller learning example.
The embedded view keeps the aircraft, canyon, occupancy grid and network display together. Replay trained flight shows the recorded frozen policy. Train from scratch uses the native companion server described below. On a static guide host, the replay works without that server and the training buttons are disabled.
Drag to orbit · right-drag to pan · pinch/scroll to zoom · T: training speed · F: fullscreen. N hides the actor panel and keeps the occupancy map.
Run the Live Trainer
The current live trainer uses the native CLI on Linux, Python 3.10 or newer, and NumPy. It does not require a new Rumoca compiler build if you already have a native executable. Nix is optional. From the repository root:
python3 -m pip install -r examples/reinforcement_learning/canyon_run/requirements.txt
cargo xtask repo modelica-deps ensure
npm --prefix packages/rumoca-web ci
npm --prefix packages/rumoca-web run build
cargo xtask docs build --book user
python3 examples/reinforcement_learning/canyon_run/serve.py \
--rumoca rumoca --port 8080
Open http://127.0.0.1:8080/user-guide/simulation/canyon-run.html. If using a
locally built executable, pass --rumoca target/release/rumoca. The companion
also needs the generated packages/rumoca-web/vendor/three_viewer.js bundle;
the repository web build supplies it. --guide-root and --vendor-root can
point to an existing built guide and its vendor directory.
Training starts only when requested. One run is allowed at a time. Fast mode
uses one CPU core, while Watch training in real time slows the same native
loop to its 0.05 s control period. Switch back to fast mode to speed learning
up. The default budget is 131,072 transitions with a 900-second wall-time
limit, including time spent watching. The native child has a 2 GiB RSS guard
and a 4 GiB address-space limit. Results, rollouts, episode records and
checkpoints go into a new directory under canyon-runs.
The view uses the same detailed airplane and canyon scene as the earlier
example. The occupancy panel shows the actual 8×8 observations. The network
panel shows the actor’s actual weights, transposed for the diagram’s
destination-row convention, with four continuous outputs. N toggles those
actor panel while keeping the occupancy map visible, P toggles paths, T switches training speed and F toggles fullscreen. Buttons expose the same visibility controls.
Drag to orbit, right-drag to pan, and scroll or pinch to zoom. Overview camera
looks down the course; Top-down camera shows its plan view. Follow camera
returns to the aircraft. These camera changes do not affect the simulated flight.
The HUD reports a rolling ratio of simulated seconds to wall seconds during live
training, including policy-update pauses and episode resets. Replay labels its
playback speed separately. The actor panel is reduced in the default view; the
episode-return overlay is hidden during replay. Live pose updates are fetched at the
control rate and visually interpolated between samples; episode resets snap to
the new start. The position readout follows that interpolated pose; other HUD values and
occupancy remain sampled simulator data.
Decorative rock erosion and animated surfaces are visualization; termination
uses the Modelica canyon geometry at the sample times.
Train or Evaluate Without the Viewer
python3 examples/reinforcement_learning/canyon_run/train.py \
--rumoca rumoca --output canyon-training --steps 131072 --wall 900
python3 examples/reinforcement_learning/canyon_run/evaluate_policy.py \
--rumoca rumoca --checkpoint canyon-training/checkpoint-131072.npz \
--output canyon-frozen
Both output directories must be new. Override --source-root if the CMM
library is somewhere other than target/cmm/CMM-a642c381.
To fly the included trained actor, use
--checkpoint examples/reinforcement_learning/canyon_run/actor.npz instead.
That actor-only export preserves the trained arrays; it does not contain
optimizer or critic state and is not a full training checkpoint.
Each training checkpoint includes the actor, critic, Adam state and random generator state. The native environment is not checkpointed, so resuming the entire simulator state is not supported. Starting another experiment from saved learning state must start a new episode and be recorded as a new run.
Controls and Observations
The aircraft and canyon physics remain those of the earlier guide. The agent now supplies all four commands at once; it does not select bank/climb bins:
| Command | Units | Bounds |
|---|---|---|
| Forward propeller thrust divided by mass | m/s² | 0 to 8 |
| Body roll-rate demand, p | rad/s | −4 to 4 |
| Body pitch-rate demand, q | rad/s | −2.5 to 2.5 |
| Body yaw-rate demand, r | rad/s | −1 to 1 |
Rates use body forward-left-up axes. Positive q is nose-down in those axes; the inner elevator controller converts that sign once. The thrust command scales to throttle using the aircraft mass and maximum propeller thrust. This is a fixed-wing adaptation of continuous thrust/rate control, not a quadrotor’s vertical collective-thrust model.
The actor and critic each read 86 values: normalized position and world velocity, a rotation matrix, body rates, a heading-aligned 8×8 occupancy grid and the previous applied controls. Inputs are clipped to [−5, 5] before the networks. The occupancy grid is computed from true canyon geometry. This is a control-learning example, without camera perception or state estimation.
The policy has two 64-unit tanh layers. It samples four Gaussian latent actions and maps them through tanh to the physical command bounds. The critic has the same hidden widths and one value output. PPO uses the stored latent actions for likelihood ratios; it never substitutes clipped commands or discrete action indices into that ratio. Reported entropy is the latent Gaussian entropy, rather than the entropy after the tanh transform.
The defaults are Adam at 3×10⁻⁴, discount 0.99, GAE parameter 0.95, PPO clip
0.2, 1,024-step rollouts, eight update epochs and 128-row minibatches. A KL
check can stop an update early. The core and its finite-difference controls
are in ppo.py and test_ppo.py.
Rewards, Termination and Reset
Reward combines forward progress, +0.1 per metre, with lateral-centering, height, command-rate and command-change costs. A crash costs 5 and finishing adds 5. The quadratic height cost coefficient is 0.1; the earlier 0.01 development variant finished but flew too close to the floor.
An episode ends at a sampled wall/floor/rim collision, speed below 4 m/s, or the 600 m finish. It is truncated after 1,200 steps, or 60 simulated seconds. The host captures the old episode’s state, reward and terminal flags before resetting. A true terminal state has zero value bootstrap. A time limit bootstraps from that old final observation, then stops GAE across the reset. The Modelica model does not reset itself underneath the learner.
Measured Development Result
A seed-1 run completed 131,072 transitions and 342 episodes, with 110 finishes. Its last 50 training episodes all finished. The final frozen policy, with learning and action noise disabled, flew 600.60 m in 34.35 simulated seconds. Its minimum sampled altitude was 6.50 m and minimum sampled lateral clearance was 4.95 m. The weights stayed unchanged during that flight.
The same policy also finished six predeclared development checks with ±1 m lateral/height offsets and ±0.05 rad heading offsets. Two initial model-rendering setup failures were retained; those two cases ran successfully after the signed-offset expressions were corrected. No policy tuning occurred between the six cases. These are small perturbations on the training course, not unseen-course, wind, sensor-noise or hardware generalization results.
The included replay records the simulated pose, controls, occupancy observations
and actuator commands. demo-flight.json binds it to the source training
checkpoint; actor.npz is a separately hashed actor-only export. The replay
is labeled as recorded data. Live training comes from the native companion
and is labeled separately.
Relationship to the Earlier Example
CanyonRun.Run remains the earlier five-action, fixed-throttle Modelica
Q-learner. This chapter’s PPO path uses CanyonRun.ContinuousControl.Run and
the Python learner. The old Q-learning results do not measure this new
controller. The native PPO demo also does not imply that PPO training has
been ported into the WebAssembly learner.
The airborne physics sensitivity examples remain available in
CanyonRun.Physics. PPO itself learns from reward and does not use those
physics gradients. For the continuous-control motivation, see
Kaufmann et al., Nature 2023.
The fixed-wing plant, observations, bounds and rewards here are an engineering
adaptation, not a reproduction of that quadrotor racing system.
Run the Frozen Actor in Modelica
The trained continuous actor can also run without a Python control loop.
CanyonRun.ContinuousControl.CanyonFly samples the same observation, calls
the shared NeuralNetwork functions, then holds its four
commands for 0.05 s. TrainedCanyonPolicy.mo contains only exported weights.
Previous commands are explicit state, so a newly computed action cannot feed
back into the observation used to select it.
rumoca sim examples/interactive/canyon_run/CanyonRun.mo \
--model CanyonRun.ContinuousControl.CanyonFly \
--source-root examples/reinforcement_learning/NeuralNetwork.mo \
--source-root examples/reinforcement_learning/canyon_run/TrainedCanyonPolicy.mo \
--source-root target/cmm/CMM-a642c381 --solver rk-like
A checked native frozen-actor flight reached 600.60 m at 34.35 simulated seconds. Its endpoint differed from the Python-driven flight by less than 1e-11 m. This checks one nominal inference flight, not a new training result or held-out generalization. Continuous PPO training still uses the Python implementation; its optimizer has not been ported to Modelica.
Read the Python network and PPO math
"""Small NumPy PPO core for four continuous, tanh-mapped control channels.
The stored Gaussian latent action defines the score-function likelihood.
The deterministic tanh/physical-unit mapping is part of the environment action;
ratios use the same latent action under old/new policies, never discrete bins.
"""
import math
import numpy as np
class MLP:
def __init__(self, inputs, outputs, rng, hidden=(64, 64), output_scale=1.0):
widths = (inputs, *hidden, outputs)
self.params = {}
for layer, (left, right) in enumerate(zip(widths, widths[1:])):
scale = output_scale if layer == len(widths) - 2 else 1.0
self.params[f'W{layer}'] = rng.normal(size=(left, right)) * scale / math.sqrt(left)
self.params[f'b{layer}'] = np.zeros(right, dtype=np.float64)
def forward(self, x, cache=False):
x = np.asarray(x, dtype=np.float64)
if x.ndim != 2 or not np.isfinite(x).all():
raise ValueError('network inputs must be a finite matrix')
activations = [x]
layers = len(self.params) // 2
for layer in range(layers):
x = x @ self.params[f'W{layer}'] + self.params[f'b{layer}']
if layer != layers - 1:
x = np.tanh(x)
activations.append(x)
if not np.isfinite(x).all():
raise FloatingPointError('nonfinite network output')
return (x, activations) if cache else x
def backward(self, activations, output_gradient):
gradient = np.asarray(output_gradient, dtype=np.float64)
grads = {}
for layer in reversed(range(len(self.params) // 2)):
grads[f'W{layer}'] = activations[layer].T @ gradient
grads[f'b{layer}'] = gradient.sum(axis=0)
if layer:
gradient = (gradient @ self.params[f'W{layer}'].T) * (1 - activations[layer] ** 2)
return grads
def gaussian_log_prob(latent, mean, log_std):
normalized = (latent - mean) * np.exp(-log_std)
return -0.5 * np.sum(normalized ** 2 + 2 * log_std + math.log(2 * math.pi), axis=1)
def actor_loss_gradient(actor, log_std, observations, latent, old_log_prob, advantage,
clip=0.2, entropy_coefficient=0.001):
mean, cache = actor.forward(observations, cache=True)
log_prob = gaussian_log_prob(latent, mean, log_std)
log_ratio = log_prob - old_log_prob
bounded_log_ratio = np.clip(log_ratio, -20, 20)
ratio = np.exp(bounded_log_ratio)
clipped_ratio = np.clip(ratio, 1 - clip, 1 + clip)
entropy = np.sum(log_std + 0.5 * (1 + math.log(2 * math.pi)))
loss = -np.minimum(ratio * advantage, clipped_ratio * advantage).mean() - entropy_coefficient * entropy
inactive = ((advantage > 0) & (ratio > 1 + clip)) | ((advantage < 0) & (ratio < 1 - clip))
inactive |= (log_ratio < -20) | (log_ratio > 20)
d_log_prob = np.where(inactive, 0.0, -advantage * ratio) / len(observations)
residual = latent - mean
inv_variance = np.exp(-2 * log_std)
grads = actor.backward(cache, d_log_prob[:, None] * residual * inv_variance)
grads['log_std'] = np.sum(d_log_prob[:, None] * (residual ** 2 * inv_variance - 1), axis=0)
grads['log_std'] -= entropy_coefficient
metrics = {'actor_loss': float(loss), 'entropy': float(entropy),
'approx_kl': float(np.mean((ratio - 1) - bounded_log_ratio)),
'clip_fraction': float(np.mean(np.abs(ratio - 1) > clip))}
if not math.isfinite(float(loss)) or not all(np.isfinite(g).all() for g in grads.values()):
raise FloatingPointError('nonfinite PPO actor loss/gradient')
return float(loss), grads, metrics
def critic_loss_gradient(critic, observations, returns):
prediction, cache = critic.forward(observations, cache=True)
error = prediction[:, 0] - returns
loss = 0.5 * np.mean(error ** 2)
grads = critic.backward(cache, (error / len(observations))[:, None])
if not math.isfinite(float(loss)) or not all(np.isfinite(g).all() for g in grads.values()):
raise FloatingPointError('nonfinite critic loss/gradient')
return float(loss), grads
def generalized_advantages(reward, value, next_value, terminated, truncated,
gamma=0.99, lam=0.95):
"""Bootstrap time limits from the OLD terminal observation; stop GAE at reset."""
result = np.empty_like(reward, dtype=np.float64)
running = 0.0
for index in reversed(range(len(reward))):
delta = reward[index] + gamma * (not terminated[index]) * next_value[index] - value[index]
continuation = not (terminated[index] or truncated[index])
running = delta + gamma * lam * continuation * running
result[index] = running
return result, result + value
class Adam:
def __init__(self, params, learning_rate=3e-4, max_gradient_norm=0.5):
self.params = params
self.first = {name: np.zeros_like(value) for name, value in params.items()}
self.second = {name: np.zeros_like(value) for name, value in params.items()}
self.steps = 0
self.learning_rate = learning_rate
self.max_gradient_norm = max_gradient_norm
def step(self, grads):
if grads.keys() != self.params.keys():
raise ValueError('gradient/parameter roster mismatch')
norm = math.sqrt(sum(float(np.sum(g * g)) for g in grads.values()))
if not math.isfinite(norm):
raise FloatingPointError('nonfinite gradient norm')
factor = min(1.0, self.max_gradient_norm / max(norm, 1e-12))
self.steps += 1
for name, parameter in self.params.items():
gradient = grads[name] * factor
self.first[name] *= 0.9
self.first[name] += 0.1 * gradient
self.second[name] *= 0.999
self.second[name] += 0.001 * gradient ** 2
first = self.first[name] / (1 - 0.9 ** self.steps)
second = self.second[name] / (1 - 0.999 ** self.steps)
parameter -= self.learning_rate * first / (np.sqrt(second) + 1e-8)
if not np.isfinite(parameter).all():
raise FloatingPointError('nonfinite parameter update')
return norm
def physical_action(normalized):
action = np.asarray(normalized, dtype=np.float64)
if action.shape != (4,) or not np.isfinite(action).all() or np.any(np.abs(action) > 1):
raise ValueError('action must have four finite normalized channels')
return np.array([(action[0] + 1) * 4, action[1] * 4, action[2] * 2.5, action[3]], dtype=np.float64)
Read the rollout and training loop
"""Development PPO training over the released Rumoca CLI, no compiler build."""
import argparse
from datetime import datetime, timezone
import hashlib
import json
import math
import os
from pathlib import Path
import shutil
import statistics
import time
import numpy as np
from environment import ContinuousEnv, MODEL, REPO, configure, sha
from ppo import MLP, Adam, gaussian_log_prob, actor_loss_gradient
from ppo import critic_loss_gradient, generalized_advantages
def preprocess(obs):
return np.clip(obs, -5, 5)
def episode_summary(rows):
selected = rows[-50:]
return {'episodes': len(rows), 'finishes': sum(r['cause'] == 'finish' for r in rows),
'last50_mean_distance': statistics.mean(r['distance'] for r in selected) if selected else None,
'last50_mean_return': statistics.mean(r['return'] for r in selected) if selected else None,
'max_distance': max((r['distance'] for r in rows), default=None),
'causes': {c: sum(r['cause'] == c for r in rows) for c in ['wall', 'floor', 'rim', 'slow', 'finish', 'timeout']}}
def collect(env, actor, critic, log_std, rng, count, carry, journal, total):
data = {name: [] for name in ['obs', 'latent', 'logp', 'value', 'reward', 'next_value',
'terminated', 'truncated', 'next_obs', 'applied', 'position']}
completed = []
obs, episode_return, episode_length = carry
for offset in range(count):
encoded = preprocess(obs)
mean = actor.forward(encoded[None])[0]
latent = mean + np.exp(log_std) * rng.normal(size=4)
normalized = np.tanh(latent)
logp = gaussian_log_prob(latent[None], mean[None], log_std)[0]
value = critic.forward(encoded[None])[0, 0]
next_obs, reward, terminated, truncated, info = env.step(normalized)
next_value = 0.0 if terminated else critic.forward(preprocess(next_obs)[None])[0, 0]
values = {'obs': encoded, 'latent': latent, 'logp': logp, 'value': value,
'reward': reward, 'next_value': next_value, 'terminated': terminated,
'truncated': truncated, 'next_obs': next_obs, 'applied': info['applied'],
'position': info['position']}
for name in data:
data[name].append(values[name])
episode_return += reward
episode_length += 1
if terminated or truncated:
_, _, z = info['position']
cause = ('finish' if info['finished'] else 'timeout' if truncated else
'floor' if z <= 0 else 'rim' if z > 30 else 'slow' if info['airspeed'] < 4 else 'wall')
record = {'global_step': total + offset + 1, 'return': float(episode_return),
'length': episode_length, 'distance': info['distance'], 'cause': cause,
'terminal_position': info['position'], 'sim_time': info['sim_time']}
journal.write(json.dumps(record, allow_nan=False) + '\n')
journal.flush()
completed.append(record)
# next_value above was computed BEFORE resetting the native world.
obs = env.reset()
episode_return, episode_length = 0.0, 0
else:
obs = next_obs
return {k: np.asarray(v) for k, v in data.items()}, (obs, episode_return, episode_length), completed
def optimize(data, actor, critic, log_std, actor_optimizer, critic_optimizer, rng, epochs, batch_size):
advantage, returns = generalized_advantages(data['reward'], data['value'], data['next_value'],
data['terminated'], data['truncated'])
advantage = (advantage - advantage.mean()) / max(float(advantage.std()), 1e-8)
metrics = {}
updates = 0
for epoch in range(epochs):
order = rng.permutation(len(advantage))
for start in range(0, len(order), batch_size):
indices = order[start:start + batch_size]
_, grads, metrics = actor_loss_gradient(actor, log_std, data['obs'][indices],
data['latent'][indices], data['logp'][indices], advantage[indices])
if metrics['approx_kl'] > .03:
return {**metrics, 'optimizer_steps': updates, 'early_stop_kl': True}
actor_optimizer.step(grads)
np.clip(log_std, -3, -.3, out=log_std)
value_loss, value_grads = critic_loss_gradient(critic, data['obs'][indices], returns[indices])
critic_optimizer.step(value_grads)
metrics['critic_loss'] = value_loss
updates += 1
return {**metrics, 'optimizer_steps': updates, 'early_stop_kl': False}
def checkpoint(path, actor, critic, log_std, actor_optimizer, critic_optimizer, rng, step):
arrays = {'log_std': log_std.copy()}
for prefix, network, optimizer in [('actor', actor, actor_optimizer), ('critic', critic, critic_optimizer)]:
arrays.update({prefix + '_' + name: value for name, value in network.params.items()})
arrays.update({prefix + '_adam_m_' + name: value for name, value in optimizer.first.items()})
arrays.update({prefix + '_adam_v_' + name: value for name, value in optimizer.second.items()})
arrays[prefix + '_adam_steps'] = np.array(optimizer.steps)
np.savez_compressed(path, **arrays)
path.with_suffix('.json').write_text(json.dumps({'global_step': step, 'rng_state': rng.bit_generator.state,
'checkpoint_sha256': sha(path), 'native_environment_resume': 'NOT_SUPPORTED; any continuation starts a fresh episode and is a new experiment'}, indent=2) + '\n')
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--output', type=Path, required=True)
parser.add_argument('--steps', type=int, default=32768)
parser.add_argument('--rollout', type=int, default=1024)
parser.add_argument('--epochs', type=int, default=8)
parser.add_argument('--batch', type=int, default=128)
parser.add_argument('--wall', type=int, default=1800)
parser.add_argument('--cpu', type=int, default=min(os.sched_getaffinity(0)))
parser.add_argument('--rumoca', default='rumoca')
parser.add_argument('--source-root', type=Path, default=REPO / 'target/cmm/CMM-a642c381')
args = parser.parse_args()
if min(args.steps, args.rollout, args.epochs, args.batch, args.wall) < 1:
raise ValueError('budgets must be positive')
configure(args.rumoca, args.source_root)
os.sched_setaffinity(0, {args.cpu})
args.output.mkdir(exist_ok=False)
for name in ['train.py', 'ppo.py', 'environment.py', 'stdio.py', 'test_ppo.py']:
shutil.copy2(Path(__file__).with_name(name), args.output / name)
rng = np.random.default_rng(1)
actor = MLP(86, 4, rng, output_scale=.01)
critic = MLP(86, 1, rng)
actor.params['b2'][0] = np.arctanh(-.5) # Initial mean = 2m/s² forward thrust.
log_std = np.full(4, math.log(.15))
actor_optimizer = Adam({**actor.params, 'log_std': log_std})
critic_optimizer = Adam(critic.params)
result = {'completed': False, 'method': 'continuous Gaussian latent PPO, tanh physical controls',
'seed': 1, 'numpy': np.__version__, 'python': os.sys.version,
'model_sha256': sha(MODEL),
'steps_requested': args.steps, 'rollout': args.rollout, 'epochs': args.epochs,
'batch_size': args.batch, 'wall_seconds': args.wall, 'cpu': args.cpu,
'gamma': .99, 'gae_lambda': .95, 'clip': .2, 'learning_rate': 3e-4,
'actor_hidden': [64, 64], 'critic_hidden': [64, 64], 'sample_dt': .05,
'purpose': 'ordinary continuous-controller development; not held-out or paper reproduction'}
(args.output / 'START.json').write_text(json.dumps(result, indent=2) + '\n')
env = None
started = time.monotonic()
total, rollout_number, episodes = 0, 0, []
try:
env = ContinuousEnv(args.output, args.wall, args.cpu)
(args.output / 'HELLO.json').write_text(json.dumps(env.hello, indent=2) + '\n')
carry = (env.reset(), 0.0, 0)
checkpoint(args.output / 'checkpoint-000000.npz', actor, critic, log_std,
actor_optimizer, critic_optimizer, rng, 0)
with (args.output / 'episodes.jsonl').open('w') as journal:
while total < args.steps:
count = min(args.rollout, args.steps - total)
data, carry, completed = collect(env, actor, critic, log_std, rng, count, carry, journal, total)
total += count
rollout_number += 1
episodes.extend(completed)
np.savez_compressed(args.output / f'rollout-{rollout_number:04}.npz', **data)
metrics = optimize(data, actor, critic, log_std, actor_optimizer, critic_optimizer,
rng, args.epochs, args.batch)
checkpoint(args.output / f'checkpoint-{total:06}.npz', actor, critic, log_std,
actor_optimizer, critic_optimizer, rng, total)
progress = {**episode_summary(episodes), 'global_steps': total,
'elapsed_wall': time.monotonic() - started, 'log_std': log_std.tolist(), **metrics}
(args.output / 'PROGRESS.json').write_text(json.dumps(progress, indent=2) + '\n')
print(json.dumps(progress), flush=True)
result.update(completed=True, global_steps=total, **episode_summary(episodes))
except Exception as error:
result.update(error=repr(error), global_steps=total, **episode_summary(episodes))
raise
finally:
if env is not None:
result['native_close'] = env.close()
result['elapsed_wall'] = time.monotonic() - started
result['ended_utc'] = datetime.now(timezone.utc).isoformat()
(args.output / 'RESULT.json').write_text(json.dumps(result, indent=2) + '\n')
files = [p for p in sorted(args.output.iterdir()) if p.is_file()]
(args.output / 'SHA256SUMS').write_text(''.join(sha(p) + ' ' + p.name + '\n' for p in files))
print(json.dumps(result), flush=True)
if __name__ == '__main__':
main()