Reinforcement Learning · Experiment: January 2026 · 6 min read
Degenerate Policy in a Highway RL Agent
By Obada Alsehli
A PPO agent learned to favor slow driving rather than the intended behavior. This write-up examines the reported training results and reward setup.
200,000 timesteps
The project README describes PPO training in highway-env for 200,000 timesteps on an NVIDIA GeForce RTX 3050 Laptop GPU, taking about 2.5 hours.
While overall reward increased during training, examining the agent's behavior revealed that accumulated reward did not reflect the intended driving objective.
100k checkpoint
The repository labels the 100k checkpoint its best highway result, but also reports similar mean reward at 100k and 200k. Those figures do not support a broad claim that more training caused a large performance collapse.
The more useful diagnostic in the README is the recorded action distribution: SLOWER dominates at the 100k checkpoint, while FASTER and lane-change actions are absent from that reported episode summary. At 200k the README reports a different mix, mostly SLOWER with some IDLE, at a similar mean reward.
Collision penalty and reward structure
The repository attributes slow driving to a mismatch between the intended speed-and-safety objective and the behavior rewarded by the documented setup. The observed policy favored repeated slowing and did not demonstrate the intended combination of speed, lane changes, and collision avoidance.
This is evidence of a policy-objective mismatch in the described simulation. The public record does not independently establish that one reward term alone caused it, so the mechanism should be read as the repository's diagnosis rather than a reproduced causal result.
Model size and proposed reward changes
The README describes a shared actor-critic network with about 17,000 trainable parameters. That size alone does not rule out capacity or optimization effects; the recorded evidence supports the narrower observation that the learned action pattern did not match the task's stated objective.
The repository proposes changes to reward design, including stronger collision penalties, non-linear speed incentives, distance-based progress rewards, and a stronger penalty for low speed. These are proposals, not demonstrated fixes in the reported run.
Recommended next checks
The project discussion recommends inspecting action distributions during evaluation rather than relying on aggregate reward curves alone.
The repository's future-work list proposes reward redesign and extra evaluation metrics, such as overtakes per episode, average velocity, lane utilization and time-to-collision. A simple heuristic baseline could make the intended behavior easier to compare, but the public project record does not show that comparison being run.
Cite this work
Obada Alsehli··n.d.
@article{alsehliwritingdegeneratepolicyinahighwayrlagent,
author = {Alsehli, Obada},
title = {{Degenerate Policy in a Highway RL Agent}},
howpublished = {\url{https://obadaalsehli.com/writing/degenerate-policy-in-a-highway-rl-agent}}
}Academic attribution & reproducible reference format
BIBTEX