Learning by Crashing Is Not an Option

Reinforcement learning (RL) agents learn optimal behavior by trial and error — exploring actions, observing outcomes, and updating their policies. In simulation, exploration is free: a virtual robot can crash millions of times to learn to walk. In physical environments (real robots, power grids, chemical plants, autonomous vehicles), exploration failures have real consequences — equipment damage, safety hazards, and economic loss. No RL algorithm currently provides provable safety guarantees during the exploration phase: the agent must sometimes try actions whose outcomes it cannot predict, and some of those actions may violate safety constraints. The tension between exploration (necessary for learning) and safety (necessary for deployment) is fundamentally unresolved.

RL has achieved superhuman performance in games and simulation but has been deployed in only a handful of physical systems (data center cooling, chip design, plasma control in fusion reactors) precisely because of the safety-during-exploration problem. Domains that could benefit enormously from adaptive, learning control — surgical robotics, prosthetic limb control, building energy management, autonomous driving — require continuous adaptation to changing conditions but cannot tolerate the exploration failures that current RL algorithms need for learning. Bridging this gap would unlock adaptive autonomy for systems where fixed control policies are suboptimal but unconstrained learning is unacceptable.

Constrained RL formulations (constrained MDPs) add safety constraints to the optimization objective but enforce them only in expectation or on average, not on every individual trajectory — meaning unsafe episodes still occur during learning. Reward shaping and barrier functions can guide the agent away from unsafe states but require knowledge of the safety boundary that may not be available a priori. Sim-to-real transfer trains the agent entirely in simulation and deploys the learned policy without further exploration, but sim-to-real transfer introduces a "reality gap" where the policy fails on dynamics the simulator didn't capture. Safe Bayesian optimization approaches (Gaussian process-based) maintain probabilistic safety bounds but scale poorly to high-dimensional state-action spaces typical of real robotic systems. Shielding approaches (runtime safety monitors) can override unsafe actions but reduce the effective exploration space, potentially preventing the agent from learning optimal behavior. The fundamental problem is that safety constraints define regions of state space that must never be entered, but optimal behavior often lies near the boundary of these regions, requiring precise exploration that current methods cannot guarantee.

A theoretical framework that formally characterizes the minimum safety-compatible exploration needed to learn an optimal policy within a given state-action space — quantifying the fundamental tradeoff rather than treating safety and exploration as independent objectives. Algorithms that can provably learn from informative but safe trajectories near constraint boundaries without crossing them, perhaps using control-theoretic barrier certificates that adapt as the agent's model improves. Transfer learning approaches that can rigorously quantify the residual uncertainty when transferring from simulation to reality, enabling targeted minimal exploration in the real environment.

A student team could implement and compare three safe RL algorithms (constrained policy optimization, control barrier function-augmented RL, and Gaussian process-based safe exploration) on a standard robot learning benchmark (OpenAI Safety Gym or Safety-Gymnasium), measuring the tradeoff between constraint violations during learning and final policy quality. Alternatively, a team could develop a Lyapunov-based safety certificate for a simple physical system (inverted pendulum, cart-pole) and demonstrate safe learning on real hardware. Relevant disciplines include control theory, machine learning, robotics, and applied mathematics.