A boiler drum is not a simple container whose water level responds neatly to a valve. It is a dynamic system in which pressure, steam demand, feedwater flow, transport delays, and the shrink-and-swell behaviour of the water–steam mixture interact. A controller must respond to what it sees now while accounting for what the process has been doing over time.
That makes boiler drum-level control an interesting test for a modern idea in artificial intelligence: can a learning-based controller use a sequence of observations—not just a single snapshot—to make decisions in a nonlinear environment?
“The challenge is not merely to make an intelligent controller act. It is to understand when its learning is reliable, when it is failing, and what the evidence actually proves.”
This research explores sequence-aware reinforcement learning for nonlinear boiler drum-level control using a Transformer-based architecture. Rather than treating the boiler as a static input–output mapping, the simulation brought together interacting process elements: mass balance, drum-level shrink and swell, water and steam properties, feedwater transport delay, and actuator and sensor dynamics.
The model was designed as a coupled simulation with ten components. Water and steam properties were represented using IAPWS formulations, while shrink–swell behaviour followed an Åström–Bell-style model. These details matter because a controller trained on an oversimplified process may learn a policy that looks successful in the model but does not cope with the dynamics that make the real process difficult.
Explore the boiler simulation
Open the interactive simulation associated with this research.
The control problem can be formulated as Markovian when the state contains all variables needed to describe the process dynamics. The research adds a history-aware, non-Markovian layer to the learned decision process: a Transformer sequence encoder considers the recent trajectory of observations rather than relying only on the current snapshot. Combined with the Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement-learning algorithm, a history of twenty observation steps provides temporal context. The encoder produces a shared representation for an actor, which proposes control actions, and twin critics, which estimate their value.
The implementation used Python, PyTorch, and Gymnasium. The resulting model contained 403,267 trainable parameters. These figures describe the implemented research system; they are not, by themselves, evidence of control quality. A sophisticated architecture still has to demonstrate stable performance against a fair and reproducible benchmark.
The training history was more revealing than a simple success story. Around episode 340, the model reached a genuine peak checkpoint. At that point, the reported mean evaluation return was 473.530, with mean absolute level-tracking error of approximately 69 mm over the evaluation episodes.
But the improvement did not hold. From around episode 360 onward, the policy underwent a sharp, sustained collapse. Episodes shortened to roughly 26 steps on average, compared with a 500-step maximum, consistent with early termination caused by simulated drum-level safety trips.
This pattern is an important reminder about reinforcement learning: a promising checkpoint is not the same as a stable final policy. Training can pass through a period of apparently strong performance and then deteriorate. Selecting a checkpoint, understanding the failure mode, and reproducing the result are part of the research—not administrative details after the interesting work is finished.
“A peak result is evidence of potential. A dependable result requires stability, diagnosis, and repeatable evaluation.”
The investigation did not establish a definitive root cause for the collapse. One complication was that the action standard deviation during training was dominated by injected exploration noise (σ = 0.15). That made it an unreliable signal for deciding whether the actor itself had become frozen or degenerate.
A dedicated diagnostic script, diagnose_actor_collapse.py , was developed to help separate possible actor behaviour from issues elsewhere in the learning system. The diagnosis of encoder-versus-actor contributions remained ongoing in the report, so the failure mechanism should not be presented as settled.
The reward function was revised to include a survival bonus and a termination penalty, addressing a narrower incentive problem associated with crashes. However, later training showed instability in mean-squared-error behaviour dominated by outliers. The observation normaliser was changed from an adaptive running normaliser to a fixed-range absolute normaliser, but a full retraining campaign to validate the effect had not yet been completed.
These are meaningful engineering steps, but they are not conclusive fixes. The responsible conclusion is that the work identified several plausible areas to investigate while leaving the overall cause of policy collapse unresolved.
PID control is the essential reference point for a practical control study. Yet a fair comparison requires a stable, clearly selected learned policy and a consistent evaluation protocol. The report identifies the formal PID benchmark against a stable final Transformer-TD3 checkpoint as pending.
For that reason, the available peak-checkpoint figures should not be turned into a claim that Transformer-TD3 has definitively outperformed PID. The meaningful comparison still needs to be completed under comparable simulated operating conditions, with performance measures and safety-related outcomes reported transparently.
The next steps follow directly from the unresolved findings: complete the actor-collapse diagnosis; test whether the fixed-range normaliser improves learning through full retraining; investigate critic and reward instability; and establish a stable checkpoint before carrying out the PID benchmark.
Further experiments can examine the effect of history length, encoder depth, attention heads, and TD3 policy delay. Multiple random seeds would help distinguish repeatable behaviour from a fortunate run. An alternative such as Soft Actor-Critic could also be evaluated, while optimisation of IAPWS property lookups may help reduce simulation overhead.
Moving beyond simulation would require a separate and much more demanding pathway: system identification informed by plant data, validation against relevant operating behaviour, independent safety analysis, and carefully controlled testing. A simulated policy is not a ready-to-deploy plant controller simply because its reward curve once looked encouraging.
Research in intelligent control is often described in terms of algorithms, neural networks, and performance metrics. But the deeper discipline is experimental honesty: knowing what has been demonstrated, what remains uncertain, and what evidence would change the conclusion.
A Transformer may help a reinforcement-learning agent interpret temporal patterns. TD3 may provide a useful learning framework. Neither removes the need for careful modelling, robust evaluation, baseline comparison, and a clear account of failure.
The goal is not to make AI appear infallible. It is to build systems whose capabilities can be measured, whose limitations can be explained, and whose use is justified by evidence. In safety-critical engineering, that standard is not a footnote to innovation. It is the foundation of it.
