Active–active failover on a plant network that was never designed for it
How we get to a 12-second RTO without asking anyone to re-cable a substation.
9 August 202611 min
The textbook answer to plant resilience is a second network. The site answer is that there is one network, it was laid in 2004, and the cost of a second one is a capital project nobody will approve for a system that has not failed yet.
What actually fails
In eleven years of incident logs the failure is almost never the cable. It is a switch that takes itself offline under load, a gateway whose certificate expired, or a server whose disk filled with the logs of the thing that filled the disk. All three are survivable on one physical network if the software stops assuming it owns the machine it runs on.
Twelve seconds
Our target is a twelve-second recovery, measured from the last accepted write. Both nodes ingest continuously; one holds the write lease and renews it every two seconds. Losing the lease is the failover, and because the standby has been ingesting all along there is nothing to catch up on — only a lease to take.
The awkward part is not the failover. It is proving to an operator, on their screen, which node is authoritative right now, without asking them to care.
- Resilience
- Networking
- Deployment