Share |
SCADA System Backup and Disaster Recovery Plan
SCADA, which stands for Supervisory Control And Data Acquisition, is the graphical user interface for an automation control system and is a critical piece of infrastructure that allows the plant operator to efficiently monitor and control the automated process. So imagine the chaos when the SCADA system suddenly dies! The operators are left blind and unable to properly control their plant. This is why it is absolutely critical to have a Disaster Recovery Plan for all SCADA systems.
It is not a case of IF a system will fail, it is a case of WHEN. SCADA systems are computer based and even the best system will fail at some point due to various reasons like;
|
Your SCADA Disaster Recovery Plan should start with asking a few questions to identify what sort of system you have;
SCADA Critical Importance
The first step in setting up your SCADA disaster recovery plan is identifying the level of critical importance for your system, which is determined by how long your plant can go without your SCADA system running. If your plant is controlling a highly sensitive process like an electricity power plant or high pressure boilers, then your SCADA critical importance is high. Alternatively if your plant is controlling a low sensitivity process like a packing line or pumping station and you can afford to do without your plant running for a while then your critical importance is low.
|
Disaster Recovery Budget
Once you have identified your SCADA system’s critical importance, you can then work out what sort of budget you can afford to implement and maintain your disaster recovery plan. Disaster recovery for a SCADA system with high critical importance is generally more expensive to implement and maintain due to the extra equipment required, alternatively the low critical importance systems are cheaper to implement and maintain.
There are several types of disaster recovery plans available and your budget will go a long way to which options you are going to utilise. |
Disaster Recovery Plans
Selecting your disaster recovery plan is an obvious choice between the functionality of each option, your SCADA system requirements and your available budget. There are several options available with each one having different pros and cons that you need to carefully consider.
Full Redundancy Plan
If you have a SCADA system with high critical importance, and you absolutely cannot live without your system running, the full redundancy architecture is the option you would go for. This is where the SCADA system has multiple servers, whether the servers are virtualised or actual physical servers, one server is actively running the SCADA system while the other servers are online and waiting idle. If the active SCADA server fails for whatever reason, one of the standby servers takes over and continues running the SCADA system. This way the plant operators are still able to control the plant and they are alerted that there has been a server failure.
The redundancy architecture generally runs multiple operator workstations also to ensure the operators always have at least one workstation to access the SCADA system from.
The full redundancy plan is going to be the most expensive option available but the one that has the highest available SCADA up time for your system which is essential for highly critical or sensitive processes. If your SCADA system’s critical importance is not so high, you can start looking at some cheaper alternatives below.
Configuration Backup Plan
The cheapest and simplest disaster recovery plan is the configuration backup. This is where the SCADA configuration (SCADA programming) is manually backed up and stored somewhere like an external disk, thumb drive or in the cloud.
The downfall of the configuration backup method is that each backup must be done manually, and you are only backing up the configuration of the SCADA system, not the whole computer itself. The configuration backup method will not protect you from actual computer disasters including hardware failures and viruses. During a disaster recovery event, you will still need to have a fully working PC with the SCADA program installed. Then the configuration backups can be used to help re-establish the system.
The configuration backup method is a good start to any disaster recovery plan, but it should not be relied upon as your only backup plan. Also, be sure to have a complete set of software installation disks ready with your configuration backups should the computer itself ever fail and you need to set up another computer to replace it.
System Imaging Plan
The next step up from configuration backups is system imaging. This is a cost effective solution where a software package takes regular images of the various SCADA computers and stores them in the cloud or on backup disks. An image is a “photo” of the computer’s hard drives which can then be used to precisely restore the computer back to a time when the image was taken. There are some fantastic disk imaging software packages available like Acronis and Easeus which can easily be set up to run on a schedule and automatically backup your SCADA computers.
When your computer’s hard drive fails, the latest image can be used to restore the system on a brand new hard drive which will get your SCADA computer up and running again exactly the way it was when the image was taken. The system images can also be used for recovery from a virus by selecting an image and restoring the system back to a known good date before the virus infected the computer.
System images are a very easy and versatile method to get your SCADA system back up and running with minimal fuss.
Spare SCADA PC Plan
Having a complete spare computer is another method of disaster recovery. This is where a complete second computer is setup with the SCADA software and configuration files, the computer is commissioned so it can communicate with the plant and then it is unplugged and stored away for future use. Don’t confuse this method with a full redundancy system because the second computer in this situation is not actually plugged in and working as an online standby unit.
The theory is that when your main SCADA PC fails, the spare PC can be plugged in its place and have you up and running again fairly quickly. The spare SCADA PC method can be an effective backup solution providing that any system changes made after commissioning have been updated on the second computer. What generally tends to happen is that the system changes that a normal process or plant goes through during its life are not updated on the second computer, and so the second computer is outdated and not at the current version when you need it the most.
SCADA Virtualisation Plan
Computer virtualisation has been around now for over 20 years and has some very nice features that suit the SCADA environment perfectly.
Conventionally, an actual computer with an operating system like Microsoft Windows runs the SCADA program that the operators use to monitor and control the plant. This is currently the most common setup in the industry and probably the one that you are using now for your plant’s SCADA system.
Virtualisation is the future of computing and it is fast becoming the way Operations Technology (OT) is run. Virtualisation is where an operating system runs on an actual physical computer in a virtualised, or hypothetical, environment with special software. Virtualised operating systems (called a machines) can run concurrently on a single physical computer and can be built with various types of operating systems (Windows Server versions, Windows Desktop versions, Linux versions etc).
The SCADA Virtualisation method is a virtual machine that is running the SCADA program. It looks and feels exactly like a ‘normal’ SCADA computer would, it is just in a virtualised environment. One of the great benefits with any virtual machine is that it is not reliant on any particular type of computer or specific hardware. Basically, as long as you have a computer with the virtualisation software loaded, you can run the virtual machine on that computer. There is no need to worry about drivers, updates or various other software packages like Word, Excel, Outlook etc.
Disaster recovery with the SCADA Virtualisation method is as simple as getting another computer, load the virtualisation software, and run the SCADA virtual machine. DONE!
You can also do some very fancy stuff with virtual machines including load sharing and redundancy protection which are great for large scale SCADA systems. We use vmware virtualisation in our business for the many and various software programs we have to run for automation systems. The Virtual machine snapshot feature built into vmware is similar to the System Imaging method we discussed earlier. We often use snapshots when we are testing a new piece of software so if anything goes wrong, we can quickly get back to before we installed the software.
|
The best part of these various disaster recovery methods we have discussed above is that you can use several methods together to create a tailored solution that suits your needs. You might want to have a SCADA Virtualisation plan with the virtualisation software pre-loaded on a Spare SCADA PC, or you might like a Full Redundancy system with System Images backing up all your SCADA servers and workstations to the cloud for off-site disaster recovery protection.
In any case, we cannot emphasis enough how important it is to back up your SCADA system, it's absolutely critical. The SCADA Disaster Recovery Plan is easily the most over looked part of any plant's maintenance routine, and failure to have an effective plan in place will have disastrous repercussions down the track when the entire plant is brought to its knees because the SCADA system has died.
If you would like assistance with your SCADA Disaster Recovery Plan,
give the team at Automated Electrics a call.
One of our SCADA experts will be able to guide you through the process and ensure you’re protected when the SCADA gremlin hits your system!
give the team at Automated Electrics a call.
One of our SCADA experts will be able to guide you through the process and ensure you’re protected when the SCADA gremlin hits your system!