Managers responsible for SCADA systems often find that their older systems are not able to meet new requirements, yet , due to a lack of planning and adequate budget, they are unable to replace them. Whats needed is a way to overcome the immediate obstacles to expansion while setting a migration path that leads to the eventual replacement of the entire system.
Stabilizing An Aging SCADA System
On many older SCADA systems, there comes a time when the cost of maintenance becomes prohibitively high. Hardware breaks routinely, and the spares inventory dwindles to non-existent for some components. Some system components are no longer manufactured, and even if the original vendor is still in business, service is often no longer available or has become unacceptable.
Usually, by the time the maintenance problems get this severe the system has been in service long enough to justify its replacement. But what can one do when its replacement has not been adequately planned and budgeted?
Finding spare parts and other sources of services are not as difficult as it may seem. The original vender, if it is still in business, is an obvious first step. If the vendor is no longer in business, a former employee may be of help. Other users of the same equipment may have service suggestions or they may have taken their equipment from service. Decommissioned systems may be an abundant source of spare components.
This can be the simplest way to keep an old system working. It doesnt involve any redesign, and the parts can often be obtained at little or no cost. Even if an alternative source for service is not found, a larger inventory of spare parts should provide the ability to withstand longer repair times and the higher non-repairable rates associated with older equipment. Finally, because it does not require a commitment to in any particular direction, it allows time to plan a lasting solution.
Keeping the system working is only part of the problem. A long-term solution calls for the ability
to expand the system as needs expand. That may involve upgrading or replacing components.
Upgrading Remote Stations
Before an upgrade can be made to remote terminal units (RTUs), their limitations must be identified. As well, requirements for the system over the next few years should be addressed. RTUs are usually limited to a fixed number of points or a fixed number of I/O cards slots. Often, space is allowed for the field additions of I/O cards or I/O terminations up to some predetermined limit.
Some RTUs can be expanded by adding another card chassis, but unless this was specifically part of the expansion plan, there are a number difficulties that may be encountered. The power supply may not handle the additional load. And even if there is room for the additional card chassis, there may not be room for the I/O terminations. These difficulties make field upgrade, at this point, a questionable proposition.
When a situation calls for RTUs to be expanded in a way that requires more than simply adding I/O cards or terminating wires, its probably not a wise idea to attempt rebuilding them in the field. Instead, the spares inventory can be used to build a RTU that meets the needed requirements and it can be used to replace the RTU in question (the old RTUs parts can then be added to the spares inventory).
If a number of RTUs are being expanded, it may be useful to purchase one new RTU configured as needed to meet the requirements of the largest site. Parts from the replaced RTU, along with parts that may need to be purchased, can be used to build the next RTU. This process can be continued until all the RTUs have been replaced. It may be possible to use some of the larger old RTUs at the smaller sites with little or no modification.
Some RTUs cannot be expanded. Once all the points are used, further expansion can only be
accommodated by adding another RTU or replacing the existing RTU with a new one.
Replacing Remote Stations
If remotes have to be replaced because spare parts are not available, some RTUs could be decommissioned and replaced with new units. The advantages of partial decommissioning are clear. First, it solves maintenance problems by removing hard to maintain equipment, while the spare parts produced will give more time to deal with the remaining old remotes. If only a few RTUs are replaced, any problems with them will be known before other new units are added. Once the process of replacing remotes has been fine tuned it can be completed as quickly a resources will allow. This is an approach that should be considered even if there is an adequate budget to replace all the RTUs at once.
This solution is not without its problems, though. The compatibility of the existing master station unit (MTU) will have to be considered. It may be necessary to have custom hardware or software designed in order for the new RTUs to properly interface with the existing system. And, the replacement RTUs should be capable of providing any additional functionality that may be required during their lifetime (about 10 or more years).
Whether RTUs are being added or replaced, the original vendor is usually a good place to start. Its there that solutions about compatibility will most easily be found.
Alternative sources can also be found. Original vendors employees, who have started small businesses to service these systems, can be of help. Some of these people will have gone on to major competing vendors, bringing their technical knowledge with them. And, of course, there are the venders that have no connection with the original vendor other than to compete with similar products.
Many of these venders, no doubt, have complementary systems or system components that will interface with older systems. And some of them have probably assisted others in doing upgrades on similar equipment. It is not uncommon to find one or two second sources for some of the more popular equipment. This is especially so true for RTUs where competing vendors will often offer their own RTUs with several protocols.
The experience gained by such an investigative exercise will give companies considering a change
the confidence to tackle other parts of the system such as the MTU.
Understanding Master Station Expansion Limitations
Its highly likely that most systems will reach a time when they will no longer accommodate additional points. There are a number of reasons this can happen including processor speed, memory, addressing limitations and/or communications bandwidth.
The most obvious sign that maximum capacity has been reached is the inability to add another point. This is usually caused by addressing limitations within the software. On many older systems, the database was placed in a common region that had to be mapped into the address space of each program that had to access the database. On 16-bit machines this generally presented a limitation of 32 kilobytes for both programs and database. By todays standards that is extremely small.
Another symptom of maximum MTU capacity is the observation of sluggish performance. Points and displays are not being updated as often as they once were, or more importantly, as often as is required for safe and efficient operation. The more points that are added, the worse the performance gets until it reaches a point of being unacceptable. This can happen without ever reaching a definite limit where not one more point can be added. But for all practical purposes, the system cannot be expanded further.
There are three common causes for sluggish performance. The problem could be that the system is approaching full utilization of the central processor unit (CPU). Sluggish performance will actually be observed long before the system reaches 100 per cent CPU utilization, and performance will continue to deteriorate as more points are added. Even if the CPU reaches 100 per cent utilization, it will most likely continue to operate, albeit very slowly.
Another cause of sluggish performance is full utilization of memory. This is common in virtual memory machines that swap programs to and from the hard disk to use memory more efficiently. Sophisticated algorithms keep track of which code sections (or pages) have not been used recently and mark those pages to be swapped out to make room for programs that are ready to run. The goal is to always have the right code in memory before it is needed. This requires enough memory so that commonly used programs are hardly ever swapped. But as points are added, they will be fixed in memory (i.e. not allowed to be swapped). This means there is less memory available for swapping. Eventually commonly used programs are being swapped so heavily that, although there is sufficient processing power available, the right code is rarely in memory when it is needed.
Another more severe manifestation of this problem, known as thrashing, occurs when two or more programs that are dependent on each other are being swapped out to make room for each other. This can lead to a lockup condition where all or part of the system stops functioning.
The third cause for sluggish performance is full utilization of the communication channels. If this is the case, the only performance problem should be that the points take too long to update. Response to operator input should not be affected unless the response requires use of the communications line.
A cursory look at the system will not always reveal the cause of the sluggish performance. To further complicate the problem, these three causes often act in concert with another. Suppose it is determined the communications line is the culprit. Increasing communications capacity will present a heavier demand on the CPU and memory resources, possibly shifting the cause of the sluggishness while having little or no effect on overall performance.
These are not the only factors that can limit master station expansion. The task of implementing
the changes can often present the most difficult obstacle to expansion. This is especially so on
older systems and those that are not of the open-architecture variety. Although they may seem to
very well documented, making changes without the assistance of the original vendor is often
impossible due to a lack of source code and adequate development facilities.
Upgrading And The Master Station
Master station upgrades fall into three basic categories: adding more remotes and points, adding new applications programs, adding or changing drivers to communicate with new types of remotes. The expense of doing such upgrades can be substantial, so the decision to upgrade the MTU must be carefully considered.
Adding more of the same type of RTUs and points would seem to be a simple task, and it is. For no less than 20 years there has been systems on the market for which adding points is a simple configuration exercise. However, if the limitations to expansion are unknown, considerable resources could be expended only to find that the system is unable to withstand the planned expansion.
For this reason, before attempting to upgrade the MTU, its best to perform a thorough analysis of the system. This will ensure the upgrade is feasible and the investment is not wasted on a system whose remaining useful life is quite limited.
The analysis should review the future requirements of the MTU. It must be determined how much will be expected from the system in order to determine if expansion is feasible or even possible. This includes more than just the number of points that will be monitored. It must also include future functionality.
The various barriers to expanding the system to meet future requirements should also be evaluated. An investment will be wasted if the upgrade prevents or limits expansion.
Cost estimates should be gained for not only the expansion at hand, but also for future expansion.
To complete these steps its likely that outside, experienced help will be needed. The original vendor may be able to assist in this process, but this would limit the number of choices available during an upgrade. An independent third party can put the various solutions into perspective.
There are a number of such firms, large and small, which could carry out such a study and recommend solutions. Many of them which specialize in this type of work also offer other SCADA related services such as specification preparation and project management.
These firms should be chosen with care. They must possess the intimate knowledge of computer systems and communications needed to assist in this situation. The best choice is a firm that can demonstrate SCADA system experience.
Adding new applications programs is a bit more difficult, and there is still the problem of ensuring the ultimate expansion can be accomplished before beginning. Even if a system supports a high level language interface, its likely it will need the help of someone very experienced in a particular system. Experience is needed with not only the internals of the system, but also with system development tools such as compiler and debugger. Again, the original vendor may be able to provide assistance in this area.
Adding new types of RTUs can be the most difficult and risky type of addition. Unless its possible to obtain new RTUs that support the existing protocol, a driver for the new protocol will be needed. On older systems this requires a high degree of specialization, but on new systems it can as simple as purchasing a driver usually at a reasonable cost installing it and going through a fairly simple configuration exercise. This sort of upgrade does not require a programmer, but someone with a good technical background (such as an engineer or technician) is usually required.
In other cases it may be necessary to add RTUs with a different protocol and mix them with existing RTUs on the same communications circuits. Usually, as long as the protocols are of the master/slave type (i.e., remotes are quiescent until the master addresses them), any number of protocols can be mixed on the same communications circuit. The real problem is getting the drives to co-operate so that they scan one at time, but there are several ways to accomplish this.
The most obvious solution is to develop drivers that co-operate by being aware of one another. This can be the best long-term solution, but it is expensive and non-standard.
On open systems, RTUs can usually be placed on and off scan and demand-scanned by means of software accessible commands. If this is available, the co-ordination can be placed on top of the drivers in a program. Quite possibly, the program will be a form of scripting language or other specialized vendor specific language meant for non-programmers.
If all else fails, a fall-back switch can be used. Fall-back switches can be purchased from a variety of vendors and can be programmed to work in a number of different configurations. One particularly effective way of using a fall-back switch is to program the device so the primary drive usually has control of the communications line.
The secondary driver can grab the line even in the middle of a message by the primary. When the secondary drops the line, it falls back to the primary. Since the secondary driver will cause frequent interruptions to the normal polling of the primary, two things must be done.
First, the primary driver must be programmed to tolerated the interruptions. In other words, alarming of communications failures must be disabled.
Then the secondary must be programmed with gaps during its polling to allow the primary time to obtain an adequate number of successful replies from each RTU.
This should not be considered a long term solution, but can be quite effective for mixing protocols
temporarily while the older RTUs are being replaced.
Replacing The Master Station
Despite the best intentions to avoid replacement of the system, sometimes it is the best choice. The entire system or just the MTU may need replacing, both of which represent major undertakings. Without adequate planning and preparation for the arrival of the new system, serious disruptions to a companys operations can occur.
Probably the most effective measure that can be taken is to make sure the existing MTU remains in service until the new MTU is fully in service. This is easier said than done. Having two MTUs fully functional at the same times usually requires that both have access to, and control of, all the RTUs.
There are, however, a few ways to do this.
A simple and effective way of handling this problem on a small system is to allow the old MTUs access to the RTUs during most of each day. Then switch the communications lines over to the new MTU when needed for testing (This may not be adequate for a system with applications programs that require continuous updates from the RTUs in order to function properly).
In such cases both MTUs can be placed on the communications line and programmed to take turns polling RTUs. This solution has its problems in that some way of co-ordinating the MTUs must be found. A fall-back switch could work well in this case, but on a very large system many switches would have to be put in to service simultaneously. And, if the communication scheme has changed radically, as is often the case when a system is being replaced, this may not be practical.
A way to get around such problems is to program the new MTU to emulate the communications of the old RTUs. Thus, as old RTUs are replaced with new ones, the old MTU is directed to the new MTU for all communications with that RTU. The new MTU can be programmed to respond to poll and command messages from the old MTU so that the old unit does not even realize that the old RTU has been replaced. The only reprogramming needed on the old MTU is to remove each RTU from its normal communications line and associate it with the line on which emulation is being done. This is usually nothing more than a simple configuration exercise, even on some of the oldest MTUs still in use.
Another possibility is to employ a front-end processor in the new system. It would be
programmed to communicate with all RTUs, new and old. It could also be programmed to
interface to both masters simultaneously in their native protocols. By the fact that a front-end
processor can provide a shared communications path to the RTUs, other benefits are derived. For
example, it is not uncommon today to find a variety of maintenance and other specialized
functions built into the protocols of RTUs. A front end processor can allow access to those
functions over existing communications circuits without burdening the master with knowledge of
the functions. In the long run, this separations of duties makes replacing sub-systems one at a time
all the more practical.
SCADA Migration Process
SCADA migration should be thought of as a process, not a goal. In other words, as a new system is being installed, its future should be considered. What can be done to make its replacement less of an ordeal? By designing the sub-system replacement in mind, there is a significant advantage of reducing risk by turning a large project into a series of smaller projects spread over an extended time. The alternative is to replace the system as one large high risk, high profile project whose commissioning must, by necessity, be compressed into an unreasonably short time frame.
By adhering to a few basic principles some clear advantages can be gained. First, research to find out which SCADA suppliers are the most successful in the marketplace. Make sure the products meet your needs, but all things being equal, pick a leader. Leaders stay in business longer. They are more likely to continue to enhance their product lines.
In designing the system, keep the applications programs separate from the SCADA functions. Require that the vendor supply a high-level language interface to the SCADA database, both real-time and historical data. Also require a SQL interface, or at least some method of exporting data to a third party relational object-oriented database, and use it to do reports on data analysis functions and long-term data storage.
This division of responsibilities can even be carried into the SCADA system itself. Consider
separating the SCADA functions into data gathering functions and man-machine interface. Putting
the data gathering functions into a client server front-end processor further strengthens this
separation. It then becomes reasonable to at least upgrade if not replace these modules
independently from one another.
Joseph Guidry is the founder of Automation Technology Consultants based in Metarie, Louisiana. (504) 887-3017 fax: (504) 455-5210. This paper was originally presented at the 1996 AM/FM International Conference XIX in Seattle (303) 337-0513.