DISTRIBUTION AUTOMATION

Power System Data Communication Architecture at BC Hydro

BC Hydro is a provincially-owned Crown corporation providing generation, transmission and distribution service to 1.5 million customers over 74,000 km of transmission and distribution lines. Generation consists of approximately 10,000 MW of hydro and 1,000 MW of gas-fired thermal generation. The BC Hydro system can be characterized as having a major load centre connected to remote generation over long series compensated EHV transmission lines.

The System Control Centre in Burnaby, BC is responsible for automatic generation control and reliability of the main 500 KV grid. The four Area Control Centres located throughout the province are responsible for the operation and maintenance procedures on the transmission, subtransmission and distribution networks in their respective areas. Only two generating plants are staffed on a 7x24 basis; all other facilities are either not attended at all or not attended for most of the week. There are about 150 Remote Terminal Units (RTU) and another 150 older remote supervisory and telemetry systems. All generating plants and major substations have separate RTUs and/or remote supervisory and telemetry systems reporting to both the System and Area Control Centres.

Almost all RTUs and older remote supervisory and telemetry systems connect to the control centres over dedicated telecommunication circuits on a one-on-one basis usually over telecommunication media owned by BC Hydro. BC Hydro has an extensive microwave telecommunication network for protective relaying, remote supervisory control and for operational voice requirements. There are also many power line carrier telecommunication links. Both the microwave and the power line carrier equipment are moving to digital technology. Some stations are not accessible by the telephone common carrier. Distribution feeder poletop RTUs use pointto-multi-point cellular radio communications.

All major stations contain digital protective relaying, digital fault recorders, sequence of events recorders or machine condition monitoring equipment which have remote dial-up access to enable operating, maintenance and planning engineers to access this equipment. Each type of equipment has its own proprietary protocol. Both corporate and third party telecommunication media are used. This arrangement is depicted in Figure 1 Typical Present Control and Data Connections. The preferred design practice for local and remote control projects is to use a LAN based local control scheme instead of the electromechanical scheme shown in the figure. The LAN uses a proprietary protocol. About 10 per cent of all stations have some form of LAN for control and/or metering.

Drivers for a Power System Data Communication Architecture (PSDCA)
The need for better utilization of power system facilities has increased significantly in the last decade and will continue, as the electric market becomes more complex and competitive. More real-time data from substations and generating plants is required to support SCADA and EMS in utility control centres. There has been a significant increase in interest by planning, operating, equipment and maintenance personnel for non real-time event and archived data. This wide interest demonstrated the need for a co-ordinated approach to accessing data by the data users. Hence, the development of a PSDCA got underway.

The need to develop PSDCA for BC Hydro was also driven by the expected growth in the use of Intelligent Electronic Devices (IEDs) with remote access capability. Moreover, there was a desire for:

• Better security methods for intranet applications than the simple passwords usually offered by manufacturers for their own devices
• Efficient use of telecommunication bandwidth -- a single WAN connection to a substation can support many IEDs
• Avoiding the need for additional telecommunication links
• Corporate wide access -- provide easier access to station IEDs for users who are already connected to the corporate IT WAN
• Taking advantage of network communication capability in IEDs
• Taking advantage of common standard protocols

The team built on earlier efforts which identified the needs of all corporate users for power system data. A draft PSDCA report was issued in the summer of 2000.

Objectives for the New Data Communication Architecture
The design team has set the following additional goals for the architecture:

• Be comprised of commercially available components (COTS)
• Permit immediate application at new stations and allow evolutionary steps toward the goal architecture at existing stations
• Not compromise the present reliability and availability of the protection, control and SCADA functions
• Separate access to Power Supply (generation) and T&D power system data
• Support the integration of protection and control functions as hardware and software technologies evolve
• Be able to upgrade a station to the PSDCA while dealing with legacy architecture in an economical manner
• Allow flexibility in the deployment of devices for large scale station replacement as well as small scale performance improvements

A detailed description of the proposed architecture is given below, followed by brief highlights of BC Hydro's vision for the main architecture components -- LAN, RTU, communication links, etc.

The Proposed Architecture
The proposed data communication architecture accommodates the needs of two different groups of users: control centres and power system data users. The control centres require real-time data which is secure and highly available. Power system data users do not require the data in real time nor do they have the same high availability requirements.

A major feature of the architecture is the physical separation of the control centre and power system data links. The former will continue to be over dedicated SCADA links back to the control centre. The second will be provided over the corporate WAN. This reflects the belief that the corporate WAN may not match the present reliability offered by the SCADA control system. See Figure 2 Goal Architecture Overview.

Under the proposed architecture, there is one station LAN based on an open standard protocol such as Utility Communication Architecture Version 2.0 (UCA 2.0). This station LAN will handle all real-time and non real-time data between the IEDs, RTU and other station equipment except for hardwired signalling between protective relays.

The RTU handles all data and control signals between the station and the control centres for realtime operation of the power system. Major substations and all generating stations will still have separate RTUs to System and Area Control Centres.

The SCADA protocol will be the Distributed Network Protocol Version 3. (DNP 3.0). Station IEDs include digital protection relays, metering devices, monitoring systems and dedicated controllers. With a standard LAN protocol, such as UCA 2.0, it should be possible to connect devices from different manufacturers. The HMI can be connected directly to the LAN, to a computer or to a programmable logic controller and will be the means for local control.

A Data Acquisition System (DAS) collects, processes and stores data from the station IEDs. The DAS is the common storage for most station data that are of interest to support personnel and may be located at the station or at a more centralized location.

A key feature of the goal architecture is the use of the Virtual Private Network (VPN) or similar technology to ensure secure remote access to the DAS and to station IEDs. The use of VPN will allow the secure flow of power system data over both the corporate telecommunications facilities and third party telecommunication networks. VPN uses authentication and encryption techniques to ensure security of data access.

The one LAN concept adopted in this architecture was a change from the draft recommendation that specified two LANs. The basis for this change appears later in the section titled "One LAN or Two LANs".

Station LAN
The station LAN is to handle remote access of both real-time control data and non-real-time data. The real-time data supports both the local Human Machine Interface (HMI) control and the remote SCADA including all control, alarming and analog metering functions.

Signalling between protective relays will still be hardwired. UCA's GOOSE messaging will not be implemented during the five year horizon for PSDCA.

The LAN connects IEDs, terminal servers and the Data Acquisition Systems (DAS) to a secure gateway which uses the VPN technology to allow remote access only to authorized users. The LAN is to use an open standard protocol such as UCA 2.0 over ethernet. Deterministic behaviour and performance were not considered to be issues for a modem ethernet installation.

For stations equipped with a corporate business WAN, the LAN will have a continuous connection to corporate users via the VPN gateway. Regular corporate business traffic would be blocked from the LAN by the VPN. For other stations, the LAN would connect to the corporate business WAN using VPN type technology over BC Hydro and third party telecommunication facilities. The Virtual Private Network (VPN) technology is described below.

Remote Terminal Units (RTU)
Ultimately, the RTUs will probably become a protocol converter and data processor or filter between the station LAN and the SCADA master. During the transition from the present electromechanical based designs, the RTUs would take many forms as the remote control evolved from hardwired inputs and outputs to LAN-based signals to IEDs. RTUs can be single, dual or even triple ported with each port having its own unique database. Major stations will have two RTUs, each reporting to a different control centre.

SCADA Communication Links
New RTUs will use the DNP 3.0 protocol for communication to the SCADA master while the existing RTUs will continue to use BC Hydro's version of the L&G 8979 protocol. RTUs will continue to have separate dedicated 4-wire data links operating at less than 9600 bps. Except for small distribution stations or fairly expensive telecommunication links, there is no intent to consider sharing a communication link with the PSD LAN access.

Station Intelligent Electronic Devices (LED)
The goal architecture assumes an increased and extensive usage of IEDs. IEDs will be installed under various upgrade initiatives and station projects. IEDs are capable of providing protection relaying, real-time metering, stored event logs and some control capability. RTUs, programmable logic controllers, Data Acquisition Systems and communication processors are needed to collect, filter and consolidate the data of interest from the IEDs for the various users.

This processing is necessary to prevent congestion on the station LAN and the communication links. Programmable logic controllers will also be required to handle inputs and outputs and control logic which cannot be found in the IEDs. One example is substation synchronizing. Terminal servers are necessary to connect low speed RS232 type signals to the ethernet LAN as well as perform the necessary protocol conversion to legacy equipment.

Data Acquisition System (DAS)
The goal architecture supports a Data Acquisition System at the station, regional or central level. The DAS automatically collects raw data from the IEDs and other sources at the station, processes it to some extent and makes the data available to users. The focus of the DAS is the acquisition of data rather than the long-term storage of data. The DAS could be a separate computer or a station programmable logic controller that also has control capability.

Data Storage
The choice of what data is stored at the station, at the region or at a central site is dependent on the amount of data, the need for access, the number of users, the frequency of access and the telecommunication facilities that are available. Some data, such as digital fault recorder data, would be stored in the IED, i.e. the fault recorder.

It is expected that most non-real-time data would be available in station or in central data archiving systems. For ad-hoc queries, authorized users would access to the individual IED directly over the VPN rather than accessing a central database. Long term archiving of data would be done at a central location rather than at the station.

Virtual Private Network (VPN)
A key focus of the new architecture was the secure access by authorized corporate users to the station IED power system data. The goal architecture proposed the use of Virtual Private Network (VPN) over both the corporate telecommunication facilities and over third party public networks. A description of VPN appears below.

One LAN or Two LANs
In parallel and independent from the PSDCA effort, BC Hydro was perusing a Distribution Protection/Control/ Monitoring initiative to upgrade a large number of distribution substations to a modem design in order to maximize the benefit of modem technology. This initiative used the draft Goal Architecture in its Request For Information from vendors. The draft architecture included two LANs in the station: one used for control and real-time data and the second used for non-real-time power system data. The responses were somewhat unexpected. The two LAN concept was interpreted in two different ways: one way requiring two physical ethernet ports on each IED connected to both LANs, and the other allowing one physical port in the IED and connection to the two LANs by means of data concentrators or multi-plexers. The PSDCA Team was requested to clarify the goal architecture.

The main issue, and perhaps the only real issue, from the team's point of view, was security of access. The concern was the intentional unauthorized attempt to access the station equipment or an inadvertent problem caused by someone with a legitimate reason to access only certain station IEDs. After some discussion within the team and with WAN and VPN experts, the goal architecture was revised to consist of a single station LAN with remote non-real-time access controlled by a security mechanism such as Virtual Private Network augmented with management procedures. The rest of the draft Goal Architecture remained unaltered.

Implementation Strategies
For the next five years, the following strategies will be followed:

• Station LAN will be implemented or expanded whenever it can be technically or strategically justified
• Continued use of the existing MODBUS Plus LAN until a new UCA 2.0 based LAN is viable
• Continued evolution of the protection and control design rather than freezing a particular design for several years
• Co-existence of new and existing LAN protocols, i.e. both UCA 2.0 and MODBUS Plus. Some stations will require a protocol converter to connect to the existing LAN
• Procure RTUs with UCA 2.0 capability as well as hardwired I/O
• Use DNP 3.0 for new station RTUs reporting to the control centre
• Use DNP 3.0 for all distribution feeder RTUs and switchgear communications
• The installation of WAN and DAS for substations will be initiated
• The installation of VPN connections will be expedited

Progress To Date
The following projects or initiatives are underway:

• Development of a DNP 3.0 interface for the present SCADA/EMS system continues
• Proactive rollout of digital metering using LAN-based designs
• Upgrade to digital protective relays which have metering and control capabilities
• Procurement of RTUs with LAN capability continues
• Deployment of the new Distribution Protection, Control and Monitoring initiative using the Goal Architecture is in planning stage
• Scoping and estimating of the WAN and DAS for substations has started

A network architecture that incorporates industry standards for PSDCA provides an opportunity to use existing network security systems. Some of the benefits of the Virtual Private Network (VPN) are:

• Standard off-the-shelf solutions available from manufacturers
• Proven technology for securing networks
• Provides a transparent layer of security for IEDs
• Security can be made very robust
• Single user logon can provide access to multiple IEDs

Other technologies are available for secure network access. Figure 3 Security Benefit vs Cost, gives an idea how VPN compares with other network security technologies.

The successful implementation of the PSDCA for corporate users relies heavily on the availability of the communications infrastructure. Over the next 5 years, the power system data network will be designed to make use of both corporate and third party facilities. For substations and generating stations served by the corporate business network, it would be convenient to install a VPN gateway at the station with all the IEDs on one side and the corporate network on the other. The VPN gateway provides the necessary logical separation between the two networks. Only a small number of stations could be handled in this fashion since these stations typically need to be reporting headquarters to justify a corporate business connection.

The majority of BC Hydro substations are not served by the corporate business network. These stations are typically unmanned. These stations will require the installation of networking infrastructure equipment such as communication lines, routers, hubs, etc. This network will probably be physically separate from the corporate business traffic to ensure a high level of security. A VPN gateway will not be required in these stations.

The communication links to these stations could terminate in a regional headquarters where the VPN gateway would interconnect to the corporate business network as shown in Figure 4. The communication links to these stations will include microwave, power line carrier, fibre, leased circuits, dial-up lines and possibly radio links.

Associated with the rollout of the VPN technology, the following guidelines or constraints will be addressed:

• Management of the VPN on a 7x24 basis: need to check the activity logs
• Authentication of authorized users
• Ensure "view only" ability to certain devices
• Still physically separate control data link
• No software downloads or change of protection relay settings

Major upgrade initiatives are expanding the use of IEDs and RTUs communicating to control centres using DNP 3.0. Station LANs are being developed with UCA 2.0. The VPN technology will allow secure access to station IED data by corporate users. A pragmatic and evolutionary approach is being taken in deploying the new standards and designs.

The new architecture supports both control centres and corporate users being able to access their own data in a reliable and secure fashion.

Acknowledgements
The authors wish to acknowledge the support from manufacturers and utilities that responded to surveys which the PSDCA Team sent out. The responses helped put a high level of confidence in the recommendations and the goal architecture.

Emile Struyk is a Senior Engineer in the Protection and Control Planning department of T&D Engineering, BC Hydro. Harry Lee is a Senior Engineer in the Protection and Control Mainten-ance department of Transmission and Distribution Engineering of BC Hydro.