Paul Cassingham and David Allen
In these times of increasingly stringent regulatory compliance reporting by publicly-held utilities like Entergy Services Inc. in New Orleans, Louisiana, the transmission and distribution (T&D) department requires an auditable and reliable source of historical data with no gaps. However, besides conventional database requirements, they also needed an auditable source for the database metadata; that is, the descriptions of the data contained in the database.
Moreover, the various departments within T&D often have divergent needs for SCADA numerical and alarm data. For instance, system planners need SCADA numerical data – typically hourly data organized by feeder – to determine future system construction requirements. Likewise, the maintenance staff needs near real-time data and alarms to closely monitor equipment in need of maintenance so that in the event of a malfunction or failure, they can react quickly to resolve any problems. Finally, the operations department needs near real-time data and related alarms as well as numerical data to ensure that the system is operating normally and to identify potential overloads or other system anomalies when planning system changes.
Lastly, the ability to meet all of these demands for data in a timely manner without allowing direct access to the operational Energy Management System data was of paramount importance to Entergy from an EMS security standpoint. After searching the marketplace for a solution that could satisfy all of these requirements, Entergy selected Pegasus RDSTM, a real-time data store, provided by Nobadeer Software Inc. of Longmeadow, Massachusetts.
The Challenge
In the 1990s Entergy developed tools in-house to pull different types of data out of the operational SCADA system, store it and/or share it to various departments. One in-house tool in particular has been used by hundreds of internal users for day-to-day equipment monitoring and viewing of historical data. A valuable feature of the in-house solution, however, was that it stored all of the data, all the time. Unlike solutions that require values to be selected and saved ahead of time, this system stored all the data so that it all could be examined anytime, after the fact.
Yet although the in-house developed tools were a popular success, they had limitations in terms of functionality and reliability. For example, there was no reasonable ability to systematically obtain and analyze the data programmatically outside the original user interface; the biggest reliability issue was a lack of data feed redundancy, which resulted in gaps in the data – a serious deficiency.
Looking for Answers
In 2003 Entergy made the decision to replace the in-house tools with an industrial quality application that would meet its functionality and reliability goals. Existing data historians were examined and found to be good general-purpose systems that require the user to re-define and remap their data into a new framework.
Given Entergy’s resource constraints and the frequency of database updates, the additional burden of remapping large data sets wasn’t deemed practical. Instead, a system that was tightly integrated with the existing SCADA system was needed to avoid undue maintenance and duplication of effort.
The Solution
After researching suitable products among qualified vendors in the marketplace, Entergy selected Nobadeer Software to develop a technical solution based on the desired functionality and the following set of real world objectives:
• Capture all of the data, all the time.
• Zero maintenance required.
• Keep unauthorized personnel out of the security perimeter.
• Provide programmatic access to the data store.
• Incorporate modern application development standards thereby providing stability, resiliency, and robustness in the platform.
After a thorough review and discussions of Entergy’s needs, coding was initiated in 2003. The resulting data storage and retrieval system – called Pegasus RDSTM – was put into service at the first of Entergy’s regional control centers in 2004, and was soon storing data at all four regional T&D centers and the central transmission operations center. Since then, the system has been expanded to include a number of reporting and data analysis tools.
Automation Breeds an Unforeseen Bonus
One of Entergy’s requirements was that the system be maintenance-free, an unforeseen result of which was the ability of the system to audit database changes. This feature has been enhanced by adding a set of reports that allow a user to compare the contents of any two SCADA databases that have been put into operation since the system started, providing a complete audit trail of database changes.
Entergy’s Doug Dollar, Project Lead in West Monroe, notes that this feature has saved his team valuable time. “The new system provides what SCADA Support Analysts were manually recording into spreadsheets,” said Dollar. “A review of the report provides a quick double check that all desired changes have been made correctly.” [Refer to Figure 1]
Figure 1: Pegasus provides a high integrity audit trail of all SCADA object actions via online screens and exported reports. The system clearly shows old vs. new values when a SCADA object is modified. In this report example, four analog data points had their Device Type changed from VCB to XFMR on 8/4/2007.
All the Data, All the Time
Every alarm, status, analog, analog limit, and accumulator in the database, including calculated values, is stored by the system. Because the data is captured periodically rather than by exception, the load on the EMS host is steady rather than a series of peaks and valleys that can wreak havoc on system performance. The stability of the system was proved during hurricanes Katrina and Rita when every data change and alarm was captured during these times of extreme system activity.
Included at each site is a fully redundant system that regularly collects the data from both the primary and secondary EMS machines. The system takes these two data feeds and decides later which one to use. [Refer to Figure 2] This approach has resulted in no substantive data losses since the system went into operation, with everything on-line and available for use. This data remains available for historical reference, forensics or other diagnostics long after any event occurs; i.e., the data is not subject to the “roll-off” that can sometimes happen with archival systems. The hardware was sized for five years of on-line storage, but keeping data for longer periods simply requires installing additional disk capacity and changing one system parameter.
Figure 2: SCADA measurements are tightly integrated with the EMS information, including the enabled host, limit pair(s), quality codes and system alarms. The screenshot shows a typical system failover from node WMSYSA (“System A”) to WMSYSB (“System B”), identifying data captured where neither node is enabled with the symbols “>> <<”. Also shown are the actual limit values for one of the displayed analogs.
Moreover, sampling data frequently and storing the median of the sampled data further enhances analog data reliability. This has the effect of removing erratic samples and results in a more statistically accurate representation of the measurement.
No Maintenance Required
To come as close as possible to meeting the zero-maintenance design goal, it’s necessary to keep track of devices over time, even when they have name changes. Name changes may occur, for example, if a Gas Circuit Breaker (GCB) is being replaced by a Vacuum Circuit Breaker (VCB). The solution is keeping an “invariant key” for each measurement in the system, and automatically remapping devices to the keys when a new SCADA database goes online. There is little if any staff intervention in this process.
Once a new SCADA database is put on-line, the system automatically stores the contents of the new database and all data collection and storage applications run though the update without further staff involvement, handling all necessary re-mapping on the fly. New data is then available for display, and points with database name changes automatically return an unbroken set of data across all time stored in the database. Since there are frequent name changes on the Entergy system, this feature is invaluable from both maintenance and data consistency perspectives.
In addition to the automatic mapping, the system also handles all necessary database partitioning, indexing, trimming and log deletion automatically, so that no database administration time is needed to maintain the system.
Although the system does occasionally require some maintenance by Entergy and/or Nobadeer, the zero maintenance target has very nearly been achieved, especially when considering the volume of data captured.
Preserving the Security Perimeter
Pegasus has shown it can reliably collect, store and provide all of the SCADA data to its internal users, so Entergy is now in the process of retiring all of the old data gathering applications from the EMS. In the future, the system will also provide information to all authorized T&D users outside of the operations arena as well via the typical corporate desktop. This helps Entergy meet the NERC Critical Infrastructure Protection (CIP) guidelines by providing a clear separation (just two logical connections) between critical and non-critical assets.
Access to the Data Store
In addition to a desktop client and the ability to move data from the client to other desktop applications like Microsoft Access and Excel, all data is available to other programs using a simple application programming interface (API). Bulk exports of data are supported by a dedicated Exporter application that uses this API. The Exporter allows the user to gather, for example, all megawatt readings from a region for a year with a simple command script. The API is also used by data mining applications that operate on the data store.
Application Development Standards
Application development used modern tools and techniques, with its processes programmed to take advantage of new multi-core CPU technologies. The system is very stable, scalable, and shows excellent performance with over one trillion records on-line. As the on-line database grows, the data retrieval time remains constant – a feature that required extensive effort to design and implement but that is invaluable in such a dynamic data environment.
Enabling Success
The storage, retrieval and audit capabilities of the system are field proven with over three years of on-line service. The ability to pull massive sets of reliable, complete data from the system is being increasingly recognized and used by the planning and maintenance organizations at Entergy. Some of the successes include:
• Automated audit tracking of the SCADA database changes
• Reducing the need for SCADA support personnel to be involved when T&D planning, maintenance or management organizations request archived SCADA data across a wide time range. Compared with the legacy system, which could only piece together data on a day-by-day basis, a reliable and flexible interface to data on daily, monthly or longer time frames is now readily available.
• Reliable and easy access to data has allowed various departments to:
- Start fine tuning alarm levels,
- Determine which transducers are out of calibration,
- Obtain data to support load balancing decisions that have a direct effect on reliability,
- More effectively support the system design team,
- And, identify “chattering” alarms to speed up repairs.
Next Steps
At this point development is focusing on data mining applications that take advantage of the tight integration of the system with Entergy’s SCADA system to provide reports and automated analysis of how well the T&D system is performing. Examples of data mining reports already developed and going into production in Q1-2008 include:
• “Sticky” circuit breakers (i.e., breakers not responding properly to operator commands). One click on this report brings up the list of breakers with problems, and selecting any of these brings up the OPEN/CLOSE history of that device for the month, showing each status change and any related operator command. [Refer to Figure 3]
Figure 3: The system provides numerous data mining reports. This example shows a drill down (detail) from the ‘No Response’ report, flagging a problem by showing that Device J0434 required multiple close commands before the breaker operation was completed.
• Devices changing state most often, sorted by frequency. This allows a quick drill down to see which field points require maintenance.
• Alarm limits most frequently violated, also sorted by frequency. This report can be used to check that alarm limits are appropriate, or quickly see which devices are most often operating beyond preset limits.
These reports also allow users to drag and drop columns so they can organize the data any way they want; for example: by substation; by device; by date; by area of responsibility; or any other column.
Additional sophisticated reporting providing mathematical analysis of analog values is under development. These reports include:
• Analogs that are mathematically erratic using statistical methods
• How long individual devices operate at or near typical operating limits, and by how much
Further analysis based on pattern recognition, applied to the power system data store, will be used to look for patterns, statistical advantages, and inference between variables. This area provides an exciting chance to break new ground in real-time analysis of the Entergy power system in the coming years.
The solution provided to Entergy is a low maintenance, industrial quality platform that enables:
• Ready access to a vast storage of historical data and events
• Forensic analysis of major electrical grid events
• Preventive maintenance insights and benefits
• High-integrity automated auditing
Being able to specify real world requirements, in addition to functional needs, and getting a solution designed and implemented around those requirements has made this project a success while also helping to alleviate Entergy’s resource constraints.
About the Authors
Paul Cassingham is Manager of Application Management for Entergy’s Transmission Business Unit and is responsible for EMS/SCADA and related applications that support engineering, operations and maintenance functions within Transmission. Paul holds a Bachelor of Science degree in Computer Engineering from Iowa State University and has over 27 years of experience with EMS/SCADA systems for Entergy.
David Allen has 30 years of experience in real-time data collection. After many years with Texas Instruments and more recently, Areva T&D, he is now a Senior Consulting Engineer with Nobadeer Software, Inc. of Longmeadow MA. David holds a Master’s degree in Computer Science from the University of Washington.
