March 29, 2024

Guns or Butter or Both

by Lee Margaret Ayers, Director T&D Utilities OSIsoft and Bill Thomas, Director of Business Systems, E&C Services, AMEC
Introduction
America needs to defend itself from terrorists, we need homeland security, but at what cost? Others see that we need to rebuild our social and economic security, but how do we pay for it? The first is driven by the horror of Sept 11, the second by the honcho’s at Enron. Economists say that we cannot do both—are they right?

This paper presents a business approach that targets both concerns: Homeland and Economic Security.

Background
Over the past year we have mobilized an army of inspectors and an array of technology in domestic and foreign, ports and airports, to inhibit if not stop terrorism. We are also beefing up monitoring around “economically vital facilities” such as power plants and refineries to prevent attacks.

Quite simply we are trying to prevent the bad guys, from using bad stuff to do bad things. However when we consider the tasks involved, are they not just the mirror image of mobilizing good people to use good stuff, to do good things? Is not protection from terror the ultimate in “asset monitoring, material tracking, pattern recognition and performance monitoring”?

Examples
In the utility business we have emergency response personnel to deal with traffic accidents that knock down power poles and black out customers. That is what a terrorist wants to do, just on a bigger scale. The bad guys want to take out power plants and transmission systems to black out parts of the Nation.

In the paper business we have the TAPPI roll numbers and barcodes to track a paper roll from its planned creation through production, transportation, to consumption. This is no different than tracking baggage or containers, as they move through our transportation networks, to enable suspicious materials to be identified and dealt with — not unlike a bad roll identified in subsequence lab tests.

Current Situation
Current economics make it difficult for the utility and transportation industries to invest in business processes and technology for economic security, but they are being forced to implement other business processes and technology for national security.

Rather than look at the costs and potential loss to the organization, why not turn this new paradigm to our advantage. In the process to “prevent attacks,” national security creates a huge opportunity to simultaneously re-energize the American economy and promote economic efficiency. We must understand that preventing terror or error can be positive outcomes of smart business solutions.

Homeland Security and Control Systems
In the testimony of Joseph M. Weiss, a Control System Cyber Security Expert, before the Committee on Government Reform’s Subcommittee on Government Efficiency, Financial Management and Intergovernmental Relations, Weiss points out that a lot of security initiatives are focused on internet activities, but that “there is a growing threat that cyber attacks on operational control systems could create a crisis for which no one is prepared.” He makes several key points about the state of control system technology of which we will review three:

  • First, many control systems (SCADA, DCS and PLCs) are of legacy design and cannot handle the bandwidth of what IT Security implies. Encryption technology is still too slow to support a real-time environment.

  • Second, a majority of installed systems were never designed to be open to the world, but the need for corporate access to control system data has forced these systems to become more open. This leaves the control system at risk from outside attacks.

  • Third, control systems lack security technologies, known as Intrusion Detection Systems (IDS). IDS technology is available, but since most control systems are proprietary applications that were implemented before security was an issue, many systems would need reengineering in order to meet security initiatives.

While all these points are true, re-engineering will take considerable time and money for installed systems. There are ways, now, to achieve security that will also service economic needs of the corporation.

Weiss points out that two ways for attack on control and other automation systems are from the device side to control system, and from corporate to the control system. The main focus of this article is the secure approach by corporate users to control systems.

Economic strength and asset management
Our gut reaction to potential breaches in security is to deny control and automation systems from any possible corporate access, but this would force utilities back to the cumbersome task of manually moving data from operations to the corporation. As of 1995-1996, when the utility industry began deregulating, utilities have struggled to move data from real-time systems to the corporation.

In their paper, “Reinventing As set Management,” Ayers and Dolezilek make the point that without real-time data, true asset management cannot occur and that “economic opportunities will not be realized without innovative concepts for understanding assets or innovative technology to support automation.” Various entities within the corporation need operating data to improve operations, reduce costs and foresee outages. Competition has forced utilities to rethink their business strategy. To this end, smart utilities use their real-time and historical operating data with an aggressiveness that redefines them as an organization. Denying access to these systems would represent a huge technological step backward.

The Internet as a band-aide strategy to security
One way vendors try to mitigate potential security breaches around control systems is to provide web views to the SCADA or automation system. The problem with this is SCADA and automation vendors are not web experts, and by providing a view to real-time data, the vendor and utility overlook the problem of managing and utilizing these data for a number of applications that cannot be serviced by a web view, not to mention every automation vendor out there is doing the same thing. As a result, the utility has to deal with multiple web views that vendors provide and multiple, un-integrated databases. Implemented in this way, Internet technology is a short-lived solution to a long-term problem when infrastructure is overlooked. These solutions also do nothing to help the long-term economic security of the corporation as data become isolated to a limited number of engineering departments. The Internet is still important for security, but as an end in itself.

Leveraging Information Architecture for Economic and Homeland Security
One technology that can be harnessed for security is the PI System(PI) from OSIsoft, the leading architecture for Real-time Performance Management (RTPM) in manufacturing and utilities. The PI infrastructure gathers information from control systems and other real-time data sources and makes the data available in highly secure format to corporate users and operations. The PI System has over 360+ standard interfaces to communicate protocols from devices and SCADA systems to secure servers at the corporate level, thereby isolating control and automation systems from interrogation and attack from outside users.

PI provides an infrastructure for managing large volumes of high-speed data and Microsoft based tools to view and analyze the data in an engineering environment. Configurable displays show real-time graphical and trended views of SCADA data along with any other data users want to look at. Many articles describe this technology and reference its benefits to utilities using it to respond to dynamic changes in the power market.

Technology that gives the real-time functionality of SCADA but in a flexible, open form, and easily integrates with other business systems has changed the way utilities do business. The ability to alarm on any type of operating condition, outside of SCADA, provides the utility with many ways to predetermine outages. A northwest utility has a web page where any corporate user can sign up for alarms tags. When SCADA alarms on a transformer, the dispatcher typically cannot determine the cause for an alarm, but now, five other people (such as the specific feeder and substation engineers) are signed up for events and when alarmed, can immediately receive pages and emails from PI. These engineers can then go to their real-time views and perform ad hoc and correlation trends to determine the cause of failure and immediately target crews to prevent events. This infrastructure provides instant economic benefits, by being able to identify and respond to major outages before they occur.

One major east coast utility has used PI to reduce their O&M budgets by migrating from a traditional maintenance programs to “Just in Time” maintenance practices. By monitoring how a myriad of assets perform in relation to the amount of revenues they generate, the utility to not only reduced maintenance cost, but also was able to treat the substation as a profit center. Decisions to upgrade or maintain the existing asset configuration became a business decision based on revenues verses performance rather than a traditional maintenance practice.

Using the infrastructure already in place for real-time monitoring, corporations are beginning to expand PI for a variety of security initiatives. They are using it to monitor the underlying information of unauthorized computer users, virus alerts, and unauthorized IP traffic. The same technology used for economic benefit is now enabling homeland security.

Revisiting Web Solutions for Security
Along with their real-time infrastructure philosophy, OSIsoft realized that a myriad of proprietary Internet solutions did not meet the needs of the corporation and so developed a versatile web solution the leverages other portal infrastructures on the market. The concept of Real-time Performance Management forces us into a new paradigm of how we consider data (or the lack of it) and traditional roles in a utility. Instead of being reactive, utilities can readily evolve to be proactive.

RTPM Technology and Security
The PI System also supports a security application called IT Monitor, which can be configured to provide alerts on anomalous behavior and thus provide an effective Intrusion Detection System (IDS). It is simply the PI infrastructure with network, server and application interfaces. The same tools for alarming and monitoring of events and business decisions are already there to identify security breaches. Some of the security functionality includes:

  • Infrastructure Security

    • Provides tamper-proof data security details with an audit trail of changes by user at both data collection, usage and reporting levels

    • Point level security and trust tables

  • Network Infrastructure Security
    (Cisco Netflow, SNMP, Packet Capture)

    • Intrusion Detection – monitoring of inbound and outbound IP traffic for early identification of unauthorised network access

    • Virus Traffic – Utilizing traffic classes on routers to define policy maps identifying Code Red and other virus traffic

    • Centralized Data - Data aggregation to centralize secure information removing duplicate data and unauthorised usage while utilising existing investment in secure utilities



Figure 1 shows how a utility lays out this technology. Again, the PI System with regard to collecting data from automation systems works in the same manner. Where the IT Monitor might look at an SNMP interface, engineers are looking at substation data via the DNP 3 protocol. IT Monitor - Security Agent, to the left of the firewall can monitor local traffic, but cannot monitor infrastructure outside of the firewall. Installing IT Monitor Remote in the remote site and enabling port 5450 on the firewall will allow IT Monitor to capture traffic, response, etc. within the remote site and transport it back to the Enterprise Server. For optimal results, when performing reliability tests, IT Monitor is installed at each end point. In this way, failures can be more accurately pinpointed. End-to-end reliability data will also save time thus reducing support costs.

Regardless of how many firewalls, proxy servers, packet filters, and levels of authentication and encryption are provided; someone or something will get through. Most popular IDS platforms do not allow for long term retention of packet logs. IT Monitor can absorb immense amounts of data and reproduce it for forensic purposes if an attack is suspected. If a port is spoofed, your ability to reproduce details of traffic flows will allow you to see whom, when and where suspect traffic entered your network.

IT Monitor measures data through a variety of interfaces. Data collection for security and health monitoring includes:

  • Simple Network Management Protocol (SNMP)

    • Polling & Trapping under one minute

  • ICMP PING Tests

    • 64k Ping tests at one minutes intervals

  • Logical Port Connection Tests

    • Port connection test

  • Physical Port Packet Interception

    • Packet logging at the router/switch

  • Windows Perfmon

    • Central & Distributed performance handles captured

  • Log File Aggregation

    • Syslog, event log file aggregation



Utilizing the same graphical interface used by corporate users of PI who look at SCADA or substation data, Figure 2 shows attack of the Nimda virus on a particular server using the same technology.

Figure 3 shows how heavy CPU users or intruders are identified. Alarms can easily be linked to either source of concern.

As shown in Figures 4a and 4b, beyond security, internal computers and networks can be monitored for system health. Power users can define their own view of IT Monitor by designing screens like the one above in minutes. These views can be used as templates for creating other views and can be converted to web/thin client views for use by more casual users. Cautionary problems are dynamically coded in yellow, while pre-alarm conditions are identified in red. Alarms can then be triggered and stored. This allows for alarm history to unveil hidden problems that cannot be solved through current state.

Summary
As organizations move from fault tolerant to fault resistant to fault prevention, it also has the architecture in place to move toward more advanced security practices. Real-time Performance Management architectures like PI give the utility an infrastructure. And much like traveling to the moon spurred the solid-state technology and an economic revolution in computer technology — by making the organization more secure and economically strong, we can realize similar economic benefits. Business drivers and the need for information are forcing utilities to become real-time organizations, and so is security.

GUNS & BUTTER: Good Understanding of National Security & Best Underlying Technology Targeting Economic Resurgence


  1. Weiss, Joseph M., before the Committee on Government Reform’s Subcommittee on Government Efficiency, Financial Management and Intergovernmental Relations, U.S. House of Representatives, July 24, 2002

  2. Ayers, L., D. Dolezilek, “Reinventing Asset Management,” Proceedings of the GITA Conference XXV, March 2001.

  3. Ayers, L., J. Baranowski, “Substations Transformers: Two Birds, One Stone,” Electric Energy Magazine, July-August 2002.

  4. Ayers, L. “Managing Your Data – From the Field to the Desktop,” Conference Proceedings DA/DSM Europe, October 1999.

  5. Ayers, L., “Temporal Data — The Undiscovered Country,” Proceedings of the GITA Conference XXVI, March 2002.