March 29, 2024

Leveraging Low-Cost Cellular to Connect Grid Assets in Texas Hill Country

by Joby Wieser, CTEC and Bryan Seal, Itron

Central Texas Electric Cooperative (CTEC) owns and operates electric distribution services in 11 counties in central Texas, around the cities of Fredericksburg, Mason, and Llano. Its service territory is roughly 4,500 square miles of classic Texas hill country – remote, expansive and pocked by springs, streams, and rivers. The unique rolling topography makes for ideal outdoor activities and a booming draw for retirees. But it also makes it tough to reliably provide network connectivity to isolated and scattered distribution substations, especially during extreme weather events, including thunderstorms, hailstorms and tornadoes – all of which occur with some frequency in the area.

Like all electric utilities, CTEC’s distribution substations contain vital data collection and control elements that CTEC needs to communicate with in order to enhance operational awareness and control. And like most other utilities, CTEC has utilized a variety of technologies over the years to get the job done.

The Lay of the Land
In 1994, CTEC enabled communications with some of its distribution substations with its first SCADA system, which used a licensed 2400bps radio network operated in a point-to-multi-point configuration. With a single 2400-baud channel shared among its various substations, the utility could only take advantage of certain SCADA functions. On average, this system would allow data refresh from all substations only every 15 minutes. In recent years, CTEC expanded its SCADA system into other distribution substations utilizing a number of different communications technologies, including phone line (POTS), DSL, radio internet providers (ISPs) and local fiber optic cable.

In 1997, CTEC deployed a one-way communications power line carrier (PLC) automated meter reading (AMR) system, followed later by the deployment of a two-way communications PLC AMR system. Both systems allowed for the retrieval of customer billing and operational data. While the one-way system utilized POTS-based communications to retrieve data from substation receivers, the two-way system allowed Internet Protocol (IP) backhaul from the substation. As with their SCADA system, CTEC utilized a hybrid approach to provide data backhaul communications from the AMR receiver located in their substation to head end software back at their headquarters in Fredericksburg.

Along the way, CTEC found that each of the communications technologies had significant downsides:

  • Dial-up, adequate for one-way AMR communications, was leveraged to provide two-way connectivity for the other AMR system by utilizing line sharing and other devices. This arrangement resulted in long distance calls to some of CTEC’s substations.
  • Digital Subscriber Line (DSL) service involved wired and wireless technologies from different service providers and required frequent troubleshooting.
  • Recurring costs often totaled more than $150 per month per substation.

An additional hidden cost directly impacted CTEC personnel and resources. Joby Wieser, CTEC’s Technical Services Specialist, estimates that he spent at least one day per week troubleshooting communications infrastructure in the field. Though mostly simple fixes – rebooting, settings, and configuration – his trips to substations would cover long distances, up to 60 miles each way. “A real headache,” he noted.


Texas Hill Country
 

Like other electricity cooperatives, CTEC faced a seemingly untenable dilemma. On one hand, it needed to modernize its connectivity to rural assets. On the other hand, it had a limited budget. Given the long distances between assets in this rural Texas landscape, fiber was too expensive to install to all but one substation near CTEC’s headquarters. Similarly, building a microwave solution to cover CTEC’s service territory would meet operational goals but was too cost prohibitive. On initial purchase, the solution had to be cost-effective, secure, reliable and have a low recurring cost.

Cost Effectively Connecting the Dots
Under Wieser’s leadership, CTEC sought proposals for solutions to, in his words, “provide a more uniform IP communications methodology.” As he considered his options, Wieser discovered Itron’s cellular grid connectivity solution. As cellular service now covers 98 percent of the U.S. population, CTEC validated that all substations in its territory had solid 3G AT&T cellular coverage except for one – a problem solved by installing an external antenna, which adequately boosted reception.

The Itron solution would give CTEC data connectivity for both the AMR and SCADA systems simultaneously at each substation and the ability to deliver that data to the different head-end data applications back at CTEC’s offices. Further, it would incorporate many tools that allow for communications link troubleshooting, such as WAN/ LAN IP packet logging and detailed system operations logs. Other tools would constantly monitor the system’s state of connectivity, automatically reacquiring a cellular connection if the connection unexpectedly dropped.

Despite the promise of operational advantages, simple questions remained. Would the solution be cost-effective to implement and reliably operate going forward? How quickly could the solution be integrated into existing infrastructure? Would the solution help CTEC overcome the operational inefficiencies that stressed its personnel and resources?

Going Cellular for Connectivity
CTEC piloted the grid connectivity solution in early 2011 and began a full-scale deployment toward the end of that year. In order to move to full deployment, CTEC had to accomplish two key goals:

  1. Establish a private AT&T connection between each substation and CTEC’s offices
  2. Ensure the existing IT architecture within each substation was optimal

CTEC and its IT management company worked directly with AT&T to establish its own private wireless IP segment (called an Access Point Name or APN) with static private IP addresses and a virtual private network (VPN) connection. With the APN and VPN established, CTEC now had a private wireless extension of its IT network available to every substation within its service territory.

The second step was to optimize the IT architecture within each substation using the routing services provided by Iron’s cellular router. Routing tables were established to allow each substation device to be connected to the Local Area Network (LAN). Like devices in each substation were assigned the same IP address (10.10.10.xxx) and each substation was differentiated according to the static WAN IP address AT&T assigned to Itron’s cellular router. This greatly simplified setup and maintenance. CTEC utilized one of the two available RS232 ports on the cellular router to connect directly with the one-way AMR receiver, allowing it to be disconnected from the substation POTS line. In the end, the solution uses a private segment to send all data – whether IP or serial – from the substation, through AT&T’s secure system and to different head end systems located at CTEC’s headquarters in Fredericksburg.

Realizing the Benefits
For CTEC, a cellular approach to grid connectivity with AT&T connectivity represents a huge improvement in data collection speeds and efficiency. Now CTEC pulls AMR data as frequently as needed and SCADA polls each protective relay every six seconds — compared with every 15 minutes prior to implementation.

Perhaps the most welcome surprise was the costs associated with implementation and ongoing operations. Based on the previous year’s operation and maintenance costs, the new solution would provide CTEC a quick return on investment. In Wieser’s words, the solution was “very inexpensive” compared with other prospective solutions and would “pay for itself in one year.” All considered, the Itron solution made connectivity simple, secure and affordable.


CTEC Substation
 

CTEC currently utilizes the solution for all 22 of its substations, reliably backhauling all AMR data. Also, in those substations that had previously utilized a mix of communications technologies for SCADA, all SCADA devices use cellular communications. In full operation since April 2012, the system has operated reliably through all seasons.

The new speed and reliability of communications has enabled two important benefits:

  • Data are now coming from substation relays every six seconds, thus providing high situational awareness.
  • CTEC can now look for abnormalities and alarms, such as a breaker trips (controlled by the relays) as well as open and close or reprogram breakers. Previously, lag time made these functions slow and unfeasible. Now, in fewer than 10 seconds, operators know that an open or closed command is successful.

The deployment of Itron’s grid connectivity solution has allowed CTEC to cost-effectively upgrade its data communications to each substation. The new technology has positively impacted the utility on numerous levels, including improved operational efficiency and reliability, enhanced data collection and SCADA capabilities, as well as significantly reduced ongoing operational and maintenance expenditures. Even after weighing the initial capital and ongoing operational expenses, the solution has effectively paid for itself within one year.

The future is bright for CTEC. The deployment has been so successful that the utility now plans to migrate the remaining SCADA units from the last of the 2400bps radio communications equipment. The addition of high bandwidth cellular communications will allow CTEC to further leverage existing substation assets for improved data exchange and system performance.

About the authors

Joby Wieser has worked for Central Texas Electric Cooperative for 26 years applying new technologies to the operation of the electric system including GIS, SCADA, AMR, Voice and Data Communications, and system planning.

 

 

 

Bryan Seal is a 31-year industry veteran having worked 23 years for The Southern Company in distribution, metering and metering communications. He joined SmartSynch in 2005 to lead SmartSynch’s engineering team. In 2012, as part of the Itron acquisition of SmartSynch, Bryan joined Itron. He is currently focused on technology solutions within the North American public power markets.