April 19, 2024

Iowa Utility Migrates Wireless SCADA Network from Analog to Digital, adds Data Ports

by By Russ Straayer, Data Comm for Business, Inc.
Dennis Hill, Telecom Manager at Northwest Iowa Power Cooperative (NIPCO) in LeMars, Iowa had a networking problem that was perplexing him. The SCADA system he is responsible for managing had analog radios and switches, but Dennis needed to migrate to a digital network.The MDS (Microwave Data Systems) analog 900 MHz radios are no longer manufactured.The pool of spare radios is very small.There are not enough analog radios around to count on for system expansion.The replacement for the analog radio is the MDS digital 9700 series.

Dennis needed to upgrade the network to run faster and with greater reliability.Digital networking is the key to more speed and greater reliability.As newer RTUs are migrated into the NIPCO SCADA network, more information can be gathered and higher transport speeds are desired on the SCADA network.

Existing Analog Switching is not Digital Compatible

The Northwest Iowa Power Cooperative (NIPCO) Iowa, had analog multidrop (MAS) MDS radios in operation for many years.NIPCO shares the radio network between SCADA and meter data collection.Dennis had developed a clever analog switching system for NIPCO.A switch at the substation is activated to select either the SCADA RTUs or the meters, using a 2800 Hz signal tone.The tone is recognized by the switch and the data path is switched from the SCADA equipment to the meter equipment.The 2800 Hz tone is sent through the analog MAS radio system.The 2800 Hz tone was selected because it was out of the frequency range of the FSK modems running over the analog radios.

When the 2800 Hz tone is active, the SCADA RTUs are switched offline, the meters are switched online.The SCADA host system is unable to collect RTU data while the meter data is collected, but the interruption lasts only a few minutes per day.After collecting the meter data, the SCADA system is switched back online by turning off the 2800 Hz switching tone.

The SCADA downtime and digital path problem

NIPCO needed to move away from this analog switching system for several reasons.First, while meter information is being collected, load management data is unavailable.Downloading meter information takes the SCADA system down for 30 minutes to an hour.During high load conditions, such as hot weather and the fall grain drying season, losing the SCADA system information for just half an hour can lead to system problems.Second, as noted earlier, NIPCO must make the transition from analog to digital radios, as the analog radios are not available for purchase in any quantity.Once the digital radios are deployed in the system, it is impossible to send a 2800 Hz tone.The 2800 Hz analog tone cannot be transmitted through the digital radio path.

The SCADA and meter equipment The NIPCO uses the LANDIS & GYR 8979 (L&G) protocol for SCADA.The meter application uses Scientific Columbus meters running at 1200 bps, using JR1 meters and JR1F protocol.Some of the sub-stations have already been upgraded to newer Jemstar meters, and all substations will, at some point in the future, be upgraded.The SCADA host and the new meters are capable of running at 9600 and 19,200 bps.NIPCO wants to take advantage of the higher speed capabilities when they switch from analog to digital radios.

The Radios

NIPCO chose to stay with Microwave Data Systems for their radios.They upgraded to the model 9710 digital radios, operating in the 900 MHz licensed band.The digital radios operate at 9600 or 19,200 bps.They have tried both speeds, and find the system operates well at either rate.

To replace the analog switches, Dennis looked at several other options.One option was to gather the meter information using the SCADA RTUs.Some SCADA equipment manufacturers use store and forward techniques to deliver the remote meter information back to the host site through the RTUs.The RTU buffers up the meter information and pass it to the host computer.The data is then passed from the SCADA host to the host computer that processes the meter data.Dennis evaluated this approach, looking at its implementation at another electric cooperative.

The Solutions

The RTU relay approach that Dennis Hill analyzed was not a solution he wanted to deploy.He found that if the remote RTU had gathered the meter data but had not yet transmitted that data to the host, and there was a power glitch, the meter data was gone.Missing meter data leads to estimated billing, rather than exact usage billing.Dennis wanted a more reliable solution.He wanted to communicate with the meters directly, not through an intermediary RTU.Dennis looked for a solution that would either allow him to switch between SCADA RTU communications and meter data collection, or would allow both SCADA and meter collection to occur simultaneously.Simultaneous operation was his preference.

Dennis looked at code activated switches.There were none on the market that did exactly what he wanted.He would have to find someone to build a custom switch or build his own.This was not the preferred solution either.With a custom switch, future repair and replacements are a problem.A third party built custom switch was an expensive solution.A switch of his own design tied the unit to him personally.If he were on vacation, or in the future, retires or is not around for some other reason, his support for the device would be gone.A standard commercial product was the preferred solution.

Dennis found a standard commercial product from Data Comm for Business, Inc.It is the DCB SCADA Multidrop Multiplexer (SMD).This device claimed to allow up to 4 channels to operate at the same time, even at different speeds and with different protocols.Dennis is a capable, but cautious engineer.He liked what he heard, but "show me" is his motto.He procured several of the units and deployed them in a test system for several months before blessing this approach.

Dennis assigned the task of testing to Tim Hansen.Tim installed the new MDS 9710 digital radios and the SMD multiplexers.The main radio tower is several miles from the host computer, requiring a link from the host computer to the host SMD multiplexer and radio.This link is provided over DSUs and a T1 link.

Multiplexed system performance
Error rates


Tim Hansen tested the system on the bench and in the first small field test for several months prior to full in-the-field implementation.Both on the bench and in the field, the SMDs worked as expected.There are virtually no errors on the system now.Initially there was a problem over the link from the host computer to the SMD/radio head end.This turned out to be an easy problem to solve.Tim discovered that while the radio and the SMDs were sending asynchronous data, the DSU was set for synchronous operation.With asynchronous data going into synchronous DSU channel, the SMDs had about a 10% error rate.No problem really, just an incorrect setup.Now with the DSU channel set to asynchronous operation, matching the radios and the SMD, the error rate through the DSUs is almost zero.As for the rest of the system, the SMDs and the digital radios, the error rate is also nearly zero.When the digital system is compared to the old analog radio system, the error rate has dropped from a percent or two to near zero. Comparing analog versus digital error rates To determine how many transmission errors were encountered when collecting meter data, Dennis built a special monitor box.The monitor box checks the host/meter data stream for ACKs and NAKs. The meter data is error free when the host sends an ACK back to the meter.When the data block is corrupted and needs to be sent again, the host sends a NAK to the meter.The box Dennis built consists of a digital comparator circuit, an LED display and a speaker.Whenever there is an ACK or a NAK, the display shows the hex character that corresponds to the ACK or the NAK.Whenever there is a NAK, the speaker beeps so you can hear when there is corrupted data.

Dennis employs this monitor box on the new digital radio links as well as the old analog links that have not yet been converted to digital.The results are striking.By a wide margin, the digital circuits have fewer errors than the analog modem circuits.The analog system beeps often, the digital system is silent.The SMD multiplexers and the digital radios provide more reliable communications (fewer data errors) and allow the SCADA system and the meter data collection to exist side by side, simultaneously.

Timeouts

The timeout parameter on the NIPCO SCADA system has been changed from about a half second to 2 seconds.With the addition of the SMD, Tim Hansen found he had best performance when he set the polling timeout to two seconds.The timeout issue, if not properly handled, may present some problems that can be difficult to troubleshoot.Older protocols, or most SCADA protocols other than DNP3, do not include the source and destination addresses or poll numbers in the poll and response.NIPCO is not using DNP3, so the SCADA polling protocol does not include source addresses or poll numbers.

The lack of addresses on the polls and responses and lack of poll numbers leads to the following potential problem if time outs are not considered.The SMD multiplexer may buffer a poll or response due to primary (radio) channel outage or heavy traffic on other channels of the multiplexer.This buffering might exceed the time of the SCADA system timeout.Suppose the SCADA system polled RTU #1, but the RTU response is buffered and delayed.The SCADA system then times out the RTU #1 poll, assumes the RTU did not respond, and goes on to poll RTU #2.If the RTU #1 response is delayed, not lost, then the next response seen by the SCADA host is from RTU #1.With older SCADA protocols, the host does not know the response came from RTU #1.The host thinks the response is from RTU #2.Now the polls and responses are out of sequence.

This out of sequence problem results in SCADA information being shifted from one location to another.Suddenly, substation data seems to shift down one location for all the substations.If the user is not aware that this is the result of timeouts and buffered data, this is a very difficult problem to diagnose and cure.Even if one knows the source of the problem, the data is reported for the wrong location.Once the source of the problem is understood, it is easy to correct.

In the case of the SMD, each drop (remote) SMD has a timer for purging old, stale data.This timer is set to match the SCADA or meter system poll timeout.When the timeouts are matched, data that is too old is purged, letting the SCADA host computer do its normal poll/response/timeout routine without the confusion of receiving old, unexpected data.As mentioned earlier, DNP3 protocol is not subject to this problem because the polls and responses have source and destination addresses and the polls and responses are consecutively numbered.Polls and responses with incorrect source and destination addresses or bad poll sequence numbers are discarded.

Converting from a single channel to multiple channels is accomplished using the DCB 4-port SCADA Multidrop Multiplexers (SMD).A four port SMD is deployed at the host site, another SMD is deployed at each remote site.With four channels, NIPCO operates the LANDIS & GYR 8979 protocol on the first channel.The meter reading application uses another channel of the SMD.

Simultaneous operation on up to 4 ports

The SMD now allows NIPCO to have simultaneous, independent operation on up to four ports. The channels do not interfere with each other.The SMD divides the MAS system channel into 4 ports by digitally sharing the radio bandwidth.One might assume that this process will greatly slow down the collection of SCADA information, but it does not.The SMD divides up the data channel without any noticeable slowing down of the SCADA system polling.

SMD polling is faster than SCADA polling

A typical SCADA system polls RTUs at a rate much below the capability of the MAS radios.In the case of NIPCO, the L&G 8979 RTUs are polled at a rate of 2 polls per second.Each RTU in a polling group of the RTUs is polled about once every 5 seconds.The SMD installed at the NIPCO host site polls up to 30 remote drop SMDs every second.This faster polling by the SMD results in a four to five time increase in the data transmission capacity of the MAS radios.This fast polling makes it possible to create additional data channels over the MAS radio links.

Rate conversion

The SMD at NIPCO also performs data rate conversion which is used to further speed up the polling process.The NIPCO MAS radios can operate at either 9600 bps or 19,200 bps, the RTUs and the host computer operate at 9600 bps, the meter system at 1200 bps.The conversion to higher speeds increases the polling rate of each channel and is transparent to both the radios and the RTUs.The SCADA or AMR system may momentarily send data to the SMD faster than the SMD sends the data over the radio system.The SMD buffers up the excess data until it can be sent through the radio system.

Migrating to the SMD System

NIPCO is migrating to their new SMD system on a systematic basis, installing the SMDs on one MAS multidrop group at a time.The first group has 11 remote sites. An SMD is installed between the SCADA host computer and master radio at the host site.At each remote site the SMD is installed between the remote radio port and the RTU.Once the SMDs are in place, they are transparent to the L&G 8979 protocol and the load management protocol.

The remote SMDs can be installed at remote drop sites before the host SMD is installed.The drop SMD is in a transparent mode until host SMD protocol is detected.This transparent mode for the drop SMD makes installation less stressful for the installers.They don’t have to take the whole system down while they rush around to all the sites and install the SMD drops and host.The 10 or 11 drops on the SCADA system remain in service as the drops are installed.Only when the host SMD is installed do the drop SMDs become active.Until then, the drop SMDsact as just a "bump-in-the-cable".

At a few sites, NIPCO does not have the need for both and RTU port and a meter port.For these sites, NIPCO deploys a single channel SMD.The remote single channel SMDs are needed because the host SMD bundles the L&G 8979 protocol in its own SMD protocol.When the data gets to the remote site, and SMD is required to un-bundle the L&G 8979 protocol.

The SMD Protocol

An SMD host unit polls each drop in numerical sequence and expects a response from the drop polled.During SMD host polling, any port data in an SMD host port buffer is also sent from the host to all drops, regardless of which drop is being polled at the time.The drops send the data out the appropriate port.The SMD drop being polled responds to the poll and includes any data pending in its transmit buffers.

When a host SMD port is defined as an Access Port, the data from the host SMD is handled a little differently.The data from the SMD host goes out the port at only one SMD drop location.This special case allows 1 to 1 SMD host toSMD drop connections.Normally, when a port is not set as an Access Port, data from a host SMD port is broadcast to all the same SMD drop ports for a 1 to many connection.

Access Switch function

With only 2 of the 4 ports used for polling communications, NIPCO has two more ports available on the SMD.Any SMD port can be configured as an Access Port.When an SMD port is set to the Access Port function, a point-to-point link is established from the host to any one of the drops.NIPCO can use the Access Port function to access and control an RTU RS232 setup port, the radio the management port, or any other equipment at the substation that has a serial management port.

Other devices may be plugged into the ports of the SMD, non-polled meters for example.The non-polled devices can be used on any of the 4 ports of the SMD.NIPCO can attach a terminal or PC to the host end SMD, then select a specific remote port to communicate with.By selecting a specific remote port, non-addressed devices can be managed while other ports are carrying polling traffic.

Ethernet/LAN/IP in the future?

With the extra ports available in the SMD, NIPCO will be able to add Ethernet bridging to the system if required. There are many new devices coming onto the market for electric utilities that utilize Ethernet ports, in lieu of, or in addition to serial ports.These devices, whether SCADA, AMR or other data collection and monitoring equipment, are typically very low data volume.These low data volume devices are an ideal fit with the SMD multiplexers.

Adding Ethernet is accomplished using an external box, a "SCADA Bridge".The SCADA Bridge connects to the SMD through a serial port.The SCADA Bridge Ethernet port connects to the LAN equipment at the host and remote ends.Bridging (versus routing) involves just plugging in the units after setting the serial port speeds, a process that avoids the more complex setup used by routers.Bridging puts the remote devices on the same LAN subnet as the host site LAN.

Conclusion

NIPCO has eliminated the periodic lack of load management data and they are able to implement the digital MAS radios.NIPCO has added 3 channels to their single channel radio system. Considering the alternatives of a parallel radio system or phone lines, the SMD multiplexer approach has saved months in time and in excess of $1,500 per remote location.Add in installation and maintenance costs and the savings are even greater.The multiplexed system has no negative impact on the SCADA and AMR systems, provides remote access, builds in room for future expansion, and is economical.