October 1999

To: CTI Readers
Cc: APC, Artisoft, Dialogic, Tripp Lite
Subject: Reviving A Phone System Stricken By Fickle Electric Power

BY Tom Keating

Our office hummed along contentedly — incongruously so — what with the glowering sky above, and the oppressive, sweltering air just beyond our walls. Yet there we were, isolated in our air-conditioned splendor, unaware that disaster was about to strike.

It lasted only a second or two. A lapse of power. So brief. So … hesitant. But so ruinous — for harm may steal upon us, all stealth, particularly when it emerges from some ozone-swathed realm.

While so brief a power outage may sound innocuous, a two-second lapse is actually worse that a power outage lasting several minutes, at least as far as computer equipment is concerned. Any computer equipment, including monitors and printers and other peripherals, will most likely be sitting there, painfully vulnerable, their switches still in the ON position, when the power returns. The problem, you see, is that when the power comes back on, it will often burst through in a dangerously strong surge. (Oh, can’t you just hear the delicate sizzle of overloaded circuitry?)

Concerned about our servers as well as our phone system, I immediately went to check their status. I knew all of our vital servers were hooked up to UPSs to protect them in the event of a power failure, so I wasn’t too concerned. First, I inspected our RAID file server. No problem there. Then, I checked our phone system, which is an Artisoft PC-PBX. To my dismay, I found the system in the middle of a reboot. I glared at the UPS connected to the Artisoft PC-PBX. “Oh, treachery, thy name is UPS!” I howled, savagely kicking the squat box.

I soon regretted my outburst. As it turned out, the UPS was almost certainly blameless. It was nearly at maximum load. And how could that be? A little poking around revealed that someone had actually connected another NT server to the UPS. Apparently, we had overloaded the UPS, and that’s what caused it to fail.

And the consequences? Well, the PC-PBX, unable to draw power from the UPS, went down, it’s switch innocently in the ON position. Then, the power came back in a surge, and…

THE BEST LAID PLANS OF MICE AND MIS
We fully appreciated the importance of power protection, particularly with respect to our phone system. We had made a point of connecting our PC-PBX to a UPS. And, coincidentally, we had just committed ourselves to evaluating a couple of new UPS products. The very week we experienced the power lapse (and surge), we had received Tripp Lite and APC UPS products for our review. (APC sent me their Back-Pro UPS, which is great for SOHO or personal computers, and Tripp Lite sent me the SmartPro 1250XL, which is targeted at mission-critical servers.) Power protection, you may gather, was much on our minds.

But as I’ve just indicated, while we did take the trouble to install a UPS system, we didn’t take enough trouble to make sure we had installed it properly. This realization gave me pause. What damage might our PC-PBX have suffered?

ASSESSING THE DAMAGE
Our PC-PBX, I noticed, booted successfully into Windows NT. Holding my breath, I checked the Event Viewer just to make sure all the Services started properly. Everything looked normal. Apparently we avoided a catastrophe. Pausing only to sigh with relief, I disconnected our PC-PBX from the overloaded UPS and connected it to the Tripp Lite review unit. (You may read the review of the Tripp Lite SmartPro 1250XL in this issue, and the review of the APC Back-UPS Pro in the November issue of CTI.)

I was done. (Or so I thought.) I made a mental note to instruct our MIS staff on proper UPS setup, and returned to my desk.

About five minutes later, I received a complaint that our phones weren’t ringing on incoming calls. I immediately investigated, and determined that half of our 48 office phones rang fine, but that the other half did not ring at all. My first thought was that we had damaged one of our PC-PBX’s telephony boards, a Dialogic MSI board. Each of the two telephony boards is responsible for connecting the PC-PBX to 24 office extensions, as well as providing ring voltage to these phones.

CALLING FOR HELP
I called our local VAR (which, incidentally, had done a fantastic job installing our PC-PBX system). The VAR, DAC Systems, confirmed my suspicions that I had a blown Dialogic MSI board. According to DAC Systems, Dialogic has encountered this “no ring-voltage” problem with other installations.

Now that we had identified the source of our troubles, I contacted Artisoft, and requested that they send us a new Dialogic MSI board. We received the board the next day.

PLAYING MR. FIX-IT
Since I figured I would need only five minutes to install the new board, I decided to shut down our PC-PBX during lunchtime. I should point out that this was only the second time we needed to shut down our Artisoft PC-PBX in the six months we’ve used it. The first time we shut down the system, we took care of an unresponsive keyboard, the problem being a faulty port on the console switcher to which the PC-PBX was connected.

While wearing an anti-static wrist strap (to reduce the risk of any static shocks), I disconnected the power cord to the Alliance Systems industrial computer, which is where the Dialogic cards resided. Before I could remove the damaged Dialogic MSI card, I needed to disconnect another power cord. This second power cord, which connected directly to the outside of the MSI board, was what carried the ring voltage to the board.

Once I had disconnected the power cords, I removed the damaged board and carefully inserted the brand-new Dialogic board that I had received. I turned the system on, and hoped for the best.

After booting into the system and checking to ensure that the Artisoft and Dialogic services had started, I made some test phone calls. All the phones rang! Success, it seemed, was at hand.

EVEN MR. FIX-IT CAN BE WRONG
My glee upon hearing the ringing of the phones soon turned to consternation when it became clear we had another problem. Once the phones started ringing, they wouldn’t stop! Even if I hung up the phone on the dialing end, the phone on the receiving end would ring and ring and ring, without pause. To stop the incessant ringing, I had to pick up the handset on the ringing phone and then hang it up.

I thought to myself, “Well, at least our employees will know when an incoming call arrives.” This cheering thought, however, soon faded as I began fuming about the new board, which had proved to be faulty.

I had little time for fuming, however. Soon, I heard a chorus of complaints about the non-stop ringing of the phones. By all accounts, the ringing was especially annoying when it issued from an unattended cubicle. In such an instance, the phone would ring until someone walked over and hung it up.

The consensus was that we would be better off if the phones didn’t ring at all. So, I reinstalled the original board. I observed ruefully that my five-minute triumph had turned into a two-hour rout. And still no end was in sight!

Anyway, after I had reinstalled the original board, the other Dialogic MSI board started to exhibit the same problem — this board’s phones wouldn’t ring, either. Now I had two malfunctioning Dialogic boards, which meant no one’s office extension would ring.

After fiddling with the board, running Dialogic’s diagnostic utility, and rebooting several times, I was able to get one of the boards working again, and fully functional. However, during the rebooting procedures, I noticed that the semi-functional board was, well, a bit flaky. Sometimes it worked, and sometimes it didn’t. So, I decided to order two new Dialogic MSI boards, just to be safe. I contacted Artisoft again, and they responded by sending two new Dialogic MSI boards via an overnight service.

OUT OF THE BLUE — A BOLT OF BLUE
This time, rather than working on the Dialogic boards during work hours, I decided to come in on a Sunday, when incoming phone traffic would be minimal. Then, I would be free to pull out both Dialogic boards and replace them with the new boards Artisoft had sent me.

While pulling out one of the faulty boards, I saw a blue arc of electric current, half an inch long, pass between the two boards! Since the boards were so close together, a soldering pin on one of the boards must have touched the other board. “Well,” I thought to myself, “if it wasn’t broken before, it’s definitely broken now!” I had that odd, giddy sensation you sometimes get when you think you may have destroyed something expensive.

I wondered how such a large electric current could pass between the two boards. Static electricity? No, the current seemed much too powerful for that. I checked the power cord running into the industrial chassis, but it was disconnected. Then, I checked the external power connector for the board I had removed. This connector had been removed. What was left? I looked at the board that was still seated. Hmmm. This board still had its power connector attached.

When I swapped boards, it didn’t even occur to me that I would need to disconnect the external power cord running to the Dialogic board still in the chassis. In all my experience, which included the installation of many ISA and PCI cards, the precaution of disconnecting the computer’s power cord was all that was necessary.

I also tried to be as careful as possible to make sure the board I was lifting didn’t touch the board next to it. Apparently, I wasn’t careful enough, as the bolt of blue current passing between the two boards would suggest. Fortunately, the boards were already fried. I was certainly glad I hadn’t damaged either of the new boards!

I should point out that I didn’t think that the power running into the Dialogic board was “active” or “live” until I actually turned the PC on. I figured some sort of relay on the Dialogic board itself would switch over and allow the current to pass only when the PC was turned on. My guess was wrong.

I don’t suppose I’m the only person who would ever make such a guess. Consequently, I would like to suggest to Dialogic that they include a “Read Me First!” note that includes a warning about disconnecting any external power cords to any Dialogic boards already residing in a chassis.

THE APOTHEOSIS OF MR. FIX-IT
After installing both the new boards, I reconnected all the power cords and turned the system on. The result? Now, not only did the phones fail to ring, but I couldn’t get dialtone on any of the 48 extensions. Things had just gone from bad to worse.
I found it highly unlikely that Artisoft sent me three bad Dialogic boards (one originally, and then two more). To troubleshoot, I took out one of the new boards and put one of the old ones back in. Now the new board worked flawlessly, and the old board was up to its usual tricks of not ringing the phones.

Next, I decided to reinstall the new board I had just removed, just to see what would happen. Granted, the two boards I now had in the chassis constituted the same pair I had tried but 15 minutes earlier, and without success. However, my gut feeling was that this combination was going to work. I rebooted the machine, and again made some test phone calls. Success! All of the phones could ring! I even dialed a special extension, which is set up to ring all the phones in the office. What did I hear? The ringing of every phone. Music to my ears. You might even say the phones raised a fanfare, in celebration of an MIS god!

THE LESSONS LEARNED
So, what did I learn?

1. Check the load on your UPS. Make sure you aren’t overloading it.
2. Test your UPS. Unplug it and see how it handles the power outage, and see how long the UPS lasts.
3. Test your UPS’s battery. A good time to check your battery is when you reset your clocks to account for Daylight Savings Time.
4. Dialogic has an excellent return policy. Lifetime guarantee on their boards, no charge! (Except for the cost of shipping the board back.)
5. Disconnect any external power sources to any telephony boards. Blue arcs look pretty cool, but trust me, they’re not worth it.
6. Respect Murphy’s Law. If something can go wrong, it will, and at the worse time — until you factor in Tom’s Law. (See below.)
7. Take heart, Tom’s Law may save you yet. Throughout the history of TMC Labs and my days working in MIS, I was wont to hear a colleague exclaim, “Tom, it wasn’t working before. But then, as soon as you came by — without even doing anything — it started working!” Creepy, isn’t it?

Don’t ask me to explain it, just trust me on this one. Murphy’s Law and Tom’s Law are actually mutually exclusive, but fortunately, in this case, Tom’s Law won out, and I was able to get the new Dialogic boards working.

If you’ve had similar problems with the Dialogic MSI board or would like to share your telephony board installation experiences or nightmares, I would love to hear from you.

Tom Keating is chief technical officer and executive technology editor for TMC. He welcomes your feedback. To contact him, send your e-mail to Tom Keating .

While I may joke about being an MIS god, I have to admit that I need to consult with some real heavy-duty figures when it comes to CTI. Let’s call these figures CTI gods. You can meet them at CTI EXPO, CTI’s equivalent to Mount Olympus. CTI EXPO, however, does have one thing on the classical home of the gods. At CTI EXPO, mortals are welcome.

As I hope this month’s column has demonstrated, we mortals could use a little divine guidance when it comes to UPS systems and telephony boards. So, fellow mortals, take heed when I tell you that at CTI EXPO you can learn about the latest UPS products, as well as UPS companies now targeting the telecom space. A case in point: APC Corporation.

At the last CTI EXPO, I met with a representative from APC and learned a great deal about APC’s high-end power protection products. (If only we had installed one of them in our office. No doubt it would have protected our PC-PBX despite our careless configuration.)

CTI EXPO is also the perfect venue to learn about the latest advances in telephony boards from Aculab, Brooktrout, Dialogic, and Natural MicroSystems, as well as a great place to meet CTI application developers. I could go on, but I think I’ve made my point. CTI EXPO is going to be even more crowded than Olympus. So, sign on while there’s still enough ambrosia to go around!