Testing in Multi-Vendor Environments (Or, How Testing is a Drain on Your Organization and What You Can Do About It)

By Paula Bernier

To the uninitiated, equipment testing may seem like an insignificant dot on the networking landscape. But if you’re involved with new product or service development, the word “testing” likely evokes “a giant sucking sound,” as former presidential candidate Ross Perot might say.

David Gehringer, vice president of marketing at Fanfare (News - Alert) Software, a four-year-old Mountain View, Calif.-based company that provides an automated testing tool called the iTest, says both equipment providers and network operators are challenged by the testing requirements they face both with new gear as well as every time there’s a new software release.

“When a carrier gets a new device, or a new rev of a device, they frankly don’t trust the manufacturers, and they can’t afford to, so they retest the device kind of by itself to make sure it functions as advertised,” he says. “And then they’ll put it into a fairly large system – and this will have upwards from 50 to several hundred of devices in it. And they’ll redo an end-to-end system test to validate that it not only continues to operate as advertised, but that it plays nice with others.”

Gehringer adds that a vendor for one service provider’s 4G trial has had a patch about every two weeks, which means that with each patch the carrier needs to go back and retest and revalidate the gear and everything that it touches to make sure it all still works together. The fact that service providers have to do that kind validation rather than handing the job off to suppliers is a major source of irritation for the carriers, he says. But Gehringer expects carriers in the future to require their vendors to provide test cases along with their network elements so they can rerun those tests.

“That is a big, significant change to how it’s been in the past,” he says.

While networking equipment companies test their gear before it hits the network, and often during the R&D process before the product is even pressed into silicon, Gehringer says some equipment from different vendors simply doesn’t “play nice” together in the network. And although some vendors will assist service providers and other network operators with a fix, others use these situations as an opportunity to point out the so-called shortcomings of other vendors’ gear in customer networks and to push their own solutions.

Of course, service providers aren’t the only ones facing these challenges. Businesses that run their own networks have similar problems. For example, Fanfare works with a bank, which has a network containing 40,000 routers from a dozen suppliers, to test patches for that gear. It’s a big job.

At the same time, many major equipment suppliers themselves use Fanfare solutions both to test the functionality of their own gear as well as to see how their products work in labs with other network elements and under various conditions. Fanfare has about 110 customers, including big service providers like AT&T, BT, NTT (News - Alert) and Verizon, as well as leading vendors like Cisco, F5, Force10 and HP.

Fanfare’s automated test software can allow those companies to assess packet processing, compare what each device in the network saw and how that relates to what the traffic generator created, determine packet loss, and then reset the lab to its predetermined state.

Feature creep on network devices, however, is creating new challenges in testing because it adds a much wider variety of permutations on a switch, router or whatever device is being tested, says Gehringer.

“We have customers who are literally running 20,000 tests a day,” he notes.

One Fanfare customer, which he categorizes as a small router company, had more testers than it did developers, he adds, “and it scared the heck out of them.”

“It takes longer to test the device than it does to develop the features that go into it,” he says.

But by automating testing, network operators and equipment suppliers can focus more of their efforts on new product development and introduction, and spend less time carrying out these tests, says Gehringer. Fanfare automated test software can take a test response from any device with an optical or RJ 45 jack that becomes part of a network, and parse that response into a very stable and repeatable piece of information. As a result, Gehringer says, users of this automated test software frequently realize productivity gains of between 500 and 1500 percent.

“If you’ve got someone who’s in their backyard digging a post hole with a trowel, you do not give them a shovel, but you back up a tractor, you’re going to radically change their productivity,” he says. “And these people have been testing manually, and we’re giving them a way to automate the testing. So it’s not just a small, incremental [improvement], it’s a big bump.”

Some potential customers are reluctant to change the way they do things and may be concerned that the move to automated testing will eliminate jobs. But Gehringer says they’re thinking about it the wrong way. Automated testing is not about eliminating jobs, it’s about helping people be more efficient and effective with their time, he says.

“I know you love your trowel,” he adds, “but you need to put it down, because once you get it done, you’re going to love it.” IT

» Internet Telephony Magazine Table of Contents