ITEXPO begins in:   New Coverage :  Asterisk  |  Fax Software  |  SIP Phones  |  Small Cells
March 2007 SIP Magazine
Volume 2 / Number 2

SIP Test Tools

By Chris Bajorek, Features Articles


Editor’s Note: Developing and deploying SIP applications and services demands superlative testing tools for proof-of-concept, load, security and interoperability testing. Professional testing labs are obviously treasure troves of such technology, so we decided to take a peek into one of the best known of such labs, CT Labs of Rocklin, California, an independent operating unit of Empirix, headed by long-time testing guru Chris Bajorek. In this article, Bajorek takes us on a guided tour of his tool collection.

Here’s a rundown on the various classes of test tools we use at CT Labs for SIP-oriented projects. Because of the range of products that we test here (from consumer to enterprise to carrier class), it’s a comprehensive list.

High Density Call Load Generators

When it comes to verifying call handling performance at rated capacities, we use a number of different call load generators. Load generators need to issue high numbers of simultaneous calls but not necessarily navigate complex telephone or voice-controlled user interfaces. A key feature to us: the ability to estimate voice quality for every active call. By the way, TDM is not dead yet: Gateways still exist in all sizes from small to very large that need to be tested at rated loads.

Specific tools we use for SIP-oriented tests:

Empirix (news - alert) Hammer NXT-IP for ultra-high density VoIP testing. The NXT-IP gives packet performance metrics as well as a voice quality score for every active test call.

• Empirix Hammer NXT-TDM for ultra high density TDM load tests. We use this mostly to test high capacity gateways.


Call Feature Testers

Call feature testers can also be used as load generators for lower density SIP products. But their primary use is to validate call and application features. For example, an IP PBX with integrated voicemail can be thoroughly regression tested using a call feature tester. Test scripts can be created that not only verify the PBX’s ability to route calls and execute a variety of call features correctly, but also can fully verify the voicemail application itself under real-world traffic conditions. We have developed scripts that will even verify the quality of the voice messages recorded and played back by emulated users under peak load conditions.

These call feature testers come in two flavors: VoIP and TDM. We still use our TDM feature tester/call generators in a surprising number of projects. One recent example: we created an automation test suite that can verify a wide range of call features for terminal adapter devices that interface POTS phones to broadband VoIP services (e.g. adapters that work with Vonage, AT&T Call Vantage, etc.).


Specific tools we use for SIP-oriented tests:

• Empirix Hammer FX-IP for VoIP feature and moderate density load testing.

• Empirix Hammer FX-TDM for TDM feature and moderate density load testing.


Network Emulators

To verify real-world performance, you cannot simply test a VoIP product in a 100% clean network environment. We have seen too many early to market products work fine on a sterile network but have serious difficulties when presented with typical impaired network conditions. To get a true picture of how a SIP or IMS device will operate under real-world network conditions, you need a device that will emulate conditions of packet loss, latency, and jitter. We use our network emulator quite often for this purpose.

Specific tools we use for SIP-oriented tests:

• Empirix Hammer PacketSphere XG


Registration Generation Tools

When you think of testing a SIP device or a staged IMS network for performance, you don’t usually think of registrations as a major traffic component. But in fact, the number of registration-oriented SIP messages that can exist on a network under certain conditions can cause serious performance problems. For example, a registrar that is serving a regional VoIP access network can be overwhelmed when power to a region has been restored after an outage. When power comes back on, all those SIP devices that were off are now alive and trying to register at the same time — which can easily swamp out an illprepared server. Our SIP registration generator has been very busy lately in a variety of performance tests.

Specific tools we use for SIP-oriented tests:

• Empirix Hammer DEX with registration template.

Device Emulation Tools

Many tests that we run in our lab involve staging the test target, a variety of supporting VoIP network devices, and emulation of the subscriber base endpoints via feature and call load generators. There are two basic ways to stage a “supporting-role” VoIP device: (1) buy a 3rd party device and integrate it into the test setup, or (2) use a tool that allows you to emulate the features and functions of that device. For example, staging an IMS network in our lab requires the inclusion of an HSS (Home Subscriber Server) device. With the tool we are now using, we can emulate the HSS and get the benefit of deeper and easier to access metrics. In addition, we only have to learn the device emulation tool interface as opposed to learning to configure and use many different third-party devices. This is quite a powerful concept.

Specific tools we use for SIP-oriented tests:

• Empirix Hammer DEX (device emulator).

Application traffic generators

When we stage certain types of tests, we not only need to generate VoIP traffic but also complementary traffic that is handled by the system under test. For example, testing a service provider-grade firewall with a SIP-oriented ALG (Application Layer Gateway) might require staging of VoIP traffic in combination with other “Internet mix” traffic such as http, POP3, ftp, etc. Other types of tests might require generation of more application-specific traffic such as IPTV or video on demand. So the goal of adding these traffic components is to create as close to a real-world traffic mix, including VoIP, as is possible in a lab setting.

Specific tools we use for SIP-oriented tests:

• Shenick diversifEye 8400 (http://www.shenick.com).


Monitoring/Analyzer Tools

We use two kinds of monitoring and analyzer tools: a SIP trace/protocol analyzer and a device resource metrics/monitoring tool. The protocol trace analyzer is an absolutely essential tool that allows you to debug low level problems with the handling of SIP messages and their associated media packet streams. The device resource monitor that we use enables us to hook into the system under test platform(s) and see in real time the effect of the various traffic loads on the device’s available internal resources (CPU usage, memory usage, disk usage, etc.).

For example, we recently completed a test on a highly scalable SIP-based VoIP communication system that was staged on a series of servers connected to a high-capacity storage device. Our resource monitoring platform allowed us to record the real-time utilization of CPU, memory, and storage resources while we varied the number of emulated subscribers and types of traffic. When the test was done we were able to graphically line up the results, explain to our customer where the bottlenecks were, and point them in the right direction for improvements.

Specific tools we use for SIP-oriented tests:

• Empirix OneSight (device resource metrics/ monitoring tool).

• Empirix Hammer Call Analyzer (protocol trace analyzer).

Security and Robustness Test Tools

This general class of tools (often referred to as protocol “fuzzers”) includes tests designed to flush out problems and vulnerabilities with lower-level SIP stacks and parser mechanisms. The need for SIP-oriented robustness validation is important now, but will be even more important as the proliferation of SIP-based communications grows and malicious VoIP denial of service attacks become more prevalent.

Using these tools you set up a special test that attempts to launch call sessions toward the system under test using SIP messages that have been “fuzzed” (intentionally corrupted) with the purpose of exposing weaknesses in the lower-level parsing and SIP message handling layer. The device under test is monitored for the appropriate response to each test case.

The first and second tools listed below take a bit of setup effort but the price is hard to beat: free.

The third listed tool — from Mu Security — is a relatively new entrant in the robustness testing marketplace; in the short amount of time that we’ve been using their platform we have come to respect its importance in terms of better preparing our customer’s products for the real world. In effect, their tester allows you to expose a wide variety of protocol and attack vulnerabilities (and not just for SIP) that could otherwise leave your product gasping for air. I strongly suggest you check out their website for more info.

The CT Labs SIP Attack Tool (not for sale) is designed to launch a range of randomized SIP messages at up to gigabit wire rates toward the system under test. The tool has proven to be extremely effective in validating products for protection against this special class of SIPspecific attacks.

Specific tools we use for SIP-oriented tests:

• PROTOS Test-Suite (http://www.ee.oulu.fi/research/ouspg/protos/testing/ c07/sip/).

• IETF SIP torture test suite (http://www1.tools.ietf.org/html/draft-ietf-sippingtorture- tests-01).

• Mu Security Mu-4000 (http://www.musecurity.com).

• CT Labs SIP Attack Generator.

Chris Bajorek is the Director of CT Labs (http://www.ct-labs.com), an independent operating unit of Empirix Inc. (http://www.empirix.com).




Today @ TMC
Upcoming Events
ITEXPO West 2012
October 2- 5, 2012
The Austin Convention Center
Austin, Texas
The World's Premier Managed Services and Cloud Computing Event
Click for Dates and Locations
Mobility Tech Conference & Expo
October 3- 5, 2012
The Austin Convention Center
Austin, Texas
Cloud Communications Summit
October 3- 5, 2012
The Austin Convention Center
Austin, Texas