SIP testing, like VoIP testing itself, was considered something exotic. Now that SIP has pretty much supplanted H.323 as the call control signaling protocol of choice, equipment vendors are scrambling to find labs and test equipment that can do full-blown tests of IP communications devices and applications running in a SIP-enabled network environment.
One of the great names in telecom-related testing over the past 10 years has been
Empirix (News - Alert)
(www.empirix.com), formerly Hammer, makers of the immensely varied Hammer product line that includes the Hammer NXT, used by next-gen network equipment makers and service providers to generate sufficient ‘real-world’ traffic to simulate real-world load and stress conditions so as to ensure proper operation of VoIP devices and next-gen applications for network deployment.
Speakeasy (News - Alert)
(www.speakeasy.net), one of the nation’s largest independent broadband services companies, has begun monitoring its VoIP network with Empirix’ Hammer XMS, a carrier-class monitoring and analysis solution incorporates signaling and media analysis and call correlation capabilities, capable of evaluating live network traffic to provide real-time network performance assessment.
Back down at the enterprise, the Hammer VoIP Test Solution for Enterprises handles pre-deployments, smoothes the way for device additions to a network, and takes care of troubleshooting. The Solution actually consists of three other Empirix products: the Hammer FXIP that generates test IP calls and assesses voice quality; the Hammer CallMaster graphical scripting and reporting tool for creating test call flows; and the Hammer Call Analyzer, a diagnostics and troubleshooting solution VoIP networks.
One fellow who’s up to his ears in Empirix test equipment is long-time testing guru Chris Bajorek, Director and Founder of
CT Labs (News - Alert)
(www.ct-labs.com), a full-service testing and product analysis firm specializing in testing services to converged communications product manufacturers and next-gen network service providers. CT Labs is one of the few organizations that can do complete testing of SIP-related products.”
“CT Labs is an independent operating unit of Empirix,” says Bajorek. “The reason for that distinction is that we engage primarily with network equipment and VoIP infrastructure manufacturers, the nature of our engagements most often are pre-release in nature and the types of tests that we do are such that the vendors would prefer that everything we discover be held in confidence. There’s some benefit to being physically apart from our headquarters in Bedford, Massachusetts.”
“The nature of the types of tests we can now do involving SIP spans up to hundreds of thousands of simultaneous SIP calls,” says Bajorek. “This gives us the capacity for doing testing on carrier-grade components used in building next-gen VoIP networks. We’ve actually done a number of interesting tests in that regard, some of which have been self-published by the manufacturers with which we’ve engaged directly.”
Bajorek elaborates: “Our SIP testing platform offers a combination of capabilities. We can test different classes of equipment. It really breaks down into three different domains: First, there’s the residential environment where we test things such as analog terminal adapters [ATAs], VoIP soft phones, hard phones and residential routers. Second, there’s the enterprise level, with IP-PBXs, contact center equipment, VoIP phones, firewalls, intrusion prevention devices, media servers, and things like that. Third and finally, at the carrier level we can test everything from softswitches to session border controllers, media servers, proxy servers, media gateways, and so forth.”
“Obviously, the one major distinction between those three product categories relates to density and capacity,” says Bajorek. “When you’re testing residential products, a full-blown load test on a terminal adapter device usually involves just two lines. That’s not too hard to handle. However, when you test at the top of the food chain and start dealing with things such as a high availability session border controller that’s designed to handle hundreds of thousands of calls at a time, you soon discover that it takes quite a bit of equipment to do that well. Fortunately, our lab can handle it.”
“So, our SIP testing facilities are a combination of testing products that we have in our lab that enables us to do tests across a wide range and down into various things within each domain,” says Bajorek. “It’s not just about testing for load and stress, which is the classic test we started out doing nine years ago. There’s a lot more to testing now, obviously.”
“Even so,” says Bajorek, “Things that we started testing for nine years ago that were important in the first generation of VoIP products, are still every bit as important. For example, it’s still extremely important to verify voice quality in VoIP-related equipment and software. Obviously, SIP is the reigning standard for VoIP phone calls and next-gen networks. So it occupies a better part of 95 percent of the work we do in the lab right now.”
“As for verifying voice quality in a SIP environment, it’s interesting to see how things have changed over time,” says Bajorek. “Starting Day One at our labs we did tests of VoIP gateways, since that was the way your call traffic got from TDM to some kind of an IP network. We do less gateway testing now, but we do a lot of voice quality testing across a wide range of devices — everything from SIP soft and hard phones to SIP PBXs, to SIP-aware firewalls and SIP-aware session border controllers. Anything that media either originates from or flows through, can potentially be negatively impacted by the device being used if traffic levels reach peak levels.”
Bajorek continues: “So what could, say, a firewall possibly do to a media stream flowing through it? The answer is, if you’re driving that firewall under high loads — any of these devices have a rated maximum number of connections or amount of actual IP traffic — then the processing power starts getting strained, and you can start imparting everything from packet loss to jitter to the media stream, which will have a direct negative effect on voice quality. We’ve developed tools and techniques in the lab for not only sending high quantities of SIP-based calls through these devices, but also other types of traffic too that we call ‘Internet mixed’ traffic. Using a special generator we can send HTTP and FTP sessions as well as SMTP and POP3 traffic — all those different kinds of traffic that might come through a normal corporate firewall when you’re going out over the net and surfing, say, HTTP pages. For a firewall device, that’s an example of where you’d want to do that kind of “Internet mixed” traffic generation in addition to just pure VoIP traffic.”
Let’s Make a Deal
“The way we bill for our testing work is usually based on the estimated number of hours for the project,” says Bajorek. “It can vary. These tests can range from quick-andstraightforward to complex-and-long. It depends on the nature of the product and whether it’s going to have an early or a late release.”
“We realized long ago that there was something that more and more was becoming an issue,” says Bajorek. “Namely, what happens if customers want to test against different third-party devices? There are different ways you can address that need. One way is that you can go out and buy one each of every kind of third-party device. You could buy four different softswitches, for example. We’ve taken a different approach. Instead of obtaining all of the thirdparty devices that could possibly connect in some way the device undergoing testing, we’ve taken the approach of using a platform to emulate different devices in our lab. We’ve virtualized the third-party devices that you’d normally need in your lab for the tests. Think of the power of doing that. When you get past the point of needing to spend some development time creating the environments that emulate what, for example, a particular type of softswitch might do, once you have the emulation, then you can run it and gain the additional benefit of being able to see intermediate states during a test to which you might not have had access if you simply used the actual third-party product.”
“Ironically, using the real product gives the test more of a ‘black box’ flavor, and you are stuck hoping that the device is doing what it’s supposed to do,” says Bajorek. “We continue to improve our device emulation platform, and it will give us more and more ability to test against many different devices that are all really staged on a single platform. This is a pretty neat, new concept. Empirix actually offers a platform that does this. We think it’s a powerful model, especially for a test lab such as us, to be able to use moving forward. Again, if a customer wants to test against different types of devices, we can now do that merely by selecting the appropriate profile and running that emulation.”
“One of the things we’re using that platform for right now is emulating large communities of access network users for purposes of doing registrations,” says Bajorek. “So let’s say you’re making a product that allows you as a service provider to build a VoIP network. One of the things you want to do is verify the performance of your staged network to handle hundreds of thousands of users having SIP phones that are registering on a SIP server. Perhaps some are doing calls, but the vast majority of your user community is just sitting there typically registering once an hour, sometimes more often, perhaps as often as every 15 to 20 seconds.”
“It turns out that when you turn on authentication and have, say, 250,000 users that are out there basically ready to receive calls, it takes a fair amount of processing power to handle those requests,” says Bajorek. “Thus, we’re using the emulation platform for emulating very large numbers of users in some of these tests. It’s a fairly new tool, but it’s turning out to be really valuable. That’s an example of something we were not able to easily do in the recent past, but it’s now possible, thanks to these new tools.”
“The development of all these new tools are driven by the new demands coming from greater numbers of deployed VoIP networks that are mostly SIP-based,” says Bajorek. “And we’re not even talking yet about what IMS is bringing to the table, but you can imagine the sophistication and complexities that will be involved.”
Trends in SIP Testing
“At the moment we’re getting many requests for validating platforms against all sorts of SIP-specific attacks,” says Bajorek. “We have a proprietary platform in our lab that allows us to simulate very high rates of attack. For example, let’s say you have a session border controller [SBC] and, among other things, you want to validate the SBC’s ability to resist attacks at very high rates — gigabit wire rate — and protect networks from those types of external attacks. We have a tool that allows us to do that. There’s quite a creative variety of different types of attacks that ‘denial of service-intending’ attackers will attempt to launch. It’s interesting to see how different devices respond to these attacks. Some manufacturers are quite surprised and others not. It’s not something that’s been done a great deal, and our industry has been in a bit of a ‘grace period’ in terms of the frequency of real-world attacks it has experienced. But such attacks are definitely on the rise. It hasn’t reached a crescendo pitch yet, but it’s coming and more and more vendors are aware that they better do something about this and make sure their products can withstand some of these attacks.”
Bajorek adds, “We’re happy to be able to provide them with some of these test capabilities, such as being able to launch a ‘SIP response spoof flood’ — that’s a mouthful — with packet rates exceeding 140,000 packets per second, which is quite a nasty attack against devices such as these. In the case of a session border controller, if it sees a SIP INVITE, for example, it can’t just ignore that and pass it on through, it has to consider ‘Okay, here’s a SIP INVITE. Is this from a valid registered user? If so, do I allow it to go through?’ So, every time the SBC sees a SIP INVITE it must deep-inspect that request and verify that it’s a valid request for service, and then let it go through. That takes processing power. Therefore, a good way to test the limits of these devices is to flood them with a bunch of packets that look valid from all external respects, but may not actually be so. It’s the job of these devices, whether they’re firewalls or SBCs, or intrusion prevention devices, to only allow the valid packets through, even if the attackers are spoofing valid addresses that in the past have been used to create real SIP calls.”
“These tests are done with two types of traffic,” says Bajorek. “One, we send legitimate SIP traffic that the devices are supposed to recognize and allow through. We even validate voice quality during these attacks. But at the same time, we generate these different types of attacks and then monitor various things: the success, or lack thereof, of the valid SIP calls; the voice quality of the media streams; and we also monitor on the protected network side to verify that none of the attack packets make it through to the far end. To the degree that they do, the test will have failed in some way.”
“It’s always interesting to see how different devices respond,” says Bajorek. “We’ve been able to demonstrate to most of our vendors where their limits are situated. That’s really the goal of any kind of test such as this.”
“These SIP attacks are really a very hot area right now,” says Bajorek. “The test requires not only the attack packet traffic but also very strong capabilities for sending the standards-based SIP calls through, and sometimes at very high call rates. We rely on the Empirix platforms for that and they do very well. Ultimately, it’s a ‘combination test’ in the sense of the equipment needed to run it.”
Freeware SIP Testing
Scott Poretsky, is Director of System Quality Assurance at Reef Point Systems (www.reefpoint.com) the company that offers such things as security gateways and the Universal
Convergence (News - Alert)
Gateway™ platform for service provider fixed-mobile convergence (FMC) networking.
“Clearly the SIP equipment vendors and the IMS equipment vendors are way ahead of test equipment vendors,” say Poretsky. “So that’s something we have to deal with, and we must come up with creative and innovative testing solutions in our labs so that we can properly test our products. There’s a great freeware tool we use called SIPp. It’s a free test tool / traffic generator for the SIP protocol available from Source Forge (www.sourceforge.net). It’s commonly used in the industry. It’s very often used in research labs, especially in universities, because it is freeware. It’s excellent but it has its limitations — it doesn’t scale well with the RTP [Real- Time Protocol] media and it doesn’t do media testing, so it’s strictly for control plane testing. I’m referring here to the SIP messaging — you set up the call but you really can’t do much with the media. It also hogs the resources of the Linux machine it runs on. The thing with SIP TCP [Transmission Control Protocol] is that it really hogs system resources. So, you really have to test with it using only SIP UDP [User Datagram Protocol]. If you’re doing SIP UDP control plane testing, it’s an excellent tool, and we did make use of it. But when you’re testing a device, if you push SIPp too hard, the resulting score you get will be lower because you’re limited by the tool itself. It’s not a limitation of the device under test’s [DUT’s] performance. That means that in a lab situation it’s not good enough to solely use SIPp.”
“Instead, there are even better commercial tools out there that are coming from companies such as
Spirent Communications (News - Alert)
[www.spirent.com] with their Landslide product, and Ixia [www.ixia.com] is also releasing interesting tools,” says Poretsky. “Where some commercial software packages fall short is with IMS (
IP Multimedia Subsystem (News - Alert)
). Until now we’ve dealt with strictly SIP test tools, and now we need to be testing IMS, which has a lot of additional functionality and complexity, where you have IPsec and SIP working in conjunction with SIP signaling that’s encrypted through IPsec tunnels. This is really generating demand for brand-new hardware technology in the test lab forms.”
“Despite the lack of IMS test tools, we worked with Spirent, that offers the Landslide product,” says Poretsky. “Landslide can take a device-under-test and actually generate its conformance for IMS and also its performance for IMS.”
“Reef Point and Spirent did a demonstration at a recent VON show that demonstrated 50,000 simultaneous SIP-enabled mobile stations calls running in conjunction with our Reef Point Security
Gateway (News - Alert)
and the Spirent Landslide,” says Poretsky. “Two IPsec tunnels are necessary coming in and out of each virtual mobile set under the 3GPP IMS standard, so 100,000 IPsec tunnels were necessary. The test was a major one since it basically proved that IMS functions and concepts as specified by the 3GPP standard actually work.”
“We then actually did SIP from the Spirent box through the Reef Point Security Gateway that has a SIP ALG [Application Level Gateway] for its firewall,” says Poretsky. “The Spirent box on one interface sent out IPsec traffic to the Reef Point Security Gateway that makes the security gateway think that there’s really 50,000 simultaneous mobile stations, not just one Spirent box. So SIP signaling ran through the IPsec tunnels encrypted, got unencrypted at our box, where authentication also occurred for access to the network. The SIP signaling gets forwarded to the PCSCF [Proxy-Call Session Control Function, a SIP proxy that is the first point of contact for the IMS terminal]. The Spirent box also emulated on another interface the P-CSCF. I believe we’re the first to do that, where you go from a mobile station with two IPsec tunnels through a firewall with a SIP ALG, to a P-CSCF. And we even did it at scale — 50,000 simultaneous mobile stations and SIP sessions. The SIP sessions were in the IPsec tunnels for the calls originating from the mobile handset. So the Spirent box had its emulated mobile handsets signal the SIP through our security gateway, which terminates the IPsec tunnels and has a SIP-aware stateful firewall. But it’s not like a session border controller because it doesn’t terminate the SIP. That’s why the Spirent box in the demo also emulated the P-CSCF out of the other interface.”
"At Reef Point, we, more than any other vendor, put very strict requirements on high performance," says Poretsky, "because our firewall can do one million stateful sessions. We support 150,000 simultaneous registered mobile subscribers, with support for up to a million simultaneous media flow connections. We're talking about a huge scale with many protocols running simultaneously and with the signaling per IMS, all running in a single chassis. Note that session border controllers support only one tenth that capacity."
So, any way you look at it, SIP testing is increasing in both scope and sophistication.
Richard Grigonis is Executive Editor of TMC's IP Communications Group.