November 2001

Empirix, Inc.
1430 Main St.
Waltham, MA 02451
Tel: 781-993-8500; Fax: 781-993-8600

Price: Starts at $35,450.

RATINGS (0-5)
Installation: N/A
Documentation: 5
Features: 5
GUI: 4.25
Overall: A

"Our customers who design and deploy VoIP network equipment are discovering that they can't ensure good voice quality without also understanding how real networks affect performance," said Gordon Eddy, Marketing Manager at Empirix. "With the new level PacketSphere integration in the VoIP 2.7 release, we've added that crucial dimension. It's easy to create realistic network conditions in the lab and flush out problems before they become very expensive to fix. The many other new test capabilities in the VoIP 2.7 toolkit provide our customers with an even more powerful solution for verifying their VoIP components and networks."

TMC Labs examined Empirix's new Transport product, which we affectionately call the "Mini-Me Hammer" due its small portable self-contained enclosure (Dolch-like box) with an embedded flat-screen display and with the same feature-set as the original Hammer IT system. The Empirix Transport features real-world testing capabilities that include testing voice (T1/E1 and analog interfaces), fax, modem, CAS, ISDN, VoIP, and SS7 signaling. It also features QoS metrics and can measure connection and speech latency, and analyze voice quality utilizing the PSQM and PAMS standards. Also, we received a pre-release beta of the Hammer VoIP 2.7 software running on the Transport system, which includes a new set of voice quality test capabilities, including front end clipping measurement for VADs, echo cancellation testing and the PAMS voice quality scoring.

INSTALLATION
There was no installation required for the Transport. It comes pre-installed with the appropriate software and hardware taking any installation complexities away from the end-user.

DOCUMENTATION
We were quite pleased with the documentation. The online help was pretty good, though the help file for the reports could have a bit more descriptive. We especially liked the step-by-step instructions in the Hammer TestBuilder Getting Started Guide. It explains from start to finish how to use the very complex and comprehensive test system in just a very short number of pages. Similarly, the Hammer VoIP Test System Getting Started Guide was also well-organized and included step-by-step instructions. For such a complex test system, we were quite surprised that the documentation so easily explained how to use the system. Indeed, the documentation has improved immensely from prior documentation we've seen from past Empirix releases.

FEATURES
One of the first features of the Hammer Transport we should point out is that that it is a lightweight, portable test platform configurable with four T1/E1 and eight analog interfaces for testing. The Transport is housed in a portable, rugged chassis with carrying case and measures just 10.75" H, 8.5" D, and 16" W and weighs only 24 pounds. The default specs include an 866MHz CPU with 256 MB RAM, 20GB hard drive, CDROM, 14.1-inch high-resolution (XGA) active matrix color display, 10/100 network card, and a floppy drive.

Creating customized test scripts, scheduling test scripts, and monitoring running scripts are the core components of the Transport system. Using scripts, you can perform real world testing with voice, fax, modem, and tone traffic, CAS, ISDN, and SS7 signaling. Testing of VoIP is available as is the ability to analyze voice quality via the PAMS and PSQM standards. An oscilloscope-like application called VQScope enables visual inspection of the sound wave forms for in-depth voice quality analysis.

Other features include:

• Monitor activity on all channels simultaneously or drill down to view detailed activity on a single channel for error diagnosis;
• Ability to listen to channel activity;
• Real-time monitoring of run time statistics;
• Pre-defined test reports for simplified test analysis;
• High-level summaries and granular channel-by-channel reporting;
• Isolate and view results from specific time periods;
• Graphical summary reports;
• Metrics for measuring the performance of gateway Voice Activity Detection and Echo Cancellation features;
• Saves test speech files above user definable PSQM score threshold for further analysis; and
• Optional Hammer Visual Basic Programming Environment.

OPERATIONAL TESTING
Using the Hammer Transport comprises five main phases:

1. Setting up the telephony boards to use a specific protocol (T1/E1, analog) to match the system under test (SUT) (setup from the Hammer Configurator);
    
2. Creating your test scripts (Using TestBuilder, an app-gen like application);
    
3. Making calls through the test scripts (Can execute scripts via the Call Profiler, the Scheduler, or execute manually);
    
4. Real-time monitoring and analysis of the calls (utilizing System Monitor, Call Summary Monitor, and the Quality of Experience applications); and
    
5. Post-analysis reporting (Reports plus VQSCOPE).

We tested each of these five phases extensively and our results are as follows:

Phase 1 -- Set up telephony boards and connect to System Under Test (SUT)
From the Hammer Configurator we selected the T1 protocol with ESF framing and B8ZS line encoding to match our system under test, namely a Gordon Gapes T1 simulator. We decided to test the Transport's functionality by utilizing the Gordon Kapes T1 simulator for a couple of reasons. First, it happened to be a T1 product we had in the lab at the time. Secondly, we've done speech quality measurements (PAMS and PSQM) on the Gordon Kapes using Agilent VQT (Voice Quality Tester) in the past so it would give us a good benchmark comparison. In any event, after choosing the T1 protocol, framing, line encoding, and clock source, we saved the configuration and applied the settings, which then downloaded the proper protocol settings into the Natural MicroSystems boards. Also, since the Transport comes with four T1/E1 spans and the Gordon Kapes only has two spans, we only applied the T1 protocol to just two of the four Transport spans. We should also mention that you could quickly test a variety of protocols since the Transport supports mixing and matching of different protocols across the various spans. We connected a T1 cable from Span1 on the Transport to Port1 on the Gordon Kapes, and then we ran a T1 cable from Port2 on the Gordon Kapes back to the Hammer Transport, connecting it to Span2. We configured the Gordon Kapes simulator to redirect calls from its Port1 to Port2 so calls were looped back to the Hammer Transport system, which is required for latency and speech quality measurements.

Phase 2 -- Creating Test Scripts
The Transport comes with several test scripts, including one designed to test speech quality and score the results based on PSQM or PAMS measurements. From the Hammer TestBuilder we designed our scripts utilizing a graphical app-gen like interface. TestBuilder utilizes a simple ladder diagram with the left side (A on diagram) acting as the originating side, the middle (N on diagram) acting as the network, and the right side (B on diagram) acting as the terminating side. Four tabs next to the ladder diagram contain different icons with actions that can be placed on the ladder diagram, such as play prompt, calculate voice quality, send digit, and more. Creating scripts was an easy and straightforward process. The Transport also comes with several useful sample scripts that run without any modification.

Phase 3 -- Making Test Calls
Just like the original Hammer IT system, the Empirix Transport supports several means of initiating test calls to your scripts. One way is to use the Scheduler. We were able to quickly schedule a script by right-clicking on a script within the Library Manager, and then choosing "Edit & Run." The Scheduler window pops up where you can then set the start time, channels, maximum number of connections, maximum test time, and other parameters.

We did have some difficult using the right-click "Edit & Run" feature if the script was modified in any way. Once you try and execute the script using the "Edit & Run" command, a message is displayed saying that we must save the script first. We then clicked "ok" and then clicked on the floppy disk icon on the toolbar to save the script. We then tried to run the script and again it gave us the same message saying we had to save the script first. We just did that! After several frustrating attempts, we finally realized what was happening. The message saying we must save the script first was changing the focus of the screen to the Library pane instead of the script pane.

Thus, when we tried to save the script we were in fact saving the Library instead of the script. Once we clicked on the script pane and then clicked the floppy disk icon, the script was successfully saved and we could execute the script. Once we knew this fact, it wasn't that hard to save the script. We believe Empirix copied the model utilized by several development programs such as Visual Basic where it has "project files" and then individual modules (forms, header files, etc.) which are saved independently according to where the "focus" is located, i.e., which screen is in the foreground. However, these development programs at least have separate "save" menu options, which allow you to save individual modules or the entire project. In addition, they also give you the option of saving the entire project if a module is modified. The Empirix software does not provide these options.

In addition to using the Scheduler to initiate calls, we also tested using the Call Profiler, which allows you to apply a traffic pattern to your test calls. For example, you can perform call patterns such as saw tooth (ramps call up then down), blast, Poisson, and others. We should mention that the Hammer also has a CLI interface for executing scripts, which we looked at, but didn't actually test.

We were most interested in the "Speech Quality" script, so this was one of our first scripts we executed. As can be seen in the ladder diagram, this script not only "calculates" the VQ (voice quality), but it also measures speech latency. Since we had connected the Transport directly to the Gordon Kapes with two T1 cables just 10 feet long, we expected the measured latency to be minimal and the voice quality score to be very good and indeed our supposition was correct. PSQM scores range from 0 to 5 with 0 being the best quality and 5 the worst. In the multi-page report, the PSQM scores for the system under test were below 1, ranging from 0.28 to 0.68 for the boy/male prompts. On the second page of the report (not shown), the girl/female prompts also had superb voice quality scores. The PSQM average for all the prompts was 0.53. The Transport displayed the speech latency as averaging 0.71ms. The low PSQM score is to be expected, as a low number indicates better quality with 0 having no degradation. Also, the very low latency (0.71ms) should be expected due to the short T1 wires and digital T1 connection to an internal test environment. Comparatively, the Agilent VQT PSQM scores for the Gordon Kapes were slightly different, averaging a 0.21 PSQM score and 0.87ms latency.

We should state that the PAMS (Perceptual Analysis Measurement System) voice quality scoring algorithm is a new feature in the 2.7 release. The software includes features to test the effectiveness of echo cancellers and VADs (Voice Activity Detectors). We ran a script to measure and score PAMS and it performed flawlessly.

Phase 4 -- Perform real-time monitoring and analysis of the calls
While the Hammer was making test calls into the Gordon Kapes system, we had the capability to listen in on the calls using the Hammer System Monitor application along with PC speakers connected to the Transport. Simply clicking on the channel allowed you to instantly "tune in" to the test call being made on that channel. The Hammer System Monitor also displays log information on the channel selected. Test engineers have the option of selecting various logging levels, including None, Failure, Normal, and Debug. When a script is running, if logging is set to Normal or Debug, you will be able to see events on the screen as they happen, such as "Place Call," "Listening for voice," "Transmitting digits," and so on. With the speakers enabled and tuned to that channel you can both listen and see exactly what is taking place while the test script is running.

In addition, the Transport also features two other real-time monitoring applications: Call Summary Monitor and the Quality of Experience applications. The Call Summary Monitor displays a graph with number of current connected calls, calls per hour, successful calls, failed calls, and other statistics. The Quality of Experience application includes real-time statistics, including a graph displaying whichever variables are selected on the top half of the screen. Our only complaint was that we couldn't seem to change the zoom level or adjust the height of the Y-axis within VQScope. According to the online help, this feature is available, but we weren't able to perform this feature as stated in the online help. In fact, the Zoom option was nowhere to be found.

Phase 5 -- Post-analysis reports
The Transport comes with several built-in reports, including Call Summary, Call Detail, Error Detail, VoIP Summary, VoIP Detailed, and more. We ran several reports and were pleased with the quality and readability of the reports.

As part of the post-analysis, the VQScope application is utilized to compare two waveforms -- that of the original canned prompts and that of the canned prompts after they've passed through the system under test to the terminating end of the Transport. Obviously the quality of the canned prompts only slightly degraded since we were testing with the Gordon Kapes T1 simulator as opposed to a VoIP gateway. We were able to select the threshold whereby the system would automatically capture and store any calls that were below a certain level of quality and then view the before and after waveforms using VQScope's oscilloscope-like interface.

When we performed a VoIP Detail report and a VoIP Summary report, we noticed that the "average speech latency" numbers in the two reports did not "jibe" with one another. They should have been exactly the same. A quick call to Empirix revealed that we had discovered a bug. In the VoIP Summary Report, zeros (0 milliseconds) are used in the calculation of the average speech latency. In the VoIP Detail Report, zeros are NOT used in the calculation. We confirmed this by dividing the speech latency average in the VoIP Detail report by the number of tests performed and then the numbers for both reports did indeed match up. Empirix is releasing a patch to fix this minor bug.

ROOM FOR IMPROVEMENT
Unfortunately, you cannot perform PAMS and PSQM scoring within the same running script. When we attempted to have both of these within the same script we received an error during the compile. We confirmed with Empirix that this is by design. Although we could schedule two separate speech quality scripts a few hours apart, this would increase testing time. Thus, we would like to be able to perform PAMS and PSQM scoring within a single script.

Although you can schedule scripts, you cannot schedule scripts using the advanced Call Profiler which features calling patterns. So we would like to see this in a future release. Another suggestion related to the scripts is moving items on the ladder diagram. First, you cannot drag and drop actions or move action items around using the mouse. Second, in order to move items, you need to highlight the arrow connecting the two actions and then click the up or down icons. We kept highlighting the action labels instead of the arrow and then clicking the up or down icons, which unfortunately doesn't work.

We had some minor usability complaints. Within the Hammer TestBuilder you cannot rename the scripts. You must do a File, Save As with a new filename, and then delete the original script file. Also, you cannot highlight one or more arrows or items within the Ladder diagram and just hit the Del key on the keyboard, although you can right-click delete items one at a time. Finally, we'd like to see a sound player included such as CoolEdit for playing back recorded test calls recorded in PCM format since the Windows Media Player doesn't support this format.

CONCLUSION
This product covers all the bases including load/stress testing, VoIP testing, speech quality analysis, real-time monitoring, and post analysis comprehensive reports. TMC Labs has always enjoyed using past versions of the Hammer product line. Now, with the portability of Empirix's Hammer Transport, lab technicians can use the Hammer Transport in the lab or on the road. Whether it's the full tower version of the Hammer or the portable version, the testing feature set is very impressive and in our opinion no telecom testing lab should be without one.

[ Return To The November 2001 Table Of Contents ]