May 2003

Finding And Fixing VoIP Problems

BY TOM GALLATIN

VoIP quality must be measured to ensure appropriate customer satisfaction. Recent state-of-the-art technology is significantly advanced by VoIP quality monitoring and testing systems based on the E-Model, an ITU G.708 standard scheme whose key contribution is that it passively measures each call more accurately without having to generate costly and intrusive test calls. Administrators may use systems that are E-Model based to proactively measure the behavior of their VoIP network, baseline the performance, and understanding the human perception of the call quality, thus predict the MOS score. Passive techniques are best for bringing the problems to light, providing accurate information that enables a rapid fix.

THE PROBLEM
Everyone wants the option of having high-quality phone calls with high availability at low cost. Voice over IP networks promise this, but are still far from delivering fully on that promise. Jitter and packet drops that can be tolerated in an IP data network are key contributors to poor quality in VoIP. To absorb most jitter, buffering is often employed but buffers can overflow and cause drops plus significant delay that is also a cause of poor perceived quality. High bandwidth (the �big pipes� solution) can eliminate much of the buffering and drops, however, bandwidth is not inexpensive and is not a panacea for all that can occur on even the most robust of IP networks. It only masks root causes leaving problems to grow. They must be found and fixed. Persistent quality problems will lead to customer dissatisfaction. It is essential to pro-actively monitor the quality of every call made in a network in order that quality issues can be detected and resolved immediately.

Before you can fix the problem, you have to identify and find it. Compounding the problem is that analog voice tools and tests don�t analyze VoIP in Ethernet networks. Some voice quality monitoring and analysis tools are based on traditional analog tests and depend on generating an active load on the network, which can give misleading results in IP networks thereby making problems difficult to find.

How can network architects get an accurate reading of the voice quality of service?

Passively measuring each call has been made possible with the ITU G.708 standard E model with extensions to recognize the role of human behavior in assessing call quality. It has become a popular standard for voice over IP quality measurements. The R-factor is a dimensionless metric based on E-Model, which can be calculated, based on observed traffic flow for every phone call on the network. The R-factor, with human behavior factored in, can be correlated to a subjective Mean Opinion Score (MOS), giving the network administrator an accurate, unbiased, independent accounting of the quality of the calls as they would be assessed by human listeners.

R-factor measurement helps solve the four major problems with existing voice quality measurement techniques:

1. Measurements can be made on the actual calls in the network there by eliminating false positives on test calls in a non-deterministic environment of IP.
2. PSQM, PESQ, and PAMS tests are based on analog network tests, and are intrusive, which means that special test calls must be made through the network in order to make measurements. This uses network bandwidth and actually reduces quality. It is only feasible to make a small number of such test calls and to get occasional snapshots of network performance. This is often not effective for non-deterministic IP networks.
3. Typical network impairments are time varying, which means that voice quality varies during a call and the perceived quality during a transition differs significantly than the actual measured quality. Active voice quality measurement tools are not able to predict the impact that time varying quality has on the human listener. Extended versions of the E-model which consider human psychological reaction to time varying impairments are able to accurately measure the human user�s experience.
4. Packet Loss and Jitter are recorded as averages as opposed to being used to actually measure the perceived quality of the call by the human listener.

PACKET LOSS BURSTINESS
Packet loss will increase when jitter buffers are minimized to eliminate delay. Uniform packet loss on a call usually will result in very little quality impact, even if the packet loss is a relatively high percentage of the total packets involved in the call. However, the degree of �burstiness� has a major impact on voice quality. This is significant because packets will usually occur in bursts due to rerouting or sudden network spikes.

As a simple example, consider a three-minute Voice over IP phone call during which there is a two-second period of time during which five percent of the packets are lost. Many voice quality analysis systems would average this packet loss over the call, assuming that a uniform five percent of packets are being lost, which would not be apparent to a listener. This would result in the incorrect prediction that the subscriber would not hear any degradation in quality. The R-factor can take the �burstiness� of the packet loss into account and make a more accurate prediction of the actual MOS perceived by the listener.
 

THE HUMAN PERSPECTIVE -- SHORT TERM MEMORY
The call quality reported by a listener depends on how near the end of a test call the degradation occurred. This is believed to be due to the �recency effect,� which relates to a human characteristic of tending to remember and penalize only the most recent events. If an event was early in the test call, and the quality improved, then the listener tends to forgive and forget, resulting in a higher test score. Moving a garbled signal from the beginning to the end of a 60-second call results in a change to the reported MOS score from 3.82 to 3.18.


COMPARISON TO PSQM / PAMS
One important measure of the effectiveness of voice quality measurement algorithms is their ability to predict the subjective quality score that people would assign.

A recent study by Telchemy Corporation compared the R-factor score generated by E-Model based systems with Human Factors extensions against the widely used PSQM and PAMS active, objective measurement algorithms. Six sets of audio files were used in the study, each containing five impaired files. Listeners were asked to rank each set of five files from best to worst and then the average rank was calculated for each file. The three voice quality measurement algorithms were then used to perform the same ranking.

The mean rank distance was calculated for each file set and for each measurement algorithm. This gives a measure of how closely the measurement algorithms predicted the ranking given by listeners. As an example, if the order given by the human listeners was 1 3 2 5 4 and the order given by one of the measurement algorithms was the same the mean rank distance would be 0; if the order given by the measurement algorithm was 1 2 3 5 4 then the mean rank distance would be 0.4.

INSTRUMENT THE NETWORK
Large enterprise networks require distributed solutions for centralized management. In these scenarios, network costs can be mitigated through a device called a tap. Taps enable monitoring, analysis, or security instrumentation to be dynamically inserted into data network links without affecting the network. They also enable use of distributed monitoring devices on different key backbones without requiring the purchase of a device for each segment.

A tap goes beyond the access granted by a switch span or mirrored port because it provides access to the actual network and to any physical errors without impacting the performance of the switch. Taps are fault-tolerant, passive to the network, and allow for the monitoring, capture, and analysis of physical errors on individual segments, or in the case of matrix switching taps, have the ability to rove between segments. Taps may provide advantages over span or port mirroring, which usually filter traffic and drop packets.

TROUBLESHOOTING
When problems occur, network administrators do the basics pretty easily. They investigate the change management system to find out �who changed what,� �ping� the usual suspects, and look at management information bases of the VoIP equipment vendor. They may even swap gear to troubleshoot the problem. If the problem is still not solved, the network administrator will be grateful for non-invasive test tools to give him information he can use. Most minimally instrumented networks will have one system deployed strategically on the critical links, which the technicians can access from anywhere. No flights, no guessing at vendor logs, just pure data from the network itself to help technicians troubleshoot the problem.

With most E-Model based test tools, measurements can be performed on each channel -- audio and video -- within every call on the IP network. These measurements go beyond what is provided by the IP gateway, switch, and router vendors. The accuracy of the QoS measurements from third-party test tools is unbiased. This measurement information can be exported as a historical baseline into a centralized repository where it can be warehoused for capacity planning, network service auditing, and customer-service applications. When trouble strikes, the network administrator may compare current traffic with baseline traffic patterns and will know if the problem was within the network rather than some customer-premises equipment or rogue application. Given the inherent chaos of IP, it is important for the network administrator to know what is really happening.

CONCLUSION
If you need to know the voice quality experienced by your customers, then you need accurate metrics. If you wait until your customers are letting you know about voice quality problems then it is too late. You need real time monitoring and analysis of voice quality to let you manage your service effectively.

Tom Gallatin is Product Marketing Manager at Finisar Corporation. Finisar Corporation is a technology leader for fiber optic subsystems and SAN/LAN performance tools, which enable high-speed data communications over Gigabit Ethernet LANs, Fibre Channel SANs, and MANs. For more information, please visit www.finisar.com.

[ Return To The May 2003 Table Of Contents ]