TMCnet - World's Largest Communications and Technology Community




[May 20, 2005]

Testing and Troubleshooting Complex New IP Services

By Jason Collins

There are many challenges involved in launching voice, data, and video services over IP networks. The high complexity of IP services makes it extremely difficult to locate and troubleshoot problems. Troubleshooting is further complicated when the strategy for operations is an afterthought in the rollout process for these services. Due to the complexity of the new services, however, it is prudent to consider from the outset the people, processes, and tools needed for troubleshooting service problems. Careful consideration of troubleshooting will assure quick problem resolution and a positive end-user experience.


In the past, the people needed for an IP service rollout were typically experts in transport technology such as switching, routing, or configuration. In most cases, the operations center staff at the applications layer is not focused on services such as voice and video. It can be expensive and difficult to find people with the right breadth and depth of skills to quickly resolve problems with voice and video. These factors drive the need for a new class of tools to help speed problem resolution and facilitate a successful service rollout.

Troubleshooting Processes

Depending on the scale of the network, well-documented and enforced processes can be of key importance to assure consistent problem resolution independent of the actual people involved. Established processes assure that a positive user experience with voice and video troubleshooting is a company asset rather than an individual employee asset. Troubleshooting processes are specific to the relevant business; however, the typical scale process involves both back office systems and a tiered operations model. In a typical three-tier model, a Tier 3 operations center watches the network, tries to proactively fix network problems, and handles end-user service issues escalated from Tier 1.

While Tier 3 centers with the right tools can effectively monitor the network, there will still be calls from end-users with issues. Ideally, Tier 1 centers would answer these problems. But with voice and video over IP, the Tier 3 centers are required to handle many issues because that is where specific expertise can be found. This method is too costly and increases the mean time to repair an end-user issue. To efficiently troubleshoot the converged services offering, many more issues will need to be handled by the Tier 1 technician than was previously possible. This approach requires that the operations center invest in appropriate tools to increase service visibility and empower the Tier 1 technician with automated analysis — ultimately reducing costs and decreasing mean time to repair.

Diagnostic Tools

Tools are necessary help support people and processes. Voice and video services over IP make these tools even more critical. In the past, performance monitoring and fault management tools have provided the information needed to proactively alert operations personnel to service issues. While these tools are still critical, converged voice and video services are driving the implementation of a new class of diagnostic tools to increase visibility and provide automated analysis.

First, voice and video over IP require additional visibility into network and service metrics. A typical problem encountered is the user who calls in with “I hear noise on many of my phone calls.” In many rollouts, the operations technician is stuck. The technician cannot see historical and transient issues; nor is the technician able to reproduce the problem. The typical IP toolbox is based on recording results from periodic pings around the network (usually every five minutes to assure minimal impact on the network and equipment itself). This technique just does not provide accurate visibility into key metrics such as jitter, loss, latency, out-of-order packets, and estimated mean-opinion-score such as voice and video quality metrics. Furthermore, it cannot provide important voice and video signaling analysis. Tools such as RMON (remote monitoring specification) could help troubleshooting if operators could turn them on without severely impacting network performance — they can’t. The need is for detailed monitoring of voice and video streams and, when necessary, the ability to actively test the service—emulating the customer as closely as possible. To solve this problem, operations center engineers should consider instrumenting the network with hardware appliances or probes to provide detailed passive monitoring and active testing of voice and video. This can be done initially at key locations to contain costs.

In addition to tools to increase visibility, there is a need for automated analysis of the data. Many operations centers find themselves with not only limited visibility into the right key parameters, but also an inability to filter through the data for important information to identify and diagnose problems. The solution is automated diagnostic analysis. Automated analysis of data must be distributed in the network (for scale) and active at centralized locations (for end-to-end diagnostics). The network hardware should be able to identify and flag issues such as out-of-order packets or issues with voice signaling setups. The inability to do this in typical IP network equipment is another reason why instrumenting the network with hardware appliances should be considered.

We discussed the need to help the Tier 1 technician with troubleshooting processes to decrease mean-time-to-repair and reduce expensive escalations. The key is diagnostic software that can collect data from multiple locations in the network and integrate data from back office tools such as fault and performance tools. Technicians need data from these multiple sources to identify problems with end-to-end service. Good diagnostic software simply replaces the manual integration and analysis of this data by a technician with automatic integration and analysis—presenting only the final result to the technician.

The thoughtful consideration of diagnostics during the design of voice, video, and data service allows the provider to improve the end-user experience and achieve a successful deployment. Spirent estimates that by carefully addressing the people and processes as well as the appropriate selection of tools, service providers can improve repair times by at least 50%—ultimately improving the end-user experience.

Jason Collins, Senior IP Market Analyst at Spirent Communications, has more than 15 years of experience in the telecommunications industry. Jason has marketed and launched IP products such as Voice over IP, IP QoS, videoconferencing and several flavors of MPLS VPNs for leading service providers. He has worked in a diverse set of organizations and roles, including a faculty research position at Georgia Tech, a next generation technology planning role at BellSouth Telecommunications, product development roles at Broadband Office, and product management and marketing roles at Zephion Networks and MASERGY Communications.

Jason holds a patent for video over DSL with two further patents pending. His experience ranges from basic research and development to profitable business rollouts and marketing.


Purchase reprints of this article by calling (800) 290-5460 or buy them directly online at www.reprintbuyer.com.

Respond to this article in our forums!

Technology Marketing Corporation

2 Trap Falls Road Suite 106, Shelton, CT 06484 USA
Ph: +1-203-852-6800, 800-243-6002

General comments: [email protected].
Comments about this site: [email protected].


© 2023 Technology Marketing Corporation. All rights reserved | Privacy Policy