Financial institutions have extensive data networks in place with best-of-class communication links, both nationally and internationally. Add to that the need for cutting-edge voice communications and CRM applications all while keeping overhead under control, and financial institutions make the ideal early adopter for converged networks using VoIP technology. The benefits of VoIP (define - news -alerts) are clear — however careful risk management is critical, as leading UK bank Abbey and BT discovered.
The VoIP Decision
In February 2003, Abbey announced the decision for a company-wide move to an integrated voice and data IP network. The new network would benefit Abbey in two areas — the initiative would lead to improved workforce efficiency as part of the bank’s cost review program, announced to the stock market in July 2002, and new applications would deliver better customer service.
Yasmin Jetha, Executive Director, IT at Abbey, said “the move to the new network would allow the company to deploy new applications, such as CRM software, more efficiently and, at the same time, help to reduce costs. Upgrading will improve both our internal and customer facing communications, increase flexibility to meet future business needs, and enable us to deliver our new strategy.”
Already a BT customer, Abbey awarded BT a £125 million, five-year outsourcing contract to install and manage a consolidated company-wide integrated voice and data telecommunications solution. Abbey selected BT based on BT’s expertise in IP infrastructure, convergence, wide and local area networks, and outsourcing. The service was based on BT Global Services’ world leading Hosted VoIP and Multimedia platform. The scope of the project required the transformation of a staggering 800 branches across the UK.
The £125 Million Guarantee
Abbey took a bold step in selecting VoIP as part of a national network upgrade. At a time when most companies were avoiding costly upgrades, Abbey based its aspirations on a young and relatively unproven technology for the scale of the envisaged rollout.
As part of the agreed risk management strategy, BT offered various service level agreement terms; one was an industry first — voice quality on the MOS (Mean Opinion Score) scale. Within the contract, it had been agreed that the voice quality would achieve an MOS of 3.7 or higher, with BT aiming to consistently deliver an MOS of 3.92 (PSTN lines typically have an MOS of 4.1).
MOS on What Terms?
Like VoIP, network management, in terms of MOS, was uncharted territory and, consequently, the terms of the contract did not specify how MOS would be calculated.
Before Psytechnics became involved, the concept was to perform a subjective test every quarter by manually surveying people within Abbey.
This proposal carried the issues of time, cost and logistics. Who would do the test? What questions would be asked? Could quarterly tests really be compared? How accurate and meaningful would the survey be? Consequently, subjective testing was ruled out as impractical on many fronts — scalability, repeatability, and difficulty in interpreting results.
Instead, BT and Abbey decided to explore the potential of objective voice quality measurement technology. BT asked if Psytechnics would act as an independent third party and perform a voice quality audit for Abbey.
Psytechnics proposed the use of the ITU standard for measuring MOS, P.862 PESQ, a test method objectively measuring voice quality, which has been adopted by the world’s leading test and measurement companies.
Education, Then a Test Plan
Abbey had limited experience with MOS, subjective testing, or objective voice quality measurement. Thus Psytechnics’ initial task was to educate Abbey. A critical part of the education was confirmation that the contractual point of an MOS of 3.7 was sensible. Psytechnics validated that 3.7 was a reasonable target both for the choice of technology and the customer experience.
Psytechnics also explained that two objective test methodologies were available — end-to end (PESQ) and continuous monitoring — each with its own benefits. Psytechnics advised end-to-end testing would be a good start; however, real-time monitoring would also be required in order for BT to be proactive in addressing customer-affecting problems.
First Test, First Problem
Psytechnics conducted MOS testing on the Proof of Concept (POC) system installed a few weeks prior. The purpose of the POC was to give Abbey the opportunity to experience the quality and feature set offered by the new system and to validate that it worked before going to field trials. For all intents and purposes, the POC acted as a real branch office. Calls could be made internally, out to the PSTN, as well as to any existing Abbey phone system.
The voice quality test for the POC system measured the quality between the different components of the new network and its interactions with the existing networks. Tests were also performed on the backup systems. As the POC system had been “up and running” for a few weeks, no problems were expected. Surprisingly, an issue was discovered.
MOS testing showed a problem for calls to existing Abbey PBX users, but, interestingly, it only happened for one in four phone calls. Analysis led the network operators to track the problem to a single media gateway, where the fault was identified and corrected within two hours and the voice quality returned to being above 3.7 for calls.
The surprise discovery of the problem on a VoIP system thought to be “field trial ready” proved that BT’s ability to provide customer-ready communications was going to require MOS information. More importantly, finding the fault using MOS measurements confirmed that BT and Abbey had made the right decision to include Psytechnics’ MOS testing in the SLA.
Moving to the Real World
The next phase was to roll out a VoIP trial system to a number of branches across the country to prove that what works in the lab can also work in the real world. The fact that all 800 sites would have identical configurations meant the field trial test plan could be reduced to only three sites. One of the benefits of VoIP is that if a problem is addressed at one site, it can be swiftly addressed at all sites.
Psytechnics performed testing between three branches and the head office in Milton Keynes. MOS tests were performed between the branch office and the PSTN connection and also between the branch office and a FeatureNet connection (Abbey’s existing PBX system).
Fifty calls were made to each site with 20 PESQ measurements per call. A total of 1000 PESQ test points per site were gathered over a two hour period. Of the 150 calls made to all branches only two calls had an average MOS below 3.7, representing a conformance metric with the service level agreement of 99.8%.
However, six calls recorded individual PESQ test points below 3.7. In other words, there were times during those calls where the voice quality momentarily dipped. Further analysis revealed the degradation coincided with single speech events and was equivalent to about three seconds of significant degradation during 22 minutes of tested speech.
Strictly speaking, the SLA target of 3.7 for all calls wasn’t achieved. Yet, in practical terms, the field trial systems were fit for purpose — even the PSTN can have three seconds of problems from time to time. The trial was deemed a success and the decision was made to continue with the full 800-branch rollout. The rollout plan called for 20 branches per night to be converted, but again, the lack of MOS measurement (customer experience) led to unsatisfactory results.
With each new branch BT installed there would be minor issues, which would first be identified by customer complaints — not the ideal way to manage a network. The BT team found that a significant amount of time was spent trying to verify whether it was a problem with the VoIP network or with the external network (such as a mobile network), or whether it was actually no problem at all.
Often, problems took a number of days to track down, but only a few minutes to rectify. The operations team confessed that the hardest part was correlating the drop in quality given in customer complaints against what actually was happening in the network. Thus, the need for continuous, real-time MOS monitoring with diagnostic information was realized.
BT decided to accelerate its plans to implement such a system for the Abbey network and approached Psytechnics for help introducing an interim real-time voice quality monitoring solution with diagnostic capability.
Real-time Monitoring Shows Real-Time Results
The decision to implement real-time 24x7 monitoring revealed more than just problems with packets and route configurations. On one occasion, the 24x7 monitoring system, PsyQ, identified a branch suffering from voice quality issues. Further investigation revealed that builders had dismantled the room in which the router had been. The VoIP system had transferred to its backup ISDN connection, which caused the drop in voice quality. PsyQ spotted the problem, while the legacy network management systems missed it.
In one case, Psytechnics witnessed BT giving a tour to an Abbey employee. The tour ended with a demonstration of PsyQ from a BT customer support agent who had been using PsyQ for approximately one week. The agent showed how PsyQ had reported some bad calls at one of the Abbey’s branch offices within the last two hours, and then he confidently drilled down into the branch data showing the quality and diagnostic information for the affected calls.
PsyQ revealed the issue was either an unreliable link or LAN congestion and so, his first task was to check the access router, which revealed no problem. He then requested a check on the branch router, which revealed a flapping link, and corrective measures were taken.
The agent was asked, “Did you need PsyQ? Surely a router alarm would have told you about the link flap?” The agent’s response was, “PsyQ lets me prioritize customer-affecting faults over the thousands of alarms raised every day. PsyQ saves me time and, in fact, no one knew about the router problem before me.” PsyQ’s MOS and diagnostic capabilities enabled a speedy solution — fault recognition to isolation and identification within 10 minutes.
This example, and most other problems spotted by PsyQ would have been considerably more difficult to characterize, quantify, and locate if the operations team relied purely on customer complaints and legacy network management systems.
What was Learned?
With the VoIP systems now live in all 800 branches, the emphasis has shifted from end-to-end testing to real-time monitoring, so that BT is instantly aware of customer-affecting problems. BT has made further investments in Psytechnics technology and is aspiring to have a deeper integration, such that it works closely with existing infrastructure and network management systems.
Refining the SLA will be a necessary step. Though an MOS of 3.7 had been declared the minimum, the SLA did not allow for exceptions. The VoIP system delivers average quality of MOS 3.92. However, a very small amount of degradation during a call could drop scores below the threshold.
The contract specified a hard limit of 3.7 although, in practice, isolated drops in single test scores do not affect overall customer opinion. Psytechnics will be publishing a guideline for MOS-based SLAs to assist both service providers and purchasers.
In terms of business, everyone is pleased with the benefits of MOS. John Blake, Head of Hosted IP Telephony for BT Global Services said, “Having MOS capability has differentiated our customer proposition from other suppliers of VoIP. Because of the reputation we’ve earned at Abbey, we have significant other prospects.” IT