TMCnet Feature
July 28, 2021

Case Study Managed Services - When getting under a customer's skin means becoming one

In an ironic twist of fate, solving complex operational issues can demand a juxtaposition of one’s role from managed service provider to end customer.

The incident described in this Case Study illustrates how, to gain a deep understanding of customer needs, one must fully adopt an end-user mindset.

Perhaps the ultimate pathway to value creation?

The Challenge

In a telecommunications network, each and every phone call, text message, data session, etc.

– generates one or more call detail records (CDRs), which are then used to charge the subscriber. Even if the value of a CDR is zero, it must be fully accounted for in the billing process. Everything must tally.

But occasionally, as in every other business or walk of life, Murphy’s Law applies: anything that can go wrong will go wrong. Admittedly, because of the vast amount of data that telecommunications devices generate every day, something is bound to give from time to time. And similar to airline disasters, the fault is most often attributable to human error.

For one particular Customer on one ill-fated Friday afternoon, Murphy raised his ugly head. Unexpectedly, the Customer’s switch (MSC (News - Alert)) started to flood with calls. This lasted for several hours before any remedial action could be taken.

The first indication of a problem was an apparent discrepancy between the switch and billing mediation. The Managed Service team arrived at this finding, simply because many of the alarms vanished as soon as the reconciliations were reset and re-run. In other words, because new data became available after the alarms were originally raised, the alarms could clear from the system.

We hypothesised that the Customer must have had an issue with their file server, causing a processing delay.

However, shortly after, our iGenuity Revenue Assurance system showed that some files were missing from one of the Customer’s switches. We received 27 alarms, whereas typically we might receive one alarm every three months.

Before informing the Customer, the Managed Service team once again diagnosed the problem, using iGenuity, and found that the missing files were in fact waiting to be processed by the file decoder – and had been backing up for many hours.

This would normally indicate that the decoder had become ‘stuck‘, which is almost unheard of on the Customer system, so the Managed Service team checked the volume of CDRs processed by the decoder.

It can be seen from this simple graph that a lot of CDRs were processed, but there was a backlog of files (although not enough to trigger a backlog alarm).

The next thing checked was whether a particular type of call was contributing to this unusual increase. Clearly, from the image below, the number of “Transit“ calls is extremely high. Under normal circumstances, this call type should be about double the number of “Mobile org. call“ calls, but not hundreds of times bigger!

The Managed Service team performed some further checks to discount a possible error with the decoder that indicated this was a real problem. As it turned out, most of the calls related to one number (+***66666107)

The Managed Service team reported this to the Customer, who was unaware of the problem.

It transpired later that day that a Customer employee had mistakenly misconfigured the routing information on the switch (MSC), generating millions of incorrect calls... and upsetting a lot of subscribers. The Customer was not aware of this until our Managed Service team reported it to them.

It took a sizeable team two days to resolve this problem. It would have taken much longer had our Managed Service team not brought it to the Customer‘s attention.

The Conclusion

This Case Study points to the merits of working with a Managed Service provider.

Going beyond our ability to access and crunch customer data for monitoring, reporting and problem-solving purposes, there are other important reasons for outsourcing some or all of the RA function.

Ironically, our innate human propensity toward laziness is a driver of positive change. By seeking to avoid a repetition of routine mistakes, we naturally find ways to get things done more efficiently.

“Mistakes are meant for learning, not repeating.”

From the Customer’s perspective, the crucial question is whether a repeat of this incident could be avoided in the future? From the supplier’s perspective, could the incident have been identified earlier?

The answer to both questions is an unequivocal “yes”, but it involves admitting to mistakes, and then making an honest effort to not letting them happen again.

Through our daily interaction with Customers, we become de facto consumers of our very own product, and therefore more determined to ensure that this product delivers the results we promised.

This approach results in a far healthier partnership between Customer and Managed Service provider, and it also supports a robust product improvement cycle. When executed well, this relationship can contribute significantly more value to both partners, and everyone wins.

» More TMCnet Feature Articles


» More TMCnet Feature Articles