The amount of data getting generated by businesses of all types has been exploding, and analyzing this information will be key to maintaining a competitive stance in the market. This is the essence of the big data movement. This phenomenon is happening across all industry verticals, but it is especially being felt in the communications industry where network bandwidth is increasing by at least 50 percent a year to accommodate all of the network traffic getting generated.

The big data opportunity is to use the network and subscriber metadata – i.e. data collected about network traffic and subscriber profiles – to help service providers make better network investments and optimize their business models. So, for service providers, big data is about generating the right business intelligence to make better business decisions.

Fixed, mobile and cloud operators continue to make significant investments in policy/DPI platforms and OSS/BSS systems to commercialize innovative subscriber-based services. These systems generate or maintain much of the network and subscriber metadata needed to enable big data analytics solutions. The challenge for suppliers of big data analytics solutions is that the metadata can be spread across many disparate, geographically dispersed platforms in a service provider’s network that are key to operationalizing the network and back-office systems that house much of the subscriber metadata. This includes a DPI platform to generate traffic statistics that identify and classify network usage by subscriber, device, network, and content in real time.  Other metadata sources such as IPDRs, OSS, BSS, DNS and DHCP logs can also be used to provide information about billing events, subscriber-specific service plans, and subscriber demographics.

But when it is feasible for all of this data to be gathered and fused together, many different analytics solutions are possible including:

• network planning and optimization based on long-term trends;
• OTT content and video analytics;
• correlation between network events and call center activity;
• marketing segmentation for campaign management;
• upsell and cross-sell opportunity identification;
• targeted advertising; and
• advanced churn detection based on inactivity.

To realize these solutions, a big data analytics platform must be tightly integrated with the DPI platform, especially since the volume of statistics generated in a large service provider network can overshadow the amount of data from the other metadata sources. This has traditionally been implemented in a batch model where data is extracted from the DPI platform in flat files or spreadsheets and then ingested by the big data analytics platform. This may work for solutions like long-term network capacity planning, but would not be effective for targeted advertising or providing enough bandwidth to an OTT video user on an economy broadband data services plan.

So the ideal is to get the metadata in as close to real time as possible from the DPI platform to the big data analytics solution. The contemporary approach to this is to use an IP protocol called IPFIX. This allows the DPI platform to stream subscriber and network metadata statistics in real time while the big data analytics platform is accessing information from other data sources to ascertain what service plan the subscriber is on, to see for example if there is a real-time upsell opportunity in progress. IPFIX also offers a rich set of customization capabilities by specifying data field mapping and the time intervals for passing information between the DPI and big data analytics platforms. By using a standard protocol it is possible for the DPI platform to easily integrate with many different big data analytics platforms to feed them Internet intelligence.

This technical approach creates a rich set of capabilities that deliver on the promise of big data analytics for service providers of all sizes and types. It puts powerful business decision-making tools in the hands of many different departments inside service provider organizations – from engineering and capacity planning, to marketing and service planning – empowering them to get the big answers they need to run their parts of the business. For them big data does equal big answers.