(This article originally appeared in the August 2010 issue of CUSTOMER INTER@CTION Solutions)
Disasters of all kinds can strike at any time, any where. As contact centers become increasingly invaluable to their organizations, they need methods and solutions that can enable them to manage through these events. That includes avoiding, or if that is not possible, preparing for and staying in operation during them, or if the worst happens, orderly shut down and quick recovery.
Dick Csaplar is senior research analyst, virtualization and storage, Aberdeen Group, a Harte-Hanks Company. In his report, “Disaster Downtime – How Much Does It Cost?” (www.tmcnet.com/32801.1) companies reported that downtime can cost as much as $1 million - $3 million per hour.
What type of downtime disasters are most likely to occur? Csaplar says “micro events,” meaning the loss of a power supply, disk drive crash, hacker, virus or accidental data deletion, are much more common disasters impacting contact centers than “macro events,” like blackouts, floods, terrorist attacks, hurricanes, earthquakes or tornadoes.
Even a small disaster can leave an unprepared facility crippled. Bill Durr, principal global solutions consultant, Verint (News - Alert) Witness Actionable Solutions, pointed to a water main break that caused six feet of water to fill the basement of the Dallas County Records building, shutting down the county’s computer system for nearly a week. Many of the county’s departments were impacted by the disaster, as access to data was limited and computers were rendered useless.
To minimize the harm from micro and macro events, the Aberdeen Group researcher points to a series of best-in-class (BiC) practices in another report, “Disaster Avoidance and Disaster Recovery: Making your Datacenter Disaster Resilient” (www.tmcnet.com/32802.1).
These include creating formal and regularly updated business continuity/disaster recovery (BC/DR) plans, tracking DR events and reporting them against the plans, having a senior manager responsible for DR performance and regular testing of them.
In putting together a BC/DR plan, a major philosophy of disaster avoidance is to eliminate any single point of failure, Csplar points out. Spreading assets across electrical grids, climate zones and storm paths can greatly reduce the effect of a macro event. However introducing more computers, networks and phone lines into the equation increases the likelihood of micro events. Contact centers are dependent on technologies that can and do fail.
“You have to look at all contingencies and measure the potential impacts,” recommends Csaplar. “That way you can have an intelligent conversation about the amount of downtime you can afford versus the amount of investment to ensure against it. In our research, we found that 33 percent of BiC organizations know their hourly cost of downtime and use it for planning and investment purposes.”
The Hosted Solutions Option
To survive macro events/disasters, contact centers need to back up applications and data off site. Employing hosted solutions, such as routing and dialing, recording, scheduling, and e-screening/training, is one strategy to avoid losing valuable software while enabling seamless service to customers from staffers’ homes and from temporary locations. Hosting can also limit micro events and prompt urgent recovery for IT reliability, as support is not the tremendous cost it can be, but is, in fact, a key selling and client retention point for providers. It also avoids potential compatibility issues that may arise if firms use live agent outsourcers as backups.
Jeremy King, executive vice president, technology and products, LiveOps (News - Alert), points out that not all contact centers have or can justify backup power. His firm protects its servers with battery UPS system and generator backups, along with multiple data centers to enable applications to function if there is a major outage. Also, most contact centers do not have dual Internet connectivity or immediate access to a range of carriers that can be switched at a moment’s notice, which LiveOps provides to protect against what he says are “fairly common” Internet or telephony outages.
inContact has built in BC/DR from the ground up. Its applications are housed on servers connected to network points of presence that are located in two facilities, in Dallas, Tex. and Los Angeles Calif. These centers are supported and monitored by a 24/7 carrier-grade Network Operations Center (NOC) located at its corporate headquarters in Salt Lake City, Utah. In the event of failure, the NOC can operate remotely.The Dallas and Los Angeles sites lie in different climatic and geologic zones, meaning if one facility is knocked out, it is highly unlikely the other will also collapse. Each center features earthquake- and explosion-resistant construction. They have redundant servers and cooling with mirrored data backup between centers via leased lines. By the same token, Salt Lake City is in a separate zone from the two server locationsInContact has generator-backed-up UPS systems, fault tolerance, redundant network components and redundant ISP connections and RAID technology. There is database and logged calls replication. Voice/data connectivity is provided by multicarrier routing agreements with Tier 1 providers. This ensures that, even when there are widespread outages on one carrier, service continues on the others.
The company also has hardened phone and data networks in its corporate offices and automated inbound office call handling. It can use home agents, technicians and other professionals to support critical customer care and operational functions, should the corporate building become compromised.
Contactual’s (News - Alert) Advanced Virtual Tenant Architecture provides a multi-platform, fully redundant architecture that is deployed in five data centers worldwide. Tenants are virtualized, so they are not tied to any particular server. If there is a need to switch platforms, recovery time is less than one minute and customers’ configuration data, prompts, recording, historical data and all other aspects are preserved.
Contactual’s solution makes it easy for contact center agents to work from any location. Its Agent Desktop is 100 percent Web-based and firewall friendly, with no plug-ins or downloads to configure and no special equipment required at the agents’ locations.
“On-premises solutions are extremely costly and time-consuming to duplicate and maintain, and typically require specialized hardware and software to support agents working remotely,” explains Kimberly Odom, senior director of marketing at Contactual. “While organizations may have a back-up call center provided by an outsourcer that can be utilized in case of a disaster, the interaction flows and systems being used are different, which hampers recovery efforts.”
Alerting and Staying in Touch with Employees, Customers
The often literally life-and-death key to coping with disasters is instructing and drilling employees on disaster plans, so they can be executed safely and effectively. This requires methods and tools that rapidly and clearly alert and inform and enable communication with staff.
Electronic reporting, such as readerboards, wall displays and desktop software that deliver operational and performance metrics are a great means of communicating impending or current disasters and what steps to take. Agents and supervisors already know to glance at these screens and lines and, in emergencies, IMs or audible alarms can be used to drive attention to them.
LCD screens are becoming a popular electronic reporting mechanism. Spectrum’s (News - Alert) VectraView software can drive disaster drill and actual event information to strategically placed units wherever they are, whether on call floors, in break, conference and training rooms and in reception areas. They can also deliver live media, such as weather warnings and news about accidents, derailments, fires and floods or other events that may impede employees’ commutes.
Disaster alerts and updates via electronic reporting can also be delivered to agents and supervisors at home or remotely or in temporary locations. Inova Solutions’ Desktop Presenter solution is preconfigured for easy deployment on desktops and laptops alike, with no on-site IT staff assistance needed. All out-of-center staff have to do is load the applications into their computers and they can connect to Inova’s LightLink middleware, which manages the reporting when they first log in; they will need to be given permission including password access beforehand.
Automated hosted outbound notification solutions can instantly alert, inform and ensure continued communication between agents, supervisors and customers, and are accessible through most any device. They can supply key numbers to call and employee or supplier identifiers, websites to log into or, when delivered by e-mail, attachments or links to documents and resources.
These tools are becoming increasingly effective. MIR 3’s notification platform, inEnterprise version 2.19, provides increased robustness and power for tighter integration with leading business continuity and IT applications via expanded API functionality. There are also improved workflow capabilities for enhanced ease of use and intuitive operations. The solution also enables call bridging into contact centers, emergency lines and conference calls.
Varolii recently introduced a comprehensive workforce continuity suite with four distinct offerings:
· Varolii First Responder quickly notifies response team members and executives of incidents, fosters cooperation with real-time status, and then effectively communicates when the incidents have subsided. Administrators can generate an immediate message and then automatically escalate to an alternative communication channel if the first try doesn’t get through.
· Varolii Enterprise Business Continuity rapidly expands communication from the first response team to a worldwide workforce, accomplishing this in minutes, compared to hours for typical manual phone-tree emergency communications.
· Varolii Employee Accountability combines outbound notifications with inbound messaging capabilities so companies can keep employees informed, collect and act upon real-time information about employees’ status, determine where assistance is needed and minimizes downtime by gaining a clear picture of staff availability.
· Varolii Pandemic Planning notifies employees of their company’s pandemic plan, supports telecommuting and other HR procedures, communicates healthcare-related information and gathers status from individual employees.
The key to effective notifications is finding and using available communications paths to personnel wherever they are – in the centers, at home or in mobile environments – and to suppliers and key customers. This requires dynamic switching to alternate routes. Equally critical is having the messages prepared ahead of time based on all possible scenarios. When these events about to or have occurred is not when one should try to compose what to say to others.
The Everbridge emergency notification system (ENS) helps organizations quickly and reliably communicate across multiple contact channels, including mobile phones, landlines, smartphones, wireless PDAs, e-mail, and text messaging, in the order specified by individual preference, until messages are delivered and confirmed by recipients. The Everbridge ENS leverages an intelligent communication platform spanning message delivery across 30-plus different contact paths. Integration between Everbridge’s Aware ENS application and iView Systems incident management and reporting product enables firms to seamlessly collect all information related to incidents in real time and trigger mass notifications based on predefined criteria. They can quickly and easily ensure that all necessary individuals are notified and aware of serious events without wasting time toggling between multiple applications.
“Because the average person experiences an 80 percent reduction in their ability to process information when a disaster strikes, we provide a message repository and scenario management capability so organizations can prepare message libraries and message maps in advance,” says Marc Ladin, vice president of global marketing at Everbridge.
Managing Dispersed Workforces
When disasters loom or strike, knowing and tracking where your agents and supervisors are and being able to schedule them appropriately becomes critical to ensuring effective response and recovery. Counting heads is no longer an option when facilities have to be evacuated and staff dispersed to temporary locations, while other centers’ employees have to be called in to handle the extra load.
Having an integrated workforce optimization (WFO) suite is vital in these circumstances, says Verint’s Durr. WFO suite solutions offer Web-based portals, such as an agent portal where frontline workers can view their schedules, access scorecards, connect with supervisors and review performance, among other day-to-day functions. These types of unified solutions help ensure the proper management of a dispersed workforce in an emergency.
“When a crisis unfolds, multiple facilities and at-home agents must function as a single, seamless entity,” Durr points out. “Technology, both physical hardware and software, can support that integration by allowing multiple organizations in disparate locations to operate in sync, helping ensure that customer and citizen service continue uninterrupted.”
Work at Home During Disasters
At-home working is a proven BC/DR solution. Distributing workforces minimizes loss of life and injuries, and severed and shrunken revenues and service from localized or regional disasters. It also provides protection from massive power outages. Yet even at-home solutions can be disrupted by events.
Alpine Access, an exclusively home-agent BPO firm, minimizes these possibilities by implementing multiple remote server access for its home agents’ Windows-based applications. Its architecture has fast application deployment, implementation and restoration to a client’s network in the event of failure. When a disrupting event is forecast or identified, the clients’ databases can be moved to a secondary server out of harm’s way and the workload transferred to remote agents who are already on the clients’ secure network to continue business as usual.
TeleTech (News - Alert) offers home agents as part of a comprehensive array on domestic, nearshore and offshore bricks-and-mortar contact center services. It recently expanded its TeleTech@Home service to provide technology and BC/DR solutions.
Todd Stott, TeleTech executive director converged communication says if home agents are geographically dispersed from the brick and mortar centers they support, as his firm recommends, they typically are not faced with the same disaster situations. For the rare occurrences they are, there should be advance planning and, if need be, investments.
Firms should encourage employees to monitor local news and weather information so that they can remain abreast of crucial evacuation or safety information, and share any emergency response information the companies receive. They should also account for all home/remote employees following disasters and provide emergency support, if needed. Home agents may also require UPS systems and wireless telephone access.
“Companies should devise a formal communications plan and utilizing alternative contact methods if traditional communications fail,” recommends Stott.
[Sidebar]
Lessons Learned From Hurricane Katrina
August 2010 marks the fifth anniversary of Hurricane Katrina, which was one of the most devastating storms to have hit the United States. According to Wikipedia, at least 1,836 people lost their lives in the actual hurricane and in the subsequent floods, with total property damage estimated at over $90 billion. While such events will continue to happen, what is important is what can be learned to avoid, or at least minimize, the effects and recover rapidly from them.
Sitel is a leading BPO firm with contact centers worldwide, including in the stricken states. That includes in Bogalusa, La., which had been severely hit, but which survived the infamous storm.
Kathleen Banashak, senior vice president, operations support, Enterprise PMO at Sitel, reports thatthe most important lesson her firm learned is that disaster communications must occur at the proper time and to the right audience.
Sitel has a communication strategy and plan that enables it to assemble a core response team within minutes of receiving alerts. All first-line managers know to immediately reach out and verify the status of every employee. Client services team members engage and alert others through mobile phones and text messaging, “which have improved our ability to do so quickly and efficiently,” says Banashak.
Sitel can also send broadcast messages to mobile phone numbers. Its IT department has set up a satellite phone solution for sites in high-risk areas to enable communication with the leadership team in an at-risk area, in the event that mobile phone services are overwhelmed.
The firm has a communications team that is not part of the recovery team, which facilitates all elements of communication. This allows the operations team to focus on response and recovery.
Sitel BC/DR plans include preparation activities, response activities and recovery activities. All leaders are trained and knowledgeable about the elements of the plan and the importance.
“In the five years since Katrina, Sitel has been able to successfully apply lessons learned to such disasters as the U.S. Hurricanes Gustav and Ike, along with the Typhoon Ondoy and Cyclone Parma that hit the Philippines and, most recently, in response to the earthquake in Chile,” says Banashak.
[sidebar]
Multilingual Communications During Disasters
The United States is becoming a multilingual country, principally Spanish, but also Chinese, Tagalog, Vietnamese and Korean, which means needing to have people who can speak these languages to connect with customers. When disasters threaten or strike, these employees may disperse to their homes or evacuation centers with their colleagues, resulting in the loss of that key capability at a time when calls flood in from victims, including those whose native language is not English.
One way to ensure language continuity is to consider partnering with a language service provider (LSP) for interpretation and translation support. LSPs have a network of highly qualified linguists located globally, in different climatic and geological zones (i.e., out of harms way), ready to take critical calls on demand.
Language Services Associates (LSA) is a large and growing LSP with 5,000 global linguists who interpret and translate, in over 200 language offerings, for more than 2,000 clients worldwide across all industry segments, including contact centers. Its services can be set up very quickly and its implementation process includes various forms of user training, from how to use dual handset telephones to tips for working with interpreters.
[sidebar]
BC/DR and UC
More standard applications are now being fitted with BC/DR-supporting features. For example the Zeacom Communication Center now includes crisis and emergency response communications systems as part of the ZCC Business Process Automation solutions.
The ZCC Executive Mobile feature has a Web-based module, which allows telephony control and access to presence from a Web-based GUI that users log into from their homes. Staff can set current “location” to “home,” which essentially redirects all extension calls to home or mobile numbers. ZCC conferencing functionality allows organizations to remain connected with their staff and teams, enabling them to collaborate with each other no matter their location. ZCC also provides outbound IVR alerts and notifications.
Alpine Access
Contactual
Everbridge
InContact
Inova Solutions
Language Services Associates
LiveOps
MIR 3
Sitel
Spectrum
TeleTech
Varolii
Verint
Zeacom
Brendan B. Read is TMCnet’s Senior Contributing Editor. To read more of Brendan’s articles, please visit his columnist page.
Edited by Stefania Viscusi
