×

SUBSCRIBE TO TMCnet
TMCnet - World's Largest Communications and Technology Community

CHANNEL BY TOPICS


QUICK LINKS




 
September 2008 | Volume 27 / Number 4
CALL CENTER Technology

Planning For The Unforeseen


Disaster planning and response, also known as business continuity, is one of those key functions that no one likes to think about, and which costs money to set up right and execute. Yet when these events happen, and they will, those contact centers that put in well-thought-out plans will either continue to operate or be back in business sooner, and with less downtime and total costs than those who had treated this matter as an afterthought.

There are a great many tools available to help contact centers respond to, mitigate the effects of, and recover from disasters. They include multiple voice/data/power feeds, home agents, contact /data rerouting to safe locations in-house or outsourced, IVR/autoattendants, outbound messages, employee intranet sites, uninterruptible power supplies (UPSes, see box) and onsite generators.





Contact centers can also take step to lessen potential damage such as by avoiding setting up in high-risk sites such as industrial areas, near airports, and in flood plains. They can site their equipment rooms and their backup power units away from potential sources of water damage e.g. not in basements. The growing popularity of home agents, cloud computing and voice networks, have made those organizations that adopted them less vulnerable to and can more quickly recover from disasters by reducing the numbers of people and assets that can be harmed by these events.

What is more important than these technologies and techniques is having a coherent business continuity plan or BCP. The plan will dictate what tools and methods you will need to carry it out. The key factor is deciding the level of importance of your contact centers to your overall functions i.e. what is the bottom-line impact of losing them. That will help you select the right response solutions.

There is not a moment too soon to consider setting up or updating your disaster plans. With the specters of global warming leading to weird and destructive weather, along with terrorism, earthquakes and volcanic activity, plus the much more common events such as fires, the next disaster is moments away from happening.

To give a complete perspective on business continuity/disaster planning and response, we present the observations and advice of a leading expert in this field, along with one of North America’s largest contact center firms that has had experience in coping with major disasters.

Attainium (www.attainium.com) Bob Mellinger, Founder and President
Every business should have a plan that covers responding to and recovery from crisis, disruptions and disasters. A good plan contains:

(1) Assumptions – what does this plan cover, under what circumstances will the plan be used or not used, what are the key business priorities, and defined recovery time objectives

(2) Response – evacuation and shelter-in-place, how to deal with specific situations like fire, tornados, earthquakes, etc. What are the rally points? Where is the shelter in place locations? Lists of staff designated to monitor and assist with emergency response and their associated responsibilities. Inventory of emergency and first aid supplies. Communications procedures

(3) Business Resumption – what needs to be done to get the contact center back online. Backup infrastructure, recovery procedures and staff. Where will staff work if the contact center facility is unavailable

(4) Schedules – plan testing & exercising, plan maintenance, staff training and awareness

(5) Staff and Vendor rosters plus any other information or documentation that could be needed if you had to operate or manage the crisis from another location

The four greatest disaster threats for contact centers can vary greatly based on geography, facilities, infrastructure, and staffing, however, the priorities for dealing with any disaster threat are the same. They are:

(1) Safety / Security of people – staff, visitors, suppliers – anyone under your roof

(2) Communications – both internal (staff, customers, stakeholders, etc.) and external (media, first responders, as well as federal, state and local officials)

(3) Continuity of Business Operations – getting the contact center back up and running as quickly and painlessly as possible

(4) Asset Protection – minimizing the damage to the facility, infrastructure, equipment and any other assets owned or operated by the contact center

The key lessons learned from recent disasters are first, having a plan. Testing, exercising and maintaining it is a close second. Many organizations have spent time, money and resources to develop a BCP, but then it just sits on the shelf - never updated. As such, the plan is of minimal if any use. An out of date plan can be a disaster in its own right. Develop a master schedule of tasks that need to be performed on the BCP – monthly, quarterly, semi-annually, annually. Assign these tasks to specific people and manage the process like a formal project.

There are steps that contact centers do to avoid or minimize the damage from disasters ahead of time. These are:

Mitigation – This is everything you do to prevent or minimize the impact of a disruption — activities aimed at reducing vulnerability or those performed in advance to decrease impact, loss, or damage. Mitigation is conducting thorough background checks on everyone in pivotal positions. It’s planning how you’ll communicate if the phones go out. It’s having a way to pay employees if the payroll system goes down for three days when checks are due. It’s investing in and implementing the best and latest cyber-security measures for your data and network systems.

There’s probably not a contact center on earth that has sufficient resources to mitigate or prepare for every potential incident or disruption; you have to concentrate on those that are most likely to occur. It is important to know, therefore, the risk of each potential disruption and to decide on which disruptions you will concentrate your resources. This universally accepted risk evaluation equation will help with this effort: Risk = Probability x Consequence (a.k.a. Impact).

Probability, logically, refers to the likelihood that something will happen, while consequence refers to its impact. For example, the probability of a meteorite hitting your contact center is likely quite small, although the consequences from such an event would be devastating. The meteorite, therefore, is probably a 100 on the consequence side and 0.0000001 on the probability side. The risks on which you want to concentrate your mitigation efforts are those in which both the likelihood of occurrence and the potential consequences are in the medium-to-high range.

Most of all, keep things in perspective. Any consequence that involves potential critical injury or loss of life must take precedence over less critical outcomes. It’s important, therefore, to prioritize the risks as well as to identify the ones on which to focus.

Convergys (News - Alert) (www.convergys.com) Carol Fox, Senior Director, Risk Management and Mike Epstein, Consultant, Business Continuity
Our contact centers serve as our clients’ face and voice to their customers. These transactions are often the only personal interaction they have to establish or enhance their customer relationship and increase a consumer’s desire to begin or continue to do business with them. Business continuity plans (BCPs) serve to maintain that critical connection through adverse events, thereby building confidence and loyalty with our clients’ customers.

A strong business continuity plan outlines the resources, actions and skills needed to respond to a diverse set of risks with effective and targeted activities. Convergys BCPs include:

* Risk assessment–Determines the general nature and potential range of adverse events, so that disaster recovery plans can be developed to meet the risks

* Emergency response–Develops procedures to activate an immediate, orderly response to emergencies, to provide for the reasonable welfare and safety of employees and to protect and/or recover critical assets and financial records

* Method to identify critical applications– Develops a methodology to identify current and future critical applications

* Critical file backup and off-site storage guidelines–Develops procedures for the retention, rotation, and retrieval of data and files

* Critical resource planning–Identifies procedures to obtain critical resources necessary for recovery, including hardware, software, space, files, human resources, power, finances, and related considerations

* Disaster recovery teams–Identifies key team personnel, establishes and assigns team responsibilities, and trains each team member to perform specific duties

* Backup processing or service centers–Identifies processing/service sites and develops procedures for alternate operating sites when the primary processing or service site is inoperable

* Restoration of facilities–Develops procedures to promptly assess damage to the primary facilities, and to make possible an orderly restoration of site operations

* Testing and maintenance–Provides periodic updates to assess the relevancy of the plan and testing strategies to maintain its validity

At Convergys, we approach potential incidents from a different direction. Rather then focusing on an event type and developing a plan for each, we begin through a risk assessment by focusing on a potential worst case scenario, and the disruption it would have on our operations at a given site. We then focus on the resource impacts that event would have, and how we care for the replacement or reallocation of those resources. This methodology allows us to develop one plan which supports response and recovery in potential less severe situations, as well as worst case.

The Convergys BCP is designed so that the basic methodology, activation, and recovery process cover many different scenarios. Not all potentially catastrophic events will result in the activation of the BCP. The objective of the plan is to complement and enhance the standard operating procedures. Therefore, the BCP will be activated only when normal response procedures are no longer sufficient to handle the situation, and the disruption is expected to be of undetermined duration.

There are several key lessons that have been learned from 9-11, Hurricane Katrina, and the Midwest floods in terms of disaster preparedness and response that should be incorporated in disaster plans.

In the first days – you are on your own. Your continuity program must be robust enough to withstand a major disruption without relying on government support, your primary suppliers and in many cases much of your staff. That’s why we rely on a modified incident command management system, bolstered by training and exercises, to provide our local incident command teams with the practice and confidence to deal with such situations. Each site team is composed of senior operations and staff managers, who are trained to deal with both disaster planning and disaster recovery. The company-wide training encompasses incident response actions, such as providing for the reasonable welfare and safety of employees, and implementing specific plans for recovery operations. Site incident command teams have the authority to procure resources, as needed, to minimize work disruption, lessening the impact to your business. Site Incident Command Teams are supported by corporate resource areas such as global risk management, facilities management, and human resources.

The lesson we learned about the importance of disaster preparedness actually came from one of our employees after Hurricane Frances in 2004. She highlighted the personal economic impact of disasters risk while thanking our site leader for being prepared, “All I have left are the clothes on my back and the items in my purse. My house is gone, my car is gone, but I have a job and my neighbors don’t.”

There are a broad range of communication tools are on the market today to enable contact centers to stay in contact with employees, customers, and users before, during, and recovering from disasters. Given the ease of access to information on platforms from traditional media to the internet and mobile phones, multiple channels can be utilized. More important than any one tool is a cohesive integrated strategy for delivering messages to various stakeholders throughout an event. At Convergys, we use internally developed channels from IVRs and online postings for site updates and employee contact to broadcast messaging for incident command activations.

5. There are also multiple strategies e.g. shutdown, going to generator power, call rerouting to look at in enabling business continuity. They generally are considered depending on the client program, and the answers to a series of questions. Who will perform the work? What vendors will be supporting the same or related business for the client? What is the geographic distribution of the work? What skills are required, and where are they available? What are the needed recovery time objectives? The appropriateness of any particular strategy is driven by the environment and client needs. The choices and implementation of IVR and or call routing options are driven by customer expectations as well as our geographic diversity. In response to Hurricane Dolly, our Brownsville, Texas contact center remained open on generator power, but employee attendance was anticipated to be light (and it was). Therefore, the planned call routing to supporting contact centers was executed so that client operations were not disrupted.

To avoid or minimize the damage from disasters ahead of time contact centers need to ‘Prepare, Prepare, Exercise, and Prepare’. While it is natural to experience stress during a disruption, panic really only occurs when people feel unprepared to deal with a situation. Protect your most valuable resource by giving them the material, training and experience to handle the situation. For those situations which simply cannot be handled, provide clear direction on what can be done, and who can provide assistance both for the business, and for your people. At Convergys, this means planning, prevention, preparedness, training, and communication, utilizing standardized processes and planning tools. One of the key elements in the BCPs is the emphasis on techniques and strategies to prevent power failures, fires, missing data, and physical destruction of equipment or facilities. We have included these areas where prevention is the objective and have integrated these strategies into our normal business functions. Constructing to certain specifications, using generator power, rerouting calls, and delivering needed supplies to our employees from unaffected sites allowed us to continue business operations even in the devastated areas. CIS

UPS Systems: On Guard for Contact Center Disaster Response Uninterruptible power supplies (UPSes), which provide emergency power from batteries and surge protection, are the business continuity guardians for contact centers, formal and home agents. Every contact center and home office should have them. Here’s why:

* UPSes prevent computer failure--and in today’s contact centers computers also include switches and routers—caused by voltage spikes by filtering the electrical supply

* UPSes provide backup power to span brief outages: typically up to 15 to 30 minutes depending on capacity based on the number of circuits and demand on them. That enables both ongoing operation and/or orderly shutdown that give enough time to save and backup data, switch on auto-attendant messages or IVR systems or reroute calls. The UPS capacity is enough to cover most outages, which are no more than five minutes.

UPSes are an essential component for generator-supplied emergency power systems, which are a necessity where contact centers provide a critical/can’t fail function and in disaster-vulnerable locations. UPSes provide the electricity that spans the gap between the loss of utility power and generator startup, and from generator shutdown to restoration. They eliminate the voltage surges and sags that occur during these transitions.

UPS capacity is typically measured in volt-amperes (VA). The best practice when deploying any emergency power is connecting only essential circuits. That means in most cases no air conditioning or fans, and only emergency lighting. Applying those guidelines as a rough guide, a 100 workstation contact center that does not have an on-site data center would typically require a UPS in the range of 40 to 80 kVA (kiloVolt-Amperes).

UPS suppliers have been developing their solutions. Emerson (News - Alert) Network Power recently expanded the capacity of its Liebert NX UPS with Softscale™ technology to 120 kVA from 80 kVA. A 200 kVA model will be coming out by the end of this year. Softscale technology allows customers to increase their UPS system capacity in 20kVA increments (40 to 80, 80 to 120, and 120 to 200) without adding additional hardware.

While the smaller NX units can be paralleled to create additional capacity, the supplier does not recommend any more than three UPSes for availability and one for redundancy for maximum uptime and economy.

Home agents should also have UPS systems for the same reasons as formal contact centers. Residences are also subject to power surges and outages that UPSes protect against.

UPS makers have been devising new more effective products for the growing home and teleworker market.

American Power Conversion (News - Alert) (APC) has made its Back-UPS® ES product line more energy efficient, smaller, and more esthetically appealing. These new units are available from 350 vA to 750 vA.

The 750vA model features SmartShedding™ Technology, which allows the product’s master outlet to sense when the computer has been turned off or has gone into sleep mode, so it can shut off power to unused peripherals plugged into the controlled outlets, saving electricity, and money.

Rebooting Your Contact Center
When your power goes down, and if you have to close your center, you need a means to restore your operations following the ‘event’. Inova Solutions has developed a high availability (HA) solution that does just that. The HA tool is housed in a virtual server that resides on top your physical redundant servers, collecting the same contact center key performance indicators. In the event of a catastrophic system failure, this virtual server restores access to the information in as little time as it takes for the redundant server to reboot.

CIS Magazine Table of Contents









Technology Marketing Corporation

2 Trap Falls Road Suite 106, Shelton, CT 06484 USA
Ph: +1-203-852-6800, 800-243-6002

General comments: [email protected].
Comments about this site: [email protected].

STAY CURRENT YOUR WAY

© 2024 Technology Marketing Corporation. All rights reserved | Privacy Policy