Webinar Transcript: Bulletproofing Your Disaster Recovery Plan with a Resilient Network
June 16, 2010
12:00 PM ET
Hello and thank you for joining us today. This is David Strauss representing Comcast Business Services, and I'll be your moderator for this special business webinar entitled Bulletproofing Your Disaster Recovery Plan with a Resilient Network.
Comcast, along with Cisco, are proud to present this first event as part of an ongoing webinar series you'll be hearing more about.
Before we get started with our distinguished guests, here are some ways for you to get the most out of this webinar. If you're just listening to us over the phone, I'd urge you to login and view the slides online. Just go to www.business.comcast.com/webinar. Click on the orange login tab and enter your email address. Then you'll be at the presentation portal.
We strongly urge you to ask some questions along the way. At any time you can submit a question through the portal. And during the last five or ten minutes, our speakers will answer your questions. Also, I'd like to remind you that you can win a new High-Definition Flip Side camera. And to be eligible you must complete the survey at the end of the webinar, and three winners will be selected.
Now onto our panel of experts.
First, we're honored to be joined by Dave Paulison. Mr. Paulison, who is the Senior Partner of Global Emergency Solutions, had a long and distinguished career in emergency services with government. And for seven years he served as our na-- served our nation as the Administrator of FEMA. In this role, Mr. Paulison managed a $9.6 billion annual budget and a 16,000-member workforce to prepare for, mitigate against, respond to and recover from domestic disasters and emergencies, whether natural, manmade or an act of terror.
Then we'll hear from Stephanie Balaouras, the Principal Analyst and Research Director for Security and Risk for Forrester Research. Ms. Balaouras leads a team of analysts who provide research and advisory services on topics such as IT infrastructure security, IT security framework and application and data security. She also provides Forrester's coverage on specific risk topics, including business continuity, IT continuity disaster recovery and backup and recovery.
Finally, we'll hear from Kevin O'Toole, the Vice President of Product Management and Strategy for Comcast Business Services. Mr. O'Toole will briefly address the benefits of a last-mile alternative with diverse network facilities for extremely robust disaster preparedness and recovery. Kevin is responsible for the lifecycle management of Comcast products serving the needs of enterprise, small business and carrier customers. He's also on the board of the Metro Ethernet Forum.
So with that, let's dive in here. And I will have Dave do the first presentation. You should see this up on your screen.
If I can figure out how to do the slides, we'll be in good shape. Thanks for being on, folks. I think the number of people that are on, which is up into the hundreds right now, shows important this issue is.
One of the things that I've found during my history as-- not only as the FEMA Administrator, but as the US Fire Administrator and Director of Fire Services in Miami Dade County, is that those businesses that prepare themselves, those businesses that are-- have planning in place, really get back in place quickly. But those that don't fail. You know, these lessons that we learned, that I've got some listed there under World Trade Center attack, the wildfires in California, the ice storms this past winter, and also just this last week the flooding in Oklahoma City shows that anything can happen.
But what I've found is that the smaller events are the ones that really have an impact on small- and medium-sized businesses because it doesn't have to be a catastrophic event. It just has to affect you. So it's the tornadoes, the floods, the fires, the power outages, the sometimes cyber attacks is what causes businesses not to survive and not have the resiliency built in to recover.
You see there I wrote down we lose up to 40% of small- and medium-sized businesses after a disaster. The truth is it's almost up to 60% that we lose, and it's for several reasons. The three things that I list is, one, lack of imagination. We cannot imagine what can happen to our business. We can't imagine what we need to do to prepare. We can't imagine how we have to keep our people informed and our suppliers informed.
Two, we have a lack of investment. Lack of investment in planning. Lack of investment in training, exercises, flood businesses. We don't invest in harding (ph) our facilities to make sure they can be returned back into business as quickly as possible, that's talking about resiliency. We have of investment in backup systems. Often, most of the people that I see that fail it don't have insurance plans up to date.
I remember walking into a dress shop in Kentucky when we had floods there, and the woman had just opened the store, had several thousands dollars worth of dresses hanging in 4 feet of water. And she looked at me and said I don't have any insurance. I don't know what I can do because I can't survive this. And she went under. And we see that time and time again.
And then also a lack of investment in employees. The-- and we'll talk about that a little bit deeper in a few minutes. Also a lack of communications. We fail oftentimes to communicate with our suppliers. Can they-- are they affected by the same disaster we are? Are they going to be able to deliver supplies to your store? A lack of communications with our customers. The customers need to know are you going to be back in business? Are you going to be open? Or they'll take their trade somewhere else. And a lack of investment in our employees.
The emergency planning part of-- for our employees is extremely critical. Two-way communication we deploy before, during and after the disaster will determine whether they're going to be back at work helping you. You know, whether you have a newsletter, you're on a-- you have a internet sys-- intranet system, emails or any type of communications, your employees need to know what's going to be expected of them should something happen. You know, you can setup a call-- a telephone calling tree or even have a password-protected thing where people can call in and say, yes, I'm okay, and, yes, I can come to work, or, no, I'm not okay.
One of the lessons that I learned, the hard lesson that I learned when I was a fire chief in Miami Dade County, Hurricane Andrew coming through South Florida. A devastating hurricane. But 250 of my firefighters lost their homes, but they had to work. And it was sometimes two or three days where they knew whether their families were alive or dead. And I knew if I didn't do something to take care of my fam-- my employees' families, I wouldn't have them next time we had a hurricane. So we took things in place to find-- we found a secure building for them where the families that were-- the firefighters' families that were on duty could come and have a safe place to stay so the firefighters didn't have to worry about that.
It's those types of things that you have to look out for because your employees are your-- the key part of your business and probably your most valuable asset.
The-- we need to make sure the employees know exactly who is going to be responsible for communicating with the employees. Do you have employees with disabilities? And so how are you going to take care of those and make sure those types of things are in place?
So I think those are where the employees are a key issue to getting you back in business again. When you consider that the number of disasters that we've had over the last several years has really increased, and I don't know if-- can't blame it on global warming or anything else. It's just a fact of life that we have more and more types of storms. They're stronger, they're bigger and affecting more people. And if you put a plan in place and have a-- you know what you're going to do, and your employees know what you're going to do during a disaster, you'll find that you'll have a much, much better chance of surviving a storm.
If I can get to the next slide.
So being prepared does make good sense. And let's talk about some of those things. The continuity of operations planning is really one of the most important things you can do. You know, I tell people to make sure you assess your company function. What are the key things, both internally and externally, that keeps your company viable? What staff is important to you? What materials? What procedures? And what equipment is absolutely necessary to keep your business operating, and you need to know what those are. You know, who are the key employees that need to be at work to keep that business open, keep it operating?
You know, do you have a business flowchart? And you also need to identify what operations are critical to your survival. One of the things that I often see is, yes, employees come back to work, but there's no payroll system in place because it's been either you've had a fire or you've had a flood. You know, who is going to make those really financial decision-making things? They have to expedited and have to work very quickly.
One thing we often don't like to talk about is about your secession management. Who is going to be in charge if somebody either cannot get in-- get to work or is incapacitated in someway? And you really would like to have somebody that is not located at the business somewhere outside that can come in and help you run that business.
You know, identify your suppliers. Identify your shippers. And what are the businesses that you interact with on a daily basis. Make sure you know who those people are, and make sure that they're-- you have a list of those where everybody can see them.
I also advise people to make sure that you have a professional relationship with more than one company. You know, we all have a vendor that we like the best and brings our supplies in. But what if that vendor cannot operate? You-- do you have a backup vendor? Do you have somebody else who can give you the same types of material to keep you in business? You know, so keep a contact list of those critical business contractors, that they're going to make sure that your supply chain is continued and running.
And what are you going to do if your building or your plan or your store is not accessible? Do you have a backup plan? Is there somebody else that you can go in business with? Is there somebody else you can work with to make sure that if your building's not accessible, where can you operate out of? Most small businesses, if they're not open within two weeks, will never open. So you need to make sure that there's a backup place for you to operate out of, even if it's out of your house. Do you have the systems, computers and things like that to continue to do that?
The crisis management procedures and (inaudible) responsibilities need to be laid out ahead of time. You know, make sure that everyone knows that is involved what they're supposed to do and make sure you have other people trained for backup help because oftentimes everybody can't come to work.
Are you in a building where there's other businesses? Have meetings with them so everybody in that building knows what the others are going to do to make sure that that system continues to work.
And then lastly on that piece of it, is make sure that you exercise that and practice that at least once a year with your employees. The key is those employees, your managers and yourself know exactly whose going to do what and who is responsible for what.
You know, I always talk about emergency supplies also. You know, even in a small business you should have some type of emergency kit with some water, food, you know, some type of a NOAA radio. I recommend keeping that in a small business so you know if it's a tornado coming or a flash flood, like we just saw last week.
Make sure your important records, your building plans, your insurance policies, all of that is duplicated somewhere else outside of the business. And the ones you do have there, make sure they're in a fireproof container.
You know, tell your employees about what emergency supplies are there and what they can depend on, when people-- what they should bring with them. You know, you've got the water, food, your radios, flashlights, first-aid kit, you know, things like that.
If you do that, if you really prepare and make sure that you have a plan in place, more than likely your business will survive if you have any type of a catastrophic event. And like I said, it could be something simple as a flood or even a fire.
You know, and now I have to do my fire thing because that's my background. You know, make sure that your building has been inspected for fire safety. Make sure you're in compliance with local fire codes. You should have smoke alarms in your building, fire extinguishers. It'd be nice if you had automatic sprinkler systems, and I know sometimes those are really expensive to put in. And some type of system to warn your employees and making sure they know what the exit routes are out of your business if you have to evacuate.
You know, those are really simple, basic steps that each of us can take to make sure that our business is going to be resilient. You can get back up operating very quickly in a couple of days if something does happen because our economy really thrives on small businesses much more than any other size of companies that we have out there.
And I know we'll answer questions at the end of the conference, so that's part of my part.
Thank you very much, Dave. Appreciate that. And now I also want to remind the audience, please don't be shy in submitting your questions. Easy to do on this portal. And now we have Stephanie Balaouras of Forrester Research, who will be presenting the shift from disaster recovery to IT continuity. Stephanie?
Great. Thanks. And thanks, everyone, for joining me today.
So if there's something that I want to leave everybody with today on my personal presentation is when you think about traditional IT disaster recovery, it really helps to begin to shift the conversation from IT disaster recovery to IT service continuity or just IT continuity. And one of the reasons why I want to emphasize that is-- well, a couple of reasons.
First and foremost is I think when we talk about disaster recovery, unfortunately that term itself tends to focus us only on catastrophic disasters. And as David just described very eloquently, it's extremely important to prepare for those catastrophic disasters. But what often happens with senior management when you're trying to build a business case for investment in IT continuity and backup and the other technology that you need to build a really resilient datacenter and computing infrastructure is they often see it as a very expensive insurance policy for things that are very rare.
The reality is when it comes to not only declared disasters where a company might be forced to fail over to another datacenter, another computing site that they might have, if you look at the causes or if you look at just major causes of computing disruptions and business disruptions, it actually tends to be really mundane events. The number one cause is actually power outages. It's followed by various types of IT failures, telecommunication failures, human error. And then it tends to be maybe extreme weather events. So, for example, severe weather, ice storms, winter storms, which can be debilitating, particularly in a region where you're not used to those types of events.
So the key is to focus, yes, on the-- on those catastrophic disasters, depending on your region where you're located, but don't lose sight of the most common causes. And to really look at the big picture holistically, which is you're trying to just overall elevate your resilience and your recovery capabilities. So more and more I've been trying to use the term IT service continuity or IT continuity as a counterpart to business continuity and try to shift the mindset away from just disaster recovery.
So a couple things I want to talk about today is, you know, from my perspective and Forrester's perspective, talk about why upgrading capabilities is a top priority for our clients and some of the trends that we're seeing in the market. Then I want to talk about specifically what I see as some of the best practices that companies should adopt when building a much more resilient IT infrastructure. And then leave you with some specific recommendations that I always leave with Forrester clients.
So with that, let's get started into the first section, which is talking about why disaster recovery continues or IT continuity continues to be a top priority for most organizations.
According to the survey data that-- and Forrester runs a lot of surveys year after year. One of the surveys that we ran, you know, we hit more than 1,000 decision makers at companies in both North America and Europe, and we asked them how much of a priority is improving business continuity and/or disaster recovery. And year after year, I would say it's one of the top five IT priorities.
As you can see on the slide here, 62% of those 1,000-plus companies that we interviewed said that it was either a critical or a high priority for their organization. And as long as I've been an analyst, which is over six-plus years, like I said, it's always been in one of the top five IT priorities is improving that resiliency, improving continuity and business continuity.
But I have seen some changes over the last couple of years as to what the drivers are, like, why is this such a top priority. And one of the things that I would say is that there has been-- we are starting to see evidence of this shift, the shift in the mentality from disaster recovery to IT continuity.
Increasingly, when you ask companies why is it such a priority, it's that it's now seen a fiduciary responsibility of any responsible company to have continuity plans in place. It's a responsibility to your own employees, to your customers, to the investors in your company. You think about your strategic partners that you-- that might rely on you. You might be a strategic supply chain partner for a larger company or a smaller company. So you think about all of the stakeholders.
All of those stakeholders are actually expecting you to have continuity plans in place. It's no longer optional. It's considered an essential management practice, a good management practice, and you actually have a fiduciary responsibility to those different stakeholders that I've described to have continuity plans in place.
And a lot of the surveys and the consulting work that we do with our clients, I would say it's somewhere between 50% to 60% of our clients are constantly responding to audit requests from stakeholders to provide proof of their preparedness. So, increasingly, you're going to find yourself having to answer questions about your own preparedness and needing to provide proof that you are, in fact, ready.
The other thing I would mention there is depending on the industry that you're in, you might also find that there are certain government and industry regulations that will require you to have continuity plans in place.
The second thing is just the changing nature of business. I mean, you know, whether you're a small organization or you're or a large organization, we're moving towards 24 by 7 operations, more globalization, a lot more competition. If you're down, that means that it's an opportunity for your competitors to steal market share from you.
And the third is the cost of downtime. It's still important. But what I'll say here about the cost of downtime is we often tend to think in terms of just lost revenue. It's a lot more than just lost revenue, and calculating the cost of downtime is a little bit more complex than that.
So when you think about cost of downtime, certainly it's lost revenue. And you've heard statistics where people often talk about, like, lost revenue per hour or per day. But it's actually more than that. When you think about, for example, like the productivity loss. If you're in an organization that has salaries employees, you're going to continue to pay the salary of your employees while you're out of business. So you think about the productivity loss that you're going to be responsible for, continuing to have to pay fully loaded salary and benefits while you're business is not operate.
You could have payment penalties or loss of discounts, essentially noncompliance penalties. If you have service-level agreements with partners and you're not able to meet those service-level agreements, you might have to pay FLA noncompliance penalties as well. So the picture is a lot bigger than just lost revenue.
There are certain things that are harder to quantify but just as important, and that's customer satisfaction. You know, companies that do recover, you will see an impact to customer satisfaction. If customers believe that it took an inordinate amount of time to recover, it might actually impact your long-term customer retention, even if you don't see a loss of customers right away. It could have impact to your partner relationships. Like I mentioned, all those different types of stakeholders that you have. You're going to have strategic business partners, suppliers, outsources or other groups that you work with, there's going to be a significant impact to them.
Impact to corporate reputation, in some cases if you look at companies that have experienced a significant outage, and, you know, that's managed to make the press. It's hit the news. You can even see it hit through their stock price for that day. So there is an impact to your reputation.
And last thing is, you know, if you didn't have good recovery plans in place or continuity plans in place, there is an impact to employee morale across the company that will impact your, you know, employee retention longer term as well. So there's some things that are easier to quantify in that it's easier to put in your business case. There are things that are harder to quantify. But I would say in some cases all the items that are on the right that are harder to quantify are just as important to take into consideration.
So those are just some of the things that I see really driving the need to improve continuity capabilities.
And I think oftentimes too, when I talk about this shift from disaster recovery to continuity, I always feel that there's perception that it's only for the largest enterprises. I'm really seeing a shift across company size, whether you might see yourself as a small or medium business. You know, you have just 100 employees, or somewhere between 100 and 500 or 400 to 1,000 or if you're above 1,000, as well as across different industries as well. It's not just financial services that have the need to be able to recover quickly after some sort of disruption. It's across all industries.
So when I kind of think of things, when I think about disaster recovery and when I think about the past, DR is a much more reactive response to those catastrophic events. There was always this mentality among senior management that investments were expensive insurance policies. You know, outside of financial services, most organizations tended to manage their downtime in hours to days, in some cases even weeks.
And what I saw was there tended to be a lack of focus or a lack of inclusion a lot of the everyday or what I would call mundane events that actually caused the majority of disruptions. Those power outages, those IT failures, the human errors, the extreme weather and not just the catastrophic events.
We found there was a lot of poor planning. There was little reporting back to senior management. And very few metrics that was actually measuring your success, specifically like how often did you test. Did your test meet your actual recovery objectives. Things of that nature.
Where I see the IT continuity, IT continuity is much more proactive. It's about limiting all types of downtime through preventative measures and having rapid response.
Management sees investments as a responsibility, again, getting back to that fiduciary advantage as well as actual competitive advantage. And like I said, across industry and company sizes, the discussion has shifted. Your target should now be in hours. So get away from this mentality about days and start thinking about hours. And depending on the industry and company size, you might even start to be thinking in minutes. It's that much more holistic view and approach where you're focusing on all likely business disruptions.
And the emphasis is not on one-time planning. The emphasis is much more on managing this as an ongoing process, so you're going to continuously maintain your plans. You're going to update them. It's going to be integrated with your change management and your configuration management practices and IT. You're going to run different types of tests. You're going to have training and awareness across the company, and it's going to be a cycle that you repeat. So each year you'll refresh your business impact analysis and your risk assessment, together with keeping your plans up to date.
So in terms of-- you know, moving on, in terms of what I see the best practices when you're actually trying to develop an IT service continuity strategy, I'm just going to quickly go over what I see some of the key technology components and just make a couple of quick comments on some of the most important ones.
I mean, obviously you're going to need two sites. And whether we're talking about full datacenters or computer rooms, or we're talking about maybe leasing space at a collocation provider, you obviously need that sort of redundancy in the site itself. You need redundancy in the actual infrastructure, so your servers, your network infrastructure, your storage infrastructure. You need some way of actually replicating or backing up or copying your information between the sites.
And there's different ways of doing that, from storage-based replication to appliance, to host, to database replication. And whether you're-- and whether or not you're using some sort of traditional backup and recovery method, you need some way of actually recovering your applications at that site. You know, larger companies will actually use clustering or rapid failover technologies. If you're a smaller company, you're going to need some way of just doing a rapid restart of your apps. And in some cases maybe you're still recovery from disk or tape.
Another important piece of it is obviously the network connectivity between the sites itself. Obviously there is an upfront capital investment that you have to make in and all the redundant site and infrastructure itself. But one of the biggest ongoing costs is actually the network. In cases it's 30% or more of your ongoing costs of a disaster recovery solution.
And it's also key to your recovery capabilities, having the appropriate amount of bandwidth and low-latency bandwidth between the sites is critical to the different types of replication technologies that you might be able to take advantage of, whether you can do synchronous or asynchronous replication, as well as how much data you could actually replicate between the two sites as well.
You know, increasingly companies aren't just protecting mission-critical applications but also business critical applications as well, and even things that they would normally consider less critical. So being able to replicate and protect more applications within your environment, especially when you think about data capacity, in some cases, growing 50% to 100% a year. Your ability to really protect yourself is contingent on how much bandwidth you have between the two sites, and your ability to optimize that bandwidth to replicate more and more larger types of data.
One other note that I'll say about the network as well is the network itself needs to be resilient, so it's important to have route diversity between those two sites in case you do have a network failure as well, so that that's pretty critical.
Now, a lot of the questions that I get, particularly from smaller organizations, is can I really afford this type of solution. I don't think advanced continuity has become more affordable. I'm not going to say it's inexpensive. But one of the reasons that it has-- a lot of the reasons it's become more affordable actually is those two sites that I showed on the previous screen, don't think of that recovery site as just sitting there idle, like you're going to make this massive investment and it's just going to sit there idle waiting for the disaster or the vent to strike.
In reality, the vast majority of companies are actually using both sites for production workloads. Maybe you use the recovery site for deferrable workloads. Or increasingly, I actually see companies use both sites as production, and each site acts as a recovery site for each other. So you're taking advantage of the investment. It's not just a sunk cost, and it's not sitting there idle.
A lot of companies are able to reduce the infrastructure investment they have to make with, you know, technologies like server virtualization, reducing server hardware costs. The cost of storage has declined rapidly. Like I mentioned, there's different types of replication technologies and the cost of them have declined.
You know, network bandwidth, the number of connectivity options and the cost of bandwidth itself has declined. But there's also technologies like WAN optimization that really help you maximize the bandwidth that you do have. You know, they're using data reduction techniques like compression and dedup, and they do protocol optimization to really help you maximize the bandwidth that you do have, so that you can replicate more data and protect more apps.
So there's a lot of ways that DR or continuity has become a lot more affordable. And I no longer view it as just the domain of larger enterprises. It is really something that small and medium businesses can be (ph) taking advantage of.
In these days, a recovery site is no longer optional, like, according to surveys to Forrester has run. And it's-- this is-- I've seen this year after year. More than 90% of companies do have a recovery site. And actually, interesting thing that you can see on the slide here is in some cases, companies will actually have more than one recovery site. And there's reasons why you might actually have more than one recovery site.
As I mentioned, you know-- well, actually I should start off and say in some cases, depending on your organization, even if you are a medium-sized business, you might actually have more than one production site. You might have multiple computer rooms or datacenters. You might have different corporate sites. You might actually have a large number of remote offices as well.
Second thing is, you know, maybe you are in financial services or insurance or another industry where you're very sensitive to any kind of data loss, and you're trying to increase the geographic distance between sites. You might actually need multiple recovery sites to achieve zero data loss as well as distance.
And the third reason, and I think it's probably the more common one, is we're increasingly seeing people move to that active-active datacenter configuration. So it's no longer a production site, recovery site. It's actually two production sites. They each act as a recovery site for each other.
Another thing I'll say about the site, and I know a lot of the questions are-- you know, again, I'm a small to medium-sized business. I can't really afford another site. You know, that's true. What I'll actually say is even amongst enterprises, we're seeing greater adoptions of collocation sites. So you don't necessary have to own-- have an internal site, although a lot of companies do, you know, but you might not have one or maybe it's not the appropriate distance away or maybe it's not resilient enough, maybe it's in a high-risk area, companies are increasingly trained to collocation providers.
So with a collocation provider it's not really outsourcing. You're simply leasing the amount of datacenter floor space that you need to support your site. And we've seen collocation adoption increase 15% in less than a year. And there's also, as the site-- as the survey data on the page shows here, you still might actually turn to a traditional disaster recovery service provider. So somebody who is not only providing the collocation site but might have access to large capital investments of inventory and different types of servers, and also has the expertise to give you some guidance and support you in the actual recovery.
And as I mentioned, why collocation? You know, it's less expensive than having to retrofit an existing datacenter or build a new one. You know, a lot of datacenters today are aging, and they're out of space and out of power.
The second thing is, these collocation datacenters are a lot more resilient than your own, in most cases. These are datacenters that need Tier 3, Tier 4 type specifications. So everything about them is N+ONE resiliency, so they're highly resilient in and of themselves. And in some cases, it's the-- not only are they more resilient, but they tend to be located in lower-risk areas. So it might give you the opportunity to have a recovery site in a lower-risk area that might be of the appropriate distance away from your current production site.
And just to give you a little bit of data, this active-active datacenter is a reality today. As you can see here, when we ask companies who do have a recovery site, you know, do you actually use that recovery site for other types of, like workloads. And you can see that more than 49% actually say that they use both sites as a production site. Another 22% say that they moved deferrable workloads to that recovery site. And a typical mode there is you might be moving things like application development and testing to that site.
So keeping the point out here, it's only 28% of companies that have a recovery site, just have it idle there. So the key is you're moving towards some sort of active-active datacenter, whether they're both production sites or whether you're moving something dev and test over to that other site.
Another common question that I get is how far apart should my sites be. I've kind of mentioned-- I've mentioned or alluded to a couple of times that your recovery sites really need to be an appropriate distance away. And a common question I get is, well, what is that distance. I always hate to say it depends, but it does depend.
You don't want your sites to be so far apart, you know, thousands of miles away, but-- not to say that thousands of miles away is wrong, but you do have to be careful about excessive site separation. The further apart your sites are, it can increase recovery time. It can increase the cost. You have to have-- take into consideration the impact on your staff. You know, you're probably going to be either manage that as a lights-out datacenter, or you're going to have to have staff at that remote site.
It could increase latency between applications. I'm not so worried about excessive site separation. I think those can be managed. What I tend to be more concerned about with clients is that their sites are too close. I continuously run into clients who have their sites just a few hundred miles apart, in some cases like a few miles apart.
If your sites are so close together that they're actually subject to the same risks or threats, you don't actually have a continuity solution in place. What you have is a really high-- a very expensive, high-availability solution in place. You know, it's almost like having that infrastructure within the same datacenter. Your sites have to be far enough apart that they're not subject to the same risks or threats. So I tend to be more worried about how close they are than, you know, whether they're too far apart.
In general, in North America, particularly in the United States, there is a movement towards greater distances. You can see on the slide here, you know, 58% of North American companies will locate their recovery site at least 155 miles away. So you see the majority moving towards 155 to a couple of hundred of miles. So generally speaking, I tend to feel comfortable with that distance. I mean, I'd still want to do a risk assessment of the geography to understand if that was far enough away. But usually I feel more comfortable with that.
You know, it tends to be the flip in the European Union in Europe, and that has less to do with risks, has more to do with, in some cases, data privacy regulations. Europe has specific data privacy regulations that say personally identifiable information can't be replicated outside of the EU or to another country. So there's much more of a desire to keep data within country, so that limits distances in Europe.
So I would say, you know, for most of the small and medium businesses that might be listening to today's webcast that are based in North America, United States, Canada, there is a movement towards greater distances. And generally I would use a rule of thumb as 155 miles plus.
And you know, how do companies achieve that greater distance? A lot of the technologies that I've been referring to, replication and backup technologies, they've become a lot more bandwidth efficient. There's technologies in the replication and in the backup, like data deduplication that reduces the amount of bandwidth, reduces the amount of data that has been replicated. There's WAN optimization technologies that a third party or your network provider can offer that will, again, address the amount of data that you have to replicate as well as address some of the protocol-- latency associated with protocol that will improve bandwidth. So there's a lot that's done with technology to help improve your ability to replicate over large distances.
A couple other things before I start wrapping up with the recommendations. You know, I talked a lot about collocation providers. You know, as I mentioned, you know, if you are a small and medium business and maybe even collocation might be too expensive for you, a lot of companies still-- as we saw on the previous slide, a lot of companies still go with the traditional and DR service provider. And I find that companies that actually outsource DR to a DR service provider, they actually tend to stay with them.
You know, as you can see on the data slide here, you know, 51% of companies that have those types of contracts in place plan to renew. A lot of companies, even if they have one of those contracts in place, will at least go to a competitive RFP to get different quotes from other providers.
You know, a lot of companies talk about bringing continuity back in-house, but it is difficult. It takes years of planning in order to bring that back in-house and really having a strategic IT plan to do that. So you'll find that a lot of companies that have gone to a DR service provider will stick with them.
You know, to do it in-house, like I mentioned, as we talked about it, it does require that capital investment. It does require that site strategy either with a (inaudible) or an internal site. You might actually lack the internal expertise to really support one or more of these advanced solutions that are based on replication and different types of failover technologies.
The other thing that I-- that companies who have brought DR back in-house often tell me is testing becomes a lot harder. If you do have that active-active model, when you go to test your continuity plan, it has a bigger impact on productivity if one of the-- if two or more of your sites are actually production sites. So that's a lot of the reasons why people would continue to use a DR service provider.
So quickly just to wrap up with a few recommendations. You know, key thing is to always manage IT continuity as a continuous process. As I mentioned, your business impact analysis, risk assessment plans, you've got to keep all of those up to date. You have to-- and particularly BIA and risk assessment, you update those annually. Plans you update continuously.
Test frequently. And there's different types of tests, from plan walkthroughs, to tabletop exercises and simulations.
Training and awareness is huge throughout the company. And in fact, people who have declared it (ph) said that that's one of the top issues that they uncovered is people really had no idea what to do or what their rules and responsibilities are.
A couple of other things that I'll finish up before I turn it back over is think about protecting all of your applications, not just your mission-critical. There are so many application interdependencies. You really can't just pick and choose which of your systems you're going to protect. If you replicate only your mission-critical apps, you might find that you can't really restore your entire business process.
And for those organizations that might be a little bit larger, don't forget about your remote sites. It's not just the core headquarters or your core computer room or datacenter. A lot of critical information and business processes are relying on your remote sites, so include them as part of your overall DR strategy as well.
So with that, I'm actually going to turn things back over at this point. And now we'll be getting ready for our next speaker. And I look forward to taking everybody's questions at the end.
Okay. Thanks very much, Stephanie. Very, very useful, helpful information.
Again, feel free to ask questions, and we will answer them shortly.
So with that, I will hand it over to Kevin.
Thanks, David, and thanks to everyone who took time out of their day to join our webinar. I'm going to spend a few minutes talking here about the challenges of getting real network diversity into your network plan, and some new choices that are emerging to help make that easier to get accomplished.
So as Stephanie pointed out, eliminating single points of failure in your network is really critical to achieving a good, survivable IT infrastructure. One of the challenges is that over the history, most CLECs really shared the infrastructure with the incumbent local exchange carrier, the phone company in their area. They shared last-mile loops. They shared interoffice facilities. They shared access at the central office. And that was particularly true for high-capacity fiber-fed transport, which was very, very constrained. If it was available from one provider in a given location, it was rarely available for-- from two different providers where you could get true network diversity in your plan.
It made relying on traditional CLEC a lot like trying to diversify your financial portfolio with mutual funds that all wind up buying the same stocks anyway. You think you're diversified because you own three different mutual funds. And then you go underneath the hood of those mutual funds, and you find out, well, you just managed to buy the same stock in the same company three times, which didn't really help.
What's interesting to me is as the world moves to cloud computing and other hosted applications, on one hand we're getting reliability and operation gains by reducing the risk in edge-based server infrastructure that might be poorly staffed or not have good reliability in disaster recovery plans for the edge-based facilities by getting them into more robust, centralized facilities. But what's actually happening is you're shifting some of the risk. You're shifting it from those edge-based environments, and you're shifting that risk onto your network. You actually would have to rethink and double down on how reliable and diverse your network facilities really are.
David noted in his presentation that communications are the key to a company's survival when there's a problem. And increasingly, we rely on these hosted services and these cloud-based services as our vehicle of communicating.
I thought it was interesting both speakers noted that this isn't just about hurricanes, right. For a small to medium-sized business, a car accident that cuts a fiber and reduces or eliminates all connectivity to hosted infrastructure is a disaster in your communications world. And so we have to consider how to mediate those-- or remediate, excuse me, those problems.
Now, as with most things in life, if there's a problem, the market will step in and find a solution for it. And I think there's something very interesting going on in the business services landscape here in the United States, and that is the emergence of cable companies as a viable alternative for the last mile for companies looking for redundant facilities or primary facilities for their IT needs.
Cable companies in general, but my company, Comcast, in particular, is emerging as a solution to this problem. Cable operates its own at-scale, high-capacity network, with reach all the way into the last mile, the last 500 feet, right to the building. There is an exception that proves every rule, but by and large, this is a totally redundant network from the phone company. It doesn’t share the facilities. It doesn't share the phone company CEO. And it's really got the scale because the cable companies are fairly large and across many territories at this point, to be one throat to choke, in many sites. You don't have to go to ten different providers to get an answer.
There's also a real strategic commitment in the cable industry, if you look at what's been going on with moving into the business services arena. So there's a lot of willingness to extend this network out to the buildings to be served and bring our very own fiber to that building. It's totally redundant.
If you look at that in total, I think you'll find that cable is the largest facilities-based alternative to the phone companies in the United States for providing that last-mile connectivity. And if you look at Comcast in particular, particularly the two charts that are up on the webinar now, you can sort of see on the left, that's a snapshot of one of our territories where the green lines represent the depths of our plan into the last mile. And you can see that cable networks are fiber deep and fiber rich.
And interestingly, our coax network is also available to businesses, and that gives access to millions of businesses today. So in the last mile you've got a good alternative. And then if you just sort of zoom back a little bit, and again, picking on my company, look at our nationwide reach, we are, today, in 39 states and the District of Columbia, with almost 600,000 plant route miles and service. So you suddenly have an at-scale carrier-grade alternative to provide truly diverse facilities for your IT infrastructure.
Of course, once you've chosen a physical network to provide your IT connectivity, you have to consider what kind of product you want to run over the top of that, what type of network service you're going to use. One that I would encourage you to think about is carrier Ethernet that's been certified by the Metro Ethernet Forum. I think it's a really great choice for your business.
Now, I'm obviously talking my own book a little bit here in two dimensions. David noted at the beginning that I'm on the board of the Metro Ethernet Forum, and Comcast Business Services offers a Metro Ethernet product. But I think if you just look at it objective, you'll find that Metro Ethernet meets many of the needs definitely articulated earlier. Just consider the core characteristics as represented in the circle up on the screen.
First, it's a standardized service, so it's ubiquitous using standard equipment that connects your lands. Second, it's very, very scalable. It's scalable really in three dimensions. The first is the size of the network. You can connect many, many endpoints together. The second is speed. You know, from 1 megabit up to 1 gigabit per second. And a third is the applications that it can support, from basic classes of connectivity up to very robust applications requiring less than 2 milliseconds of (inaudible) for example.
It's also very, very reliable. The entire network is built with the notion that this is carrier grade, and that we have to recover from incidents without impacting the end users. There's been robust service management built into the specifications by the Metro Ethernet Forum, so the ability to monitor, diagnose and manage your network across many sites is intrinsic to the product that's delivered. And finally, it does have very good class service choices, so you have a lot of granularity in terms of your bandwidth and quality of service option. You can really meet the need of any application, in terms of speed, quality, latency, whether you're just doing typical internet access or you're managing voice over the top of your network.
I think as a standard solution, standards-based solution, it's particularly well suited for thinking about redundant facilities because the barriers to interoperability between Provider A and Provider B, are actually reasonably low. You know, when you consider that Ethernet's been a proven technology in the LAN for over 40 years, what you really can think of is, is this capability as a 100-mile-wide LAN. Now, nothing's perfect in life, but if you buy Metro Ethernet compliant, MEF9, MEF14, MEF18 compliance solutions, I think you'll really be on your way to a simpler life in terms of planning your disaster recovery.
Obviously I'd like you to consider Comcast business class, but more generally I'd really suggest you visit the most recent forum website, which you can find at MetroEthernetForum.org, and learn about MEF certified products and who you can get those from.
So wrapping up, just, again, highlight that the move to hosted and cloud services really increases your risks if you have a single-threaded network. You'd definitely have to think about multithreading your network with truly diverse facilities that aren't reliant on one critical point of failure in the last 500 feet, the last mile or the interoffice facilities. And that's going to take a lot of work. You've got to find those facility-based alternatives to limit your hidden exposure to shared facilities where you thought you had redundancy, and you didn't, in fact.
And, again, I'd really encourage you to look at cable, bet it Comcast in our service territory, or other cable companies in other service territories, as a great alternative in that last mile. And then I'd really encourage you to look at MET certified internet solutions because they really are robust and can meet the needs of almost any application.
With that, David, I will pass it back to you.
Great. Thanks very much, Kevin. So wonderful presentations by our three speakers. And so with that, with about ten minutes remaining, let us get to some of the questions that a number of you have sent in.
So for Stephanie, you say a-- 90% of enterprise and S&B (ph) have DR sites. What is-- where'd you get that information from and number of employees per annual-- per year, annual revenue, et cetera.
Yes, so this is actually from Forrester Surveys. So we run an annual IT infrastructure, which essentially an IT hardware survey each year. It goes-- it's a really large survey. It goes out to several thousand panelists. And the types of individuals that answer these questions are IT decision makers, and in some cases, depending on the topic, you know, we further sub-segment them by their decision making or influence in a particular area. So in this case it would be disaster-recovery related decision makers.
The survey data that I was particularly using was companies that are around 1,000 companies, so I would say they're more small, medium enterprise, with some slightly larger companies mixed in there. And I broke it down, as you'll-- for everybody who has a copy of this presentation afterwards, you'll see breakdowns by North America versus Europe.
So yes, I mean, the data, just, you know, 90% of companies, of those companies that we surveyed do have recovery sites in place. And if you further break that down, if you look back at the data, you will find that there is a large percentage of companies that do use an internal site. I think it was between 50% to 60%. You had another 20% or so that were using (inaudible). And then the remaining are still using traditional DR service providers, (inaudible) you know, SunGard, IBM, HP, you know, other vendors in that category as well.
So there has-- amongst larger enterprises, there was a time where there was movement towards bringing disaster recovery back in-house, and a lot of the reasons were because of some of the technology shifts. You know, if you looked at a lot of the things that I went over, reduction in costs of a lot of infrastructure, the active-active datacenter strategy, reducing cost of bandwidth, et cetera, it did make it easier to bring DR back in-house. And for larger companies, you know, they like the comfort of managing to their own recovery objectives than using a third party.
But I am starting to see the traditional DR service provider market change a bit. There's-- they're reducing their costs. They're trying to become more easier to do business with. They're starting to offer you a range of cloud-based recovery services from online backup to things that are based on host-space replication. The idea being it is true cloud. It would-- you know, these are multi-tenancy services that you would-- it would be like pay per use. So you would, you know, pay for some amount, you know, per server and per storage, and it would be per month. And it would be significantly less lengthier contracts in place for them.
Okay. Now, in your answer you've answered a number of the questions that have come in. One was certainly around cloud computing and what impact it's having on IT continuity. So thank you for that.
Dave, in your remarks, you talked a lot about obviously natural disasters and its impact on smaller businesses. Do you want to comment a bit on larger businesses and what you've seen, whether it was in Dade County or at the national, federal level and the impact that disasters can have? I mean, I suspect much of what you suggested and recommended applied to larger organizations as well.
It does. The-- what I have seen though is a lot of the larger organizations do have disaster plans in place and tend to be exercising those and keeping their employees informed. Now, some done. But sometimes some events are so catastrophic-- I mean, look what BP is going through right with a-- you know, what could be considered a cross between a manmade natural disaster. So those types of catastrophic events can really affect any company, regardless of the size.
You know, with Hurricane Andrew in South Florida when it came through, it took ten years for that area to start coming back because the businesses did not recover. And if there's no business in place, there's no place for people to work, and, therefore, they're going to move out and move somewhere else. And we saw a significant shift in the demographics of that area because of that one storm, and primarily because the businesses couldn't respond.
So regardless of the size of the business, all the things that I talked about, they really do apply.
Okay. Very good. Thank you. Thank you. Stephanie, let me come back to you with one of the questions we have here about the decision one makes as far as-- your datacenter, whether it's your own or an independent one, and the decision about the transport to and from the datacenter, which speaks a bit to I guess what you had noted in the research around the distance, right, between datacenters. So how important is the decision? And what are the criteria that an IT decision maker should make in choosing what-- you know, what provider is providing the transport, if not multiple providers?
Yes, it is important. I mean, the one thing, back to route diversity, just because you have multiple providers doesn't guarantee you route diversity. You know, as Kevin mentioned, I mean, they could be going through some more infrastructure in central offices. So it's something that you have to work with your provider to make sure that you, in fact, achieve. And you can achieve route diversity with a single provider as well.
So it is important. I think you need to view your provider as a partner. I mean, not only do they help you make the decision about the appropriate type of transport and the amount of bandwidth that you'll need, but particularly with disaster recovery, it's going to require a fair bit of integration with the infrastructure that you have within your datacenter.
So do they have strong relationships with the different types of storage vendors who might be supporting the replication or host-based replication providers. Because when it comes to determining the amount of bandwidth, it's actually a pretty tricky formula that is hard to figure out yourself. You're going to need to have your replication partner together with your network service provider to sort of get together and figure that out.
It's a function of the amount of data that you need to replicate. So how you're replicating the data, whether you want to replicate it synchronously or asynchronously and your sensitivity to any-- to latency, as well as the distance between the sites. So a combination of all of those things will determine how much bandwidth between the sites.
So, to me, that network provider isn't just a supplier. You've got to treat them as a strategic partner in your IT environment.
Okay. Very good. Thank you, Stephanie. Kevin, there's a question for you. Could you briefly compare the internet service provided over cable with internet service provided via WiMAX?
Sure, David. Let's actually compare three things; WiMAX, instead of the classic cable (inaudible) solution, and the Ethernet solution that's being offered now.
If you consider WiMAX, you know, WiMAX is a wireless technology, which introduces the usual questions around wireless, which is signal propagation. You know, how close are you to the tower? How strong is the signal that you can see? And also what kind of in-building penetration can you get from the service?
So it does have the virtue that it is an alternate last mile or last 500 feet. There's some service quality questions that come into play. And then, remember, all wireless solutions eventually wind up back on a landline network, and so you really have to consider who is providing the backhaul for that landline network. Is that, again, going back through that same telephone company's central office, or is it actually being taken out to the internet via a truly redundant network.
If you think about the typical cable business services solution that we provided off of our (inaudible) infrastructure, our cable model infrastructure, that is a point-to-internet solution only, so we don't provide point to point, like T1 replacements, off of that. It's really a great vehicle for getting to the internet for a small to medium business. In that environment we have, at Comcast, 50-megabit service very broadly deployed across the United States now. It is definitely not a service that, today, is offered with a hard SLA, which brings us into Metro Ethernet. So I think business class internet is very good as your primary internet access vehicle and very high bandwidth.
If you're looking for point-to-point connectivity with hard SLAs and things like latency and bandwidth and jitter (ph) controls, as I mentioned earlier in my presentation, you're now over in the Metro Ethernet solution, which is typically fiber-fed to your building. And we can provide scalable service up to a gigabit per second and provide, again, the various classes of service, basic class of service for things like web access and the internet; preferred class of service for maybe mission-critical applications and then premium class of service for voice or video or other real latency and jitter-dependent applications.
So I think you really have three things you need to consider as you look across that continuum.
Very good. Thank you, Kevin. Well, we're at the top of the hour. So I, again, want to thank all of you in the audience, as well of course our presenters. Hopefully everyone has found this very helpful and useful and valuable in your disaster planning moving forward.
I'd also like to remind everyone that if you complete the survey that will popup momentarily as this webinar ends and send that in, you'll be eligible to win one of the Flip cameras, as noted earlier.
So, again, thank you to everyone. Appreciate everyone's participation today.