Municipality Disaster Recovery Checklist and Best Practices

We’ve written a great deal about data backup and disaster recovery best practices over the years. Yet, stories are often
an impactful way to understand the importance of a solid data backup and disaster recovery strategy for a municipality.

While the disaster stories we outline in this guide are fictional, they are based in truth by combining scenarios from many, many cities and towns we’ve helped, talked to, and heard about during our more than 25 years serving municipalities.

For each story in this guide, we present you a scenario. Each story contains data backup lessons that may relate to an incident you’ve experienced—or will experience. Finally, we include a checklist to help ensure you are ready for a disaster.

Case Study 1

How Permanent Data Loss from Ransomware Took Down a City

Imagine that a city employee who works in the finance department opens their email in the morning. As they check their email, they see one message that seems to come from the city manager. Without thinking, the employee clicks on a file attachment assuming it’s an important document related to a meeting that day.

This employee is not technically savvy, so they are not too alarmed when they see something downloading onto their computer. A window pops up that asks the employee to accept something. The employee clicks “yes.”

Within seconds, a chill goes down their spine. Something is wrong. A red screen appears on the person’s computer demanding a ransom in bitcoin. The employee tells their supervisor, and the supervisor places a call to their IT support vendor.

Logging into the financial server to make sure it’s working properly, the IT support person now gets a chill down their spine. They cannot access any data on the financial server because it’s also infected with the ransomware.

Panic ensues. Unfortunately, many recent security patches had not been applied. Plus, the city runs a free version of some antivirus software that’s only updated when the IT vendor sends someone on site.

“Thank goodness there’s a data backup of the server,” says the city manager. But when the IT support vendor tries to restore the financial data from the backup…that backup doesn’t work. At all. “But we’ve been backing it up manually at least once a week,” says the city manager.

“Have you tested the backup?” asks the IT support person.

"No,” says the city manager. Everyone now realizes a nightmare scenario became real. The city’s financial data is lost. Permanently.

Some variation of this story is all too common for many cities. The good news? Cities can easily prevent a devastating ransomware attack by addressing some of the errors committed in this story.

Error #2

Reactive IT support not maintaining and monitoring servers and computers

The IT support people in our story weren’t getting paid to do ongoing, proactive IT support. Thus, they only reactively addressed issues when the city called on them for an onsite visit. Plus, it appeared that they did not have a process in place for regularly updating the antivirus software and testing the city’s data backups. Experienced IT professionals need to regularly audit antivirus software to confirm that it’s installed on every machine and that virus definitions (which help detect known viruses) are up-to-date.

Error #3

An employee clicked on an email attachment

You might have thought we’d mention this error first. Your employees are on the front line, targeted by thousands of cyberattacks a week. Everyone is busy, and anyone can make a mistake by clicking on a malicious email attachment or website. That’s why you need a strong foundation in place—business-class antivirus software, regularly tested data backups, and proactive IT support—to stop as many cyberattacks as possible. And, even if an employee clicks on something malicious, you need to be able to recover from ransomware.

Because ransomware can still get through strong defenses, employee training is a must. Train your city staff about common sources of ransomware and viruses such as email attachments, websites, online software, and games. With training, you can make your employees more aware about online threats that are easy to avoid if they know how to spot them.

When is offsite data backup not offsite data backup? The following story offers an example—and a warning—to cities.

A city was already backing up its data onsite using an extra server. If the server failed at city hall, the other one would take over to restore the city’s data. However, some department heads urged the city to also consider an offsite data backup plan in case of a major disaster. The city manager researched some options and brought in a few IT experts to talk about possible solutions.

After some outside IT experts reinforced and reiterated the idea of creating both an onsite and offsite data backup plan, the city took a shortcut. The city manager didn’t like the idea of sending data off to a data center. He viewed it as unnecessarily expensive. Plus, he wanted control—to “see” the data when he wished. And so the city nixed the idea of offsite data backup located far away from the city.

As a result, the city worked around these parameters to build an “offsite” data backup plan. Working with their local IT vendor, the city set up a backup server in a building they owned located just down the block from city hall. The city manager argued that this building was separate from the city hall building and, thus, “offsite.” if something destroyed city hall, this server would contain all their data. Problem solved.

Or was it?

Does this scenario seem unlikely? That’s what all cities, businesses, organizations, and people often think…until after the disaster strikes.

Let’s look at the errors in our story and how your city can avoid them.

Error #1

The city's definition of "offsite" is not really offsite

Offsite does not mean down the block. It does not even mean two blocks away. True offsite data backup means many many miles away. When your data is stored in a geographic location far away
from your city, it’s likelier to be protected from a localized disaster such as a tornado.

We often recommend that you send offsite data to at least two data centers (for example, one on the East Coast and one on the West Coast). Then, if a city block is destroyed, your data is safe and accessible from multiple data centers. Your city can start operating within hours of the disaster while you are in the process of ordering
new servers.

Error #2

An improper risk assessment focused too much on cost instead of the cost of a disaster

Sure, it might be cheaper to set up another server in a building down the block. It’s also cheaper to buy health insurance with high deductibles that doesn’t cover serious medical conditions. In each case, the costs are astronomical when a disaster hits. Cheaper isn’t better and cost is a poor tool to judge a data backup solution’s ability to mitigate risk.

What’s the cost of losing your data? How will your community be impacted if all city records are lost? That’s the cost you
should assess. From there, you can make a better case for investing in a disaster recovery solution that mitigates risks by
storing data in a geographical location far from your city.

On the surface, a coastal city did some correct things to back up its data. The city had a few servers in a physically secure basement room that were well-maintained by IT staff. One of the servers backed up important data. In case a server failed, the backup server would run until the city could replace the original server.

A long time had passed since the city last experienced a hurricane. When a hurricane finally seemed eminent, the city was ordered to evacuate until the massive storm passed. The city manager and IT staff didn’t think much about the servers other than placing them upon concrete blocks in case of flooding. As long as the city implemented its emergency action plan and evacuated everyone safely, the city manager assumed its information technology would remain safe.

After the hurricane passed, city staff returned to find that no massive devastation occurred but they did experience heavy flooding. The IT staff had placed the servers upon concrete blocks as a precautionary measure, but they learned an incredibly hard lesson in hindsight.

Located in a basement room, the servers sat below sea level. Although the rest of city hall experienced moderate flood damage in places, the basement had filled up to dangerously high levels. All the servers – including the backup server – were rendered unusable by the flooding.

With a sinking feeling, the city manager realized all critical data – including financial, public safety, document management, email, and website data - was gone. The only backup server got destroyed along with the others.

Now, the mayor, city council, the media, and public would be asking questions.

Sure, the city manager and IT staff made a bad decision to place servers in a basement room below sea level. But their errors go deeper than this poor choice of physical location for the servers.

Let’s look at the errors in the story above.

Error #2

Lack of offsite data backup

While locating the servers on a higher floor may have prevented this immediate flooding disaster, it’s still not a full disaster recovery plan. Anything can happen to your technology onsite. To guarantee full recovery of your data after a disaster, you need an offsite data backup component to your emergency plan.

We recommend storing your data offsite in geographically dispersed locations (such as in centers both on the East and West coasts). Then, even if the worst disaster wipes out your buildings, you will be able to recover and access your data.

Error #3

Lack of technology planning

The lack of offsite data backup also signifies a larger issue-a lack of planning. The city had developed an emergency plan and used it in the case of the hurricane. But when was the plan developed? When was it last updated? Did it include technology-related scenarios? What was the plan to protect data in case of a disaster?

First, the city needed to update its emergency plan and include technology. That would have addressed technology-related gaps in the city’s data backup, disaster recovery, and business continuity plans. Second, the city needed regular technology planning meetings (at least annually) and ongoing monitoring to ensure that data backups were tested and working. This regular monitoring and planning would help the city adapt to changes (such as new technology, more staff, building changes, etc.) and ensure that the risk of data loss is minimal.

Disaster Recovery Checklist

While disasters can take many tragic forms, the way to recover from those disasters follows some predictable principles that you can apply now. Follow the checklist below to ensure that you can recover your data after a disaster and begin to help residents immediately.

Use onsite local data backups to lessen time to recovery for smaller incidents (such as a server failure).
Use offsite data backup to plan for worst-case scenarios (such as a natural disaster or ransomware). Offsite means storing your data backups far from your geographical location.
Monitor your data backups. It’s important to identify problems with your onsite and offsite backups before a disaster occurs.
Regularly test your data backups. If you don’t test your backups, you won’t know if you will be able to recover after a disaster.
Encrypt your backup data at rest and in transit—such as when you’re sending data backups to your data center or cloud provider. Make sure your decryption keys are stored both onsite and offsite.
Use enterprise-grade antivirus and endpoint detection and response (EDR) to prevent and detect attacks.
Proactively monitor and maintain your IT hardware, software, and network equipment. This includes software patching to eliminate cyber vulnerabilities.
Periodically train employees about ways to spot phishing attacks and common cyberattacks. (95% of cyberattacks begin in an email.)
Create a disaster recovery plan that clearly outlines how your municipality will recover your data and restore operations after a cyberattack or other disaster.

Conclusion

Is a disaster waiting to happen at your municipality? Implementing all these best practices may feel overwhelming. VC3’s Manage Essentials IT Services help prevent data loss by providing your municipality onsite data backup for quick recovery after events like a server failure, after a major incident like a natural disaster or ransomware, real-time monitoring to quickly address data backup issues, and quarterly testing to verify your disaster recovery. Have additional questions? Concerns about your current data backup and disaster recovery strategy? Contact us to talk with an IT specialist!

We Make IT Easy

Municipal Disaster Recovery Checklist

Municipal League Partnerships

2024 Managed IT Services Cost & Pricing Guide

Disaster Recovery Checklist and Best Practices for Municipalities

Case Study 1

How Permanent Data Loss from Ransomware Took Down a City

Short on time?

Download the PDF to read later 👇

Error #1

Lack of business-class antivirus software

Error #2

Reactive IT support not maintaining and monitoring servers and computers

Error #3

An employee clicked on an email attachment

Case Study 2

Permanent Data Loss After Tornado Destroys City’s “Offsite” Backup

Error #1

The city's definition of "offsite" is not really offsite

Error #2

An improper risk assessment focused too much on cost instead of the cost of a disaster

Error #3

A need to “see” the data and keep it close

Error #4

A lack of a disaster recovery plan

Case Study 3

Here Comes the Flood: How a City Permanently Lost Its Data

Error #1

Locating servers in a flood-prone area of your building

Error #2

Lack of offsite data backup

Error #3

Lack of technology planning

Disaster Recovery Checklist

Conclusion