If You Are Doing Incident Response, You Are Doing It Wrong
I’d been thinking about this for awhile, but conversations with Rob Lee and then a presentation with him really helped me clarify my thinking on this issue. Here goes:
If you are doing incident response, you are psychologically, if not operationally, in a reactive rather than proactive mode. To do it right, incident response needs to be part of your ongoing daily business process. True incident response only occurs during major breaches. As part of your incident management, you proactively – days, months, even years in advance – address the issues that might create a need to respond to an incident.
By managing incidents rather than responding to them, you:
- Reduce the severity of the incidents that do occur.
- Reduce the number of incidents that do occur.
- Shift from responding to incidents to managing incidents as part of your normal operations
- Reduce unforeseen expenses related to incident investigations
- Increase your visibility within the business, and thus the support for your organization
- Strengthen security posture (Thank you to Corey)
- Reduce stress on your staff and increase their job satisfaction (unless they are adrenalin junkies)
An incident management mindset depends on accepting a truism:
Compromise Is Inevitable – Something truly malicious has been in, is in, and will be in your environment.
If you accept that compromise is inevitable, why wait for it to happen? Why not get ahead of it, reduce its impact, and increase your resilience?
Which leads me to my second point – traditional emergency management has been doing this for decades. If you do a search for “Emergency management cycle”, you will find many images similar to the following:
Tornados, earthquakes, fires, automobile accidents, heart attacks, and many more emergencies happen daily. Rather than treating these as one off incidents that require all hands on deck, emergency services plan, recruit, train, and respond in a very calm, business like manner because it is their normal business. (I speak from 15 years of emergency management experience. Find a fire fighter in your organization and run this past them.) When a fire engine rolls up to a fire, does everyone jump out, run around, and add to the chaos? No, they respond in a very consistent, calm, and methodical manner.
Take a hard look at ICS – Incident Command System. FEMA has several short, online courses to familiarize you with it. Step back and think about how it might apply to your organization. The modular, scalable nature of ICS enables effective response to incidents by multiple agencies. Sounds like something that might apply to a breach investigation? (You don’t need to buy into all the labels. Just think about the core concepts.)
In closing, let me ask you to think on two points:
- Manage incidents, and the entire lifecycle, in a way that enables you to treat incidents as part of your normal operational tempo.
- Pay attention to how traditional emergency management works and learn from them. An enormous amount of thought and effort has been invested in emergency management already. Build on that rather than try to recreate everything.
Integriography: A Journal of Broken Locks, Ethics, and Computer Forensics 
The chief strength of using the emergency management model is summed up in your first bullet, that there is a goal of decreasing the severity of incidents. Executives, especially, seem to always want the number of incidents to go down, because they can measure that (and thereby have the illusion of control), measuring the severity of incidents is a lot harder, and then you get into metric gymnastics of does the incident business load = the # of incidents x the severity of incidents?
The emergency management doesn’t fit as well in several areas.
First is language. Mitigation is a word that is great on a graphic, but really doesn’t carry the proactive load that it can carry. I don’t mean to pick on emergency management here alone, getting words understood across the disciplines involved in a computer security incident can be challenging. Even within IT, an “incident” is not the same in the ITIL and information security communities. Response and recovery are ones that also may not be immediately obvious. In the SANS Incident Response Step by Step guide, they talk about “Containment” which originated more in the days of viruses and worms, but helps guide people toward the idea that “Response” addresses. The SANS guide also had an interesting last step. It was identified, because it was the most often skipped, and that was a “Lessons Learned”. From my understanding, this would be one of the activities of “Mitigation”, but really, after the incident, how many people want the additional expense of bringing the parties together again while the incident is fresh to analyze the progress of the incident. Some emergency management teams do have scribes, and resource people, and structured processes for briefings, so that Lessons Learned wouldn’t be so difficult, so it depends on how supportive your managers are (and what levels of management they are at.)
The second limitation has to do with an underlying assumption of many emergency management programs, and that is that there is some place where the emergency is not happening that you can tap for additional resources. If there is an ice storm, then line crews can be drawn from someplace further south to help out. With Heartbleed, people did help each other out, but mainly by creating detection tools. Sharing of processes for response and recovery were really limited.
Heartbleed is a good example of the 3rd limitation, and that is that the emergency is obvious. High winds, not much flooding, ripped apart houses, then it is most likely a tornado. With Heartbleed getting accurate information and accurate detection tools (with low false negatives) took some real effort. Especially in IT, getting accurate information is difficult. The news outlets hire technologists which can translate information into things that people can understand. It is hard to get good technical information out of an article that has been through the “convert it to the masses” transformation. Add to that, the people who post something, with good intent, but limited knowledge, and probably limited testing. Add to that the “relevance” spin that search engines put on things, and you will get information which agrees with the information that you already have (not credible information, or the information that completes the information you have.) Read “The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think” by Eli Pariser if you don’t know what I mean.
The last limitation is that traditional emergency management first responders usually have some sort of “good samaritan” legal protection. If you move the beam to try to get someone out, and it causes an unforeseen shift that kills a few others, you will undoubtably get sued, but people understand that you are using your best judgment to deal with a bad, and possibly worsening situation. Your management, or legal counsel, may advise against communicating what you have discovered out of fear of being sued if the information is wrong, or misapplied, or doesn’t work in all cases. Laws regarding IT are way behind, and seem to be getting further behind.
One other observation is that emergency management seems to often involve “frequentist” views of the risk of emergencies. People in the information security field seem to be leaning more to a Bayesian view of the risk of emergencies. I have to admit that I am thin on this observation. I work in an area where information security is part of risk management, and they seem to apply the things that don’t fit (like there is a likelihood of an incident, and that other incidents handled well or not have no effect on the likelihood on this incident) and don’t apply things that do cross well between traditional risk management and information security (like avoiding meaningless ROI estimation, when was the last time your insurance group calculated ROI of buying insurance). Information security, and information security failures affect each other. Classical emergencies, not as much. A flood along the Mississippi, probably doesn’t have a direct effect on losing air traffic control radar in the NYC-Washington corridor.