ITS 4350 - Disaster Recovery

Chapter 8, Incident Response: Recovery and Maintenance


This lesson is about chapter 8. Objectives important to this lesson:

  1. Recovery
  2. Maintenance
  3. Forensics (most of the chapter is about this)
  4. Preserving technical data
  5. eDiscovery and anti-forensics
Chapter 8

This chapter begins with the next chapter in the horror story about a student testing a worm program. Not much is added in this installment, other than the beginning of the CSIRT staff involvement. The situation is "contained", and the staff will move into a Recovery phase.

The Recovery phase is still part of the incident response process. It starts with figuring out what went wrong. How were we compromised? Why weren't we protected from what happened? We need to find the answers to several questions:

  • What was the nature of the attack? If we don't know this, we can't say with certainty that the attack is over or that we have actually contained it.
  • What was damaged/lost/compromised?
  • Are controls still functioning? If so, how did the attack get by them?
  • Which systems were involved?
  • What was special about those systems? Were they attacked at random? Were they vulnerable in some way that we can address?
  • Was the infrastructure part of the problem? Did the firewalls fail to stop traffic that could have been stopped?

The text warns us that we may be in too much of a hurry or too inexperienced to know that a "reimage the machines and restore the data" approach may not solve the problem. If we were attacked by an entity that means to do it again, then reimaging and restoring is only setting the stage for another act of violence. If we have contained the problem, but not removed it, we will see the same effect. We need to find the problem. The vulnerability that was exploited needs to be eliminated or reduced. If logs, monitors, and incident detectors tell us what happened, we have to make a plan to guard against it happening again. Then we need to implement that plan.

Once a plan to avoid the same kind of incident has been put in place, we can begin bringing systems back online. If we have no other way to remove a virus/worm, we may have to reimage and restore data (where possible), but we should do so as part of testing our theory about the event, and this version of our plan to prevent a repeat of it. Version? Always have some part of your mind that assumes you are wrong. It is hard to do, but don't get overconfident. Remember Deming's cycle:

  • plan - plan for recovery based on your understanding at the moment
  • do - try out the plan: do it on a few devices at a time
  • check - see what happens with your solution
  • act - if the solution works, expand your application of it; if it fails, learn why and start over
  • *

So, let's assume you found the virus and removed it. You have amended firewall rules, email filters, and educated your staff. While you were doing that, you should have restored the services that were brought down during the incident, either by the event or by your staff. Sending an announcement to managers to share with their staff, honestly worded so as to avoid rumors about hiding the truth, should help staff believe that the situation is normal again.

This takes us to the Maintenance phase. The text is unclear as to whether we are doing maintenance of our incident response plan, or maintenance of our organization. You could say that the steps we take could do both.

  • Incident review - The text calls it an after-action review, which is a little grandiose. Review what happened, what we did, and whether what we did was right or wrong. Also review what we should do in a similar situation in the future.
  • Lessons learned - What changes should be made to the incident response plan? What changes need to be made to our environment or our organization?
  • Historical record - File the review documentation where it can be read again when it is needed. Also, use it as a training aid when running drills with your staff.
  • Closure - It's a little late in the process to enjoy the Yogi Berra quote in the text. In my opinion, we should be muttering "It ain't over 'til it's over*" every time we have to repeat the PDCA cycle in the previous bullet list. Assuming we did, and we learned to be careful, we can celebrate the actual closure of of the incident at this point.

Most of the text is about the rules, procedures, and policies that relate to an organization, but page 319 opens a new topic: what do we do when we encounter evidence of a crime? Who do you call? The simple answer is to start with local police, and to be referred to other agencies that relate to the nature of the crime. The role of the IT staff may be to preserve the evidence so that it is not made invalid in some way. Page 322 starts a long section of the chapter about Forensics,

Forensic concerns can be contrary to IT operating principles. Our first instinct should be to repair a device or restore it to proper operating status. In a situation in which there may be something to prove in a courtroom, we are required to isolate, contain, and otherwise NOT change the state of a device whose condition points to a crime. This is not something that every technical person is trained for. As such, it may be necessary to immediately consult with senior staff or specialists who may advise us, or may begin a forensic investigation themselves.

The text presents a list of concerns that apply to collection or seizure of devices, data, or other items that may be submitted as evidence in a court.

  1. Does the organization have a policy that allows for a search for evidence? (The text notes, correctly, that an exhibit submitted to a court is not evidence until it is accepted as such by a judge. The word "evidence" is used here in the general sense, something that seems like it may be needed to prove a case.)
    If there is such a policy, does it apply to all staff, and have the staff associated with the potential evidence agreed to the policy?
  2. Is there a reasonable cause for a search to be made?
  3. Is the scope of the search permissible?
    Items 2 and 3 are based on the fourth amendment to the United States Constitution. As explained by this user friendly web page, the amendment relates to personal property, not to property owned by the party conducting the search. It is still required, however, that a search be made in a context that is justified by the conditions/situation/accusation that led to the search. You can't search for something in a place you would not expect to find it.
  4. Verify that the organization owns the "containers" to be searched, such as computers, phones, file folders, external storage devices, and network devices.
  5. The search must be authorized by appropriate management, by an audit requirement, or by an actual subpoena.

The text offers over six pages of relevant discussion about law and about employee expectations of privacy, which are grayer when the employee is using organization equipment for acceptable personal use, or using personal equipment for the benefit of the employer.

The text remarks that the plan for a forensic investigation (looking for evidence, and presumably looking for the truth) should include three aspects:

  • How much it will cost to conduct the investigation? - This is a frequent topic when there is a request for information from email. Setting up a holding area, restoring archived mail to that area, and examination by expert staff all cost more than daily operations cost.
  • How long will the investigation take? - The text mentions that making our own staff available to do it may be less expensive, and take less time, than granting an outside agency access, and holding their hand while they figure out our system.
  • What parts of the data to be examined are sensitive? - Sometimes the answer is "all of it", which will elevate the cost, and extend the time to do it, since care must be taken to protect the rights of persons whose personally identifiable information may be contained in the data.

Beginning on page 332, twenty pages of the chapter detail the actions to be taken by first responders and by analysts who will examine the data on seized devices and files. The rules about preservation of a chain of evidence are the same as those you may be familiar with from television and movies. The level of detail in this section is better for reference for most staff who will never have to do the job, but may have to cooperate with those who will.

The chapter ends with a brief discussion of two terms:

  • eDiscovery - The text explains that discovery is a legal process in which the attorney for one side requests pertinent information about the case from the other side. (Sounds like those lawyer shows we saw with surprises in the courtroom were a little uninformed.) eDiscovery is about collecting, searching, and cataloging potential evidence from electronic data.

  • anti-forensics - Anti-forensics would be any effort on the part of a perpetrator to hide evidence from a forensic investigation.


The assignment for this week has not yet been determined.