Computer Incident Drill

Document Sample
Computer Incident Drill Powered By Docstoc
					             Computer Incident Response

                   a tabletop exercise

                                       Facilitated by Jim Leinweber
                                             sponsored by BadgIRT

10/27/2006               BadgIRT - CSIRT drill                        1
1.      basics of incident response                         (20 minutes)
2.      break and preparation                               (3 minutes)
3.      table top exercise                                  (20 minutes)
4.      postmortem                                          (15 minutes)

      We will need an audience volunteer to participate!

Materials today:
    – these slides
    – example red booklet [circulating]            (checklist, policy chunks, …)
    – participant scenario packets                 (network, tools, hints, …)

Copies of all of these are available at:
10/27/2006                      BadgIRT - CSIRT drill                              2
                           Why have a drill?
• our best efforts at prevention will eventually fail
• we don’t want to invent process during a crisis
       – a drill improves our disaster recovery planning
       – a drill improves our business continuity planning
• it’s a security best practice
       – all organizations should do this, and most commercial ones
         already do
       – For HIPAA health care components, it’s a legal requirement
• some of us have few real incidents to learn from

• But don’t drill in a vacuum…
       – drill last, after you have policies & procedures in place

10/27/2006                        BadgIRT - CSIRT drill               3
                       What are we defending?
1. Integrity of our data
       –     We’re a university. Data quality is everything.

2. Availability of our systems
       –     it’s our mission as IT professionals

3. Confidentiality of our data
       –     often a legal mandate (FERPA, HIPAA, IRB, …)

4. Our Reputation
       –     two words: Arthur Anderson

10/27/2006                         BadgIRT - CSIRT drill       4
                       Why are we defending?
• The bad guys are out there,
       – they’ve been trying to get in since 1988
       – viruses, worms, sql injection, password guessing, …
• distributed denial of service attacks (DDOS)
       – multiple times per week at the UW-Madison for ~ 5 years now
• a tsunami of e-mail SPAM
       – 80% of inbound e-mail
• identity theft is now a very profitable criminal enterprise
       – 3% of web credit card transactions are fraudulent
       – lots of recent database compromises (UC Berkeley, Lexis/Nexis, …)
• botnets, networks of remotely controlled zombie computers
       – used for spam or DDOS
       – a least 5 million current victims worldwide. Maybe 70 million?
       – universities are popular targets due to high bandwidth & weak

10/27/2006                         BadgIRT - CSIRT drill                     5
                           How are we defending?
1.   prevent
       bank: vault, two keys to access safety deposit boxes, …
       computers: configure securely, patch, firewall, antivirus, file integrity, …

2. detect
       bank: alarms (windows, vibration, …)
       computers: monitor logs & networks

3. react
       bank: police
       computers: incident response – by us!

•        Today we’re going to practice reacting
•        We have to fix our own problems

10/27/2006                                BadgIRT - CSIRT drill                       6
                          Incident Response teams
•    CSIRT “Computer Security Incident Response Team”
       – an organized group of people who cope with computer and network security
         problems in an organization. These started in 1989 …
       – “Research and Education Networking Information Sharing and Analysis Center
•    lots of other groups, e.g. SANS, NANOG, UNISOG, NIST, CISecurity, …

•    FIRST: 180+ teams worldwide
       – CERT/CC (post-DHS: us-cert) after 1988 Morris worm
       – teams from governments, industry, academic, …
             • 4 from the Big Ten
•    BadgIRT – Badger Incident Response team
       – UW-Madison’s FIRST liaison
       – 10 campus expert volunteers
•    DoIT– Computer Information Security Office (new!)
       – about 6 FTE under Jim Lowe
       – plus separate ResNet staff for student dorms

•    Us: ad hoc, but pre-planned & (soon!) practiced

10/27/2006                             BadgIRT - CSIRT drill                          7
                      An Incident Response Process
1.     Identification
       –     is there a problem?
       –     is it a computer security problem?
2.     Coordination
       –     exactly what is the problem? how big is the problem?
              •   what systems are affected? Are other organizations involved?
       –     what is our plan to solve the problem? Who do we need on the team?
       –     what outside resources do we need to invoke?
3.     Mitigation
       –     stop the spreading & regain control
       –     fix the systems which were intruded upon
       –     recover lost or damaged data
       –     resume normal operations
4.     Investigation
       –  how, exactly, did the bad guys get in?
       –  what was the sequence of events?
       –  how complete was the cleanup?
       –  if prosecuting, supply the evidence to law enforcement
5.     Education – learn & improve
       –  improve prevention, detection, and reaction; update policies and procedures
       –  share outcomes

10/27/2006                                    BadgIRT - CSIRT drill                     8
             Think about your threats – and responses
• pre-planning protections & likely responses
       – systems with nasty forensics should be better protected
       – have some ideas about dealing with malware, unacceptable use,
         compromised networks, privacy leaks, damaged servers, …
• You can’t know all the details in advance, but you can:
       –     have good, tested backups of your data
       –     have an inventory of your systems and maps of your network
       –     have procedures for rebuilding systems from scratch
       –     have communication plans with phone trees, wiki’s, e-mail, lists
             of cell phone numbers and IM names, etc.
• Policies and procedures matter
       – server documentation standards
       – departmental acceptable use policy

10/27/2006                           BadgIRT - CSIRT drill                      9
         Think about your people - a few basic roles
• public affairs
       – media contacts; communicate with students, staff, customers,…
• human resources
       – policy violations, work rules violations, e-harassment,…
• help desk
       – correlate symptoms, escalate, communicate
• system administrators
       – identify problems, mitigate them
• faculty / staff / lab supervisors …
       – consult on systems & customer decisions, workarounds; communicate
         with students & staff
• incident handler
       – lead technical response
             • default choice: your security officer
             • a role, not a person: in 24x7 issues, this will hand off 2-3 times per day

10/27/2006                               BadgIRT - CSIRT drill                              10
                            Identifying an incident
•    start with non-intrusive stuff
       – trouble reports from users, sysadmins, …
       – off-host log files (consolidated windows events, unix syslog, AV console, …)
       – network traffic ( can get IP flow data)
•    on-host, try not to change things
       – your security officer should have up to date copies of forensics CD’s such as
         Helix or F.I.R.E
       – sophisticated intrusions may be memory-resident only
•    look for anomalies
       –     CPU, disk, memory usage – and discrepancies
       –     log files
       –     network traffic
       –     beware of rootkits
•    consider alternate hypotheses
       – runaway applications?
       – user or sysadmin error?
       – innocuous but unusual circumstances?

10/27/2006                              BadgIRT - CSIRT drill                            11
                            Coordinating an incident
•    how big is the problem?
       – if we face a blended threat, did we identify all the propagation vectors?
       – how do we identify all the involved / compromised hosts, accounts, …
       – do we escalate to our college or campus?
•    who do we need on the response team?
       – sysadmins, application experts, supervisors, users, …
       – what is our escalation path, and what outside resources do we need?
•    choose system shutdown options
       – disconnect from network? (at switch port? yank cable?)
       – emergency memory dump?
       – yank power?
              • normal shutdown destroys lots of important evidence
•    will we need temporary protection?
       – firewall changes
       – reduce application access or functionality?
•    look for collateral damage
       –     did the primary attacks have side effects?
       –     what are the business continuity issues?
       –     are there privacy violations?
       –     do we need to notify major customers?
10/27/2006                                  BadgIRT - CSIRT drill                    12
             Coordination: when we need outside help
• respond to legal demands, such as subpoenas
       – get UW Legal, they have lawyers
• request law enforcement assistance
       – get DoIT Security, they have contacts with UWPD, FBI, SS, etc.
• computer forensics
       – get DoIT security, they have disk capture & analysis tools
       – get Campus P&S, they have evidence lockers
• denial of service attacks
       – get DoIT NOC
              • they can block traffic upstream, and have more clout with ISPs
• media queries
       – get UW communications, they have writers & contacts
              • particularly if there are privacy law violations
• work discipline issues (porn, harassment, unacceptable use, …)
       – get Human Resources (technical staff can’t handle people issues)

10/27/2006                                 BadgIRT - CSIRT drill                 13
                        Mitigating an incident
• change compromised passwords & secret keys
• clean or rebuild compromised hosts
       – first you have to find them all
       – rebuild is safest, due to rootkits & multiple compromises
• close the entry vectors
       – patching
       – more secure configurations
• restore lost data
       – from backup
       – re-construction from paper / off-line originals
• verify validity of data before returning systems to service
• step up log & network monitoring to verify it’s really over

10/27/2006                        BadgIRT - CSIRT drill              14
             Investigating an incident (afterwards)
• try to reconstruct the sequence of events
• what vulnerabilities were exploited?
• what was the first evidence demonstrating the problem?
   – want the first available, not just the first actually
• what malware was involved?
• were any policies violated?
• were any SOP’s ignored?
• was this incident anticipated in your threat model?
       – if so, did the anticipated response work?

• if law enforcement is involved, submit the final evidence

10/27/2006                       BadgIRT - CSIRT drill        15
                 Education - learning from an incident
• what went well?
• what could have gone better?
• how do we improve?
       –     what could prevent similar incidents in the future?
       –     could we detect the problem faster?
       –     how could we recover with less effort?
       –     do we need more or different tools?
       –     do we need more training? Who, on what?
• what did we have to improvise?
       –     which SOP’s need revision?
       –     which policies and procedures need revision?
• document!
       –     what happened (write an occurrence report)
       –     lessons drawn
• share results with campus, peers and security community

10/27/2006                                      BadgIRT - CSIRT drill   16
              Why you want a CSIRT Red Booklet
• there isn’t time to read a real book during an incident
• you don’t want to improvise process during an incident
• participants benefit from checklists
       – of CSIRT roles, processes, escalation paths, and organizational policies

• Red cover reminds people it’s for emergencies
• 8-15 pages
       – long enough to be comprehensive in outline
       – short enough to cope with during a crisis - be succinct with details

• an example template is circulating
       – it still needs a lot of work, but it’s a start …

10/27/2006                             BadgIRT - CSIRT drill                    17
                         Running today’s drill
• I’ll toss out a scenario event

• starting with the handler, participants describe what they
  would do next
       – phone someone, make notes, drive somewhere, investigate
         something on a computer, …
       – if only one person wants to act, we’ll go right on
       – the handler can propose actions by staff not in this room
       – if multiple people want to act or collaborate, we’ll take turns until
         everyone is done

• Then another scenario event will occur
       – some events can be probabilistic, either for outcome or duration
10/27/2006                         BadgIRT - CSIRT drill                     18
                                   Facilitator roles
• Guide the scenario
       – there is background, and a rough timeline of anticipated events

• answer questions
       –     about the scenario
       –     about what people not present might do
       –     about what the results of your investigations are
       –     about use of UW resources

       (some answers may wait for the postmortem)

• In general, the participants are on their own
       – but the facilitator may intervene if
              • something vital to our goals is being overlooked
              • a discussion is losing focus

10/27/2006                               BadgIRT - CSIRT drill             19
• We need a volunteer to play the role of incident handler

• BadgIRT Team members today will be playing:
       –     Jim Leinweber:   the facilitator
       –     Jeff Savoy:      campus network investigator (himself)
       –     Bruce Labuda:    an over-eager graduate student
       –     Eric Giefer:     a slightly clueless faculty member

• If the Incident Handler wants advice from techpartners,
  he or she may ask the audience.

I’ll try to keep helpful stuff on-screen as we go …

10/27/2006                         BadgIRT - CSIRT drill              20