Docstoc

Malware analyzer detection in virtualized environments

Document Sample
Malware analyzer detection in virtualized environments Powered By Docstoc
					EUROSEC 2011

Gábor Pék , Boldizsár Bencsáth and Levente Buttyán

                Laboratory of Cryptography and Systems Security
                Budapest University of Technology and Economics




nEther: IN-GUEST DETECTION OF
OUT-OF-THE-GUEST MALWARE ANALYSERS
Short Summary

 We successfully achieved
  In-guest detection of an out-of-the-guest
   malware analysis framework (Ether)
       In-guest timing attack
       Detection based on CPUID information


  Detecting hardware assisted virtualization
      (can be a bit of information for analysis )
       Detection based on errata in Intel CPUs

                         Gábor Pék, CrySyS Lab.     2
3/31/2012
Goals in Malware Analysis

  Analyser: dissecting and figuring out the
   operations of the analysed program

  Author of the malware: thwarting the
   analysis of the code and hiding its real
   intents, operations, execution




                     Gábor Pék, CrySyS Lab.    3
3/31/2012
What is Malware Analysis?
  Analysing malware
       Static (entire program, thwarting disassemblers)
       Dynamic (one control path)  we focus on this
  Two types of dynamic analysis: Native and
   Virtualization based
  Main tricks of detecting dynamic analyzers
       Timing information
       Special data structures, e.g., PEB
       Single-step debugging (trap flag)
       Exception handling
                         Gábor Pék, CrySyS Lab.            4
3/31/2012
HW Assisted Virtualization

  New and higher CPU privilege level (Ring -1)
  Native instruction execution
  Intel VT
       VMX root mode for VMM/Hypervisor
       VMX non-root mode for guest OS
       VMX transitions: VM Exit / VM Entry
  Rich feature set and control of operation
  Xen, KVM

                        Gábor Pék, CrySyS Lab.    5
3/31/2012
Ether – Malware analysis via
HW Virtualization Extensions
  Transparent, out-of-the-guest malware analysis
   platform based on Xen and Intel VT
  Transparency of Ether: the malware cannot
   detect Ether
  Transparency requirements as of the Ether paper:
       Higher privilege of analyser environment
       No non-privileged side effects
       Same instruction execution semantics X
       Identical exception handling
       Identical notion of time X

                           Gábor Pék, CrySyS Lab.     6
3/31/2012
Advantages of Ether and
challenges to detect it

  No in-guest memory presence
  Hide of changes made on CPU registers
  Memory protection: modifies only shadow
   page tables
  Privileged instruction handling
  No instruction emulation
  Controlling timing (e.g., RDTSC instruction)

                    Gábor Pék, CrySyS Lab.        7
3/31/2012
Contributions

  Design and implementation of an application
   framework to detect Ether based on multiple
   feature tests
  Feature tests for Ether and Intel VT
        A practical in-guest timing attack against Ether
        Detecting Ether via CPUID information
        Detection of HW assisted virtualization
         utilizing CPU errata


                         Gábor Pék, CrySyS Lab.             8
3/31/2012
System Overview




            Gábor Pék, CrySyS Lab.   9
3/31/2012
Various uses of RDTSC
 Different behaviour of sensitive instructions
  (e.g., RDTSC) in VMX non-root mode
     Normal operation                Operation of Ether

            Guest OS       Guest OS              …   Guest OS



                           rdtsc faketime (FT) rdtsc FT+Y
            rdtsc TSC
                            VM Exit VM Entry



               CPU
                             Virtual             …    Virtual
                              CPU                      CPU

                        Gábor Pék, CrySyS Lab.                  10
3/31/2012
Facts about Ether

  Alleged operation: FT = TSC, Y = TSC_OFFSET
       but TSC_OFFSET is disabled
  Real operation: Monotonic increase of FT for
   every RDTSC call (Y =1)
  There can be external RDTSC calls during an
   analysis
       The TSC difference between two RDTSCs of the
            analysed program = # of RDTSCs of the Guest
            during analysis (~9-171)

                            Gábor Pék, CrySyS Lab.        11
3/31/2012
Practical implementation of
in-guest timing attack
 Call an RDTSC and store it
 Create a loop of non-sensitive instructions
  (e.g., nop)
 Call an RDTSC and compare it with the stored
  value (diff)

      if (diff < length of the loop)
           Ether is present
      else
           Ether is not present

                     Gábor Pék, CrySyS Lab.      12
3/31/2012
CPUID information

  CPUID instruction: processor identifcation
   and feature information
  Allegedly: Ether has no in-memory presence
  Reality: The TSC bit returned by CPUID is
   unset under Ether
  Other bits of information
       PAE and PSE are disabled




                        Gábor Pék, CrySyS Lab.   13
3/31/2012
CPU Errata
  Design deficiencies of CPUs
  Some of them are unpredictable
  Cause unexpected system behaviour
  Several have ”No Fix ” status
  Xen creates virtualized CPUs for privileged
   instructions
  We have an erratum using MSRs (AH4)
  The access of MSRs are privileged  VM exit
  Errata are not emulated by virtual CPUs
  Bingo, we have a new feature test
                    Gábor Pék, CrySyS Lab.       14
3/31/2012
Detecting Intel VT

   Erratum AH4                   Number of updates
      # of tests    Native                    Xen       Xen + Ether
             100     59                             0       0
            1000     650                            0       0
            10000   4232                            0       0
        100000      20870                           0       0




                           Gábor Pék, CrySyS Lab.                     15
3/31/2012
Future Work

  Fundamentality of these problems
  Updating the theoretical model and practical
   implementation of Ether
  Finding more feature tests against other out-
   of-the-guest approaches (e.g., Azure)
       Proving that perfect transparency has practical
            limitations




                          Gábor Pék, CrySyS Lab.          16
3/31/2012
Thanks for Your Attention!

 Questions?

 pek@crysys.hu
 boldi@crysys.hu
 buttyan@crysys.hu

 CrySyS Lab. http://www.crysys.hu
 Budapest University of Technology and Economics

                     Gábor Pék, CrySyS Lab.        17
3/31/2012

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:19
posted:4/1/2012
language:English
pages:17