HBGary's Active Defense
Strengths and Weaknesses against Mandiant MIR
HBGARY CONFIDENTIAL
Contents
Ability to find unknown malware. This means that the FBI has not contacted the customer. This means that Mandiant has
not sent the customer a 'Victim Notification'. (Penny) ............................................................................................................... 4
Ability to detect malware based upon behavior traits (Penny) ................................................................................................... 4
Evaluating the Digital DNA capabilities for finding APT (Bob/L3)................................................................................................. 4
Ability to white list known good software (Penny) ...................................................................................................................... 5
Ability for a level 1 or 2 to perform scans and IOC queries (Penny) ............................................................................................ 5
Ability to scan for variants (Penny) .............................................................................................................................................. 5
Speed and scope of scans (Penny) ............................................................................................................................................... 5
Ability to define a hierarchical structure for organization of hosts/servers (L3) ......................................................................... 5
Ability to group objects/hierarchical structures (L3) ................................................................................................................... 5
Ability to apply commands/queries/reports against these structured objects (L3) .................................................................... 5
Ability to scale to 120+ organizational units and 100,000 systems. (L3)...................................................................................... 6
Ability to randomize a wait time between when a scan finishes and when the results are returned to the server, so that
network traffic returning to the server is throttled over time. (GREG)........................................................................................ 6
Ability to provide complex queries in XML and initiate/monitor jobs programmatically (L3) ..................................................... 6
Ability to programmatically control the Active Defense system from a 3rd party enterprise system (Greg) .............................. 6
Ability to provide query /job results in XML formats. (L3) ........................................................................................................... 6
Ability to schedule “chron” jobs. (L3)........................................................................................................................................... 6
Ability to support multiple concurrent threads (e.g. Multiple jobs, from multiple analysts) (L3)................................................ 7
Ability to queue a scan against a host that is offline, and initiate the scan on the target host when it comes online (Greg) ..... 7
Ability to throttle scans (Greg) ..................................................................................................................................................... 7
Ability to specify a 'safe window' in which to run scans (Greg) ................................................................................................... 7
Ability to complete a scan even when a laptop has been taken out of the network (Greg) ........................................................ 7
Ability to scan physical memory at the end node, without copying any memory snapshots over the network (Greg) / Ability
to scan PHYSICAL memory concurrently (Penny)......................................................................................................................... 7
Ability to collect system metadata and events (Hardware, Software, Configuration Files/Info, Event Logs, Processes, Files,
Executables, DLLs, etc.) (L3) ......................................................................................................................................................... 8
Ability to provide Audit Logs of Agent Activities/Data Collections (L3) / Audit logging of all actions/events (attributable to
specific authenticated analysts and/or chron jobs) (L3) .............................................................................................................. 8
TFA to control/attrribute Administrative/Analyst Access (L3) ..................................................................................................... 8
Support for OpenIOC or similar capability XML Schema (L3) ....................................................................................................... 8
Ease of installation/deployment/uninstallation (L3).................................................................................................................... 8
Support for wake-up call (Greg) ................................................................................................................................................... 8
Ability to deploy agents directly from console (Matt) ................................................................................................................. 8
Ability to deploy with 3rd party mechanisms (Greg) ................................................................................................................... 9
System impact when idle, and when scanning (L3) ..................................................................................................................... 9
Performance impact of running multiple concurrent queries (L3) .............................................................................................. 9
Ability to search for indicators including (but not limited to) filename, location, hash, size, registry key (L3) ........................... 9
Ability to pull files, registry values, memory dumps, deleted files, process/port listings, or filesystem dumps from a machine
(L3) ............................................................................................................................................................................................... 9
Ability to pull multiple files & folders at once (Matt)................................................................................................................... 9
Ability to pull a disk image (Matt) .............................................................................................................................................. 10
Ability to stream a disk image to a secondary location (Matt) .................................................................................................. 10
Ability to stream physical memory dump to a secondary machine on the network as opposed to the local host under analysis
(Matt) ......................................................................................................................................................................................... 10
Pull down full system information as part of a basic scan (Matt) .............................................................................................. 10
Ability to construct complex queries based off of multiple indicators (L3)................................................................................ 10
Speed of running simple or complex queries across single or multiple hosts (L3) / Ability to scan concurrently (Penny) ........ 10
Ability to scan raw disk/memory (L3) ........................................................................................................................................ 10
Ease of entering indicators to scan for (automated methods preferred) (L3) ........................................................................... 11
Output reporting and ability to export data in common formats (automated methods preferred) (L3) .................................. 11
Ability to find unknown malware. This means that the FBI has not contacted the customer. This
means that Mandiant has not sent the customer a 'Victim Notification'. (Penny)
This is a significant strength of HBGary over Mandiant. HBGary is empowering customers to run their own incident response
teams and clean up their own network. HBGary is a product company first, and only recently started offering services.
Mandiant is the complete opposite, offering services primarily and then trying to build a product second. This is reflected in
everything. Mandiant's product is hard to use and requires a low level expert user - HBGary's product is easy to use, even for
an entry level analyst. Mandiant's product is typically sold with Mandiant's services, and Mandiant personnel operate the MIR
appliance for the customer. HBGary's product is designed so customers don't need outside services and can become self
sufficient. Finally, Mandiant's product is clearly written by security consultants who also have coding skills - it just doesn't have
the polish and design of a real professional product, has lots of bugs, and alot of 'organic' non-directed design work that makes
it an overall complicated experience. HBGary's Active Defense, on the other hand, it written by professional developers all of
which have a long career in product development - and the Active Defense product is designed on elegant, well thought out
architecture. It's just a higher quality product, and anyone who doesn't have a prior bias and uses them side by side for any
length of time will figure that out.
Ability to detect malware based upon behavior traits (Penny)
Active Defense uses behaviors to detect suspicious programs. All executable code is detected in physical memory, even
injected code or code that doesn't have a module on disk. All of this code is disassembled down to machine language, control
flows are calculated, and large directed graphs are built. This is fed into the Digital DNA system that can detect software
behaviors using rules. On a typical host, this process will cover millions of data points. Nothing in the security industry gives
you such a comprehensive and low level scan of the executable code running in your enterprise.
Evaluating the Digital DNA capabilities for finding APT (Bob/L3)
Well, this is hard to gauge. It would be best if the AD system is tested on a statistically relevant number of machines - a few
hundred at least. HBGary believes that DDNA is detecting over 70% of unknown malware in zero-knowledge testing (no prior
signatures). This is based on HBGary's feed processor work. A large number of malware in this set are external non-targeted
(botnets, etc).
In the category of APT specifically, the number of actual samples to work with are smallish (a few hundred collected over
several years) - of this set, we have typically detected them about 50% of the time going in, and the other half we scored low.
For the low scoring samples, we subsequently analyzed them and discovered new malware tricks that we had to add to our
DDNA trait set. All DDNA traits are generic - HBGary never creates a sample-specific signature. Given that HBGary has been
doing this for several years now, HBGary believes that DDNA is probably 60-80% effective at detecting unknown APT.
It should be noted that once Active Defense is deployed, a set of IOC's will be developed over time that are specific to the
attackers in the customer's environment. These IOC's augment the DDNA system for detection. And, if any APT malware
samples are collected (found via IOC or otherwise) that score low on DDNA, HBGary has an extremely fast turn-around on
DDNA updates. HBGary will reverse engineer any low scoring APT sample and update DDNA with behavioral traits, without
using signatures.
Note: this is a strength against Mandiant. All detection capability with Mandiant is based on IOC's - they have no generic
capability to detect suspicious program behavior.
Matt's Note: A/D at its best; MIR has nothing here except maybe very brilliantly crafted searches
Ability to white list known good software (Penny)
Active Defense allows suspicious programs that have a high scoring DDNA to be whitelisted.
Ability for a level 1 or 2 to perform scans and IOC queries (Penny)
Active Defense has a very simply interface to specify IOC's - anyone who can create a Google Advanced Search can create an
IOC in Active Defense. Mandiant, until recently, required the user to know XPATH and XML in order to craft a query. Now they
offer an IOC builder, but that was only after they saw how easy Active Defense was making it, and they had to compete. Note:
It is not clear if the IOC builder is integrated into MIR, however - it might be an external tool that you then have to copy the XML
back into MIR.
Ability to scan for variants (Penny)
Because Digital DNA is not based on binary-specific signatures, it has the ability to detect multiple variants of the same
malware. For example, HBGary has been able to detect multiple APT backdoors that had different command-and-control
systems in place, but otherwise were based on the same core codebase.
Speed and scope of scans (Penny)
This is addressed in more detail below, but Active Defense is architected from the ground up to minimize impact on the
network while leveraging concurrent parallel scanning at the end node for large scalability. In terms of scope, this includes
Digital DNA, physical memory, timeline / event analysis, low level disk/raw volume, and live running system objects. All other
strengths aside, this single factor makes Active Defense a clear choice for customers who know time is a precious resource,
have hit the limit of their team, and have to scale up. Active Defense is all about saving time.
Ability to define a hierarchical structure for organization of hosts/servers (L3)
A single AD server will support apprx. 20,000 connected clients. HBGary does not support a tiered hierarchy of AD servers
where you can manage multiple child AD servers from one master. To deploy against 100,000 nodes, you will need five AD
servers, and you would organize the 100,000 clients across those five AD servers in some logical manner that makes sense.
Mandiant will claim that they support a tiered hierarchy for MIR - they put this in their data sheet. Be extremely cautious of
this claim. In reality, this 'feature' of MIR has a history of not working, or has been buggy in the past.
Note: you are going to find this alot with Mandiant. They claim to support many features, but in reality many features are
buggy and during training they will recommend to their customers that these features 'be avoided'.
Ability to group objects/hierarchical structures (L3)
Machines can be grouped in a hierarchy, with groups and sub-groups. HBGary lets you apply policies and initiate actions
against a group and optionally, all subgroups of a group.
Ability to apply commands/queries/reports against these structured objects (L3)
Scan policies (the equivalent of an IOC scan) can be applied to groups and subgroups. This concept applies also to
administrative tasks, such as updating an agent or sending a wake-up call.
Ability to scale to 120+ organizational units and 100,000 systems. (L3)
This will require approximately five to seven AD servers, depending on how many nodes are in an organizational unit and
whether they can be broken across server boundaries. Scan policies can easily be copied across these servers because of XML
import/export features. AD allows you to import your machine lists from active directory, or from XML formatted lists of
machines. AD allows you to deploy directly from the AD server, or use a third party deployment mechanism.
Ability to randomize a wait time between when a scan finishes and when the results are returned
to the server, so that network traffic returning to the server is throttled over time. (GREG)
AD supports this feature. It can also be tuned depending on the number of nodes under management. A 10,000 node
deployment should have a wait time of around 10-15 minutes. A 20,000 node deployment should use a wait time of around 30
minutes.
Ability to provide complex queries in XML and initiate/monitor jobs programmatically (L3)
AD supports compound queries with multiple AND / OR relationships. These queries can be specified using a GUI interface that
is intuitive and resembles 'Google Advanced Search' in many ways. These queries can also be specified in XML. They can be
imported/exported using XML from the web console interface. They can also be specified and delivered directly to an end-
node agent using a job.xml file. The direct-to-agent interface is a fully documented and supported SDK that allows a customer
to manage Active Defense deployments using their own management system or product. Several commercial security
companies, as well as government agencies, have successfully integrated Active Defense directly into existing frameworks or
products using the SDK.
Ability to programmatically control the Active Defense system from a 3rd party enterprise system
(Greg)
HBGary makes the SQL database schema available to customers so they can integrate with Active Defense with other enterprise
frameworks and home-grown systems. By interfacing through SQL, almost any data can be queried in any format from the AD
system. Furthermore, almost any manipulation or action can be initiated. This would be an advanced form of use, but it is
supported and would be of value if a customer has custom developed frameworks already in use.
Ability to provide query /job results in XML formats. (L3)
From the Active Defense web console, scan policies (IOC queries) and report queries can be exported in XML, CSV, XML, PDF,
TXT, and HTML formats. If the customer wants to retrieve results directly from an agent without using the Active Defense web
console, that can be done using the results.xml file that is produced at the end node. Direct communication with a deployed
agent is documented and supported via HBGary's Partner SDK, and this communication is primary through a job.xml file (input)
and a results.xml file (output).
Ability to schedule “chron” jobs. (L3)
Active Defense allows Digital DNA scans and IOC scans to be scheduled.
Ability to support multiple concurrent threads (e.g. Multiple jobs, from multiple analysts) (L3)
Active Defense has a robust queuing system. Multiple analysis will have no problem running jobs and using the system
concurrently. The interface is web based, there is no native client. HBGary has typically had 4+ analysts using the system at
once, and there is no artificial upper limit on this.
Ability to queue a scan against a host that is offline, and initiate the scan on the target host when
it comes online (Greg)
Active Defense queues all jobs. If a host is offline, it will pick up the job when it comes online. The status of all jobs is reported
in the Active Defense web console, including how many hosts have checked in and completed the scan, and how many hosts
remain outstanding.
Note: we know that historically this is a weakness of Mandiant, they can only scan hosts that are currently online
Ability to throttle scans (Greg)
AD has throttle settings for the end node scans. These apply to both physical memory scans and IOC queries against disk.
Note: We know this is a weakness with Mandiant, with MIR you cannot adjust that setting
Ability to specify a 'safe window' in which to run scans (Greg)
AD gives the user an optional 'safe window' feature, which is a time-span in which scans are allowed to run (for example, only
between midnight and 6AM). Scans will never be running outside of this safe-window.
Note: this is not available in MIR
Ability to complete a scan even when a laptop has been taken out of the network (Greg)
If a laptop picks up a job from AD, and then the user takes the laptop home, the job will still run and complete. The results will
be queued and the AD agent will wait until it can check in again. When the laptop is reconnected to the enterprise network,
the laptop will check in and return the job results.
Note: we know this is a weakness of Mandiant
Ability to scan physical memory at the end node, without copying any memory snapshots over the
network (Greg) / Ability to scan PHYSICAL memory concurrently (Penny)
AD performs all scanning at the end node. Only the results.xml file is brought back, which is a small file that only contains
meta-data and results. No memory is copied over the network unless the user specifically requests it. This allows AD to scale
to a very large network without impacting network performance or bottle-necking on data transfers. This makes AD very fast
and efficient.
Note: We can't be sure, but this may be an issue for Mandiant. We have never been able to get a clear answer on when data
needs to be copied over the network, and when analysis is at the end node. It seems there are multiple ways to do things with
MIR and it's not very clear when it occurs at the end node or when data gets copied to the MIR appliance first.
Ability to collect system metadata and events (Hardware, Software, Configuration Files/Info, Event
Logs, Processes, Files, Executables, DLLs, etc.) (L3)
Active Defense can collect a great deal of information from a host.
Digital DNA for all live running modules (from physmem)
Physical memory snapshot
All modules and processes, including path information
Live extracted file from physical memory (DLL, EXE, SYS, injected)
Events recovered from event log
Master File Table ($MFT), including all attributes
Internet browsing history (index.dat, temporary internet files, etc)
Any file from disk, even deleted files or those that are locked/in-use (forensically sound)
Ability to provide Audit Logs of Agent Activities/Data Collections (L3) / Audit logging of all
actions/events (attributable to specific authenticated analysts and/or chron jobs) (L3)
Active Defense has a log of activity, but it's purpose is primarily for debugging and status, not auditing. It may not be as robust
as that provided by Mandiant MIR.
TFA to control/attrribute Administrative/Analyst Access (L3)
Active Defense doesn't support this. HBGary has numerous customer requests to add this feature. It will be added at a future
data. This is likely a strength of Mandiant.
Support for OpenIOC or similar capability XML Schema (L3)
Scan policies (IOC's) can be imported and exported using XML. Active Defense doesn't support OpenIOC. Note: HBGary would
support a non-partial IOC language, but OpenIOC is actually Mandiant's. That said, converting between Mandiant's OpenIOC
and HBGary's IOC XML is trivial.
Ease of installation/deployment/uninstallation (L3)
This doesn't get any easier. Active Defense is much easier to use than Mandiant's MIR. Even Mandiant's own employees
wouldn't be able to argue with this.
Support for wake-up call (Greg)
Active Defense allows you to wake up agents. This is not required, as agents will check in on a periodic schedule, but is an
option if the user is in a hurry.
Note: we know this is a weakness of Mandiant.
Ability to deploy agents directly from console (Matt)
Active Defense allows agents to be deployed over windows networking. This requires domain admin credentials.
Note: we know this is a weakness of Mandiant
Ability to deploy with 3rd party mechanisms (Greg)
Yes, Active Defense can be deployed via EnCase Enterprise, McAfee ePO, SMS, Bigfix, or any method that allows a single EXE to
be executed with command line options.
System impact when idle, and when scanning (L3)
No impact while idle. Throttled impact when scanning. Scanning is never going to be zero cost, but HBGary makes every effort
to throttle and minimize impact to the end user. HBGary also supports 'safe scan window' (see above). HBGary also will not
initiate a scan immediately when a user logs onto their workstation, and instead will randomly wait for a period of time before
beginning a queued job, giving the workstation time to startup and load programs before a scan begins.
Performance impact of running multiple concurrent queries (L3)
Active Defense runs each queued query in succession at the end node. If multiple search terms are being used in a single query,
Active Defense only performs a single pass for all search terms. This single-pass / multiple search term algorithm is one of the
reasons Active Defense is so fast.
Ability to search for indicators including (but not limited to) filename, location, hash, size, registry
key (L3)
All of the above no problem. Active Defense allows scanning based on file contents as well, and status such as deleted, and file
access times. Registry keys and values can be scanned as well. Physical memory objects can also be scanned, including live
running binaries. The running operating system can also be queried, including process and parent process, virtual memory, and
paths.
Matt's note: A/D should allow for this feature; but will run into the same problem that Mandiant had which prompted such
things as implementing XPATH to pre-filter results before bringing them back to the controller. Another example is file hashing;
files would be pre-filtered to a file name, size range, or path in order to prevent having to hash all the files on the system (which
would be time and resource intensive).
Ability to pull files, registry values, memory dumps, deleted files, process/port listings, or
filesystem dumps from a machine (L3)
Any scan policy hit (IOC hit) can be pulled from the machine. These are not pulled over the network unless the user specifically
requests them. Typically, a user would scan first, get the results, and then make select pull-downs for files of interest.
Regarding the above, Active Defense doesn't collect port listings automatically, this may be a strength of Mandiant. All the
other things listed are covered, including deleted files, memory dumps, or any arbitrary file on disk (locked or otherwise).
Active Defense can also extract the volatile memory for a single live running executable, it is unclear if this is supported by
Mandiant.
Ability to pull multiple files & folders at once (Matt)
Active Defense only allows single files at a time. There is a feature in the pipeline to support multiple files and folders. This
may be a strength of Mandiant.
Ability to pull a disk image (Matt)
This is not supported by Active Defense. This may be a strength of Mandiant. For what it's worth, the whole concept behind
Active Defense is that you shouldn't have to pull entire disk images anymore.
Note: MIR runs at peak network usage ~ 2MBps on the hosts NIC during a disk image
Ability to stream a disk image to a secondary location (Matt)
Active Defense doesn't have drive imaging.
Matt's Note: Similar to memory acquisition having a feature to DD image a volume (or even disk) to a specified location would
be useful in some situations for forensic response. Mandiant does have this but the larger the collection to more disastrous it is
for them. If you can enable this feature and stream the dd image live to a path that the user can specify, you are going to win
big vs Mandiant.
Ability to stream physical memory dump to a secondary machine on the network as opposed to
the local host under analysis (Matt)
This is not supported by either Mandiant or HBGary to the best of my knowledge. HBGary has added this to our feature
pipeline.
Pull down full system information as part of a basic scan (Matt)
Matt's Note: A/D can pull some system information but it should pull full system specs (volumes, mounted drives, user groups,
etc) along with Live Response data (network data like connections/ports, PIDs, etc)
Ability to construct complex queries based off of multiple indicators (L3)
Yes, yes, yes.
Speed of running simple or complex queries across single or multiple hosts (L3) / Ability to scan
concurrently (Penny)
Active Defense runs all jobs concurrently and in parallel. A scan is sent to all end nodes that are targeted, the scan runs at the
end node, and all results are sent back. You can scan 10,000 machines and have your results in less than 60 minutes. This
puts Active Defense in a class by itself for performance. EnCase can't touch it. It's very likely that Mandiant can't touch it
either.
Ability to scan raw disk/memory (L3)
Yes, yes.
Note: MIR does do some of this but they don't have DDNA to automate analysis and detection of anomalies
Ease of entering indicators to scan for (automated methods preferred) (L3)
This is easier with AD than with Mandiant. Users don't have to know XPATH or XML - it's like using Google Advanced Search.
Automated methods are covered direct to agent, or direct to AD console by XML, or by direct interaction with the SQL
database.
Note: Mandiant has an OpenIOC editor which is a little difficult to master
Matt's Note: A/D has a simple query builder but it does not allow for as many types of IOC variables to input or search for -
Mandiant might be stronger in coverage of things that can be specified
Output reporting and ability to export data in common formats (automated methods preferred)
(L3)
Active Defense can export the data in XML, CSV, HTML, TXT, or PDF. There is no automated delivery mechanism for report
results, but this is planned in the near-term feature list. Automated delivery might be a strength of Mandiant.
Note: MIR does everything in XML and the raw data is stored in AFF format