IEEE-pentest by burmesepentester


									  Building Security In
  Editor: Gary McGraw,

                 Software Penetration Testing

                                      uality assurance and testing organizations are                      small and predefined allotment of
                                                                                                          time and resources to the effort) as a
                                      tasked with the broad objective of assuring that a                  final security checklist item at the
                                                                                                          end of the life cycle.
                                      software application fulfills its functional busi-                       One major limitation of this ap-
                                                                                                          proach is that it almost always repre-
                                      ness requirements. Such testing most often in-                      sents a too little, too late attempt to
                                                                                                          tackle security at the end of the de-
                 volves running a series of dynamic functional tests to ensure                            velopment cycle. As we’ve seen, soft-
                                                                                                          ware security is an emergent prop-
BRAD ARKIN       proper implementation of the appli-         component will perform function-             erty of the system, and attaining it
Symantec         cation’s features. However, because         ally as desired. However, it’s unrea-        involves applying a series of best prac-
                 security is not a feature or even a set     sonable to verify that a negative            tices throughout the software devel-
SCOTT STENDER    of features, security testing doesn’t       doesn’t exist by merely enumerating          opment life cycle (SDLC; see Figure
Information      directly fit into this paradigm.1            actions with the intention to pro-           1).1 Organizations that fail to inte-
Security             Security testing poses a unique         duce a fault, reporting if and under         grate security throughout the devel-
Partners         problem. Most software security de-         which circumstances the fault oc-            opment process often find that their
                 fects and vulnerabilities aren’t related    curs. If “negative” tests don’t un-          software suffers from systemic faults
GARY             to security functionality—rather,           cover any faults, we’ve only proven          both at the design level and in the im-
MCG RAW          they spring from an attacker’s unex-        that no faults occur under particular        plementation (in other words, the
Cigital          pected but intentional misuses of the       test conditions; by no means have we         system has both security flaws and se-
                 application. If we characterize func-       proven that no faults exist. When ap-        curity bugs). A late lifecycle penetra-
                 tional testing as testing for posi-         plied to security testing, where the         tion testing paradigm uncovers prob-
                 tives—verifying that a feature prop-        lack of a security vulnerability is the      lems too late, at a point when both
                 erly performs a specific task—then           negative we’re interested in, this           time and budget severely constrain
                 security testing is in some sense test-     means that passing a software pene-          the options for remedy. In fact, more
                 ing for negatives. The security tester      tration test provides very little assur-     often than not, fixing things at this
                 must probe directly and deeply into         ance that an application is immune           stage is prohibitively expensive.
                 security risks (possibly driven by          to attack. One of the main problems              An ad hoc software penetration
                 abuse cases and architectural risks) to     with today’s most common ap-                 test’s success depends on many fac-
                 determine how the system behaves            proaches to penetration testing is           tors, few of which lend themselves to
                 under attack.                               misunderstanding this subtle point.          metrics and standardization. The
                     One critical exception to this                                                       most obvious variables are tester skill,
                 rule occurs when the tester must            Penetration                                  knowledge, and experience. Cur-
                 verify that security functionality          testing today                                rently, software security assessments
                 works as specified—that the applica-         Penetration testing is the most fre-         don’t follow a standard process of any
                 tion not only doesn’t do what it’s not      quently and commonly applied of all          sort and therefore aren’t particularly
                 supposed to do, but that it does do         software security best practices, in         amenable to a consistent application
                 what it’s supposed to do (with regard       part because it’s an attractive late life-   of knowledge (think checklists and
                 to security features).                      cycle activity. Once an application is       boilerplate techniques). The upshot
                     In any case, testing for a negative     finished, its owners subject it to pen-       is that only skilled and experienced
                 poses a much greater challenge than         etration testing as part of the final ac-     testers can successfully perform pen-
                 verifying a positive. Quality assur-        ceptance regimen. These days, secu-          etration testing.
                 ance people can usually create a set of     rity consultants typically perform               The use of security requirements,
                 plausible positive tests that yield a       assessments like this in a “time             abuse cases, security risk knowledge,
                 high degree of confidence a software         boxed” manner (expending only a              and attack patterns in application de-

532          PUBLISHED BY THE IEEE COMPUTER SOCIETY   ■ 1540-7993/05/$20.00 © 2005 IEEE   ■ IEEE SECURITY & PRIVACY
                                                                                                                    Building Security In

sign, analysis, and testing is rare in cur-                Security                  External                    Static         Penetration
rent practice. As a result, security                    requirements                  review                   analysis           testing
findings can’t be repeated across dif-
                                                    Abuse                   Risk            Risk-based                      Risk
ferent teams and vary widely depend-                cases                 analysis                                        analysis            Security
                                                                                           security tests
ing on the tester. Furthermore, any                                                                                                            breaks
test regimen can be structured in such
a way as to influence the findings. If
test parameters are determined by in-
dividuals motivated not to find any               Requirements              Design        Test                Code          Test                 Field
                                                 and use cases                           plans                            results             feedback
security issues (consciously or not), it’s
likely that the penetration testing will
result in a self-congratulatory exercise        Figure 1. The software development life cycle. Throughout this series, we’ll focus on
in futility.                                    specific parts of the cycle; here, we’re examining penetration testing.
    Results interpretation is also an
issue. Typically, results take the form
of a list of flaws, bugs, and vulnera-         A better approach                          perform most of the grunt work
bilities identified during penetration         All is not lost—security penetration       needed for basic software security
testing. Software development or-             testing can be effective, as long as we    analysis. Of course, a tool-driven
ganizations tend to regard these re-          base the testing activities on the se-     approach can’t be used as a replace-
sults as complete bug reports—thor-           curity findings discovered and              ment for review by a skilled security
ough lists of issues to address to            tracked from the beginning of the          analyst (especially because today’s
secure the system. Unfortunately,             software lifecycle, during require-        tools aren’t applicable at the design
this perception doesn’t factor in the         ments analysis, architectural risk         level), but such an approach does
time-boxed nature of late lifecycle           analysis, and so on. To do this, a pen-    help relieve a reviewer’s work bur-
assessments. In practice, a penetra-          etration test must be structured ac-       den and can thus drive down cost.
tion test can only identify a small           cording to perceived risk and offer        Second, tool output lends itself
representative sample of all possible         some kind of metric relating risk          readily to metrics, which software
security risks in a system. If a soft-        measurement to the software’s secu-        development teams can use to track
ware development organization fo-             rity posture at the time of the test.      progress over time. The simple met-
cuses solely on a small (and limited)         Results are less likely to be miscon-      rics commonly used today don’t
list of issues, it ends up mitigating         strued and used to declare pretend         offer a complete picture of a system’s
only a subset of the security risks           security victory if they’re related to     security posture, though, so it’s im-
present (and possibly not even those          business impact through proper risk        portant to emphasize that a clean bill
that present the greatest risk).              management.                                of health from an analysis tool
    All of these issues pale in com-                                                     doesn’t mean that a system is defect
parison to the fact that people often         Make use of tools                          free. The value lies in relative com-
use penetration testing as an excuse          Tools should definitely be part of          parison: if the current run of the
to declare victory. When a penetra-           penetration testing. Static analysis       tools reveals fewer defects than a
tion test concentrates on finding and          tools can vet software code, either in     previous run, we’ve likely made
removing a small handful of bugs              source or binary form, in an attempt       some progress.
(and does so successfully), everyone          to identify common implementa-
looks good: the testers look smart            tion-level bugs such as buffer over-       Test more than once
for finding a problem, the builders            flows.2 Dynamic analysis tools can          Today, automated review is best
look benevolent for acquiescing to            observe a system as it executes as well    suited to identifying the most basic
the test, and the executives can              as submit malformed, malicious, and        of implementation flaws. Human
check off the security box and get            random data to a system’s entry            review is necessary to reveal flaws in
on with making money. Unfortu-                points in an attempt to uncover            the design or more complicated im-
nately, penetration testing done              faults—a process commonly referred         plementation-level vulnerabilities
without any basis in security risk            to as fuzzing. The tool then reports       (of the sort that attackers can and
analysis leads to this situation with         the faults to the tester for further       will exploit), but such review is
alarming frequency. By analogy,               analysis.3 When possible, use of these     costly. By leveraging the basic
imagine declaring testing victory by          tools should be guided by risk analy-      SDLC touchpoints described in this
finding and removing only the first             sis results and attack patterns.           series of articles, penetration tests
one or two bugs encountered dur-                   Tools offer two major benefits.        can be structured in such a way as to
ing system testing!                           First, when used effectively, they can     be cost effective and give a reason-

                                                                         ■ IEEE SECURITY & PRIVACY              33
     Building Security In

                 able estimation of the system’s secu-       though this problem is much harder        should be targeted to ensure that
                 rity posture.                               than it seems at first blush7). By iden-   suggested deployment practices are
                     Penetration testing should start        tifying and leveraging security goals     effective and reasonable and that ex-
                 at the feature, component, or unit          during unit testing, we can signifi-       ternal assumptions can’t be violated.

                                                                                                       Integrate with the devel-
 Penetration testing can be effective, as                                                              opment cycle
                                                                                                       Perhaps the most common problem
 long as we base the testing activities on                                                             with the software penetration testing
                                                                                                       process is the failure to identify
 the security findings discovered and                                                                  lessons to be learned and propagated
                                                                                                       back into the organization. As we
 tracked from the beginning of the software                                                            mentioned earlier, it’s tempting to
                                                                                                       view a penetration test’s results as a
 lifecycle.                                                                                            complete and final list of bugs to be
                                                                                                       fixed rather than as a representative
                 level, prior to system integration.         cantly improve the greater system’s       sample of faults in the system.
                 Risk analysis performed during              security posture.                             Mitigation strategy is thus a criti-
                 the design phase should identify                Penetration testing should con-       cal aspect of the penetration test.
                 and rank risks as well as address in-       tinue at the system level and be di-      Rather than simply fixing identified
                 tercomponent assumptions.4,5 At             rected at the integrated software sys-    bugs, developers should perform a
                 the component level, risks to the           tem’s properties such as global error     root-cause analysis of the identified
                 component’s assets must be miti-            handling, intercomponent commu-           vulnerabilities. If most vulnerabili-
                 gated within the bounds of con-             nication, and so on. Assuming unit        ties are buffer overflows, for exam-
                 textual assumptions. Tests should           testing has successfully achieved its     ple, the development organization
                 attempt unauthorized misuse of,             goals, system-level testing shifts its    should determine just how these
                 and access to, target assets as well as     focus toward identifying intercom-        bugs made it into the code base. In
                 try to violate any assumptions the          ponent issues and assessing the secu-     such a scenario, lack of developer
                 system might make relative to its           rity risk inherent at the design level.   training, misapplication (or nonexis-
                 components.                                 If, for example, a component as-          tence of) standard coding practices,
                     Testers should use static and dy-       sumes that only trusted components        poor choice of languages and li-
                 namic analysis tools uniformly at the       have access to its assets, security       braries, intense schedule pressure, or
                 component level. In most cases, no          testers should structure a test to at-    any combination thereof could ulti-
                 customization of basic static analysis      tempt direct access to that compo-        mately represent an important cause.
                 tools is necessary for component-           nent from elsewhere. A successful             Once a root cause is identified,
                 level tests, but a dynamic analysis tool    test can undermine the system’s as-       developers and architects should de-
                 will likely need to be written or           sumptions and could result in an ob-      vise mitigation strategies to address
                 modified for the target component.           servable security compromise.             the identified vulnerabilities and any
                 Such tools are often data-driven tests      Dataflow diagrams, models, and             similar vulnerability in the code base.
                 that operate at the API level. Any          high-level intercomponent docu-           In fact, best practices should be de-
                 tool should include data sets known         mentation created during the risk         veloped and implemented to address
                 to cause problems, such as long             analysis stage can also be a great help   such vulnerabilities proactively in
                 strings and control characters.6 Fur-       in identifying where component            the future. Going back to the buffer
                 thermore, the tool design should re-        seams exist.                              overflow example, an organization
                 flect the security test’s goal: to misuse        Tool-based testing techniques are     could decide to train its developers
                 the component’s assets, violate inter-      appropriate and encouraged at the         and eliminate the use of potentially
                 component assumptions, or probe             system level, but for efficiency’s sake,   dangerous functions such as str-
                 risks.                                      such testing should be structured to      cpy() in favor of safer string-han-
                     Unit testing carries the benefit of      avoid repeating unit-level testing.       dling libraries.
                 breaking system security down into          Accordingly, they should focus on             A good last step is to use test result
                 several discrete parts. Theoretically, if   aspects of the system that couldn’t be    information to measure progress
                 each component is implemented               probed during unit testing.               against a goal. Where possible, tests
                 safely and fulfills intercomponent               If appropriate, system-level tests    for the mitigated vulnerability
                 design criteria, the greater system         should analyze the system in its de-      should be added to automated test
                 should be in reasonable shape (al-          ployed environment. Such analysis         suites. If the vulnerability resurfaces

                                                                                                                     Building Security In

in the code base at some point in the        7. R. Anderson, Security Engineering:
future, any measures taken to prevent           A Guide to Building Dependable Dis-
the vulnerability should be revisited           tributed Systems, John Wiley &
and improved. As time passes, itera-            Sons, 2001.
tive security penetration tests should
reveal fewer and less severe flaws in        Brad Arkin is a technical manager for
                                            Symantec Professional Services. His pri-
the system. If a penetration test re-       mary area of expertise is helping organi-
veals serious severity flaws, the “rep-      zations improve the security of their
resentative sample” view of the re-         applications. Arkin has a dual BS in com-
sults should give the development           puter science and mathematics from the
                                            College of William and Mary, an MS in
organization serious reservations           computer science from George Washing-
about deploying the system.                 ton University, and is an MBA candidate
                                            at Columbia University and London Busi-
                                            ness School. Contact him at barkin@

P     enetration testing is the most
      commonly applied mechanism
used to gauge software security, but
                                            Scott Stender is a partner with Informa-
                                            tion Security Partners. His research inter-
it’s also the most commonly misap-          ests are focused on software security, with
                                            an emphasis on software engineering
plied mechanism as well. By apply-          and security analysis methodology. Sten-
ing penetration testing at the unit         der has a BS in computer engineering
and system level, driving test cre-         from the University of Notre Dame. Con-
ation from risk analysis, and incor-        tact him at
porating the results back into an or-
                                            Gary McGraw is chief technology officer
ganization’s SDLC, an organization          of Cigital. His real-world experience is
can avoid many common pitfalls. As          grounded in years of consulting with
a measurement tool, penetration             major corporations and software pro-
                                            ducers. McGraw is the coauthor of
testing is most powerful when fully
                                            Exploiting Software (Addison-Wesley,
integrated into the development             2004), Building Secure Software (Addi-
process in such a way that findings          son-Wesley, 2001), Java Security (John
can help improve design, implemen-          Wiley & Sons, 1996), and four other
                                            books. He has a BA in philosophy from
tation, and deployment practices.
                                            the University of Virginia and a dual PhD
                                            in computer science and cognitive science
References                                  from Indiana University. Contact him at
 1. G. McGraw, “Software Security,”
    IEEE Security & Privacy, vol. 2, no.
    2, 2004, pp. 80–83.
 2. B. Chess and G. McGraw, “Static
    Analysis for Security,” IEEE Secu-
    rity & Privacy, vol. 2, no. 6, 2004,
    pp. 76–79.
 3. B.P. Miller et al., Fuzz Revisited: A
    Re-Examination of the Reliability of
    Unix Utilities and Services, tech.
    report CS-TR-95-1268, Depart-
    ment of Computer Science, Univ.
    Wisconsin, Apr. 1995.
 4. D. Verndon and G. McGraw,
    “Software Risk Analysis,” IEEE
    Security & Privacy, vol. 2, no. 5,
    2004, pp. 81–85.
 5. F. Swidersky and W. Snyder, Threat
    Modeling, Microsoft Press, 2004.
 6. G. Hoglund and G. McGraw,
    Exploiting Software, Addison-Wes-
    ley, 2004.

                                                                           ■ IEEE SECURITY & PRIVACY    35

To top