wordwide infrastructure security report 07

( VO L U M E I I I ) WORLDWIDE INFRASTRUCTURE SECURITY REPORT SEPTEMBER 2007 Worldwide Infrastructure Security Report, Volume III Table of Contents OVERVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 KEY FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 DEMOGRAPHICS OF SURVEY RESPONDENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 SURVEY METHODOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 MOST SIGNIFICANT OPERATIONAL THREATS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 ATTACK VECTORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 SCALE AND EFFECTIVENESS OF ATTACKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 FREQUENCY OF ATTACKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 ATTACK TARGETS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 ATTACK DETECTION AND TRACEBACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 ATTACK MITIGATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Access Control Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 BGP Real-Time Blackhole Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Scrubbing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 DDoS Managed Security Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 LAW ENFORCEMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 INFRASTRUCTURE PROTECTION TECHNIQUES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Application and Anti-Spoofing Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Protection of Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Infrastructure Address Hiding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Occurrence of Internal Compromise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Handling Vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Infrastructure Shortcomings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 COMPOSITION OF SECURITY TEAMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 EMERGING THREATS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 ISPs: BOTS & BOTNETS, AV & MALWARE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 ABOUT THE AUTHORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1 Worldwide Infrastructure Security Report, Volume III List of Figures Figure 1: Sustained Attack Size – Gbps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Figure 2: 2007 Respondent Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Figure 3: Most Concerning Threat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Figure 4: Attack Vectors – Largest Bits-Per-Second Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Figure 5: Attack Vectors – Largest Packets-Per-Second Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 Figure 6: Largest Attacks Observed – Past 12 Months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Figure 7: Customer-Impacting Attacks – Per Month . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Figure 8: Infrastructure-Impacting Attacks – Per Month . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Figure 9: Attack Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Figure 10: Frequently Attacked Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Figure 11: Attack Detection Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Figure 12: Flow Types Utilized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Figure 13: Flow Use by Respondents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Figure 14: Flow Sampling Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Figure 15: Flow Deployment Inhibitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Figure 16: Attack Mitigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Figure 17: DDoS Managed Security Service Offerings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Figure 18: DDoS Managed Security Service SLAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Figure 19: Number of Law Enforcement Referrals – Past 12 Months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Figure 20: Application of Anti-Spoofing Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Figure 21: Filter Traffic From Customers/Broadband . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Figure 22: Routing Protocol Transport Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Figure 23: Infrastructure-Related Vendor Vulnerability Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 24: Dedicated Security Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 25: Personal Protection From Internet Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Figure 26: Largest Observed Botnet – Past 12 Months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Figure 27: Is Botnet Command and Control Server Cleanup an Action the Service Provider Should Take? . . . . . . . . . . . . . . . . . . . . . 28 Figure 28: Is Infected Host Cleanup an Action the Service Provider Should Take? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2 Worldwide Infrastructure Security Report, Volume III Overview Arbor Networks®, Inc., in cooperation with the Internet security operations community, has completed the third edition of an ongoing series of annual operational security surveys. This survey, covering a 12-month period from July 2006 through June 2007, is designed to provide data useful to network operators so that they can make informed decisions about their use of network security technology to protect their mission-critical infrastructures. It is also meant to serve as a general resource for the Internet operations and engineering community, recording information on trends and employment of various infrastructure security techniques. Operational network security issues—the day-to-day aspects of security in commercial networks—are the primary focus of survey respondents. As such, the results provided in this survey more accurately represent real-world concerns than theoretical and emerging attack vectors often speculated about elsewhere. Key Findings Fear the Zombies: ISPs now rank the millions of zombie computers across the Internet as the single largest threat facing network services availability and operational security. Zombie computers, also known as bots, are assembled to constitute bot networks (botnets) that are controlled by one or more individuals or organizations, and used for an array of nefarious activities. ISPs reported observing an increase in the types of activities for which botnets are employed, from spam relay, to supporting entire phishing systems, to click fraud and launching distributed denial of service (DDoS) attacks. DDoS Attacks Go Pro: While mid-level DDoS attacks have plagued the Internet since 2000, survey respondents report a widening gap between common mid-level “amateur” attacks and multi-gigabit “professional” efforts involving tens of thousands of zombie hosts. Most surveyed ISPs reported significant improvements in the sophistication and coordination of zombie army attacks. Attacks Outpace ISP Network Growth: During the last two years, most Tier 1 and Tier 2 providers completed significant investments in backbone infrastructure—upgrading links from OC-12/48 (2 Gbps) to OC-192 (10 Gbps). However, surveyed ISPs reported sustained attack rates exceeding 24 Gbps—more than double the size of these recently upgraded links. Figure 1 shows the largest sustained attack rate (Gbps) reported each year since 2001. Providers Can Help: This year’s survey found a significant increase in the number of providers offering managed DDoS detection and mitigation services. Over one-third of surveyed providers reported currently available DDoS managed security service offerings; another one-third indicated they plan to roll out such services in the next 24 months. Sustained Attack Size — Gbps 2001 30 25 20 2002 2003 2004 2005 2006 24 17 Gbps 15 10 5 10 2.5 0.4 0 1.2 Figure 1: Sustained Attack Size — Gbps Source: Arbor Networks, Inc. 3 Worldwide Infrastructure Security Report, Volume III Demographics of Survey Respondents Survey participants included 70 self-classified Tier 1, Tier 2 and other IP network operators from North America, South America, Europe and Asia. All survey participants are directly involved in network security operations at their respective organizations. This is a 27 percent increase in respondent pool size compared with the 2006 edition of the survey. Survey respondents included a majority of Tier 1, Tier 2, large content and hosting providers, as well as a cross-section of academic and other network types, including large broadband consumer access operators. In addition, a large number of “hybrid” network operators responded to the survey. Hybrid networks represent large-scale, globally distributed enterprise networks that have multiple Internet access interconnections and provide their respective organizations with traditional end-user services. From the respondent pool, 13 percent selected Other as their organization type, with free-form input indicating small ISP or hosting provider, voice over IP (VoIP) services provider, wireless ISP or Internet Exchange Point (IXP) providers. Survey Methodology This edition of the survey consisted of 87 free-form and multiple-choice questions, representing an array of issues facing network security operators today. Questions addressed such topics as threats against backbone infrastructure and individual customers, techniques employed to protect network infrastructure itself and mechanisms used to detect and respond to security incidents. All data represented here is provided in an aggregate and anonymous manner, and is provided with the permission of the respondents. Individual respondents were typically senior network security architects or operations engineers at their respective organizations. Standard mathematical methods to weight responses have been applied where incomplete answers were provided for a given question. 2007 Respondent Distribution Tier 1 Academic 35% 30% 25% 20% 15% 10% 5% 0% Tier 2 Large Content Enterprise/Hybrid Other Large Hosting Figure 2: 2007 Respondent Distribution Source: Arbor Networks, Inc. Arbor Networks intends to continue conducting this survey annually and sharing all results with the global Internet security and operations communities. Our goals are: 1) to continually refine the questionnaire in order to provide more detailed and relevant information in future editions; and 2) to increase the scope of the survey respondent pool to provide greater representation of the global Internet network operations community. 4 Worldwide Infrastructure Security Report, Volume III Most Significant Operational Threats Unlike the previous two editions of the survey, bots and botnets took the top spot as the most significant operational threat, with DDoS attacks coming in a close second. Bots and botnets were not an option in the first edition of the survey, however, they were added in the 2006 edition due to the obvious growing concerns among the network operations community. Since a majority of today’s DDoS attacks are believed to be perpetrated by botnets, concerns about botnets largely imply concerns about DDoS attacks as well. Nonetheless, it is interesting that this year a much larger percentage of the respondent pool believed bots and botnets to be a larger threat than DDoS itself, perhaps providing some indication that botnet activity, beyond just that of DDoS, is more frequently impacting network security operations. Most Concerning Threat BGP/Route Hijacking DNS Poisoning ID Theft 35% 30% 25% 20% 15% 10% 5% 0% DDoS Compromise Infrastructure (e.g., root servers) Worms Bots/Botnets Primary Figure 3: Most Concerning Threat Source: Arbor Networks, Inc. Secondary DDoS attacks are again featured as a rather prominent concern of network operators, some seven years after the initial flurry of attacks that garnered a large amount of news and research attention. Sixty-four percent of respondents reported DDoS as a top concern in the 2005 edition of the survey, compared to 46 percent last year and 24 percent this year. Two of the primary reasons for this decrease are, as discussed above, the introduction of the superseding “Bots/Botnets” category, and the increase in availability, deployment and maturity of infrastructure security and attack mitigation techniques, as discussed later in this report. Respondents were asked to order the list of threats from most concerning to least concerning, with the primary and secondary threats as illustrated in Figure 3. Of the eight threats provided in the pool this year (versus six last year), DNS poisoning-related threats were considered the least concerning, while in the previous edition of the survey, BGP/Route Hijacking and related threats garnered the least amount of concern. The following percentages represent respondent concern with each of the associated threats: 29% – Bots/Botnets. 24% – DDoS. 19% – Infrastructure (e.g., root servers). 11% – ID Theft. 9% – Worms. 3% – Compromise. 1% – BGP/Route Hijacking. 1% – DNS Poisoning. 5 Worldwide Infrastructure Security Report, Volume III Concerns about security and availability of Internet infrastructure, and in particular, the root and top-level domain (TLD) name servers, were duly noted. With regard to the worm threat, as previously noted, the chief concerns are related to the potential worm payload and propagation vectors, which lead to network congestion and its impact on network availability and performance, rather than the end-system infection itself. Infected host detection and cleanup by ISPs is discussed in later sections of this report. Attack Vectors In this version of the survey, we split questions regarding the largest observed attacks into two categories: largest bits-per-second (bps) attacks and largest packets-per-second (pps) attacks. This was done because many of the respondents in the previous editions of the survey indicated that their largest bps attacks were not necessarily their largest pps attacks. For example, TCP SYN flooding attacks are usually quite small from a packet-size perspective, yet a large number of pps may be generated with a typical goal of exhausting the target TCP connection state. Whereas, with attacks aimed at exhausting the bandwidth resources of a given target and perhaps upstream interconnections as well, larger packets in the form of UDP or ICMP protocols might be employed, demanding lower pps frequencies. The primary DDoS attack vectors representing the largest sustained bits-per-second attacks over the past 12 months are illustrated in Figure 4. In the previous edition of the survey, TCP SYN floods were the most prominent with 27 percent of the responses, followed closely by UDP floods at 26 percent, and application-layer attacks (included in the Other field) at 18 percent. Attack Vectors — Largest Bits-Per-Second Attacks 43% UDP Flood Application-Layer 7% 5% 19% 4% 18% Figure 4: Attack Vectors — Largest Bits-Per-Second Attacks Source: Arbor Networks, Inc. TCP SYN Reflective/Amplification ICMP Flood IP FRAG Other 4% In this year’s survey, 43 percent of respondents reported that the largest bps attacks they observed over the past 12 months employed UDP flooding. Application-layer attacks, which were given their own category for the first time this year, came in second at 19 percent, just ahead of TCP SYN attacks. Seventeen percent of the respondents indicated that the largest bps attacks they observed were 2 million packets per second (Mpps) or larger. Some additional information on the scale of these bps attacks is provided in the next section. 6 Worldwide Infrastructure Security Report, Volume III Attack Vectors — Largest Packets-Per-Second Attacks 26% UDP Flood Application-Layer TCP SYN 6% 41% 2% 17% Figure 5: Attack Vectors — Largest Packets-Per-Second Attacks Source: Arbor Networks, Inc. Reflective/Amplification ICMP Flood 4% 4% IP FRAG Other The largest reported pps attack data does not look that different from the largest reported bps data, with the obvious and intuitive exception of TCP SYN attacks moving up into the second spot at 26 percent. It is interesting to note that application-layer attacks were represented in the high teens with regard to percentage of respondents in the pps category. It will be interesting to see how this changes in the coming years as attack sophistication evolves. One conclusion that can be drawn from this data is that brute-force (e.g., host state and resource consumption or bandwidth exhaustion) attacks are still the most prominent of the large attacks. Scale and Effectiveness of Attacks Thirty-six percent of the respondents reported observing sustained attacks of 1 Gbps or larger over the past 12 months. This percentage is less than that reported in previous versions of the survey due to the fact that the latest survey had a larger respondent pool representing smaller network types. The largest reported attack in this edition of the survey exceeded 24 Gbps. Only two respondents reported attacks with sustained bits-per-second (bps) ranges greater than 20 Gbps; the other was 22 Gbps. The largest bps attack reported in the previous edition of the survey was 17 Gbps. The 24 Gbps attack was reported to be a DNS reflective amplification attack1 against a hosting services provider. Figure 1 illustrates growth in the largest sustained DoS attacks over the past six years. Most individual core Internet backbone links today are no larger than 10 Gbps, with standardization efforts and some products coming to market for 40 and 100 Gbps links. As such, most of the larger attacks today still easily inflict collateral damage on infrastructure upstream from targets themselves. 1 http://www.us-cert.gov/reading_room/DNS-recursion121605.pdf 7 Worldwide Infrastructure Security Report, Volume III Largest Attacks Observed — Past 12 Months 20+ Gbps 500 Mbps-1 Gbps 25% 20% 15% 10% 5% 0% 10-20 Gbps 100-500 Mbps 4-10 Gbps <100 1-4 Gbps No Answer Attack Size – Bits-Per-Second Figure 6: Largest Attacks Observed — Past 12 Months Source: Arbor Networks, Inc. Reflective amplification attacks exploit IP address spoofing (i.e., the reflective part) and protocol query/response behaviors by spoofing queries with a given target as the source address. This elicits much larger responses (i.e., the amplification part) from third-party systems (e.g., DNS resolvers), which respond to the target with the large payloads. Attacks of this nature employed against Internet root and TLD name servers in early 2006 achieved a 1:76 amplification factor. With such levels of amplification, a small number of well-connected hosts are capable of generating large amounts of attack traffic, easily overwhelming most organizations connected to the Internet today. We discuss some of the anti-spoofing techniques that operators can implement later in the Infrastructure Protection Techniques section (page 20), and survey where and why anti-spoofing mechanisms, which would partly mitigate this attack type, are not universally deployed today. Survey respondents reported a wide-range of packets-per-second attack rates, as shown below. 27% – Less than 500 Kpps. 19% – 1 to 2 Mpps. 17% – 500 Kpps to 1 Mpps. 11% – 8 Mpps or greater. 4% – 2 to 4 Mpps. 3% – 4 to 8 Mpps. The aggregate of the attacks on Estonian targets during April and May of 2007 were reported to be 4 Mpps sustained at peak. About 15 percent of our respondents saw attacks this size or larger over the past 12 months. 2 The last survey reported the largest pps attack as a UDP flood at an aggregate sustained rate of 22 million packets per second. As provided earlier in the Key Findings section (page 3), the scale of attacks has continued to increase at a considerable rate over the past six years. Yet most organizations and end sites are still connected to the Internet at speeds of 1 Gbps or less. As a result, attacks frequently overwhelm Internet connection throughput to enterprises. In order to effectively mitigate an attack, the network operator must take some action on the network side of the connection. In the Attack Mitigation section (page 15), we discuss what these mitigation actions typically are and briefly discuss the managed security services (MSS) offerings that many services providers are rolling out to address this growing threat. Given the growing number of bots, the virulence of today’s attack vectors and malware, and the rate of broadband penetration globally, it is clear that attack scale and effectiveness will continue to evolve. 2 Unfortunately, when adding the new question group for packet-per-second attacks, we neglected to add a free-form input field to include the single largest observed packet-per-second attacks over the past 12 months—just what the ranges were. As a result, this year’s survey does not provide insight as to the single largest packet-per-second attack. 8 Worldwide Infrastructure Security Report, Volume III Frequency of Attacks Actionable attack frequency remained somewhat constant on aggregate this year. However, the distribution of attack frequency was somewhat wider yet again, an effect we anticipated as we continue expanding the respondent pool. Figure 7 indicates the number of attacks per month that impact customers and network infrastructure. Customer-Impacting Attacks — Per Month 2005 2006 Current 70% 60% 50% 40% 30% 20% 10% 0% 500+ 100-500 10-100 <10 Survey Respondents NA/Other Number of Attacks Figure 7: Customer-Impacting Attacks — Per Month Source: Arbor Networks, Inc. Infrastructure-Impacting Attacks — Per Month 2005 80% 2006 Current Survey Respondents 70% 60% 50% 40% 30% 20% 10% 0% 500+ 100-500 10-100 <10 NA/Other Number of Attacks Figure 8: Infrastructure-Impacting Attacks — Per Month Source: Arbor Networks, Inc. We do expect the number of customer-impacting attacks to increase in the future, as more providers begin to offer DDoS managed mitigation services. Prior to offering a DDoS managed security service, many providers do not actively look for and detect customer-impacting attacks unless the customer calls in to report the issue. Also, as with previous versions of the survey, these numbers are somewhat skewed by responses from network operators with large numbers of lower-speed subscribers. 9 Worldwide Infrastructure Security Report, Volume III Our survey defined actionable attacks as those that require some response from the network operator, either on behalf of the customer (as a result of their request or a service offering) or in the event of collateral damage to the infrastructure. This distinguishes these attacks from attacks that may go undetected, or attacks that are detected but just considered “tolerable noise” on the network. Of the attacks that target infrastructure, the most common attack vectors were: 47% – TCP SYN. 29% – UDP Flood. 24% – ICMP Flood. 14% – Other. 11% – IP Fragmentation. 9% – TCP RST. Beyond the Other category, the order of these attack types remained the same as in the previous editions of the survey. The Other responses included services scans (e.g., SNMP, SSH), password brute forcing and application-layer attacks. Respondents again indicated that the majority of these types of attacks, which impact customers or their infrastructure and require some action on their part, appear to have been performed by botnets. Attack Targets When respondents were asked what the primary targets of actionable attacks were, they responded as Figure 9 illustrates. Respondents were allowed to select multiple answers. Specific customer services continue as the top target, with network services stepping up to the second target of choice. Free-form Other responses included “multi-mode attacking everything,” “spread out IP space” and “worm spread/scans to various existent and non-existent addresses.” Attack Targets Core Routers/Backbone Interfaces Network Services (e.g., DNS, NTP) 70% 60% 50% 40% 30% 20% 10% 0% Customer Aggregation Links Internet Infrastructure (e.g., root servers) Specific Customer Services Other Figure 9: Attack Targets Source: Arbor Networks, Inc. 10 Worldwide Infrastructure Security Report, Volume III In addition to what is being attacked, we asked network operators which types of customers or sites seem to “attract” attacks the most frequently. Figure 10 illustrates the distribution of responses to this question. Again, respondents were permitted to select multiple responses within this section of questions. Frequently Attacked Targets Religious Commercial 40% 35% 30% 25% 20% 15% 10% 5% 0% Political Banking Pornography Other Gambling Figure 10: Frequently Attacked Targets Source: Arbor Networks, Inc. Many of the more prominent attacks were driven by extortion, particularly those targeting gambling sites. With online gaming projected to be a $48 billion market in the U.S. by 2010, and with single events such as the FIFA World Cup or the NFL Super Bowl generating as much as $2 billion in U.S. online gaming transactions, most ISPs expect extortion attempts to become more common. DDoS motives extend beyond extortion according to surveyed ISPs. For example, many gambling sites have various rules surrounding what happens if a given player loses connectivity while a game is active, or if the site becomes unavailable. If a player loses his or her connection to a site while a hand is at play, the entire hand might be voided for all players. So if you are dealt off-suited two and three hole-cards and the ante is rather large, you might attempt to exploit your gaming site’s policies by breaking some player’s connectivity to the site, or attempting to take the site itself offline. Other examples might include religious or politically motivated attacks, such as those on Estonia after the removal of a Soviet World War II memorial, or attacks on various Denmark sites triggered by “Cartoon Rage.” The Other responses included: - Hosting services, customers and broadband access customers. - Social networking, Web forum and community sites. - Toll-fraud attacks. - Domain-hosting companies. - Gaming sites and gamers. - Anti-spam and anti-bot vendors and services. Many of the commercial, banking and other attack types are driven by a number of factors, including extortion, competitiveness and other motivations. 11 Worldwide Infrastructure Security Report, Volume III Attack Detection and Traceback Respondents were asked to identify the tools and techniques they employ for attack detection and traceback. Figure 11 illustrates the distribution of responses and compares it to responses in the previous edition of the survey. Attack Detection Techniques 2006 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Commercial Customer Calls In House/ SNMP Open Source NA/Other Current Figure 11: Attack Detection Techniques Source: Arbor Networks, Inc. A notable increase in open source tools that can help to enable attack detection was observed, perhaps in part because of the larger number of smaller respondents, as well as the wider availability of more open source tools that enable attack detection. Commercial tools (which include the Arbor Peakflow® platforms) did experience an increase within responses from Tier 1 and Tier 2 operators, although, as expected with a larger pool of respondents, their deployment appeared to drop on aggregate. The percentage of network operators that rely on customers calling is slightly greater as well, largely for the same reason. Customer calls hold the number two spot among secondary attack detection mechanisms. Well over 80 percent of the respondents also indicated that they do have the capability to trace attacks back to ingress network interface(s). Most of the open source and commercial tools that provide attack detection and traceback are enabled by flow-based protocols such as Cisco’s NetFlow or the emerging IETF IP Flow Information Export (IPFIX) standards. Typically, flow-based protocols provide summarized network and transport layer information regarding microflows transiting a given router or switch. As with the previous edition of the survey, respondents were asked which of these protocols they used, with responses being more indicative of network equipment vendors than anything else. The types of flow information utilized by respondents are listed in Figure 12 (note that respondents were allowed to select multiple responses). Only 11 percent of the respondents reported using no flow data at all. Various versions of NetFlow, primarily supported by Cisco gear, were clearly the most prominent. IPFIX—a newer flow information protocol designed by the IETF, which found its roots in NetFlow v9’s extensibility—is just completing the standardization process, so deployed hardware support is not widely available. 12 Worldwide Infrastructure Security Report, Volume III Flow Types Utilized NetFlow v9 sFlow v4 IPFIX 80% 70% 60% 50% 40% 30% 20% 10% 0% NetFlow v5 sFlow v2 No Flow Used sFlow v5 JFlow/cflowd NetFlow Other Figure 12: Flow Types Utilized Source: Arbor Networks, Inc. Respondents were also asked to identify their various uses of flow data and to select all categories that apply. Figure 13 details those responses. Security and traffic/peering analyses are the traditional and most prominent uses of flow data. However, compliance, auditing, billing and accounting are all gaining in popularity as well. The Other responses included troubleshooting and research. Flow Use by Respondents Accounting Peering/Traffic Analysis 100% 80% 60% 40% 20% 0% Billing Compliance Auditing Other Security Figure 13: Flow Use by Respondents Source: Arbor Networks, Inc. At lower speeds with processor-based routers, all packets were traditionally counted in flow when generating flow information. However, with the increase in speeds, complexity of accounting functions and other factors, “raw” or 1:1 sampling is not supported in many routers and switches today. If it is supported, it is often only at very low packet-per-second switching rates. As a result, many deployments today sample packets that are subjected to flow data collection. Figure 14 provides information on common sampling frequencies among respondents utilizing flow-based protocols. 13 Worldwide Infrastructure Security Report, Volume III For the most part, flow sampling at the customer edge seems to be mostly at the same frequency as the peering/inter-provider edge of the network. This edition of the questionnaire did not afford respondents the option to select 1:10000 sampling frequencies on the customer edge. We suspect this is in large part the cause of the variance between the 1:10000 and Other/NA responses between Customer Edge and Peering Edge. Flow Sampling Rates Customer Edge 30% 25% 20% 15% 10% 5% 0% s1:1 s1:10 s1:100 s1:500 s1:1000 s1:10000 NA/Other Peering Edge Figure 14: Flow Sampling Rates Source: Arbor Networks, Inc. Finally, because of the variance in vendor and hardware support for flow, we asked respondents what their primary concerns and inhibitors of flow deployment are. The distribution of responses is illustrated in Figure 15. Flow Deployment Inhibitors Lack of Vendor Support Security of Flow Information Lack of Full Support (e.g., TCP Flags) 35% 30% 25% 20% 15% 10% 5% 0% Expense of Dedicated Hardware (e.g., AS PIC) Lack of Utility Other Concern of Backhauling Flow Legacy Hardware Figure 15: Flow Deployment Inhibitors Source: Arbor Networks, Inc. 14 Worldwide Infrastructure Security Report, Volume III There is a rather wide array of responses regarding what inhibits flow deployment. Most of these are surrounding challenges with legacy hardware and vendor support, or requirements for dedicated hardware to enable flow data collection. The Other responses included: - Lack of personnel/resources to implement. - Taxation of resources/ports. - Bugs, historical problems/experience. - Missing features (e.g., multicast/IPv6). - Impact to performance. - Utilization by staff. The nearly 10 percent of respondents who cited lack of utility corresponds quite closely to the “No Flow Used” responses above. Attack Mitigation The primary and secondary mitigation responses remained the same as the previous survey. Access control lists (ACLs), also commonly referred to as packet filters, kept the top spot. BGP destination-based real-time blackhole (RTBH) routing maintained its place in the second spot. Scrubbers (e.g., the Arbor Peakflow SP Threat Management System) inched up slightly as well. Attack Mitigation ACLs Scrubber 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% BGP Destination-based RTBH Other BGP Source-based RTBH No Answer Primary Figure 16: Attack Mitigation Source: Arbor Networks, Inc. Secondary The unfortunate downside of this constancy is that ACLs and BGP destination-based RTBH routing are techniques that in most cases result in the target IP address being taken offline, thereby effectively completing the attack. Most often this is the case when the attack is IP-based, or when the scale of the attack is resulting in collateral damage to the infrastructure and all the traffic towards the destination needs to be discarded at ingress in order to minimize collateral damage that would otherwise affect the network or other customers. 15 Worldwide Infrastructure Security Report, Volume III As previously provided, the primary reasons for destination-based mitigation techniques, rather than source-based, include: - Lack of infrastructure capabilities for enabling large numbers of packet filters in the router’s forwarding path (either lack of support for the sheer number of entries required to mitigate attacking sources, or intolerable degradation in the performance of the router’s forwarding capabilities). - Inability to verify the authenticity of attacking source IP addresses (e.g., filtering based on sources when spoofing is employed may resulting in or enable other DoS vectors). The following subsections will provide a brief explanation of the various mitigation options employed above. Access Control Lists Access control lists (ACLs), also commonly referred to as packet filters, normally operate at the network (Layer 3) and transport (Layer 4) layers. This enables traffic discrimination and action (e.g., discard or rate-limiting) based on any combination of a packet’s header characteristics, such as: - Source and/or destination IP address. - Protocol types (e.g., TCP, UDP, ICMP). - Source and destination port numbers (many well-known, primarily on the server side, such as TCP port 80 is normally HTTP). - ICMP type and codes, etc. The disadvantage of destination-based ACLs is that they effectively complete the attack for the service in question. For example, if miscreants launch an attack on a Web server, they would normally direct attack traffic to server_ip_address:tcp:80. In order to mitigate the attack, a network operator using ACLs would define something like the following (this is a sample Cisco IOS ACL that defines a rule for dropping all TCP traffic destined for port 80 on host 1.1.1.1): access-list 101 deny tcp any host 1.1.1.1 80 access-list 101 permit ip any any The network operator would then apply the ACL to the interface(s) from which the attack traffic ingresses the network—or egresses the network bound for the customer, depending on the traceback capabilities and magnitude of the attack. The result is that all traffic destined to the Web server’s HTTP port will be discarded, thereby effectively completing the attack. There are several reasons why destination-based ACLs are a primary mitigation technique—even while they negatively impact the victim’s services. For additional information, see Effectively Completing the Attack posted on the Arbor blog.3 BGP Real-Time Blackhole Routing BGP destination-based real-time blackhole (RTBH) routing accomplishes essentially the same thing as destination-based ACLs, except that you lose some of the more granular definition capabilities that are available with ACLs (e.g., being able to filter on transport layer protocol types or port numbers). Instead, RTBH routing employs the BGP routing protocol to announce a BGP route set to all or some subset of routers in the local routing domain. This BGP route set instructs the routers to drop all traffic for a destination host or set of IP addresses. While you lose some of the granularity that ACLs provide, BGP RTBH routing allows you to employ BGP as a signaling protocol and manage the policy application from a single point in the network. On the other hand, ACLs require per-device and per-interface configuration, and often times require the network operator to augment or specify existing ACLs and processing sequences for each interface. As a result, BGP is much more efficient in distributing the policy set to the routers in the network. Destination-based blackhole routing also allows you to exploit the routers’ native forwarding capabilities, and typically results in little or no forwarding performance degradation, such as what might be incurred with ACLs. Visit the referenced Web sites for additional information and references on the configuration and benefits of BGP RTBH routing.4 3 4 http://asert.arbornetworks.com/2006/04/effectively-completing-the-attack-and-this-posting/ http://www.ietf.org/rfc/rfc3882.txt and http://www.nanog.org/mtg-0402/morrow.html 16 Worldwide Infrastructure Security Report, Volume III Another attack mitigation technique employed by network operators surveyed includes source-based BGP RTBH routing. Source-based BGP RTBH routing techniques with unicast reverse path forwarding (uRPF)5 in order to use BGP to disseminate source-based packet drop policy sets. One additional mitigation technique based on BGP is described in BGP flow specification.6 BGP flow specification, or “flow spec” for short, exploits BGP signaling capabilities and extends the protocol to enable the dissemination of information beyond just IP addresses to include network and transport layer information (e.g., protocol types, port numbers, source and destination information, etc.) and other traffic attributes. From there, a network operator can take actions such as discarding traffic; setting quality of service (QoS) precedence-based, rate-limiting traffic; or even diverting traffic based on the match rules. When it comes to mitigating DDoS attacks, many providers are beginning to replace ACLs with BGP flow spec, as well as optimizing scrubbing and other services. An example of a BGP flow spec deployment is provided in the North American Network Operators’ Group (NANOG) presentation.7 Scrubbing Scrubbing typically involves diverting or “off-ramping” traffic to devices that “scrub” the traffic for a given set of Internet destinations. These devices only allow legitimate traffic to be forwarded on through the network towards the target. Scrubbing usually requires specialized network devices that perform the scrubbing function. It is quickly becoming a more common technique for effectively dealing with DDoS attacks. Scrubbing is possible as many attacks tend to exhibit discernible characteristics that allow discrimination between malicious and legitimate traffic flows, although these characteristics are only discernible if dedicated traffic analysis and filtering hardware is deployed. Off-the-shelf routers and switches rarely provide these capabilities, and if they do, it is usually at a considerable premium. Scrubbers also often employ functions that enable spoofed source detection, such as TCP SYN proxy techniques. Unfortunately, scrubbing typically results in some change to the data path in the network (likely increasing packet propagation delay on a given path) and requires dedicated capital investments by the network operators. In addition, diverting traffic to the scrubbing devices and then “on-ramping” legitimate traffic once it has been scrubbed is often a complex exercise. The percentage of respondents who suggested that they are deploying scrubbing devices in the first two editions of the survey was about 7 percent, a number which increased to only 9 percent in this version of the survey. The numbers that have deployed scrubbing infrastructure and are planning to deploy scrubbing infrastructure for managed security services (MSS) increased quite considerably over the past year. We believe this increase is largely attributable to the ability of network service providers to offset scrubber capital investment costs and even generate new top-line revenue opportunities with DDoS managed mitigation services. We will dive deeper into MSS in the next subsection. However, it is worth calling attention here to the fact that while most Tier 1 and Tier 2 providers surveyed are either in process or have already deployed DDoS MSS, their primary DDoS attack mitigation techniques for DDoS non-managed security services customers have remained relatively fixed over the past three years. This data seems to suggest that unless you are an MSS customer, it is unlikely that you will be able to benefit from your provider’s investment in this infrastructure. http://www.ietf.org/rfc/rfc3704.txt and http://asert.arbornetworks.com/2006/05/gone-arent-the-days-of-spoofing/ http://tools.ietf.org/id/draft-marques-idr-flow-spec-04.txt 7 http://www.nanog.org/mtg-0610/lozano.html 5 6 17 Worldwide Infrastructure Security Report, Volume III DDoS Managed Security Services As discussed in the previous section, opportunities involving managed security services (MSS) are causing many network service providers to invest a great deal in scrubbing infrastructure. With more mission-critical services being converged onto IP-based networks, and more revenue being tied to customer network availability, a DDoS MSS market has been born—purely out of necessity. Many organizations generate a majority, and often times all of their revenue, from Web or other network service transactions. Any disruption of network service has a direct impact on the financial performance and stability of these organizations. As a result, many enterprises have demanded that their service providers offer “clean pipes” services. These enterprises now consider a subscription to such services as an everyday cost of doing business on the Internet. Figure 17 illustrates the number of respondents who currently offer DDoS detection and/or mitigation services to customers. DDoS Managed Security Service Offerings Detection & Mitigation Mitigation (only) 30% 25% 20% 15% 10% 5% 0% Will in Future Other Detection (only) No Answer Figure 17: DDoS Managed Security Service Offerings Source: Arbor Networks, Inc. Well over the majority of Tier 1 and Tier 2 respondents fell into the “currently offering” or “plan to offer” categories above. The Other category largely included hybrid enterprises that deployed their own scrubbing infrastructure and/or subscribed to MSS from their provider(s). We also asked respondents who currently offer or plan to offer DDoS MSS if they were offering service level agreements (SLAs) associated with those services. Figure 18 indicates those responses. This data indicates that a considerable investment in DDoS MSS infrastructure has occurred over the past 12 to 18 months. With the scale of attacks today, far less than 1 percent of the largest end sites on the Internet could even consider throwing bandwidth and other resources at the DDoS problem. Therefore, network services subscribers must work with their providers to obtain effective DDoS mitigation solutions that do not jeopardize business continuity. Another upside for the service providers offering these services is that the same scrubbing infrastructure that protects customers can be used to protect their own infrastructure and network elements, with revenue-generating services offsetting the required capital investments. 18 Worldwide Infrastructure Security Report, Volume III DDoS Managed Security Services SLAs Detection & Mitigation Mitigation (only) 60% 50% 40% 30% 20% 10% 0% Will in Future None Currently Detection (only) No Answer Figure 18: DDoS Managed Security Service SLAs Source: Arbor Networks, Inc. However, even if mitigating attacks with large-scale dedicated scrubbing infrastructures, many respondents indicated that inter-provider cooperation and upstream mitigation of attack flows are absolute necessities. Otherwise, scrubbing capacity will be quickly overwhelmed. The Arbor Fingerprint Sharing Alliance 8 is one example of solutions that automate the inter-provider communication problem. Law Enforcement As with the previous edition of the survey, very few respondents indicated that they referred attacks to law enforcement organizations during the survey period. Number of Law Enforcement Referrals — Past 12 Months 10-50 60% 50% 40% 30% 20% 10% 0% Less Than 10 None No Answer Figure 19: Number of Law Enforcement Referrals — Past 12 Months Source: Arbor Networks, Inc. 8 http://www.arbornetworks.com/en/fingerprint-sharing-alliance.html 19 Worldwide Infrastructure Security Report, Volume III Reasons for this lack of referrals varied. Some of the reasons cited include: - Customer privacy/request. - Lack of forensic detail. - Too many attacks to bother with. - Belief in utility of reporting. However, the primary reason cited for not referring attacks to legal authorities was this: because the attacks were targeted at customer end sites, the network operators and service providers felt it was the end site’s responsibility to report the attacks to law enforcement, not the service provider’s. The majority of the feedback provided suggested that network operators were more than happy to provide whatever information they had about attacks to law enforcement—if requested by law enforcement organizations on behalf of the victim. When survey respondents were asked if they believe law enforcement had the power and/or means to act upon information provided by network operators, the responses varied, with 26 percent responding Yes, 44 percent responding No and 16 percent responding Other/Depends. Other feedback collected in this section of the survey was generally positive regarding the strides that law enforcement has been recently making on this front. The FBI’s Operation: Bot Roast 9 is an example of some of the activity in the space over the past year. Infrastructure Protection Techniques Again in this edition of the survey, we asked questions regarding several well-known infrastructure security techniques. In particular, we asked whether network operators might use some techniques to lessen the probability of source IP address spoofing on the Internet, whether they see a need to harden the transport connections associated with their routing protocols, and, if they have been compromised at any point in the past, what vulnerability types led to the incident. Application of Anti-Spoofing Techniques IETF BCP 38/RFC 282710 provides an overview on anti-spoofing measures that should be employed by network operators. Essentially, it recommends that a network operator should not accept packets into the network on a given interface for source address s unless destination address s is considered reachable through that interface. This policy is traditionally implemented by provisioning ingress ACLs on each interface, statically defining which destinations are considered reachable through the interface, and therefore, which source address should be permitted to ingress the network through that interface. Unicast RPF, as referenced in the Attack Mitigation section (page 15), provides a mechanism to automate guidelines provided in BCP 38. There are multiple modes in which uRPF may be implemented to provide for anti-spoofing measures, depending on the topology of the network and network equipment in use. Note that some of these modes permit spoofing within address spaces known within the local routing systems, and are therefore less effective than more explicit BCP 38 and “strict-mode” uRPF style policies. Furthermore, some types of uRPF can create a false sense of protection because, even when implemented, they can still enable attacks such as the reflective amplification attacks described earlier in this report. BCP 84/RFC 3704 11 provides some additional information on these techniques. http://asert.arbornetworks.com/2007/06/who-ya-gonna-call/ ftp://ftp.rfc-editor.org/in-notes/rfc2827.txt 11 ftp://ftp.rfc-editor.org/in-notes/rfc3704.txt 9 10 20 Worldwide Infrastructure Security Report, Volume III Application of Anti-Spoofing Techniques Broadband/Dial-up Edge 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% BCP 38 uRPF Loose uRPF Strict None Other Dedicated Customer Edge Peering Edge Figure 20: Application of Anti-Spoofing Techniques Source: Arbor Networks, Inc. The Other responses under Broadband/Dial-up Edge included: - Static MAC/ARP entries. - DHCP verify. - Depends on hardware support. - Cable source-verify dhcp. - Non-applicable/content-only network. The Other responses for Dedicated Customer Edge included ACLs for multi-homed, depends on hardware support and non-applicable/content-only network. The Other responses for the Peering Edge included explicit ACLs and bogons only. Because we asked a more general set of questions in the previous edition of the survey, it is not particularly useful to compare these responses to those of the earlier surveys. However, we can assure you that where only uRPF loose-mode or no anti-spoofing measures are applied, spoofing and reflective amplification attacks will continue to fester. Along the same lines, we asked survey respondents if they implement any other types of filtering TO or FROM broadband or customer networks. Figure 21 indicates filtering FROM customer networks. While we have some concerns about the application of anti-spoofing techniques discussed earlier in this section, we are actually quite impressed at the responses to this set of questions regarding the customer/broadband edge. We suspect that anti-spoofing at these edges may fill some of the void from the provider edge, as well as impact compromised host spam relay and other malicious activity vectors. 21 Worldwide Infrastructure Security Report, Volume III Filter Traffic From Customers/Broadband Filter Bogon IPs Filter or Rate-Limit ICMP 70% 60% 50% 40% 30% 20% 10% 0% Filtering SMTP Access To Filter Other Ports or Protocols Filter DNS to Internal Resolvers Other Figure 21: Filter Traffic From Customers/Broadband Source: Arbor Networks, Inc. We also asked respondents if they apply other types of filtering as default policy TO their broadband or customer networks (e.g., tcp/80, tcp/25, tcp/445, etc.). One example of why an organization would apply this type of policy is to restrict the services the customer or subscriber could offer from their home/office networks. One respondent indicated that they filter access to all ports less than 1024. Another respondent indicated that they statefully filter access to all customer ports on egress, enabling client-initiated transactions only. Nearly 12 percent of respondents indicated that they explicitly filter access to well-known ports such as tcp/25, tcp/80, tcp/443, etc. Finally, 16 percent of respondents indicated Other, with free-form entries that included: - Filter access to some “special” ports. - Special exploit-driven ACLs. - Port 25 return filters from authorized SMPT server only. - “We offer full IP access, we will not filter any port.” - For abusers, yes. Not for all customers. - Only by customer request. - 135, 137, 139, 445, 1433, 1434. - As a content/hosting provider, we manage custom-tiered ACLs for all of our customers. So it appears that there is quite an array of filtering and a full spectrum of techniques employed for special-purpose filtering—and it all appears to be largely pain-driven. Protection of Routing Protocols Network operators were also asked whether they used TCP’s MD5 signature option12 to protect BGP transport connections in the network, and whether they employ the MD5 protection mechanisms available with their interior gateway routing protocols (IGP) as well. Responses are illustrated in Figure 22. 12 http://www.ietf.org/rfc/rfc2385.txt 22 Worldwide Infrastructure Security Report, Volume III Quite a large number of network operators use MD5 mechanisms to protect external BGP sessions in particular, and well over half of the respondents also use it to protect internal BGP sessions and their IGPs. A small number of network operators also reported using IPSEC for this purpose. We also asked respondents what percentage of eBGP peers support the TCP MD5 signature option. Most of the respondents indicated that well over half of their peers supported it. Routing Protocol Transport Protection Yes 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% TCP MD5 iBGP TCP MD5 eBGP IGP MD5 IPSEC iBGP IPSEC eBGP No No Answer Figure 22: Routing Protocol Transport Protection Source: Arbor Networks, Inc. A great deal of research and engineering work is underway in the IETF and Internet network operations communities to further enable routing protocol protection, well beyond that of just transport connection protection discussed here. However, given the number of attacks that have targeted routing protocols in the past, and the degree of concern among operators regarding where routing protocols fall in the most significant threats list, the protection of routing protocols does not appear to be a chief concern of network operators today—most likely because it is the least painful of the threats at the moment. Infrastructure Address Hiding When respondents were asked if they implement any type of “hiding” 13 of all or some subset of infrastructure addresses, 30 percent said Yes while 53 percent said No. Occurrence of Internal Compromise When network operators were asked about what vulnerability was used to carry out attacks that target their own infrastructure, responses were as follows: 41% – External password attack. 37% – None. 31% – Host compromise through vulnerability. 21% – Insider threat. 1% – Other (SNMP read open in default config). The most alarming of these is perhaps the “insider threat” response at 21 percent. 13 http://tools.ietf.org/html/draft-ietf-opsec-infrastructure-security-01#section-6 23 Worldwide Infrastructure Security Report, Volume III Handling Vulnerabilities In this edition of the survey, we also added a question that asked respondents how they handle vendor vulnerabilities for infrastructure devices. Respondents were allowed to select multiple responses. The results are provided in Figure 23. Infrastructure-Related Vendor Vulnerability Handling Emergency Maintenance Use IDS/IPS 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Protect with ACLs at Perimeter Other Protect with Filter (e.g., Flow Spec) Figure 23: Infrastructure-Related Vendor Vulnerability Handling Source: Arbor Networks, Inc. Given the respondent organization type, it is clear why IDS/IPS is not a more common mitigation option. The Other responses included configuring netfilter/iptables on routers, depends on vulnerability and depends on recommend actions from vendor. Infrastructure Shortcomings We also asked survey respondents what the most critical missing or limited vendor security feature was. Responses were as follows (multiple responses were permitted): 50% – Management of ACLs on individual elements and across routers. 41% – Number of ACLs or impact of performance. 33% – Flexibility of ACLs. 24% – Reporting on dropped traffic (BGP RTBH). 24% – Reporting on dropped traffic (ACLs). 16% – Management of individual and multiple scrubbing systems (e.g., Arbor Peakflow SP Threat Management System). 3% – Other. The Other responses included the lack of indent automation and lack of support for source-based real-time black hole routing. 24 Worldwide Infrastructure Security Report, Volume III Composition of Security Teams This section always proves to be a favorite data set for respondents, as it typically helps them demonstrate to management where their staffing and related expectations fall in relation to other organizations. When asked what the size of their dedicated security group was, responses are shown in Figure 24. Dedicated Security Staff 50 or more 2-4 80% 70% 60% 50% 40% 30% 20% 10% 0% Tier 1 Tier 2 Content Hosting Enterprise Academic Other 10-49 Just Me 5-9 No Dedicated Organization Type Figure 24: Dedicated Security Staff Source: Arbor Networks, Inc. We were somewhat surprised by some of the self-classified Tier 1 responses to this question, both with the wide spectrum of responses and, in particular, with several indicating that they had no dedicated security staff. This applies similarly to Tier 2 respondents. When respondents were asked how their security group fits into the overall organization, 64 percent said it was part of network engineering, 9 percent said it was part of a dedicated abuse department, 13 percent said it was an independent organization and 14 percent said it was contained within some other group. Emerging Threats When respondents were asked if they have specific tools or mechanisms to monitor and detect threats against VoIP, 20 percent said Yes, while 60 percent indicated that they do not have any tools to detect threats against VoIP. When asked if they have tools to mitigate threats against VoIP, only 11 percent of the respondents said Yes, while 67 percent said No. Respondents were also asked if they had specific tools to monitor and detect threats against DNS infrastructure. While 36 percent of the respondents said Yes. 44 percent said No. When asked about the availability of specific tools or mechanisms to mitigate threats against DNS, 33 percent of the respondents said Yes, while 46 percent said No. When respondents were asked if they feel ISPs are in a position to effectively mitigate attacks considering the complexity of Internet attacks and the use of encryption and tunneling, 14 percent said Yes, 10 percent said No and 73 percent said Somewhat. 25 Worldwide Infrastructure Security Report, Volume III Personal Protection from Internet Threats Personal OS FW Commercial AV Tunnels/VPNs 80% 70% 60% 50% 40% 30% 20% 10% 0% Hardware FW OpenSource Flow Tools Credit Reporting Service OpenSource AV Commercial Flow Tools Figure 25: Personal Protection From Internet Threats Source: Arbor Networks, Inc. We also asked respondents how they protect themselves personally from Internet-related threats. As indicated in the chart above, most respondents selected multiple answers and indicated that layered security is the only viable approach given today’s security landscape. ISPs: Bots & Botnets, AV & Malware We asked respondents an array of questions ranging from botnet sizing to distribution of AV and malware. Most of the information in this section is shared as is, with very few author conclusions provided. As with the rest of the survey, it is simply meant to be somewhat representative of the network operator perspective on the issue. Figure 26 illustrates the responses when asked what the largest observed botnets were over the past 12 months. Largest Observed Botnet — Past 12 Months >150k 100k-150k <1000 50k-100k No Answer 10k-50k 1k-10k 25% 20% 15% 10% 5% 0% Figure 26: Largest Observed Botnet — Past 12 Months Source: Arbor Networks, Inc. 26 Worldwide Infrastructure Security Report, Volume III When respondents were asked if any trends in botnet sizing have been observed over the past 12 months, responses varied widely. These responses included: - Smaller or adaptive and targeted bots preferred over larger ones. - Smaller, better employed, more firepower and better protected at the command and control level. - Smaller botnets with more firepower, more effective attack vectors. - Peer-to-peer botnet control seems to have enabled bots to grow larger again, since locating the mast controller is more difficult. - Better organized, peer-to-peer command and control. - Botnets now seem to be smaller, Internet relay chat (IRC) controlled and country distributed in order to generate different flows, using different entry points or other functions when attacking a given victim. - Organization is improving, increases in non-IRC command and control (e.g., HTTP with encoding/encryption, peer-to-peer). Increase in the use of MD-5 and SHA-1 to protect “important” bot verbs and counter RE attempts. When asked what activities respondents had observed botnets performing over the past 12 months, responses were as follows: 71% – DDoS attacks. 64% – Sending spam. 37% – Components of phishing systems. 34% – Used as open proxies. 16% – Storing ID theft information. 6% – Other. The Other responses above included recon and brute-force password guessing, credit card checkers, hosting and installing various .exes, keylogging and testing IP blocks for targets. We also asked respondents if they believed DDoS attacks would decrease as bots are employed for more revenue-generating opportunities. Fifty-nine percent of the respondents said No, while 41 percent said Yes. When asked to identify the most effective tools and techniques for measuring and detecting botnets, respondents answered: - Flow-based tools. - Still looking! - Taking advantage of stupid behavior by bots (e.g., naively choosing to query DNS RRs that could be filtered). - DNS request logs, flow data and darknets. - Snort, darknet reporting. - Not actively doing it due to legal issues (span 14 countries). - IDS, honeypots. - NetFlow, abuse complaints. - Packets—and the human brain! - Watching command and control transactions. - Information sharing. - Flow data, DNS mining. 27 Worldwide Infrastructure Security Report, Volume III When network operators were asked if they believed that infected host cleanup was an action the service provider should take, the responses were rather split, with 41 percent saying Yes, 30 percent saying No and 29 percent saying It Depends. We also asked respondents if they believed botnet command and control (C&C) server cleanup is an action that service providers should take. Fifty-nine percent of respondents said Yes, 17 percent said No and 24 percent said Maybe. Is Botnet Command and Control Server Cleanup an Action the Service Provider Should Take? Yes 14 12 10 8 6 4 2 0 Content Edu Hosting Tier 1 Tier 2 Other No Maybe Figure 27: Is Botnet Command and Control Server Cleanup an Action the Service Provider Should Take? Source: Arbor Networks, Inc. Is Infected Host Cleanup an Action the Service Provider Should Take? Yes 8 7 6 5 4 3 2 1 0 Content Edu Hosting Tier 1 Tier 2 Other No Maybe Figure 28: Is Infected Host Cleanup an Action the Service Provider Should Take? Source: Arbor Networks, Inc. 28 Worldwide Infrastructure Security Report, Volume III When respondents were asked if they follow-up and take action with infected host lists to an origin AS, obtained either directly from attacks or from dark IP reporting, 50 percent said Yes, while 30 percent said No. Among these 30 percent, many cited belief in utility or lack of resources as the primary reason for not following up. Respondents were asked if they felt they were hindered by legal or regulatory issues when connecting to or diverting command and control server traffic. Only 20 percent said Yes, while 49 percent said No. One might presume that those 49 percent have yet to experience some precedent changing their perspective, although we hope this is not the case. Some additional discussion on this topic can be found here.14 We also asked respondents if they track recurring attack or abuse sources across multiple events (e.g., spam, phishing, botnet C&C, DDoS). Surprisingly, 36 percent said they did track such host activity across multiple events, while a solid 50 percent said they did not. One might surmise that destination-based attack mitigation techniques, for example, which are meant to minimize response resources, would only exacerbate the host recurrence problem. With such techniques, malicious traffic is dropped only at the local network ingress, and when the attack stops the policy is removed. Subsequently, or in parallel, the attacking bots are either used to launch other attacks, or to perform other malicious activities. This is yet another reason why we believe it is critical to contact the ingress network and ensure that infected hosts are sanitized. Respondents were asked if they null-route or take other actions to disrupt activity from known compromised hosts entering their networks—or if they divert traffic to emulated C&C systems for analysis. Sixty-nine percent of respondents said Yes, 20 percent said No. Forty-three percent of the operators surveyed said they have a group responsible for tracking and keeping up-to-date with current malware threats, while 57 percent said they did not. When asked if they currently subscribe to a service to obtain lists of well-known malware distribution sites, 31 percent said Yes, while 69 percent said No. We asked respondents if they believed that tracking malware distribution sites and filtering access to them was a role service providers should take. Forty percent of the respondents said Yes, 21 percent said No and 39 percent responded Maybe. When asked if respondents believe that malware (either infected hosts or distribution sites) is a concern for service providers, 77 percent said Yes, while 14 percent said No. When respondents were asked if they had any partnerships with anti-virus vendors to enable customer offerings, 44 percent said Yes and 47 percent said No. Conclusion In the year since the last survey, providers have invested hundreds of millions of dollars in their backbone and infrastructure security. While DDoS and emerging attack vectors like botnets continue to pose a threat, ISPs are generally more optimistic. More than half of the surveyed providers now believe ISPs can effectively mitigate most Internet attacks against their backbone infrastructure and customers. In the arms race between ISPs and hackers, ISPs believe they are ahead. Most providers reported that DDoS attacks continue to be relatively unsophisticated “brute-force” flooding efforts. The majority of ISPs now say they have the commercial or in-house tools and procedures to address the majority of these attacks. As one observed, “We’ve been able to take advantage of stupid behavior.” Providers also credit improved information-sharing between ISPs and the growth of dedicated security teams within their organizations with giving ISPs the upper-hand. Many of the surveyed providers now even view DDoS/botnet threats as revenue opportunities with close to three-quarters either currently offering or planning managed security services. But all of this ISP optimism about infrastructure security should be tempered by the survey data on emerging critical infrastructure. More than half of the surveyed providers said they had no means to either detect or mitigate attacks against DNS and close to 90 percent have no means to protect critical VoIP infrastructure. And while ISPs routinely address 1-2 gigabit DDoS attacks, few providers have the capability to mitigate 10+ Gbps attacks. It should be interesting to see if this and the other trends mentioned in this report continue as we move forward to the fourth annual report in 2008. 14 http://asert.arbornetworks.com/2007/03/botnet-cc-quandry-infiltrate-or-extirpate/ 29 Worldwide Infrastructure Security Report, Volume III About the Authors Danny McPherson, Chief Research Officer, Arbor Networks danny@arbor.net As Arbor’s Chief Research Officer, Danny McPerson leverages his 14+ years of experience in the Internet network operations, security and telecommunications industry to contribute to Arbor’s overall strategy and product architecture. McPherson is a common contributor within the Routing, Operations, and Internet Areas of the Internet Engineering Task Force (IETF) and global network operations community. He is also a member of the Internet Architecture Board, the FCC’s Network Reliability and Interoperability Council (NRIC) and the MPLScon Advisory Board, and currently co-chairs the IETF's PWE3 working group. McPherson has authored a significant number of books, Internet protocol standards, network and security research papers and other documents related to Internet routing protocols, network security, Internet addressing and network operations. Dr. Craig Labovitz, Chief Scientist, Arbor Networks labovitz@arbor.net Dr. Craig Labovitz brings to Arbor extensive expertise in distributed denial of service attacks (DDoS), networking protocols, network engineering and Internet research. Dr. Labovitz has served as a backbone network designer and engineer, as well as a network researcher and scientist for Microsoft Corp. Previously, he spent nine years with Merit Network Inc. and the University of Michigan as a senior backbone engineer and Director of the Research and Emerging Technologies group. His work at Merit included design and engineering on the NSFNet backbone and Routing Arbiter projects. Dr. Labovitz also served as the director of several multi-million dollar National Science Foundation (NSF) network architecture and routing protocol research grants. Dr. Labovitz received his PhD and MSE from the University of Michigan. Mike Hollyman, Manager of Consulting Engineering, Arbor Networks mhollyman@arbor.net With more than 12 years in the network, security and telecommunications industries, Mike Hollyman brings extensive knowledge of service provider and large enterprise network design and security experience to Arbor. He provides leadership to the Arbor sales organization through his management of the Consulting Engineering team for North American service providers. Prior to joining Arbor, Hollyman was a network and security consultant, both independently and through his own consulting company. Prior to consulting, he worked as a network engineer for OneSecure, Qwest Communications and the University of Illinois. 30 Corporate Headquarters 430 Bedford Street Lexington, Massachusetts 02420 Toll Free +1 866 212 7267 T +1 781 684 0900 F +1 781 768 3299 Europe T +44 208 622 3108 Asia Pacific T +86 10 8529 8885 About Arbor Networks Arbor Networks provides network security and operational performance for the world’s global business networks—protecting networks from the service provider cloud to the enterprise core. Arbor Peakflow products deliver network-wide visibility, anomaly detection and scalability to defend against current and future threats, including worms, data theft, botnets and more. Arbor Peakflow enables businesses to harden their networks, maintain business continuity and prevent the loss of customer confidence. Arbor is headquartered in Lexington, MA, with a research and development office in Ann Arbor, MI, and overseas headquarters in London and Beijing. www.arbornetworks.com Copyright ©1999-2007 Arbor Networks, Inc. All rights reserved. Arbor Networks, the Arbor Networks logo and Peakflow are all trademarks of Arbor Networks, Inc. All other brands may be the trademarks of their respective owners. WISR/US/0907