Worldwide Infrastructure Security Report US sept06

( VO L U M E l l ) WORLDWIDE INFRASTRUCTURE SECURITY REPORT SEPTEMBER 2006 Worldwide Infrastructure Security Report, Volume II Table of Contents OVERVIEW AND KEY FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 DEMOGRAPHICS OF THE SURVEY RESPONDENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 SURVEY METHODOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 MOST SIGNIFICANT OPERATIONAL THREATS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Attack Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Size and Frequency of Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Attack Frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Attack Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 ATTACK DETECTION AND TRACEBACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 ATTACK MITIGATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 LAW ENFORCEMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 INFRASTRUCTURE PROTECTION TECHNIQUES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 USE OF FLOW DATA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 COMPOSITION SECURITY TEAMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 EMERGING THREATS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 VoIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 ABOUT THE AUTHORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 List of Figures Figure 1: Respondent Organization Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Figure 2: Primary Concerns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Figure 3: Attack Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Figure 4: Largest Observed Attack Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Figure 5: Customer Impacting Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Figure 6: Infrastructure Impacting Attacks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Figure 7: Attack Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Figure 8: Primary Mitigation Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Figure 9: Law Enforcement Organization Engagement (Six Months) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Figure 10: BCP38/uRPF Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Figure 11: MD5 Use with Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Figure 12: Dedicated Security Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 i Worldwide Infrastructure Security Report, Volume II Overview Arbor Networks, Inc., in cooperation with the Internet network security operations community, has completed the second edition of an ongoing series of annual operational security surveys. The survey, covering the second half of 2005, is designed to provide data useful to network operators so that they can make informed decisions about their use of network security technology to protect their mission-critical infrastructure. Operational network securities issues—the day-to-day aspects of security in the commercial networks—are the primary focus of survey respondents. As such, the datasets provided in this survey more accurately represent real-world concerns than the theoretical attack vectors often speculated about elsewhere. Key Findings - Distributed Denial of Service attacks (DDoS) remain the most significant ISP security threat. After the initial flurry of well-publicized DDoS attacks six years ago, the majority of surveyed operators spend more resources addressing DDoS today than any other security threat, including worms and botnets. - Attack firepower grows. Respondents report- increased frequency and magnitude of multi-gigabit, supra-backbone DDoS attacks. ISPs now regularly report attacks beyond the capacity of core backbone circuits in the 10-20Gbps range. - Zombies rule. Despite the best efforts of firewall, IDS and OS vendors, compromised end systems available to participate in DDoS or other illegal activities number in the millions. And there is no end in sight. - ISPs finish the job. Lacking more advanced infrastructure/tools, the majority of ISPs surveyed mitigate attacks by filtering all traffic to the victim. While this is successful in protecting ISPs backbones from collapse under DDoS, the mitigation “cure” may be worse than the original DDoS. - No law enforcement engagement. Despite an average of 40 attacks per month that directly impact their customers, most attacks go unreported by ISPs because the majority believes law enforcement cannot effectively assist them. Demographics of the Survey Respondents Participants included 55 self-classified tier-1, tier-2, and hybrid IP network operators in North America, Europe and Asia, all of whom are directly involved in network security operations at their respective organizations. (See chart 1.) This is a 65% increase in the respondent pool size compared with the previous survey. Survey respondents included a majority of tier-1 ISPs, large content and hosting providers, and a cross-section of tier-2 networks, ultimately including large broadband and dial-up network operators worldwide. In addition, a large number of “hybrid” network operators responded to the survey. Hybrid networks represent large-scale globally distributed enterprise networks with multiple Internet access interconnections, and, in addition, provide their organization with traditional end-users’ services. Survey Methodology This edition of the survey consisted of 65 multiple choice and free response questions, more than doubling the 32 questions of the previous edition, and covering the major issues faced by network security operators today. Questions addressed such topics as observed threats against backbone infrastructure and individual customers; techniques employed to protect network infrastructure; and mechanisms used to detect and respond to security incidents. All data has been made anonymous and is presented with permission from the respondents. Individual respondents were most often senior network security architects or operations engineers at their respective organizations. Standard mathematical methods to weight responses have been applied where there are incomplete responses. 3 Worldwide Infrastructure Security Report, Volume II Respondent Organization Type Tier 1 Hosting 30% 25% 20% 15% 10% 5% 0% Tier 2 Enterprise/Hybrid Content Academic/Other Figure 1: Respondent Organization Type Source: Arbor Networks, Inc. Arbor Networks intends to conduct this survey every twelve months and share the results with the international network security community. Our goal is to continually refine the survey questionnaire in order to provide more detailed information in future editions, as well as to increase the scope of the respondent pool to provide greater representation of the global community of Internet network operators. Most Significant Operational Threats As noted above, more than five years after the initial flurry of DDoS news articles and research papers, DDoS remains the number one concern for large IP network operators. Forty-six percent of the survey participants said DDoS is the most significant operational security issue they face today. While this number is down from sixty-four percent reported in the previous edition of the survey, the difference is mostly attributable to the fact that a compelling new category, simply labeled bots and botnets – was added as an option. Bots and botnets registered a close second, with 31% of the respondents listing them as their primary operational security concern. There is some overlap in the current threats pool, which included the following list of threats and the associated percentage of respondents identifying them as their primary concern: 46% – DDoS 31% – Bots and Botnets 7% – Worms 6% – Compromised Infrastructure 6% – DNS 4% – BGP Route Hijacking Seeing that many of these threats, DDoS in particular, are typically perpetuated by botnets, we intend to rephrase the survey questions to avoid this overlap in future editions of the survey. Respondents were asked to order the above pool of threats from most concerning to least concerning. (See Figure 2) Of the six threats mentioned in the pool, BGP route hijacking, which has gained a considerable amount of attention in the research and operational community of late, was considered the least significant threat. Apart from the hybrid network operators responsible for end host cleanup, the chief concerns related to worms resulted from the potential worm payload and propagation vectors leading to network congestion, rather than the infection itself. Infected host cleanup is becoming more of a concern for broadband and dial-up network operators as well. 4 Worldwide Infrastructure Security Report, Volume II Primary Concerns BGP Hijack Worms 50% 40% 30% 20% 10% 0% DNS Compromise DDOS Bots Primary Figure 2: Primary Concerns Source: Arbor Networks, Inc. Secondary As we will discuss later in this report, fighting DDoS has evolved from an exotic, rare occurrence to a daily chore for most ISP operations groups. Most providers have deployed both dedicated detection and mitigation mechanisms as well as NOC operational procedures. An increasing number of ISPs are offering commercial DDoS mitigation and detection services, such as AT&T’s “Internet Protect”, Verizon’s “WAN Defense”, and Colt’s “IP Guardian” service. Attack Vectors The primary DDoS attack vectors reported for the largest observed attacks over the past six months are provided in Figure 3. Note that this data only represents the largest single attack observed by each respondent over the past six months, without consideration for frequency of occurrence of the given attack vector or other issues. SYN attacks led the pack, with UDP floods a close second. RST and Fragmentation attacks were also reported. Eighteen percent of respondents reported “other” attack types. Because there was no free-form entry in that field, the exact details of what comprises “other” attacks was not provided. We intend to revise this entry and provide the data in the next edition of the report. Moreover, 18% of the respondents gave no answer to this particular question. One possibility, which is revealed later in this report, is that a slightly larger number of respondents either do not currently have tools in place to detect attacks, or the tools that they currently use only provide Layer 2 visibility (e.g., SNMP ifOctets, etc.). As such, they have no Network or Transport Layer visibility and therefore cannot properly classify the attacks beyond Layer 2 (i.e., bits per second, packets per second). Attack Vectors 27% SYN 18% 2% 9% 18% 26% Figure 3: Attack Vectors Source: Arbor Networks, Inc. RST FRAG UDP OTH NA 5 Worldwide Infrastructure Security Report, Volume II Finally, this data clearly reinforces the fact that brute-force attacks remain the most predominant attack type on the Internet today. Dumb attacks that simply fill host connection state tables or overwhelm network link capacity are far more popular than complex Application Layer attacks that require non-spoofed sources and consume more attacker resources. Given the ease of recruitment and proliferation of botnets to date, simple bandwidth and host resource consumption attacks remain sufficiently damaging. Size and Frequency of Attacks Thirty-five of the fifty-five respondents reported observing attacks of 1 Gbps or more. Nine of these reported that the attacks were 4-10 Gbps and another ten of the thirty-five respondents reported attacks larger than 10 Gbps. Figure 4 illustrates reported attack size over the past six months versus “Ever.” In addition, this edition of the survey explicitly distinguishes “Ever” from the previous edition to provide additional context given the larger respondent pool in this edition. Largest Observed Attack Size Past 6 Months 60% Ever 1H2005 Previous Ever Survey Respondents 50% 40% 30% 20% 10% 0% 10+ Gbps 1-4 Gbps 100-500 Mbps NA/Other Attack Size Figure 4: Largest Observed Attack Size Source: Arbor Networks, Inc. We can observe here a discernable increase in attack sizes and that attack sizes are still on the increase. Most interestingly, the following details on attack magnitude were reported: - 17 Gbps was largest sustained attack reported - 22 Mpps UDP flood was reported - 14 Mpps SYN flood was reported Arbor Networks explicitly verified the above numbers with the respondents who provided them. Interestingly, each of these attacks were experienced by large content and hosting provider networks that have multiple Internet access interconnects via several service providers, and are well-distributed across the set of Internet links. After speaking with the upstream providers of the respondents reporting these attacks we found that they believe these numbers were plausible given their fraction of the aggregate connectivity provided to the network in question and the size of attacks they’ve observed towards targets in those networks. This seems obvious, given that no single service provider would likely see all the traffic from a well-distributed attack. The magnitude of these attacks is incredible when you consider that a 14 Mpps SYN flood can nearly fill an entire OC192 (10 Gbps) circuit with a minimum packet size. Any one of these attacks, or even a fraction thereof, can create significant pain for even the largest ISP networks in the world today. The current survey question set didn’t allow for associating packets-per-second with the largest attacks, nor did it distinguish largest packets-per-second attack versus largest bits-per-second attack – assuming that they’re different. Arbor Networks intends to capture this distinction in future editions of the survey. 6 Worldwide Infrastructure Security Report, Volume II Attack Frequency Attack frequency remained somewhat constant over the past year. Figures 5 and 6 indicate the number of actionable attacks per month targeting infrastructure and customer elements, respectively. Note that the infrastructure impacting attacks figure is inclusive of attacks that targeted customers but resulted in collateral damage to the infrastructure and therefore required some action on behalf of the network operator. Customer Impacting Attacks 1H2005 60% Current Survey Respondents 50% 40% 30% 20% 10% 0% 500+ 100-500 10-100 < 10 NA/Other Attack Size Figure 5: Customer Impacting Attacks Source: Arbor Networks, Inc. Infrastructure Impacting Attacks 1H2005 70% Current Survey Respondents 60% 50% 40% 30% 20% 10% 0% 500+ 100-500 10-100 < 10 NA/Other Attack Size Figure 6: Infrastructure Impacting Attacks Source: Arbor Networks, Inc. On average, the previously reported number of ~40 actionable customer impacting attacks per month decreased. It appears that the wider distribution of respondent types in this edition of the survey is likely attributable to an increase in the NA/Other responses. Again, a large number of attacks appear to be associated with networks that have a large user install base. For example, large residential broadband and dial-up providers report a higher frequency of attacks than typical backbone services providers. 7 Worldwide Infrastructure Security Report, Volume II Actionable attacks, as illustrated in figures 5 and 6, are attacks significant enough to require some response, either to protect the customer, or to protect the network itself. This distinguishes them from attacks that are undetected, or are detected but just considered “tolerable noise” on the network. Of the attacks that target infrastructure directly, the most common attack vectors in order of appearance were: SYN, UDP Flood, ICMP Flood, Fragmentation and RST, the majority of which appeared to be performed by botnets. The number of actionable infrastructure impacting attacks per months remains at about 2 attacks per month on average. This number wasn’t affected significantly by the increase in number of smaller respondents in this edition of the survey. Attack Targets As previously noted, DDoS was the primary concern of respondents with bots and botnets a close second. All respondents reported observing attacks or other malicious activity performed by botnets. When asked about the largest botnet observed over the past 6 months, 11% of respondents reported observing bots sized in the 100-150k range. Another 6% reported observing botnets in the 50-100k range, 15% in the 10-50k range, 7% in the 110k range, and another 15% reported observing botnets of less than one thousand nodes. 8% of the respondents reported observing botnets larger than 150k. When asked about any observable trends in botnet sizing, individual responses included: - More smaller botnets - Command and control channels are harder to connect to and better monitored by botherder - More obfuscated command and control (IM, HTTP w/unregistered DNS, stripped down IRCds) - More variations of older botnets - Better organized, more difficult to take down, more capabilities, more flexible, and better packed to resist analysis - Larger botnets (now that we have ability to see them as we didn’t before) - Using more public IRC servers (e.g., EFnet, etc.) - Use of SSL-encrypted servers - More bots, botnets, and firepower - Better organized - Bots are better at hiding and more difficult to remove One respondent made a new and disturbing observation: “I have noticed a trend towards marketing rather than sizing. The degree of organization [for botnets] has been very high for some time now, and sizes are uniformly gargantuan today. But the marketing of these herds is really disturbing. I have seen emails sent to customers advertising botnet availability & costs (by attack, or by time period, or by activity ["blast your affiliate numbers overnight!"].” Given that the survey authors have received many of these marketing emails ourselves, we can confirm the truth of this observation. Survey respondents were also asked to comment on observed botnet employment over the past six months. Here are some of the more common and notable responses: - Spam as proxy or relay - DDoS - ID Theft - Espionage - Form-logging - Phishing - Address Harvesting - Open Proxy - Scan and Sploit - SSH Brute Force Attacks - More Marketing - Lifting CD Keys 8 Worldwide Infrastructure Security Report, Volume II Of the above, spam, Phishing and DDoS were the most detectable. Respondents also noticed bots employed for multiple activities, rather than single functions. This illustrates the evolution of the command and control capabilities of botnets over the past few years, enabling multitasking and flexibility of execution sets, even within a single botnet. Attack Detection and Traceback Here respondents were asked to identify the tools and techniques they employ for attack detection and traceback. Large tier-1 and tier-2 providers rely primarily on customer calls or commercial flow-based tools to detect attacks (see Figure 7). Only two respondents, consistent with the previous edition of this survey, indicated that they had no support for tracing DDoS attacks across their infrastructure to the network ingress points – although several respondents admitted that the traceback was a manual and tedious process. Attack Detection Commercial Open Source 40% 30% 20% 10% 0% Cust Call NA/Other SNMP Figure 7: Attack Detection Source: Arbor Networks, Inc. When respondents were asked whether they currently offer any type of DoS-related services to customers, they responded as follows: 49% - No Answer 20% - Detection and Mitigation 12% - Other 6% - For Detection 4% - For Mitigation When asked about DoS-related SLAs, they responded as follows: 65% - No Answer 24% - Other 7% - Detection and Mitigation 4% - Mitigation 0% - Detection Only 9 Worldwide Infrastructure Security Report, Volume II Attack Mitigation Destination-based Access Control Lists (ACLs) and destination-based BGP (Border Gateway Protocol) blackhole routing remained the primary attack mitigation techniques. Primary Mitigation Techniques ACLs Scrub 40% 35% 30% 25% 20% 15% 10% 5% 0% BGP DST Other BGP SRC NA Figure 8: Primary Mitigation Techniques Source: Arbor Networks, Inc. Challenges cited with regard to performing any IP sourced address techniques for attack mitigation included: - Lack of infrastructure capabilities for enabling a large number of packet filters in the routers’ forwarding path (either lack of support for sheer number of entries required or an intolerable amount of packet forwarding performance degradation) - Inability to verify authenticity of attacking IP source addresses Access Control Lists ACLs, also commonly referred to as packet filters, normally operate at the Network (Layer 3) and Transport (Layer 4) layers. This enables traffic discrimination and action (e.g., discard or rate-limiting) based on any combination of a packet’s characteristics, such as: - Source and/or destination IP address - Protocol types (e.g., TCP, UDP, ICMP) - Source and destination port numbers (many well-known, primarily on the server side, such as TCP port 80 is normally HTTP) - ICMP Type and Codes, etc. The offshoot, of course, with destination-based ACLs is that they effectively complete the attack on the service in question. For example, if a miscreant launches an attack on a web server the attacker would normally direct attack traffic to server_ip_address:tcp:80. In order to mitigate the attack a network operator using access control lists would define something like the following (this is a sample Cisco IOS access control list that defines a rule for dropping all TCP traffic destined for port 80 on host 1.1.1.1): access-list 101 deny tcp any host 1.1.1.1 80 access-list 101 permit ip any any 10 Worldwide Infrastructure Security Report, Volume II The network operator would then apply this definition to the interface(s) from which the attack traffic is ingressing the network – or egressing the network bound for the customer, depending on the traceback capabilities and magnitude of the attack. The result is that all traffic destined to the web server’s HTTP port will be discarded, thereby effectively completing the attack. There are several reasons why destination-based ACLs are a primary mitigation technique – even while they negatively impact the victim’s services. For additional information on this see “Effectively Completing the Attack.” 1 BGP BGP destination-based blackhole routing accomplishes essentially the same result as destination-based ACLs, except that you lose some of the granular definition capabilities that are available with ACLs, for example, the ability to filter on Transport Layer protocol types or port numbers. Instead, destination-based blackhole routing employs the BGP routing protocol to announce to all or some subset of routers in the local routing domain a BGP route set that instructs the routers to drop all traffic for a destination host or IP address space. While one loses some of the granularity that ACLs provide, BGP blackhole routing allows network operators to employ BGP as a signaling protocol and manage the policy application from a single point in the network, whereas ACLs require per device and per interface configuration and, oftentimes, augmentation of the existing ACLs on each interface. As such, BGP is much more efficient in distributing the policy set to the routers in the network. Destination-based blackhole routing also allows network operators to exploit the routers’ native forwarding capabilities and typically results in little or no forwarding performance degradation as might be incurred with ACLs. For additional information and references on the configuration and benefits of BGP blackhole routing consult the following papers from the Internet Engineering Task Force (IETF)2 and The North American Network Operators’ Group (NANOG)4. Scrubbing Other attack mitigation techniques employed by network operators surveyed included source-based BGP blackhole routing and “scrubbing”. Source-based BGP blackhole routing couples normal blackhole routing techniques with unicast RPF (reverse path forwarding) in order to use BGP to disseminate source-based packet drop policy sets. More information on uRPF can be found at the IETF site4 and at Arbor Networks5: “Scrubbing” typically involves diverting traffic to devices that “scrub” the traffic for a given set of Internet destinations and allow only legitimate traffic to be forwarded through the network towards the victim. This usually requires specialized network devices and is quickly becoming a more common technique for effectively dealing with DoS attacks, as many common attacks tend to exhibit discernible characteristics that allows discrimination between malicious and legitimate traffic. However, scrubbing typically results in some change to the data path in the network and requires that the network operator make dedicated capital investments. In addition, diverting traffic to the scrubbing devices and then “on-ramping” legitimate traffic once it’s been scrubbed is often a complex exercise. While the percentage of respondents who have suggested that they’re deploying scrubbing devices between the first half of 2005 and the second half of 2005 remains fixed, at about 7%, we suspect that between the increase in respondents, many from smaller and more hybrid networks, and the wider-scale deployment of scrubbing systems in larger networks, there is indeed growing operational deployment of these systems. http://asert.arbornetworks.com/2006/04/effectively-completing-the-attack-and-this-posting/ http://www.ietf.org/rfc/rfc3882.txt 3 http://www.nanog.org/mtg-0402/morrow.html 4 http://www.ietf.org/rfc/rfc3704.txt?number=3704 5 http://asert.arbornetworks.com/2006/05/gone-arent-the-days-of-spoofing/ 1 2 11 Worldwide Infrastructure Security Report, Volume II Law Enforcement When network operators were asked how many attacks per month they refer to law enforcement agencies, the number remained constant with that of the first half of 2005. Law Enforcement Organization Engagement (6 months) 51+ None 70% 60% 50% 40% 30% 20% 10% 0% 10 to 50 NA/Other <10 Figure 9: Law Enforcement Organization Engagement (Six Months) Source: Arbor Networks, Inc. In dividing the number of attacks over the past six months by the number of LEO engagements, only about ~1.5% of all actionable attacks are reported to law enforcement. Reasons cited for not engaging law enforcement agencies on a more regular basis included: - Lack of forensics detail - Belief in the utility of reporting incidents to law enforcement - Customer privacy/confidentiality requests - Too many attacks/to high a frequency to bother with Respondents were explicitly asked, “Do you believe that law enforcement has the power and/or means to act upon information provided by service providers,” with results as follows: 29% – Yes 38% – No 15% – Other 18% – No Answer It remains evident that network operators are primarily concerned with mitigating attacks to the ingress (optimal) point of their network, and the resources required to trace attacks further upstream and enable cleanup of compromised hosts doesn’t even come close to justifying the investment required by the service provider. This is a disturbing but understandable artifact resultant of the complexity and resources required to take down evermore globally distributed and resilient botnets, and with SPs more and more focused on ROI, a trend that only economics and law enforcement success with criminal prosecution with effect. 12 Worldwide Infrastructure Security Report, Volume II Infrastructure Protection Techniques New to this edition of the survey were questions related to several well-known infrastructure security techniques. In particular, questions regarding whether network operators might use some techniques to lessen the probability of source IP address spoofing on the Internet, whether they see a need to harden the transport connections associated with their routing protocols, and, if they have been compromised at any point in the past, what factors led to the incident. BCP38/uRPF Application Customer Edge 60% Peering Edge Survey Respondents 50% 40% 30% 20% 10% 0% 53% 45% 33% 16% 11% Yes No 20% 18% 4% NA Other Figure 10: BCP38/uRPF Applications Source: Arbor Networks, Inc. The IETF provides an overview of anti-spoofing measures that should be employed by network operators6. Essentially, BCP38 recommends that a network operator not accept packets into the network on a given interface for source address s unless destination address s is considered reachable through that interface. This policy is traditionally implemented by provisioning ACLs on each interface, statically defining which destinations are considered reachable through the interface, and therefore, which source address should be permitted to ingress the network through that interface. Unicast RPF, as referenced in the mitigation section earlier in this document, provides a mechanism to automate guidelines provided in BCP 38. There are multiple modes to which with uRPF may be implemented to provide for anti-spoofing measures, depending on the topology of the network and network equipment in use. Note that some of these modes permit spoofing within address spaces known within the local routing systems and are therefore less effective than more explicit BCP 38 and ‘strict-mode’ uRPF style policies. In hindsight, we should have asked two separate questions regarding anti-spoofing measures employed by the service provider. We could have asked whether BCP 38 style ACLs were provisioned and where, and whether or not and which mode of uRPF is used at what points in the network. Network operators were also asked whether they used TCP’s MD5 Signature Option7 to protect BGP Transport connections in the network, and whether or not they employ the MD5 protection mechanisms available with their Interior Gateway Protocol (IGP) as well. Responses are illustrated in Figure 11. Quite a large number of network operators use MD5 mechanisms to protect external BGP sessions, in particular, and well over half of the respondents also use it to protect internal BGP sessions and their IGPs. A small number of network operators also reported using IPSEC for this purpose. 6 7 http://www.ietf.org/rfc/rfc2827.txt http://www.ietf.org/rfc/rfc2385.txt 13 Worldwide Infrastructure Security Report, Volume II When network operators were asked about attacks that target vulnerabilities in their own infrastructure, responses were as follows: 36% – External password attack 36% – Vulnerability used to compromise systems 16% – Insider threat Lack of BCP/poor security practices implementation SNMP open access MD5 Use with Routing Protocols TCP MD5 no IPSEC 100% 80% 60% 40% 20% 0% iBGP eBGP IGP no TCP MD5 MD5 IPSEC no MD5 Figure 11: MD5 Use with Routing Protocols Source: Arbor Networks, Inc. We also asked survey respondents to identify the most critical missing or limited vendor security feature, and responses were as follows (multiple responses were permitted): 42% – ACL Impact/Performance 38% – ACL Flexibility 66% – Management of ACLs on individual interfaces and across multiple routers 27% – Management of individual and multiple load-balanced Intelligent filtering/scrubbing devices 2% – Provisioning and network management in general 5% – Other Use of Flow Data Over the past several years, most ISPs have adopted router flow export as a significant additional network data source. Flow export systems include Cisco’s NetFlow, Juniper’s cflowd and JFlow, Foundry’s SFlow and similar data mechanisms from other vendors. When network operators were asked about their use of flow data, their responses were as follows (note that multiple responses were permitted): 78% – Security 73% – Peering & Traffic Analysis 35% – Auditing 22% – Accounting 16% – Billing 7% – Compliance 2% – Troubleshooting 14 Worldwide Infrastructure Security Report, Volume II When asked which versions of flow data they use for the above purposes, responses were as follows: 82% – NetFlow v5 16% – NetFlow v9 13% – NetFlow Other 15% – JFlow 2% – sFlow v2 2% – sFlow v4 7% – sFlow v5 0% – IPFIX 5.5% – Do not use flow information When asked about when using flow data if they trust the accuracy of that information as compared to SNMP, responses were as follows: 40% - about the same as SNMP at higher sampling frequencies 26% - about the same at 1:1 (no sampling) 18% - no answer 4% - use for billing 4% - not even close 9% - other When asked about sampling frequency for flow information at the customer and peering edge of the network, responses ranged widely from 1:1 to 1:10000. Composition of Security Teams When respondents were asked about the size and structure of their network security organizations, responses varied considerably (see Figure 12). Note that the one response indicating 50+ dedicated security staff is clearly an outlier. When the respondent (a large North American dial-up and broadband services provider) was asked for further explanation they clarified that this number represents both the infrastructure security team, as well as the abuse and internal security departments. Well over half of the respondents indicated that the dedicated security team is actually part of the Network Engineering organization, while about 25% of the respondents stated that the team was independent from network engineering and other organizations. Dedicated Security Staff 50 or more Just me 40% 30% 20% 10% 0% 49 to 10 No Dedicated 9 to 5 No Answer 4 to 2 Figure 12: Dedicated Security Staff Source: Arbor Networks, Inc. 8 http://www.uscert.gov/reading_room/DNS-recursion121605.pdf 15 Worldwide Infrastructure Security Report, Volume II Emerging Threats The final section of the survey included questions on emerging ISPs threats, ranging from DNS to VoIP attacks. About half of the surveyed ISPs indicated they had deployed mechanisms to detect both DNS and VoIP threats. Unlike brute force DDoS flooding attacks, application levels attacks (including DNS and VoIP) masquerade as legitimate traffic. As one ISP noted, “Attacks typically consist of replay of a legitimate-looking set of query packets, at a higher-than-legitimate volume. Other than volume, we have little way of detecting them currently.” Most ISPs report a significant rise in recursive DNS attacks over the last year. 8 Some ISP described scripts that query DNS server logs at random sampling intervals to spot potential malicious activity. Several others monitor DNS traffic and payloads with commercial monitoring tools. VoIP Many providers are still in the early stages of planning or deployment of commercial VoIP services. While few reported attacks against VoIP infrastructure, providers are wary. One lead security engineer noted “[That VoIP is] vulnerable is an understatement.” Several said that they use Session Boarder Controllers which include built-in DDoS mitigation mechanisms. Other providers monitor VoIP with commercial flow monitoring tools. Others include firewalls, rate-limiting, ACLs, Arbor detection, and border controllers to prevent most attacks. Conclusion Despite two decades of predictions about the Internet’s imminent collapse9, the Net has proven remarkably resilient to failure, technological change and the most sinister plots global miscreants can devise. But as the Internet matures, the future of the network is as often decided by business considerations as it is technological advances. While most ISPs responding (69%) believe they will have technology to keep pace with hackers, many respondents question whether they can pay for this continued arms race. As one provider noted, “ISPs are operating at very thin margins, [and] putting resources towards mitigation of threats requires more revenue. Customers are only interested in cost per bit and not interested in footing the bill for enhanced threat protections.” Another complained, “Politically, it is popular for customers to point to the ISPs and ask why they are not blocking the attack of the day while at the same time customers select Internet service based solely on the lowest price.” While the old Internet academic culture of cooperation between providers continues to prevail, some ISPs question whether an increasingly competitive and consolidated ISP market may play into the hands of attackers. For example, universal deployment of customer access filters would go a long way toward resolving the threat spoofed DDoS, but, as noted earlier in this survey, large numbers of ISPs have yet to deploy these basic mechanisms. Mitigation of large-scale DDoS and zombie army threat requires inter-provider cooperation. Today this cooperation is achieved through direct back-channel communication between security engineers with inter-personal relationships at different providers, and grassroots efforts by network security vendors such as Arbor Networks’ Fingerprint Sharing Alliance.10 While this works better than nothing, one security engineer warned, “It’s not the end answer. Eventually, the security guy/gal will not be available when needed. Our community has never been able to adequately address this well-understood and fundamental problem - and the mind reels at the implications...” 9 http://www.infoworld.com/cgi-bin/displayNew.pl?/metcalfe/bm120495.htm http://www.arbornetworks.com/fingerprint-sharing-alliance.php 10 16 Worldwide Infrastructure Security Report, Volume II About the Authors Danny McPherson, Chief Research Officer, Arbor Networks danny@arbor.net As Arbor's Chief Research Officer, Danny leverages his 14+ years of experience in the Internet network operations, security and telecommunications industry to contribute to Arbor's overall strategy and product architecture. Prior to joining Arbor, McPherson was Director of Emerging Technology at Amber Networks. He has served as network architect for global service providers such as Qwest, MCI, and Genuity. McPherson is a common contributor within the Routing, Operations, and Internet Areas of the Internet Engineering Task Force (IETF) and global network operations community, and is active within VPN & MPLS standardization and deployment efforts. He currently chairs the IETF PWE3 Working Group and is a member of several IETF Area directorates and Internet research groups. He is also a member of the FCC's Network Reliability and Interoperability Council (NRIC) and the MPLScon Advisory Board. McPherson has authored a significant number of books, Internet protocol standards, network and security research papers and other documents related to Internet routing protocols, network security, Internet addressing and network operations. Dr. Craig Labovitz, Director of Network Architecture, Arbor Networks labovitz@arbor.net Dr. Labovitz brings to Arbor extensive expertise in distributed denial of service attacks (DDoS), networking protocols, network engineering, and Internet research. Dr. Labovitz has served as a backbone network designer and engineer, as well as network researcher and scientist for Microsoft Corp. Previously, he spent nine years with Merit Network Inc. and the University of Michigan as a senior backbone engineer and Director of the Research and Emerging Technologies group. His work at Merit included design and engineering on the NSFNet backbone and Routing Arbiter projects. Craig also served as the director of several multi-million dollar National Science Foundation (NSF) network architecture and routing protocol research grants Dr. Labovitz received his PhD. and MSE from the University of Michigan. Please direct questions and comments to Danny McPherson, danny @arbor.net, and Craig Labovitz, labovitz@arbor.net mailto:labovitz@arbor.net 17 Corporate Headquarters 430 Bedford Street Lexington, Massachusetts 02420 T +1 781 684 0900 F +1 781 768 3299 Europe T +44 208 622 3108 Asia Pacific T +86 10 8529 8885 About Arbor Networks www.arbornetworks.com Copyright ©1999-2006 Arbor Networks, Inc. All rights reserved. Arbor Networks, the Arbor Networks logo, Peakflow, and ArbOS are all registered trademarks of Arbor Networks, Inc. All other trademarks are the property of their respective owners. PDF0806 Arbor Networks® provides core-to-core network security and operational performance for the world’s global business networks—protecting networks from the service provider cloud to the enterprise core. Arbor Peakflow® delivers unmatched network-wide visibility, anomaly detection and scalability to defend against current and future threats, including worms, data theft, botnets and more. Arbor Peakflow enables businesses to harden their networks, maintain business continuity and prevent the loss of customer confidence. Arbor is headquartered in Lexington, MA, with a research and development office in Ann Arbor, MI, and overseas headquarters in London and Beijing.