SPECweb2005®™ in the Real World: Using Internet Information Server (IIS) and PHP Samuel R. Warner James Steven Worley Intel Corporation IBM Corporation 2700 156th Ave. NE; Suite 300 3039 Cornwallis Road Redmond, WA, 98007 Research Triangle Park, NC 27709 performance achieved with the SPECweb2005™/JSP™ kit is quite unrealistic; with top result translating to approximately 40000 Abstract users able to be sustained under load against a In this paper we examine using the SPECweb2005™ single webserver. Helping the IT customers benchmark to analyze various systems using IIS as the understand the performance achieved using web server and PHP to serve dynamic content. Using IIS SPECweb2005™/JSP™ and what the equivalent and PHP provides results for a representative “real performance would have been using world” web server configuration. Results will be SPECweb2005™/PHP will insure compared between systems with various configurations. SPECweb2005™ remains the benchmark of choice for the IT customer purchase process on webserver workloads. I. Introduction Today’s SPECweb® benchmark has brought a new I.A. Overview of the dimension to the ability of computer equipment SPECweb2005™ Benchmark manufacturers (OEM) to show the relative performance of the various systems manufactured in the Web Serving SPECweb2005™ is the Standard Performance Market. With each alteration in the benchmark come Evaluation Corporation benchmark for evaluating new opportunities and with the advent of the performance of World Wide Web Servers. It SPECweb2005™ the opportunities are becoming quite is the latest in the family of SPECweb® apparent. While SPECweb2005™ can be used for benchmarks superseding SPECweb99™ and research efforts in universities and within computer SPECweb99_SSL™. There are three workloads equipment manufactures to evaluate alternative that are part of the benchmark, technologies, another avenue for its use is in the purchase SPECweb2005_Banking, process by Information Technology (IT) professionals. SPECweb2005_Ecommerce, and These IT purchase evaluation are started with a request to SPECweb2005_Support, where each workload ask the various OEMs to show their ability to handle the represents the characteristics of the market specific requirements of the end customer, often using a segment represented. The SPECweb2005™ benchmark that closely matches the market domain of the benchmark also comes in two implementation customers use. Today SPECweb2005™ has two variants, one that uses JSP™ and the other that alternatives to enable performance evaluation, either Java uses PHP. Each workload variant of Server Pages (JSP™) or Perl Hosted Programming SPECweb2005™ enables measuring the (PHP), SPEC intended to enable the customers to maximum number of simultaneous user sessions understand the performance of these software competing that a web server is able to achieve while still alternatives in their end environments. However, the meeting specific QOS metrics and error rate result is that the logic of SPECweb2005™ is not complex requirements for the market segment represented. enough to accurately represent end user usage, and that While the individual sub-metric scores indicate we have caused a problem. Our problem is that the the total number of simultaneous user sessions the Rank Web Server Chips Cores Processor Result 1 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 37,001 2 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 37,001 3 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X7350 Processor 30,261 4 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon E7330 Processor 26,119 5 2 (x86_64) Rock Web Server v1.4.6 (x86_64), Rock JSP/Servlet Container v1.3.1 4 Intel Xeon Processor X5365 26,077 6 2 (x86_64) Rock Web Server v1.4.3 (x86_64), Rock JSP/Servlet Container v1.3.0 4 Intel XEON 5365 processor 22,332 7 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 2 AMD Opteron 8222SE 22,254 8 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 2 AMD Opteron 8220 21,470 9 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon Processor X5355 20,387 10 4 (x86_64) Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 2 AMD Opteron 8220 20,235 11 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X5355 Processor 19,931 12 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon Processor X5355 19,661 13 4 (x86_64) Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 2 Intel Xeon 7140M Processor 18,981 14 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X5365 Processor 18,931 15 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X5355 Processor 18,917 Table 1- Top 15 SPECweb2005/JSP™ Scores server can support, the overall SPECweb2005™ metric for a compliant result is the geometric mean of the three II. SPECweb2005™ & Real sub-metrics, normalized to a reference platform score. A score of 100 represents the same overall performance of World the reference platform, while a score of 20,000 In today’s Web Server world, the complex represents a score 200 times that of the reference combination of number of users supported, platform. content development language, and sub-category of the web server market all add to the complexity For SPECweb2005™ the benchmark clients run the of determining which Web Serving software and application program that sends HTTP requests to the hardware to use. Many IT customers look server and receives HTTP responses from the server. towards performance measurements made using The benchmark does not require a specific choice of web industry benchmarks as a way to evaluate the server software. The only requirement is that the web complex combination and make a purchase server support HTTP 1.1 and SSL (HTTPS). Of course, decision. Sadly the IT professional doesn’t have the implementation of the web server will lead to much spare time to understand the subtleties of a differences in observed performance on the same system benchmark, and often simply uses the simple under test. Today’s performance using SPECweb2005™ terms to describe the benchmark combination that on the latest hardware with Rockweb and the JSP™ kit is he would like OEMs to evaluate. The end result achieving performance which is roughly 370x better than of the IT customer purchase process is that the the reference platform used. OEMs are constrained by competitive pressures to produce leading results coupled with the lack knowledge about the SPECweb2005™ I.B. Overview of Top Fifteen Results alternatives available, is causing the publications for SPECweb2005™ performance to diverge The top fifteen results for SPECweb2005™ as of from real world IT customer performance. November 2007 were submitted with either the Sun™ Java[TM] System Web Server 7.0 or Rock Web Server v1.4. These results are published on the SPEC.org website at http://www.spec.org (see Table 1). Thirteen of the top fifteen results were with the Rock Web Server. Although the Rock Web Server is undeniably fast it has a relatively small customer base compared to Apache™ HTTP Server or Microsoft®’s Internet Information Services (IIS) The consistent trend for web servers has been that under load the total number of connections supported has been in the range of 200-2000 sustained customer connections. As processing capacities have improved, so have the concerns for security along with the additions of complexities in formulating the responses for web server requests.1 Many of the papers written from 1999 to today, show the connection loads at constant levels, at a time when processing capacities and software performance have continued to increase. One paper shows connection loads in ranges of 400 in a 1 second interval in the 2000 timeframe, when SPECweb99™ results were showing connection loads ranging from 400- 4000. During the years of SPECweb99™ publications we can find results that range from 400 to 25000 for the five years of the publication. Meanwhile, webserver connection rates remained in the 1000s. Thus while SPECweb99™ results achieved a 62x change in total connection able to be sustained, the end-users were only seeing roughly observing a 4x change in total performance. TFigure 1 - 50x Sustained Connection in two years 1 Various papers are showing connection loads for end-user scenarios ranging from 100s to 1000s of connections sustained during time intervals of one second. http://www.cs.bu.edu/techreports/1999-001-dpr-cluster-load- balancing.pdf http://22.214.171.124/search?q=cache:GtJ7HQvTTfYJ:www.cs.bu.e du/faculty/crovella/paper- archive/usits99.ps+connection+loads+to+webservers&hl=en&ct=cln k&cd=1&gl=us http://actapress.com/Content_Of_Proceeding.aspx?ProceedingID=39 1 Distributing Requests by (around k)-Bounded Load-Balancing in Web Server Cluster; OK and PARK IEICE Trans Inf & Syst.2006; E89-D: 663-672 SPECweb2005/JSP Scores since inception 50x change in Connections sustained in two years 45000 70 40000 Memory(GB), Number Netwokrs, Number of Sockets 60 35000 50 SPECweb2005/JSP Scores 30000 40 25000 20000 30 15000 20 10000 10 5000 0 0 Sep-05 Mar-06 Aug-06 Jun-07 May-07 Jun-07 SPECweb2005/JSP Score # sockets Memory (GB) # of Nets Web Server Creator Windows Linux Solaris CGI FastCGI Servlet ASP.NET IIS Microsoft Yes No No Yes Yes No Yes Rock Web Server Accoria Networks Yes Yes Yes Yes Yes Yes No Sun Java System Sun Microsystems Yes Yes Yes Yes Yes Yes No Table 2 - Comparison of Web Servers published with the existing SPECweb2005™ This is part of the problem the web server market is benchmark. The IT industry would be served encountering with SPECweb2005™ as well. The well by this benchmark as a result, as they would published results for SPECweb2005®/JSP™ on two and be able to understand how the new equipment four socket systems has begun to achieve levels intended to be purchased will help them in their approaching 50,000 users simultaneously connected to a environment. Perhaps with this added signal system for a duration of ½ hour. This is in sharp understanding the IT industry would also be able contrast with a majority of the industry, where the adjust their requests for equipment performance published connection load from research papers tends to studies to be specifically done using the be in the 2000 to 3000 user range.2 SPECweb2005®/PHP kit. By studying the SPECweb2005®/PHP kit’s performance, II.A. Analysis of Currently Published and comparing the performance achieved with that SPECweb2005™ Results achieved on SPECweb2005®/JSP™, we should be able The present Request for Information (RFI) and bridge the gap between what the real world is Request for Proposal (RFP) processes have led encountering for connection loads, and what could be many OEMs to develop publications for SPECweb2005®. Yet the RFI and RFP processes 2 http://dirt.cs.unc.edu/packmime/docs/INFOCOM04.pdf are also forcing OEMs to use software http://whitepapers.techrepublic.com.com/whitepaper.aspx?docid=32 combinations that enable leading performance 6508 http://www.mu.jisc.ac.uk/reports/viewreport.php?reportid=23 results. Competition is great a way to insure that and many other papers with similar connection rates. the final purchased product meets the needs of the Rank Web Server Chips Cores Processor JSP Results PHP Results 1 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 na 2 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 na 3 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X7350 Processor na 4 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon E7330 Processor na 5 2 (x86_64) Rock Web Server v1.4.6 (x86_64), Rock JSP/Servlet Container v1.3.1 4 Intel Xeon Processor X5365 na 6 2 (x86_64) Rock Web Server v1.4.3 (x86_64), Rock JSP/Servlet Container v1.3.0 4 Intel XEON 5365 processor na 7 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 2 AMD Opteron 8222SE na 8 4 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 2 AMD Opteron 8220 na 9 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon Processor X5355 20387 1083 10 4 (x86_64) Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 2 AMD Opteron 8220 na 11 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X5355 Processor 19931 1083 12 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon Processor X5355 19661 1083 13 4 (x86_64) Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 2 Intel Xeon 7140M Processor 18981 1143 14 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X5365 Processor na 15 2 (x86_64) Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 4 Intel Xeon X5355 Processor 18917 1083 Table 3 - – Comparison of JSP™ to PHP performance end customer. Competition constrained by a benchmark supports Linux and Solaris™ and IIS supports that closely matches the end-user environment helps only Windows. The major difference in dynamic produce an informed purchase, with benchmarks content is that IIS supports ASP.NET while Rock providing a means to mimic the end-user environment. Web and Sun™ Java System supports servlets Unfortunately the present sets of published results are not (JSP™).4 constrained (see Figure 1). The trend with the SPECweb2005®/JSP™ results is the connection rates have climb at a rate that exceeds prior SPECweb99™ II.A.2 Which web servers currently rate of 62x in 5years. The top fifteen results show a used in submissions connection rates that are at least 10-20x higher than The choice of which Web Servers is used when connection loads that are normally seen by WebServers3. preparing results for a submission is largely up to The unhindered trend will see connection loads of 90000 the OEM. As new hardware/software within another year. components become available, or a possible customer develops an RFI, the OEM has to make a choice about how to achieve results that will II.A.1 Comparison of WebServers differentiate the value to the end consumer. The Of the top fifteen SPECweb2005™ results thirteen of contributing components of the choice for a Web those results were using the Rock Web Server v1.4. The Server to use depends on whether the results are Rock Web Server is a proprietary, non-open source web prepared as market collateral to show the value of server developed by Accoria Network. The number of a hardware/software component, or whether the customer for this server product is currently not able to in response to an customers RFI process be measured by any Web Server market analysis company. Of the other top fifteen SPECweb2005™ As new hardware or software components are results two were submitted using the Sun™ Java System made available marketing material that can aid in Web Server 7.0. This is a proprietary, non-open source the education of the end-user community on the web server developed by Sun™ Microsystems. The web value of the new product is prepared. The OEMs server chosen for the results presented in this paper is develop most of this marketing material and do so Internet Information Services (IIS) a proprietary, non- via a variety of benchmark publications and open source web server developed by Microsoft®. whitepapers. Market conditions are the primary variable that regulates the decision of which Web Table 2 shows a comparison of the operating systems and Server to use in these cases. The easiest way to dynamic content supported by each of the web servers insure that the new components maximum value mentioned above. Sun™ Java Systems supports is shown is to use the best possible combination Windows, Linux, and Solaris™. Rock Web Server of hardware and software, otherwise the OEM 3 4 See other referenced materials earlier in this paper http://en.wikipedia.org/wiki/Comparison_of_web_servers Figure 2 - Market Share Data in year 2004 from a variety of market research firms % Internet Market Share 100 80 60 40 20 0 Alexa 1000 Media 100 Hot NetCraft NetCraft Metrix 500 Non-active Active Market Research Firm Apache IIS Netscape Zues would not only need to educate the consumer on the new software, this translates into the server software product but on the value of the information supplied stack that produces the highest result regardless of using less than ideal components. the number of customers of the end product, During an RFI process, often the IT consumer requests Presently the net result of the various decision for SPECweb2005™ explicitly, with out understanding making processes is that two Web Servers have there are two kit variants and the relative value of the been used in nearly all of the submissions. results to their purchase decision. The OEM upon receipt Accoria Network’s RockWeb has shown itself to of the RFI is left with the choice of producing a set of be quite capable to achieve industry leading results that might be more relevant to the consumer and results, where thirteen of the fifteen top results lower in performance, or to achieve industry leading use this product, and has been used in a majority results. Left with the increased cost of educating the IT of the submissions by the OEMs. Sun™ consumer on the value of the more relevant result versus Microsystems’ Sun™ Java System has been used simply achieving the highest result possible, the decision in the other set of submissions that comprise the becomes governed by the increased costs and risks other two of the fifteen submissions. Oddly, the associated with producing results that are more pertinent net result of the decision making process is that to the end users environment and likely to be neither Apache™ or IIS, both recognized in their miSun™derstood. use in the lion share of websites by a majority of the Web Server analysis companies, have been Time to market, cost of producing a result, and the costs used in any submission in the last two years. associated with any addition education needed to realize the value of the final information, all combine in to a benchmarking results producing dynamic that gravitates II.A.3. JSP™ use in submissions toward the best possible software and hardware Presently, all of the results submitted for combination available. In the case of Web Server SPECweb2005™ have used the JSP™ kit. While Developer 7-Oct Percent 7-Nov Percent Change Apache 68,155,320 47.73% 76,028,287 50.76% 3.03 Microsoft 53,017,735 37.13% 53,679,916 35.84% -1.29 Google 7,763,516 5.44% 7,910,879 5.28% -0.15 lighttpd 1,541,779 1.08% 1,505,122 1.00% -0.07 Sun 2,262,019 1.58% 619,262 0.41% Yes JSP™ is a widely used language it is not the primary the market place. One simple solution would be content development language used.5 Once again, the for the IT industry to explicitly require the primary reason is that in order to achieve leading industry SPECweb2005®/PHP publications for the RFI results an OEM needs to use a software stack that has the process. Yet at the same time, the ability to spend lowest overhead to generate a response. The current the time to study the true relevance of the implementation of SPECweb2005™ in JSP™ has lower SPECweb2005®/PHP kit compared to the real overhead relative to the PHP kit. workloads used in the IT industry is limited at best. Hopefully the content in the paper will help to address the discrepancy. II.B. Analysis of Real World Workloads While the industry results on SPECweb2005®/JSP™ II.B.1 Web Server Statistics have improved by ~50x since the inception, at the There are numerous papers that discuss web publications from various universities show that the true server statistics found in the real world. Many of connection loads have improved are trending at levels at these look at connection rates or connection loads least 1/10th those observed in publications. Similarly on of web servers. Others look at response times. 6 SPECweb99™ the same disconnect occurred where the The Aversa and Bestavros paper show a TCP performance achieved last publications was 62x higher connection load of between 496 and 663 requests than the initial publications, while other publications served. The mean response time is between were only showing a 4x change in performance. The 0.92s and 0.26s depending on whether load Web Server industry that uses the SPECweb2005™ balancing is used or not. The Grottke publications to evaluate hardware and software paper7showed a maximum connection rate of 390 combinations to purchase should understand this, as there connections/second (c/s) for their analysis. is an alternative available. Additionally, the Cao paper8 measured connection rates of between 0.18 c/s and 34 c/s on one Using the SPECweb2005®/PHP kit on equivalent network and between 2.41 c/s and 230 c/s on systems used for the SPECweb2005®/JSP™ publications another. From these papers it is obvious that we are finding that the change in connection loads connection rates on real web servers and networks observed is not as high. The Intel® Xeon® X5355 are generally much lower than those measured by processor achieving performance in a range from 18000 the top SPECweb2005™ results. For this reason to 20000, using the SPECweb2005®/PHP kit achieves it seems reasonable to look at a scenario where around 1000 connections. Oddly enough, this is the top connection rates are measure to be in the approximately the same level of connections being 2000-3000 connections/second range. observed in various university publications. 6 Having SPECweb2005™ scores that more closely match “Load Balancing a Cluster of Web Servers: Using Distributed Packet Rewriting”, Luis Aversa and Azer the IT industry would help in the purchase evaluation Bestavros cycle. Equivalently the computer manufacturing industry 7 “Analysis of Software Aging in a Web Server, Michael would be better served by helping understand the true Grottke, Lei Li, Kalyanaraman Vaidyanathan, and Kishor S. benefit being realized from the new products launched in Trivedi 8 “Stochastic Models for Generating synthetic HTTP Source Traffic, Jin Cao, William S. Cleveland, Yuan Gao, Kevin 5 See IDC publications in press in June 2007 Jeffay, F. Donelson Smith, Michele Weigle Table 4 - Latest data on Market Share via NetCraft Table 5 - Dual Scket Class Server Details SPECweb2005 Score Support Bank Ecommerce Simultaneous Connections Geomean X5350 964 1900 750 1400 1259 X5355 1082 2200 800 1600 1412 II.B.2 Discussion of use of IIS II.B.3 Discussion of use of PHP Because the Sun™ Java System Web Server results were A variety of marketing companies have analyzed run on Solaris™ 10 and the other results use Rock Web the percentage of the market share for the various Server v1.4, it is useful to look at results for a widely content development languages. These research used web server. For this paper Microsoft® Internet firms show levels from 35 to 75%, depending Information Server (IIS) was chosen. According to primarily on whether the analysis includes all various Web Server market analysis companies known websites or newer websites actively being approximately 35% of web servers on the internet use developed.9 The consistent trend in the market IIS. A quick interpretation of the trend across the set of place has been that website content development Web Server market analysis reports also reveals that IIS has been increasing from ~25% in 200110 to tends to be used in Websites with more traffic. The ~50% PHP content developed in 2007.11 There important point though is none of these web server are a variety of reasons a developer would market analysis companies are able to measure the gravitate to one language over another. market share represented by RockWeb, and this the Familiarity, ease of development, comprehension webserver product used in all but a handful of the present of the API set available, and time available to SPECweb2005™ submissions. develop, all are components. For this paper we won’t try to understand why PHP is chosen, While Netcraft data doesn’t automatically split out active simply we are finding from various publications and inactive sites, we can see from the latest data that this development language is used. available that around 36% of the Web Server market is IIS (see Table 3). Observing top five Web Servers used, III. SPECweb2005®/PHP RockWeb doesn’t appear. This in combination with the broader set of data in Figure 2 would leave one to wonder Results and Analysis how the IT industry correlates the performance reported The external publications not using from SPECweb2005™ submissions to their own SPECweb2005™ show that at real customer sites environment. the number of connections per webserver is in the range of 2000-3000 connections range. At the same time, SPECweb2005®/JSP™ publications are showing connections ranges approaching 40,000. By concentrating a few of these 9 http://www.nexen.net/chiffres_cles/phpversion/php_statistic s_for_april_2006.php#global 10 http://www.imakenews.com/badblue/e_article000044504.cf m 11 See IDC publications in press for June 2007. SPECweb2005 Score Support Bank Ecommerce Simultaneous Connections Ge 7140M 1141 2100 900 1750 1490 Table 7 - Four-Socket class server details submissions and producing results using the SPECweb2005®/PHP kit we find promising data though, PHP JSP as the number of connection drops dramatically to around 1000-2500 users depending on the workload used. The SPECweb2005 Avg Resp Avg Resp Geomean and reference platform pull the arithmetic value Time Time down making the result unit-less, yet the individual (sec) (sec) Support 5.133 6.334 workload results still retain the number of simultaneous Ecommerce 1.432 1.511 Banking 0.652 1.366 connections. The representativeness of Table 6 - SPECweb2005™ Response Time comparison SPECweb2005®’s core design for the three workloads, coupled with an implementation language that has III.A Dual-Socket Results equivalent PathLength12results in connection loads more similar to the connection loads encountered by end users. The dual-socket system chosen to use was a The key value in a benchmark is its ability to accurately Intel® Xeon® 5355 processor dual socket predict for the market segment intended the value of the platform. Here the equivalent results can be seen new hardware or software component, and with the in Table 4. The Support workload achieved the SPECweb2005®/PHP kit there appears to be more highest connection load, with Banking producing similarity. the lowest connection load. Each workload has different characteristics, and based on the Web Another, interesting result from using the Server and Operating System combinations the SPECweb2005®/PHP kit is that the hardware bottlenecks encountered can differ. Yet with this requirements to achieve the peak result have decreased Web Server and Operating System combination dramatically. While each customer tends to by unique the performance achieved is with in the range of combinations of components on their systems, the approximately 1000-2000 users for a dual socket standard webserver doesn’t tend to have the same level of class system (see Table 5) hardware as that required to presently achieve the leading SPECweb2005®/JSP™ results. Rather, the systems The Support and Ecommerce workloads achieved purchase tends to more closely match what are processor utilizations that exceeded 95%, and the considered the default configurations available by most Banking workload achieve processor utilizations OEMs. When we compare the hardware requirements of above 80%. The responses times for Ecommerce the SPECweb2005®/PHP results we find that they are are nearly equivalent to the closer to these default OEM configurations. SPECweb2005®/JSP™ results, and the similarly for the Support which were approximately 0.83x Though there are many ways to produce a result at that of JSP™. The response times for Banking connection loads in the range of 2000-3000 using were quite a bit lower than that of the JSP™ SPECweb2005®. By looking further into the workload, achieving 0.47x that of JSP™. The SPECweb2005®/PHP results we should be able to Banking workload using the observe if the system under test was loaded to the SPECweb2005®/PHP kit has software maximum possible load, or simply under-loaded to serialization, in part due to the session state achieve the result. By doing this we help insure that the having higher processing and IO requirements benchmarks ability to predict end user performance, even that limits the processor utilization achievable. in scenarios where market conditions motivate OEMS to The key issue is that across the three workloads achieve peak results, will still produce customer relevant the request response times are nearly the same, or results. with understood reasons for the differences. (see Table 6) 12 Instructions retired by the processor per request Table 8 – Summary of Hardware used to Achieve SPECweb2005 Results kit variant Memory Network physical disks SPECweb2005 Score 4Socket PHP 32 3 4 1083 JSP 64 17 58 18981 2Socket PHP 16 2 4 1083 JSP 32 9 58 20387 JSP™ results require more than 5-18 subnets III.B Multi-Socket Results worth of unique 1Gb conduits into the server to Equivalent characteristics observed in two-socket supply enough bandwidth for the number of systems are observed as one measures a four socket sessions required for the benchmark. For PHP system. When we used an Intel® Xeon® 7140M class that number drops to two 1GbE conduits into the system the connection rates improved slightly for server for the information presented in this paper. Ecommerce and Banking, and had a slight degradation on All results presented used an optimum memory the Support workload. (see Table 7) configuration of sixteen 1GB memory DIMMS for a total of 16GB of memory. The systems Equivalent to the two sockets, the Banking workload had under test used four 80G 15K RPM SATA drives. high software serialization which inhibited the ability of One drive was used for each of the following; the the system to achieve higher than 70% processor operating system, paging space, PHP session state utilization; while Support and Ecommerce achieved information, and benchmark data files. Besides higher than 90% utilization. the increased bandwidth for the network to achieve a top performing result, the JSP™ III.C Comparison of configurations to submissions also required more client systems to simulate the load, and up to ten times the storage. published results The reason for the increased number of physical SPECweb2005™ offers two different implementations of disks is to help provide enough disk I/O capacity the logic that comprises the workloads for the market in order to accommodate the increased demand segments using the technologies of PHP and JSP™13. for content on disk. The large data set for content Currently, all submissions for SPECweb2005™ have grows as a function of the number of users used JSP™ to serve dynamic content. One of the major supported by a SUT (see Table 8) determining factors for each Original Equipment Manufacturers (OEMs) choice to use the JSP™ kit is that the path lengths, or number of instructions required to generate a response, for JSP™ to serve the dynamic IV. Conclusion content are much shorter than for PHP to perform the By using the SPECweb2005®/PHP kit a new same task. When using the PHP variant of system can be evaluated at a level of performance SPECweb2005™ the results are lower as a direct result with respect to connection loads and response of the number of instructions required to generate a times that more closely matches a real world response (see Table 6). The net result of this is PHP scenario. Present results using achieves approximately 10-11x higher performance than SPECweb2005®/JSP™ are able to show system the reference platform. performance, and show improvements achievable from the new hardware. However, the There is a significant contrast between the hardware performance gains realized, both in terms of the requirements required for a top performing JSP™ result connection loads and corresponding response time and a top performing PHP result. The top performing effects are more have a better representation to the end user scenarios. 13 As of Dec 2007 publications made using these two kit variations are identified by either SPECweb2005/PHP or SPECweb2005/JSP™ respectively.