ColdFusion MX 6.1 Performance Brief
August, 2003
Copyright © 2003 Macromedia, Inc. All rights reserved. The information contained in this document represents the current view of Macromedia on the issue discussed as of the date of publication. Because Macromedia must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Macromedia, and Macromedia cannot guarantee the accuracy of any information presented after the date of publication. This white paper is for information purposes only. MACROMEDIA MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. Macromedia may have patents, patent applications, trademark, copyright or other intellectual property rights covering the subject matter of this document. Except as expressly provided in any written license agreement from Macromedia, the furnishing of this document does not give you any license to these patents, trademarks, copyrights or other intellectual property. Macromedia ColdFusion MX are either trademarks or registered trademarks of Macromedia, Inc. in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Macromedia, Inc. 600 Townsend Street San Francisco, CA 94103 415–252–2000
ColdFusion MX 6.1 Performance Brief Executive Summary
Contents
Executive Summary........................................................................................................ 2 Statistics Summary .....................................................................................................2 Defining Application Performance ................................................................................. 3 Performance Enhancements in ColdFusion MX 6.1 .....................................................3 Test Configuration and Application..............................................................................4 Test Results .................................................................................................................... 6 Windows 2000.............................................................................................................6 Windows 2003.............................................................................................................7 Red Hat Advanced Server 2.1.......................................................................................8 Solaris 9 ......................................................................................................................9 CFMAIL Performance Gains .......................................................................................10 Summary ........................................................................................................................11 Appendix........................................................................................................................12 Test Data – Windows 2000 ........................................................................................12 Test Data – Windows 2003 ........................................................................................12 Test Data – Red Hat Advanced Server 2.1 ..................................................................12 Test Data – Solaris 9..................................................................................................13 Email Performance Test Notes ...................................................................................13
August, 2003
Page 1
ColdFusion MX 6.1 Performance Brief Defining Application Performance
Executive Summary
Due to enhancements to the runtime engine, ColdFusion MX 6.1 is capable of processing significantly more page requests than any previous release of ColdFusion, as measured by reductions in response time. As a result of these enhancements, customers can now significantly increase their server capacity and provide improved response times to the end users.
Statistics Summary
• Windows 2000: Approximately 23 times faster than ColdFusion 4.5, 3 times faster than ColdFusion 5, and 2.5 times faster than ColdFusion MX • Windows 2003: Approximately 3 times faster than ColdFusion 5 • Linux: Approximately 5 times faster than ColdFusion 5 • Solaris: Approximately 4 times faster than ColdFusion 5
Page 2
August, 2003
ColdFusion MX 6.1 Performance Brief Defining Application Performance
Defining Application Performance
A high performing application is able to deliver content to users quickly. In this brief, application performance is analyzed by measuring page response time: the elapsed time between the submission of a request (the clicking of a submit button, the manual entry of a URL, or the clicking of a link by a user) and the successful completion of that request. A lower response time per request (completing the request more quickly) allows the application to deliver more content overall to more users in a given period of time (throughput). There are three basic factors that influence response time performance: • Web application server architecture and configuration • Network infrastructure • Web page design All three factors are of considerable importance, and no Web application will function efficiently if any factor is neglected. However, since network bandwidth and Web page design are not functions of the web application server, this brief focuses solely on the changes in page response time that result from enhancements to ColdFusion. This brief illustrates application runtime performance gains in ColdFusion MX 6.1 by comparing the average page response times in a test application across multiple past versions of the product with ColdFusion MX 6.1. The application used in these tests makes very intensive use of the CFML language, as would be the case in many large-scale content-management type applications. Applications that are more dependent on database transactions or I/O operations may experience less significant performance gains than the application tested for this paper, as those factors are less dependent on the application server.
Performance Enhancements in ColdFusion MX 6.1
Extensive engineering effort went into internal performance enhancements for the version 6.1 release of ColdFusion MX. The entire ColdFusion MX codebase was analyzed by running real-world applications on special ColdFusion MX 6.1 installs built with instrumentation. In all, nearly one hundred customer applications consisting of approximately 2.4 million lines of CFML code were used for analyzing CFML usage patterns and to look for bottlenecks in realworld customer settings. Where issues were revealed, the server runtime code was then tuned to eliminate them wherever possible. As each bottleneck was uncovered and removed, each change was measured individually to gauge its overall performance impact on the server. Most of the changes were small, resulting in incremental improvements, sometimes as small as 1-2%, but their cumulative effect was dramatic.
August, 2003
Page 3
ColdFusion MX 6.1 Performance Brief Defining Application Performance
In addition to internal tuning, ColdFusion MX 6.1 also includes upgrades to various integrated third-party components that directly impact runtime performance. ColdFusion MX 6.1 was optimized for and ships with the new Sun 1.4.2 Java virtual machine, or JVM (also known as the Java 2 Platform, Standard Edition version 1.4.2). With the 1.4.2 release of the Sun JVM, many performance and stability enhancements from Sun are now available to ColdFusion customers. Performance was a major goal for Sun Microsystems with the release of 1.4.2, and by working directly with Sun and the 1.4.2 JVM early in its release cycle, optimal integration of ColdFusion MX 6.1 and the 1.4.2 JVM was achieved. ColdFusion MX 6.1 also provides support for the latest versions of several vendor operating systems, including Microsoft Windows Server 2003, SuSE Enterprise Server 8.2, Red Hat Enterprise Server AS 2.1, Sun Solaris 9, IBM AIX 5L for Power v 5.2, and HP-UX 11i. This allows ColdFusion customers to take advantage of performance improvements in the latest operating system versions. For example, Windows Server 2003 was measured as three times faster than Windows NT 4 on the same hardware for serving dynamic web content. Finally, ColdFusion MX now includes version 3.2 of the DataDirect Connect for JDBC drivers, which provide additional performance optimizations for database connectivity. While these results illustrate performance improvements in deployed applications, significant improvement has also been made in the product’s compiler performance. The ColdFusion compiler’s job is to convert CFML source code into Java bytecode. In ColdFusion MX 6.1, dramatic increases in compilation speed were realized by reengineering the compiler to compile CFML directly into Java bytecode without producing any Java source code. The result is a 10-fold increase in compiler performance, and a much better development experience, making it easier for developers to build and debug applications without having to wait for pages to recompile.
Test Configuration and Application
ColdFusion MX 6.1 may be deployed in a number of different configurations depending on needs and server environment. The two main types of deployment in the Enterprise edition of the product are standalone, or server configuration, and J2EE configuration. In the server configuration, the underlying J2EE application server (Macromedia JRun) is not exposed to the user or application administrator. In the J2EE configuration, ColdFusion is deployed on an existing installation of a standard J2EE application server such as Macromedia JRun (a full license for which is included with ColdFusion MX Enterprise 6.1), IBM WebSphere, SunONE, or BEA WebLogic. Since this latter deployment option was not available in prior versions of ColdFusion, the standalone configuration was used in this testing. That allowed ColdFusion MX 6.1 to more closely resemble the deployment of ColdFusion 4.5, 5, and MX.
Page 4
August, 2003
ColdFusion MX 6.1 Performance Brief Test Results
Testing for this brief was performed in Macromedia’s Enterprise Testing facility in Newton Massachusetts. The application used in the test is the iBuild sample application, which originally shipped as a sample of the Macromedia Spectra application framework. The application represents a typical content-centric Web application found on the Internet or intranets, and makes extensive use of CFML tags and custom tags. While actual performance gains experienced with other applications will vary depending on scope and complexity, it is important to note that the iBuild application was not performance-tuned specifically for this paper, and as a result, real-life customer applications may or may not realize similar results with similar applications and server configurations. For the purposes of testing, simultaneous HTTP requests were generated to simulate 30 no-think-time virtual users. This simulated activity at thresholds of between approximately two to six million requests per day. Each stress test was performed multiple times to confirm accuracy. Averaged performance numbers were used for this document. In real-world Web applications, most of the session time is spent waiting for the user to do something. This is particularly true for content-centric sites, where a large portion of time is spent reading web pages. A large Web site or application may have hundreds of open user sessions, while only a few dozen are actively making requests to the application server. Recreating realistic user pauses and mistakes is very difficult, so for the purposes of this test, user pauses were removed altogether. Thus, 30 simultaneous virtual user sessions does not represent 30 simultaneous users. While the exact ratio of open sessions to active sessions is difficult to generalize because of the disparity in Web applications, a test using 30 virtual users would roughly translate to 600 open user sessions.
August, 2003
Page 5
ColdFusion MX 6.1 Performance Brief Test Results
Test Results
Windows 2000
Performance evaluation was conducted between ColdFusion 4.5, ColdFusion 5, ColdFusion MX, and ColdFusion MX 6.1 on Windows 2000 with the following specifications: • • • • • Windows 2000 SP3 Microsoft Internet Information Server (IIS) 5.0 Hewlett-Packard Proliant DL380 G3 Server 2 x 2.8 GHz Xeon MP processors (hyper-threading enabled) 2560 MB RAM
The back-end database for each test suite was a separate Compaq 6500 with four 500MHz Pentium III Processors running Microsoft SQL Server 2000. All tests were performed using 30 no-think-time virtual users (VUs) simulated with Segue SilkPerformer 5.1.
Page 6
August, 2003
ColdFusion MX 6.1 Performance Brief Test Results
Windows 2000 Response Time Gains
3 2.5 Avg. Resp. Time (sec.) 2 1.6022 1.5 1 0.5875 0.5 0 ColdFusion 4.5 ColdFusion 5 ColdFusion MX ColdFusion MX 6.1 1.5311 13.6171
Product Version
Figure 1: Response time reductions in ColdFusion MX 6.1 compared to previous releases on Windows 2000
As illustrated in figure 1, the optimizations in ColdFusion MX 6.1 resulted in a significant reduction in response time compared with all previous releases of ColdFusion. Pages were delivered approximately 2.7 times faster in ColdFusion MX 6.1 than in ColdFusion 5 (1.6022/0.5875=2.7271) and more than 23 times faster than in ColdFusion 4.5 (13.6171/0.5875 = 23.17).
Windows 2003
Performance evaluation was conducted between ColdFusion 5 and ColdFusion MX 6.1 on Windows 2003 with the following specifications: • • • • • Windows Server 2003 Web Edition Microsoft Internet Information Server (IIS) 6.0 Compaq Proliant 1850 Server 2 x 500MHz PIII Xeon processors 1024 MB RAM
August, 2003
Page 7
ColdFusion MX 6.1 Performance Brief Test Results
The back-end database for each test suite was a separate Compaq 6500 with four 500MHz Pentium III Processors running Microsoft SQL Server 2000. All tests were performed using 30 no-think-time virtual users (VUs) simulated with Segue SilkPerformer 5.1.
Windows 2003 Response Time Gains
7 Avg. Resp. Time (sec.) 6 5 4 3 2 1 0 ColdFusion 5 Product Version
Figure 2: Response time reductions in ColdFusion MX 6.1 compared to previous releases on Windows 2003
ColdFusion MX 6.1
Figure 2 illustrates how the average page response time with ColdFusion MX 6.1 on Windows Server 2003 was approximately one third of the average page response time of ColdFusion 5. Pages were delivered approximately 3 times faster in ColdFusion MX 6.1 than in ColdFusion 5 (6.0227/2.0681=2.912).
Red Hat Advanced Server 2.1
Performance evaluation was conducted between ColdFusion 5, ColdFusion MX and ColdFusion MX 6.1 on Linux with the following specifications: • • • • • RedHat Enterprise Linux AS 2.1 Apache 1.3.27 Compaq Proliant DL360 Server 2 x 733 MHz PIII Xeon processors 2048 MB RAM
The back-end database for each test suite was a separate Compaq 6500 with four 500MHz Pentium III Processors running Microsoft SQL Server 2000. All tests were performed using 30 no-think-time virtual users (VUs) simulated with Segue SilkPerformer 5.1.
Page 8
August, 2003
ColdFusion MX 6.1 Performance Brief Test Results
Linux Response Time Gains
Avg. Resp. Time (sec.) 16 14 12 10 8 6 4 2 0 ColdFusion 5 ColdFusion MX Product Version
Figure 3: Response time reductions in ColdFusion MX 6.1 compared to previous releases on
ColdFusion MX 6.1
Linux
As illustrated in figure 3, the optimizations in ColdFusion MX 6.1 resulted in a significant reduction in response time compared with all previous releases of ColdFusion on Linux. Pages were delivered approximately 5 times faster in ColdFusion MX 6.1 than in ColdFusion 5 (13.525/2.7271=4.959) and more than 2.7 times faster than in ColdFusion MX (7.3461/2.7271 = 2.693).
Solaris 9
Performance evaluation was conducted between ColdFusion 5, ColdFusion MX and ColdFusion MX 6.1 on Solaris with the following specifications: • • • • • Solaris 9 iPlanet 6.0 SP5 Web Server Sun E220 Server 2 x 450MHz SparcV9 processors 512 MB RAM
The back-end database for each test suite was a separate Compaq 6500 with four 500MHz Pentium III Processors running Microsoft SQL Server 2000. All tests were performed using 30 no-think-time virtual users (VUs) simulated with Segue SilkPerformer 5.1.
August, 2003
Page 9
ColdFusion MX 6.1 Performance Brief Test Results
Solaris Response Time Gains
Avg. Resp. Time (sec.) 8 7 6 5 4 3 2 1 0 ColdFusion 5 ColdFusion MX Product Version
Figure 4: Response time reductions in ColdFusion MX 6.1 compared to previous releases on
ColdFusion MX 6.1
Solaris
As illustrated in figure 4, the optimizations in ColdFusion MX 6.1 resulted in a significant reduction in response time compared with all previous releases of ColdFusion on Solaris. Pages were delivered approximately 4 times faster in ColdFusion MX 6.1 than in ColdFusion 5 (6.9996/1.7838 =3.92) and more than 2.5 times faster than in ColdFusion MX (4.3391/1.7838 = 2.43).
CFMAIL Performance Gains
While the ability to send dynamically generated email from ColdFusion has been a core feature of the product since its inception, ColdFusion MX 6.1 dramatically improves the performance and stability of this feature. Email delivery capability has been increased by as much as 50 times more than ColdFusion 5 and the server can automatically failover to a back-up mail server if the primary server becomes unavailable. In testing performed in Macromedia’s Enterprise Testing facility, a dual processor Hewlett-Packard Proliant DL380 G3, ColdFusion MX Enterprise 6.1 was able to deliver 1,241,379 dynamically generated email messages per hour (see figure 5). This dramatic improvement in throughput was achieved by through changes in the mail delivery architecture. ColdFusion MX and earlier versions had a single thread for delivering mail to the mail queue and a single thread for delivering mail from the queue to the SMTP server. ColdFusion MX 6.1 provides multiple delivery threads. The number of threads used for each of these functions can now be configured in the ColdFusion Administrator and the ColdFusion server now pools the connections for reuse. In many cases, adjusting these thread-level settings high enough can result in sending mail faster than can be processed by the mail server.
Page 10
August, 2003
ColdFusion MX 6.1 Performance Brief Appendix
Note: Connection pooling and multiple delivery threads, in addition to the option to spool messages to memory (instead of disk) are features of ColdFusion MX Enterprise 6.1.
Email Messages per Hour
1,400,000 1,200,000 Number sent 1,000,000 800,000 600,000 400,000 200,000 0 ColdFusion 5 ColdFusion MX Product Version
Figure 5: Number of dynamically generated email messages sent per hour from ColdFusion MX 6.1 compared to previous versions.
1,241,379
16,285
25,904 ColdFusion MX 6.1
Summary
ColdFusion MX 6.1 enables developers to build and deploy Internet applications and web services that perform with exceptional speed and stability on servers under load. The combination of enhancements to runtime performance and compiler performance makes ColdFusion MX 6.1 the best version of ColdFusion for both developer and user experience.
August, 2003
Page 11
ColdFusion MX 6.1 Performance Brief Appendix
Appendix
Test Data – Windows 2000
Response Time Data, 30 Virtual Users (VU)
Product Version # Seconds
ColdFusion 4.5 13.6171
ColdFusion 5 1.6022
ColdFusion MX 1.5311
ColdFusion MX 6.1 0.5875
(lower number indicates better performance)
Test Data – Windows 2003
Response Time Data, 30 Virtual Users (VU)
Product Version # Seconds
ColdFusion 5 6.0227
ColdFusion MX 6.1 2.0681
(lower number indicates better performance)
Test Data – Red Hat Advanced Server 2.1
Response Time Data, 30 Virtual Users (VU)
Product Version # Seconds
ColdFusion 5 13.525
ColdFusion MX 7.3461
ColdFusion MX 6.1 2.7271
Page 12
August, 2003
ColdFusion MX 6.1 Performance Brief Appendix
(lower number indicates better performance)
Test Data – Solaris 9
Response Time Data, 30 Virtual Users (VU)
Product Version # Seconds
ColdFusion 5 6.9996
ColdFusion MX 4.3391
ColdFusion MX 6.1 1.7838
(lower number indicates better performance)
Email Performance Test Notes
The following configuration was used for the email performance testing: • dual processor Hewlett-Packard Proliant DL380 G3 • disk spooling disabled • 30 delivery threads used • “spool interval” of 15 seconds • dedicated SMTP server running Qmail software
August, 2003
Page 13