Java web services CXF performance comparison

							The Apache CXF web services stack builds on some of the same technologies
as the Apache Axis2 and Metro stacks discussed in earlier articles of
this series. Like Axis2, it uses the Apache WSS4J WS-Security
implementation. Like Metro, it predominantly uses JAX-WS 2.x web services
configuration and JAXB 2.x data binding (even using the same reference
implementation of JAXB as Metro, though versions may differ between the
two stacks). Beyond these common components, however, the stacks differ
in many ways, including their processing engines and WS-SecurityPolicy
configuration handling.
Past articles in this series have compared Axis2's performance with
Metro's, both for simple message exchanges and with different WS-Security
configurations. In this article, you'll see how CXF performance compares
with the most-recent releases of both Axis2 and Metro.
About this series
Web services are a crucial part of Java technology's role in enterprise
computing. In this series of articles, XML and web services consultant
Dennis Sosnoski covers the major frameworks and technologies that are
important to Java developers using web services. Follow the series to
stay informed of the latest developments in the field and aware of how
you can use them to aid your programming projects.
Looking at performance
Just like the earlier articles on web services performance ¡ª "The high
cost of (WS-)Security " and "Metro vs. Axis2 performance" ¡ª this article
takes the approach of measuring the time required to execute a particular
sequence of requests when both the client and the server run on a single
system. This approach does a great job of comparing web services
processing overhead, because it eliminates the impact of network
latencies and overhead from timing results. Assuming the client code
isn't significantly slower than the server, the figures are also good
representatations of actual server performance under load.
This article also uses the same test application as the earlier articles,
a seismic data retrieval service. The service uses a database of more
than 93,000 earthquakes that occurred worldwide over a period of years.
Requests to the service specify a time range and a geographic coordinate
range, and the service returns all earthquakes within the specified
range. See "The high cost of (WS-)Security" for full details of the test
application and a sample request/response message pair.
As in the previous articles, two sets of request sequences are used for
the performance tests. The first set uses 1,000 requests with query
parameters adjusted to match a small portion of the entire earthquake
database (returning only 816 matching earthquakes for the 1,000
requests). The second set uses 100 requests adjusted to match a larger
portion of the database (returning 176,745 matching earthquakes for the
100 requests). These two request sequences emphasize different
performance characteristics of web services stacks. The first one shows
how quickly stacks process requests with little data, and the second
emphasizes the speed of processing data volumes. Each request sequence
was run multiple times in different security configurations with only the
best time for each configuration kept in the results.
The tests were run on a Mandriva 2009.1 64-bit Linux system with an
Athlon X2 5400+ processor and 4GB of RAM, using a Sun (Oracle) Java
1.6.0_18 32-bit JVM (which gave slightly better performance than the 64-
bit JVM for a given heap size). The server code was run on Tomcat 6.0.20,
configured to use 1024MB of heap, with the client code using 512MB of
heap. The web service stack versions were:
CXF 2.1.7Metro 2.0Axis2 1.5.1 with the 1.5 release of RampartIf you want
to try the tests on your own hardware and JVM, download the code.
Back to top
Performance without WS-Security
Figure 1 shows the measured test times for Axis2, Metro, and CXF without
any WS-Security usage. As you can see from the chart, Metro is
significantly faster than Axis2 and CXF (about 25 percent faster) for the
case of many requests with small responses, and about the same speed as
the Apache stacks for the lower number of requests with larger responses.
(In all of this article's charts, shorter bars are better because they
indicate faster times.)
Figure 1. Test times without securityThese results show some interesting
differences from the earlier comparison between Metro 1.5 and Axis2
1.5.1. The timings suggest that, at least for the data used by the test
application, Metro 2.0 is faster than Axis2 and CXF in per-request
processing. In data conversions to and from XML, the three stacks all run
at about the same speed. This is to be expected in the case of Metro and
CXF, since they both use the JAXB reference implementation for
conversions. Judging from these results, the Axis2 Databinding Framework
(ADB) binding implementation used by default with Axis2 runs about as
fast as JAXB.
Back to top
Performance with WS-Security
The next figures show test times for the following security
configurations:
plain: No security (the same values as in Figure 1)username: WS-Security
plain-text UsernameToken on requestssign: WS-Security signing of body and
headers, with timestampsignencr: WS-Security signing of body and headers,
with timestamp, and encryption of bodyFigure 2 shows the measured test
times for 1,000 requests with small responses:
Figure 2. Small-response measured timesFigure 3 shows the relative times
(normalized to the CXF results) for the same 1,000 requests with small
responses:
Figure 3. Small-response normalized timesAxis2 is consistently the
slowest of the stacks across all security configurations for this test
case. Metro is consistently the fastest, though the difference between
Metro and CXF is much smaller than the difference between CXF and Axis2:
Metro is about 10 percent faster than CXF across the different security
configurations, whereas Axis2 is more than twice as slow as CXF.
Figure 4 shows the measured test times for 100 requests with larger
responses:
Figure 4. Large-response measured timesFigure 5 shows the relative times
(normalized to the CXF results) for the same 100 requests with larger
responses:
Figure 5. Large-response normalized timesFor this second test case, Axis2
is again much slower than Metro and CXF (again about half the speed of
CXF), and the difference between Metro and CXF in the smaller-response
messages is more than reversed, with CXF about 15 percent faster overall.
CXF 2.1.7 vs. 2.1.6
These test times reflect a noticeable performance improvement for CXF
between versions 2.1.6 and 2.1.7. Code tuning is partly responsible for
this, but a more significant portion of the improvement is due to the fix
for a problem discussed in "WS-Security with CXF." CXF 2.1.6 did not
process a UsernameToken WS-SecurityPolicy configuration unless a
<sp:TransportBinding> policy or some form of encryption or signing policy
was present. Normally such an added policy would be present when
UsernameToken is used ¡ª especially a plain-text UsernameToken because
without either transport- or message-level encryption the username and
password can be observed in transit. But it's perfectly valid from a
policy standpoint to use UsernameToken on its own (as in this article's
username configuration), and the fix implemented for CXF 2.1.7 handles
this case. As part of the fix, CXF 2.1.7 skips adding WS-Security
handling to the response message flow in this particular case.
Removal of WS-Security from the message flow in the username
configuration improved CXF's results by several percentage points overall
compared to the same tests run with CXF 2.1.6. That improvement between
versions is, unfortunately, somewhat artificial. If the usernaeme test
case used either transport- or message-level encryption, WS-Security
handling would be present in the response message flow and cause the CXF
timing results for that configuration to be noticeably slower. Hopefully,
future CXF versions will extend the policy analysis so that WS-Security
is configured in either request or response message flows only when it is
needed, thereby extending the performance benefit to a wider range of use
cases (including those in which signing or encryption is needed in only
one direction).
Back to top
Conclusions
Based on the test timings reported in this article, it looks like Metro
2.0 is faster in basic request/response processing than either Axis2
1.5.1 or CXF 2.1.7. When it comes to WS-Security processing, though,
Metro's XWSS library is no faster overall than the WSS4J library used by
both Axis2 and CXF.
Probably the most interesting aspect of these results is the implication
for Axis2. Earlier tests showed that Axis2 was much slower than Metro in
WS-Security processing. These test results show that the difference is
not due to the WS-Security implementation code, so it must be due to the
way Axis2 (and the Rampart security module) handle messages being passed
to and from WSS4J. This confirms the results discussed in "Metro vs.
Axis2 performance." Axis2 also requires significantly more memory to run
the tests than the other stacks, most noticably for the signenecr
configuration (Axis2 needed about 80MB of memory on the client; CXF ran
fine with 48MB). These issues reinforce the earlier impression that Axis2
wastes processing time and memory doing unnecessary conversions between
different message representations as part of the WS-Security handling.
The performance overhead of using WS-Security signing and encryption is
substantial for all the stacks tested. This overhead can be problematic
for services with heavy usage. In the next article, you'll learn about
the WS-Trust and WS-SecureConversation add-on technologies, which can
reduce the WS-Security overhead when clients make extensive use of a
particular service.
Back to top
Download
DescriptionNameSizeDownload methodSource code for this article
j-jws14-src.zip
4.86MB
HTTP
Information about download methods