Sun responds to Microsoft on WSTest
Posted By: Dennis MacNeil on August 03, 2004 @ 05:30 PM
On June 23, 2004, Sun published a whitepaper showcasing superior JavaTM 2 Enterprise Edition (J2EETM) web services performance when compared to Microsoft's .NET. See http://java.sun.com/performance/reference/whitepapers/WS_Test-1_0.pdf for the original whitepaper.

Microsoft published a response on July 14th at http://www.theserverside.net/articles/showarticle.tss?id=SunBenchmarkResponse refuting our claims and stating that in their tests .NET performance was higher than Java Web Services..

Issues with Microsoft's Response

In their response, Microsoft make several statements that we shall examine here :

Database interactions

Microsoft claims that Sun's tests are over simplistic and do not provide customers with useful information because they do not include any database interactions.

In our paper, we made clear that the purpose of the test was to measure the performance of the web services infrastructure. We quote : “To avoid the effects of other platform components, the web service methods perform no business logic but simply return the parameters that were passed in.”
WSTest is a micro-benchmark with a specific purpose to measure the basic performance of the underlying web services infrastructure.

Lightweight application server

Microsoft claims that by using Tomcat instead of a commercial J2EE application server, the results are somehow irrelevant.
Our purpose in using Tomcat (which is bundled with JWSDP) was to make it easier for a customer to run the benchmark. Considering that this benchmark makes very light use of web server functionality, we believe that the web server used is immaterial. Note that Tomcat is used only as a http server; the actual web services functionality is implemented within JWSDP.

Doculabs benchmark

Microsoft makes reference to the Doculabs web services benchmark report that was published more than a year ago. This report is irrelevant for the purposes of this discussion as it did not test any J2EE compliant web services implementations, as at the time of publication, there were no such products. (The products tested were not JAX-RPC 1.1 compliant).

Tuning

We discovered that we did not increase the number of network connections on the .NET client side. We added the following to machine.config, and re-ran the tests using J2SE 5.0 Beta2 and found that the measured throughput and response times obtained are still superior to the .NET results

Conclusion

The only way to see eye to eye on a benchmark, is if it is developed by consensus in an industry standards organization that is experienced in developing benchmarks. We would like to once again invite Microsoft to participate in such an effort at SPEC. See http://www.spec.org/benchmarks.html#esp and http://www.eweek.com/article2/0,1759,1522049,00.asp.

Dennis MacNeil

Senior Product Marketing Manager
Java2 Platform, Enterprise Edition
Sun Microsystems
--
修改:dev FROM 61.49.126.*
FROM 61.49.126.*