Response to IBM's Whitepaper Entitled Benchmarking and ...
47
pages
English
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
47
pages
English
Ebook
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
MICROSOFT
Response to IBM’s Whitepaper Entitled
Benchmarking and Beating Microsoft .NET
3.5 with WebSphere 7
7/2/2009
This document is a response to an IBM Benchmark Rebuttal Document to our original benchmark results
published at http://msdn.microsoft.com/stocktrader. Microsoft stands behind the original test results
published there. This document is the Microsoft response, point-for-point, to the IBM response to our
original results, and includes new benchmark data as further evidence. Contents
Executive Summary ....................................................................................................................................... 4
Introduction .................. 6
The Microsoft Findings.. 8
Response to IBM Rebuttal ............................................................................................................................ 9
IBM Friendly Bank Benchmark .................. 9
Microsoft Comments on IBM Friendly Bank Rebuttal Benchmark ........................... 9
IBM CPO StockTrader/.NET Benchmark.................................................................................................. 11
Microsoft Comments on IBM CPO StockTrader Rebuttal Benchmark .................... 12
Summary ..................................................................................................................................................... 19
Appendix A: Microsoft .NET StockTrader and WebSphere Trade 7 Test ...
Voir
Response to IBM’s Whitepaper Entitled
Benchmarking and Beating Microsoft .NET
3.5 with WebSphere 7
7/2/2009
This document is a response to an IBM Benchmark Rebuttal Document to our original benchmark results
published at http://msdn.microsoft.com/stocktrader. Microsoft stands behind the original test results
published there. This document is the Microsoft response, point-for-point, to the IBM response to our
original results, and includes new benchmark data as further evidence. Contents
Executive Summary ....................................................................................................................................... 4
Introduction .................. 6
The Microsoft Findings.. 8
Response to IBM Rebuttal ............................................................................................................................ 9
IBM Friendly Bank Benchmark .................. 9
Microsoft Comments on IBM Friendly Bank Rebuttal Benchmark ........................... 9
IBM CPO StockTrader/.NET Benchmark.................................................................................................. 11
Microsoft Comments on IBM CPO StockTrader Rebuttal Benchmark .................... 12
Summary ..................................................................................................................................................... 19
Appendix A: Microsoft .NET StockTrader and WebSphere Trade 7 Test ...
MICROSOFT
Response to IBM’s Whitepaper Entitled
Benchmarking and Beating Microsoft .NET
3.5 with WebSphere 7
7/2/2009
This document is a response to an IBM Benchmark Rebuttal Document to our original benchmark results
published at http://msdn.microsoft.com/stocktrader. Microsoft stands behind the original test results
published there. This document is the Microsoft response, point-for-point, to the IBM response to our
original results, and includes new benchmark data as further evidence. Contents
Executive Summary ....................................................................................................................................... 4
Introduction .................. 6
The Microsoft Findings.. 8
Response to IBM Rebuttal ............................................................................................................................ 9
IBM Friendly Bank Benchmark .................. 9
Microsoft Comments on IBM Friendly Bank Rebuttal Benchmark ........................... 9
IBM CPO StockTrader/.NET Benchmark.................................................................................................. 11
Microsoft Comments on IBM CPO StockTrader Rebuttal Benchmark .................... 12
Summary ..................................................................................................................................................... 19
Appendix A: Microsoft .NET StockTrader and WebSphere Trade 7 Test Results for IBM’s Revised Test
Script ........................... 20
Hardware Tested ..................................................................................................................................... 20
Application Server Hardware .............. 20
Database Server Hardware ................................................................................................................. 20
Methodology and Scripts ........................ 21
Testing Buys and Sells ............................................................................................................................. 21
.NET Buy LoadRunner Script ................... 25
Benchmark Results .................................. 29
.NET StockTrader LoadRunner Summary ................................................................ 30
WebSphere 7 Trade LoadRunner Summary ........................... 31
.NET StockTrader HTTP Responses/Second ............................ 32
IBM WebSphere 7 HTTP Responses/Second .......................................................................................... 33
.NET StockTrader Pass/Fail Transactions per Second ............. 34
IBM WebSphere 7 Trade Pass/Fail Transactions per Second ................................................................. 35
.NET StockTrader Transaction Response Times ...................................................... 36
IBM WebSphere Trade 7 Transaction Response Times .......................................... 37
.NET StockTrader Application Server CPU Utilization during Test Run ................... 38
IBM WebSphere 7 Trade Application Server CPU Utilization during Test Run ....... 39
.NET StockTrader Application Server LoadRunner Summary on Completion of Test ............................ 40
IBM WebSphere 7 Trade Application LoadRunner Summary on Completion of Test 41
Appendix B: Web Service Tests using no HTTP Server ............................................................................... 42 Application Server Hardware .............................................................................................................. 42
.NET WSTest EchoList Front Ended with IIS 7 vs. IBM WebSphere 7 WSTest EchoList Front Ended with
IBM HTTP Server ..................................... 43
.NET WSTest EchoList without IIS 7 (self-hosted WCF HTTP Service) vs. IBM WebSphere 7 WSTest
EchoList without IBM HTTP Server (in process WebSphere port 9080) ................................................. 44
WebSphere Tuning – Windows Server 2008 .......................................................... 45
IBM HTTP Server Tuning Windows Server 2008 ................. 45
.NET Tuning ............................................................................................................................................. 46
Executive Summary
In late April of 2009, Microsoft released a comprehensive benchmark report entitled Benchmarking IBM
WebSphere 7 on IBM Power6 and AIX vs. Microsoft .NET on Hewlett Packard BladeSystem and Windows
Server 2008. Recently, IBM has circulated a rebuttal document within many enterprise accounts, and
that rebuttal is entitled Benchmarking and Beating Microsoft .NET 3.5 with WebSphere 7. The IBM
rebuttal document was created by the IBM SWG CPO Performance Team at IBM. This document is the
Microsoft response to that rebuttal document.
IBM’s rebuttal document centers around two new benchmarks, and makes several false claims, as noted
below. We stand behind all of our original findings, and point-for-point respond to IBM’s claims.
IBM did not publish any Java or .NET source code to their Friendly Bank or CPO StockTrader
rebuttal benchmark workloads. Microsoft follows a full disclosure policy and publishes all
source code and full testing details. This information is available at
http://msdn.microsoft.com/stocktrader. IBM needs to publish all source code for their counter
benchmarks.
IBM’s Friendly Bank benchmark uses an obsolete .NET Framework 1.1 application that includes
technologies such as DCOM that have been obsolete for many years. This benchmark should be
fully discounted until Microsoft has the chance to review the code and update it for .NET 3.5,
with newer technologies for ASP.NET, transactions, and Windows Communication Foundation
(WCF) TCP/IP binary remoting (which replaced DCOM as the preferred remoting technology).
IBM makes several false claims about the .NET StockTrader:
o IBM claim: The .NET StockTrader does not faithfully reproduce the IBM Trade
application functionality.
Microsoft response: this claim is false; the .NET StockTrader 2.04 faithfully reproduces
the IBM WebSphere Trade application (using standard .NET Framework technologies
and coding practices), and can be used for fair benchmark comparisons between .NET
3.5 and IBM WebSphere 7.
o IBM claim: The .NET StockTrader uses client-side script to shift processing from the
server to the client.
Microsoft response: this claim is false, there is no client-side scripting in the .NET
StockTrader application.
o IBM claim: The .NET StockTrader uses proprietary SQL.
Microsoft response: the .NET StockTrader uses typical SQL statements coded for SQL
Server and/or Oracle; and provides a data access layer for both. The IBM WebSphere 7
Trade application similarly uses JDBC queries coded for DB2 and/or Oracle. Neither
implementation uses stored procedures or functions; all business logic runs in the
application server. Simple pre-prepared SQL statements are used in both applications.
o IBM claim: The .NET StockTrader is not programmed as a universally accessible, thin-
client Web application. Hence it runs only on IE, not in Firefox or other browsers.
Microsoft response: In reality, the .NET StockTrader Web tier is programmed as a
universally accessible, pure thin client Web application. However, a simple issue in the use of HTML comment tags causes issues in Firefox; these comment tags are being
updated to allow the ASP.NET application to properly render in any industry standard
browser, including Firefox.
o IBM claim: The .NET StockTrader has errors under load.
Microsoft response: This is false, and this document includes further benchmark tests
and Mercury LoadRunner details proving this IBM claim to be false.
Introduction
In late April of 2009, Microsoft released a comprehensive benchmark report entitled Benchmarking IBM
WebSphere 7 on IBM Power6 and AIX vs. Microsoft .NET on Hewlett Packard BladeSystem and Windows
Server 2008. Recently, IBM has circulated a rebuttal document within enterprise accounts, and that
rebuttal is entitled Benchmarking and Beating Microsoft .NET 3.5 with WebSphere 7. The IBM rebuttal
document was created by the IBM SWG CPO Performance Team at IBM. This document is the Microsoft
response to that rebuttal document. Ultimately, vendor competition around middle tier software
performance and pricing is healthy for customers, and we believe ongoing exploration of application
server performance and pricing is a key part of that competition.
It is important to note that in all Microsoft-driven middle-tier application server benchmarks involving
Microsoft .NET and IBM WebSphere, a policy of full disclosure is followed; as is required in both the IBM
WebSphere End-User License Agreement and the Microsoft .NET End-User License Agreement (see
“Benchmarking” Clause in the respective EULAs). Full disclosure is extremely important, as it allows
customers and competing vendors to fully analyze the results, and even replicate the testing on their
own such that fully informed responses can be made. Full disclosure was adhered to in the original
Microsoft benchmark entitled Benchmarking IBM WebSphere 7 on IBM Power6 and AIX vs. Microsoft
.NET on Hewlett Packard BladeSystem and Windows Server 2008. This means that along with the
benchmark results as documented in the paper, any customer or competing vendor is able to download
at a publicly posted Web site the following materials:
All source code used in the benchmark workloads (both the Java and .NET implementations)
Detailed benchmark results
Details of the benchmarking software used, and test script flow(s) used for all workloads
Breakout of all software and software versions used in the tests
Detail on the precise hardware used in the tests
Charts showing the test bed setup, including network diagr
© 2010-2024 YouScribe
