INTERNETWORK EXPERT .ORG

Egenera criticized Cisco’s math for UCS pricing in an article titled “A Closer Look at Cisco UCS pricing“.

After reading the first few paragraphs of this article it became quite clear that Egenera needs a closer look at their own math. Let’s examine the flaws in the article…

About the price of management software for a “legacy” Blade deployment of 20 chassis eGenera said:

If Management software is a chassis-based price ( $7,000 from above) – with 20 chassis this total should be $140,000 – however Cisco shows $554,400 – a $414,000 mistake. Cisco’s $554,000 comes out to $27,700 per Blade chassis, or $1,732 per Blade. No that’s a pretty big mistake.

OK, lets be honest here – YES – the “legacy” system is HP BladeSystem. Given that, to achieve the dynamic provisioning and stateless computing capabilities included in Cisco UCS, an HP BladeSystem customer would require not only a Virtual Connect Enterprise Manager (VCEM) license of $7,000 per chassis, they would also need another management platform called HP Insight Dynamics VSE with a per-chassis license cost of $20,000. The Cisco UCS comparative pricing is correct in showing roughly $27,000 of management costs per HP Blade chassis to achieve what is already included in Cisco UCS (dynamic provisioning, stateless computing, “logical servers”). The extra $700 per chassis is likely the pair of HP Onboard Administrator modules required per HP Blade chassis for device level management. Management modules per chassis are not needed with Cisco UCS, all device level management and provisioning is built in to the UCS Manager.

Moving on. Egenera claims that Cisco showed UCS pricing of a system that had no redundancy:

Also, the Cisco configuration for 320 blades includes 40 Extenders. We know this because if you divide the 8 Blade “Extender” cost into the 320 Blade configuration extender cost – it equals 40. We also know that each Cisco Blade chassis holds 8 blades, so 40 chassis’ are needed to support 320 blades. SO, there is ONE Extender in each Cisco Blade chassis in this cost comparison. That’s ONE extender for each of the 40 blade chassis. That means that the extender is a single point of failure for each blade chassis in this configuration. In contrast, the (HP) “legacy system” is priced with full redundancy.

Bottom-Line: Cisco is not providing a fair comparison. Advantage HP.

WRONG. The 8 Blade “Extender” cost was showing the cost of (2) Fabric Extenders! Each fabric extender is $1999. Therefore (2) fabric extenders in a single chassis with 8 blades is $3998, and (2) fabric extenders in 40 chassis is $159,920. Exactly as shown is the Cisco UCS pricing webcast.

Bottom-Line: Cisco provided a fair pricing comparison and came out 31% better. Advantage Cisco.

Egenera then tries to focus on bandwidth but gets it wrong:

The Extenders have only four 10Gb/s FCoE ports for attachment to the fabric switch. So, that amortizes to 5Gb/s for each Blade – “IF” the ports on the Cisco Blades can be teamed or configured as Active/Active

Correct that each fabric extender has four 10GE uplinks, and each chassis will have (2) fabric extenders as described above, which results in 80Gb/s of bandwidth to a chassis of 8 blades. That’s 10Gb/s to each blade.

Egenera also asserts that Cisco was unfair with respect to bandwidth in a 320 Blade deployment:

There is another factor at play. EACH Fabric Interconnect (aka Nuovo Switch) supports 40 Fabric connections to the Cisco Blade chassis. In order to support 320 Blades, each Cisco Blade chassis can only have ONE 10Gb/s connection to EACH of the redundant Cisco 61000 Fabric Interconnect switches. So, this results in EACH Cisco Blade chassis having a maximum of TWO(redundant) 10Gb/s fabric connections shared by 8 Blades. So this calculates to 20Gb/s for 8 Blades – or 2.5Gb/s per Blade. This, this tells us that the Cisco 320 Blade solution has a 2.5Gb/s fabric capacity. Now, for the “Legacy” (HP), recall the blades have redundant (2x) 10Gb/s Ethernet ports and redundant (2) 4Gb/s Fiber Channel Ports. The redundant (2) 10Gb/s Ethernet Switches in the “Legacy” system have a total of 16×10Gb/s uplink ports – so there is enough fabric and switch bandwidth for a true 10Gb/s Ethernet throughput for each Blade. The redundant (2) Fiberchannel Switches have 8x 4Gb/s fiber channel ports. Which is an additional 0.5Gb/s of Fiber channel through put per Blade in the Legacy system.

Bottom line: This is an unfair cost comparison of a 10Gb/s “Legacy” (HP) system to a 2.5Gb/s Cisco UCS system.

Quite the contrary, Cisco was actually overly fair to HP in this comparison (in my opinion). True, in a 320 blade single UCS system you will have (2) 10GE redundant uplinks from each of the 40 chassis. Keep in mind that Cisco never mentions bandwidth in this price comparison because it’s not about bandwidth — we are strictly looking at the cost to scale to 320 blades managed in a single system.

Why is this overly fair to HP? The price comparison Cisco showed in the 320 Blade scenario was overly fair to HP by not including the 10GE networking costs required for the 20 HP Blade chassis, where as the Cisco UCS pricing did include the networking costs (included in the UCS Manager price). The 320 Blade comparison was not factoring bandwidth, it was factoring extreme scalability, and thus Cisco was fair enough to leave out 10GE network costs for the HP solution while still showing those costs for its own solution. A truly fair comparison would have also included the cost for (40) 10GE network ports in the HP scenario, but Cisco was kind enough to leave it out.

Bottom Line: Cisco was more than fair with the 320 Blade cost comparison.

The article finishes off with even more bad math…

If a customer wants a full 10Gb/s fabric, each Blade Chassis will need two Fabric Extenders ($3,998 each) AND must use all 8×10Gb/s uplinks on the Fabric extenders. This also means the 40 Port Fabric Interconnect switch will only support –Five– blade chassis’ – not 40. So achieving a 10Gb/s fabric version of the Cisco UCS is limited to 40 Blades maximum.

Bottom line: If a customer is interested in full bandwidth 10Gb/s connections to each of the Blades, the entry cost is $138,182 for the networking only, and only for 40 slots. That’s $3,304 per Blade slot for infrastructure. And then the blades themselves are additional costs!

First of all, again each fabric extender is $1999 each. If the goal is 10GE to each blade then yes all (4) 10GE uplinks from fabric extender #1 links to Fabric Interconnect #1, all (4) 10GE uplinks from fabric extender #2 link to Fabric Interconnect #2. So each Fabric Internconnect has (4) ports used for each blade chassis. If I have a 40 port Fabric Interconnect, that means I can have TEN blade chassis in a full bandwidth system – not five as Egenera claims. Therefore achieving a “full 10Gb/s fabric” could be done with TEN blade chassis each with 8 blades — that’s 80 Blades each with a full 10Gb/s managed in a single system.

So lets assume the price of the UCS fabric interconnect in this scenario is $138,182. Actually, Egenera left out the cost of the fabric extenders for 10 blade chassis which would be $39,990. So lets add $39,990 to $138,182 and arrive at the number $178,162 which includes the price of the UCS Manager (Fabric Interconnect) and the fabric extenders for 10 chassis at full bandwidth. If a customer is interested in full bandwidth 10Gb/s connections to each of the Blades, the entry cost is $178,162 for the unified networking of 80 blade slots. That’s $2,227 per Blade slot for storage and IP infrastructure.

Now lets look at the networking costs for HP BladeSystem under the same 80 Blade scenario.

For HP, a customer would need 5 chassis to network 80 blades. For the network piece this would require (10) Ethernet modules, and (10) Fibre Channel modules, (2) of each in the (5) chassis. To provide 10GE and 4G FC to each of the 16 blades in the HP chassis, each chassis would then link up to (16) 10GE network ports for Ethernet and (16) FC ports for storage, requiring (80) 10GE network ports and (80) FC ports.

So lets look at the equivalent network costs on the HP side to provide 10Gb/s to 80 blades (to be fair):

(80) 10GE Ethernet ports = $70,000 (based on two Nexus 5020)

(80) 4G FC ports = $32,000 (based on four 24 port 9124 fabric switches)

(10) Ethernet blade switches = $120,000 (based on $12,000 per VC-eth module)

(10) FC blade swtiches = $94,990 (based on $9,499 per VC-fc module)

Total HP networking costs for 80 Blades = $316,990 / 80 Blades = $3,962 per Blade

Toal Cisco UCS networking costs for 80 Blades = $178,162 / 80 Blades = $2,227 per Blade

Bottom Line: For 80 Blades of 10Gb/s per blade –> Advantage CISCO (by almost a factor of 2)

Disclaimer: The views and opinions expressed are those of the author as a private individual and do not implicitly or necessarily reflect the views and opinions of the author’s employer (Cisco Systems, Inc.).

So you just lit up your new high-speed link between Data Centers but are unpleasantly surprised to see relatively slow file transfers across this high speed, long distance link — Bummer! Before you call Cisco TAC and start trouble shooting your network, do a quick calculation of what you should realistically expect in terms of TCP throughput from a one host to another over this long distance link.

When using TCP to transfer data the two most important factors are the TCP window size and the round trip latency. If you know the TCP window size and the round trip latency you can calculate the maximum possible throughput of a data transfer between two hosts, regardless of how much bandwidth you have.

Here is how you make the calculation:

TCP-Window-Size-in-bits / Latency-in-seconds = Bits-per-second-throughput

So lets work through a simple example. I have a 1Gig Ethernet link from Chicago to New York with a round trip latency of 30 milliseconds. If I try to transfer a large file from a server in Chicago to a server in New York using FTP, what is the best throughput I can expect?

First lets convert the TCP window size from bytes to bits.  In this case we are using the standard 64KB TCP window size of a Windows machine.

64KB = 65536 Bytes.   65536 * 8 = 524288 bits

Next, lets take the TCP window in bits and divide it by the round trip latency of our link in seconds.  So if our latency is 30 milliseconds we will use 0.030 in our calculation.

524288 bits / 0.030 seconds = 17476266 bits per second throughput = 17.4 Mbps maximum possible throughput

So, although I may have a 1GE link between these Data Centers I should not expect any more than 17Mbps when transferring a file between two servers, given the TCP window size and latency.

What can you do to make it faster?   Increase the TCP window size and/or reduce latency.

To increase the TCP window size you can make manual adjustments on each individual server to negotiate a larger window size.  This leads to the obvious question:  What size TCP window should you use?  We can use the reverse of the calculation above to determine optimal TCP window size.

Formula to calculate the optimal TCP window size:

Bandwidth-in-bits-per-second * Round-trip-latency-in-seconds = TCP window size in bits / 8 = TCP window size in bytes

So in our example of a 1GE link between Chicago and New York with 30 milliseconds round trip latency we would work the numbers like this…

1,000,000,000 bps * 0.030 seconds = 30,000,000 bits / 8 = 3,750,000 Bytes

Therefore if we configured our servers for a 3750KB TCP Window size our FTP connection would be able to fill the pipe and achieve 1Gbps throughput.

One downside to increasing the TCP window size on your servers is that it requires more memory for buffering on the server, because all outstanding unacknowledged data must be held in memory should it need to be retransmitted again.  Another potential pitfall is performance (ironically) where there is packet loss, because any lost packets within a window requires that the entire window be retransmitted – unless your TCP/IP stack on the server employs a TCP enhancement called “selective acknowledgements”, which most do not.

Another option is to place a WAN accelerator at each end that uses a larger TCP window and other TCP optimizations such as TCP selective acknowledgements just between the accelerators on each end of the link, and does not require any special tuning or extra memory on the servers.  The accelerators may also be able to employ Layer 7 application specific optimizations to reduce round trips required by the application.

Reduce latency?  How is that possible?  Unless you can figure out how to overcome the speed of light there is nothing you can do to reduce the real latency between sites.  One option is, again, placing a WAN accelerator at each end that locally acknowledges the TCP segments to the local server, thereby fooling the servers into seeing very low LAN like latency for the TCP data transfers.  Because the local server is seeing very fast local acknowledgments, rather than waiting for the far end server to acknowledge, is the very reason why we do not need to adjust the TCP window size on the servers.

In this example the perfect WAN accelerator would be the Cisco 7371 WAAS Appliance, as it is rated for 1GE of optimized throughput.

WAAS stands for:  Wide Area Application Services

The two WAAS appliances on each end would use TCP optimizations over the link such as large TCP windows and selective acknowledgements.  Additionally, the WAAS appliances would also remove redundant data from the TCP stream resulting in potentially very high levels of compression.  Each appliance remembers previously seen data, and if that same chunk of data is seen again, that data will be removed and replaced with a tiny 2 Byte label.  That tiny label is recognized by the remote WAAS appliance and it replaces the tiny label with the original data before sending the traffic to the local server.

The result of all this optimization would be higher LAN like throughput between the server in Chicago and New York without any special TCP tuning on the servers.

###

Brocade announces intent to buy Foundy — Why would Brocade do this?

The writing was on the wall … Either become a Ethernet company or get crushed by Cisco.

What is the new Cisco Nexus 5000? — The industry first switch to deliver unified server I/O, providing Fiber Channel and IP traffic over a single 10G Ethernet port to the server.  Nexus 5000 delivers very low latency wire speed lossless Ethernet service to the server.