Relentless Competition Drives Down Ethernet Switch Costs

If you want to see what real competition might look like at some point in the future of the server racket, look no further than the Ethernet switch market, where switch ASICs and the companies that build switches alike have to fight for every dollar and make it up in volume every year without pause.

A switch has a much longer lifespan in the datacenter than does other equipment – something akin to five or six or sometimes even seven years compared to maybe three or four years for the average server. But that could be changing as architectural advances and aggressive use of new chip etching and packaging techniques are bringing down the cost of switch ASICs at the same time competition between the Big Five datacenter switch vendors is getting more intense and other innovative ASIC suppliers are entering the fold and whitebox switch makers are making headway among hyperscalers, cloud builders, and more than a few large enterprises.

This pricing pressure and the lower cost of delivering high bandwidth are two big reasons why growth in port counts for Ethernet datacenter switches (as distinct from generally lower speed switches used to link users to each other in the corporate, government, or academic campus) is higher than the revenue increases they are able to garner. This is exactly the opposite thing that is happening in the server space, where the high cost of CPUs, memory, flash, and GPU accelerators is driving up average selling prices while unit volumes are exploding. The lowering network costs are, in a way, helping to fund the server buildout, especially as companies are also figuring out to use higher radix switches and to flatten out their networks so there are fewer switching tiers. (Facebook, with its new Minipack switches and its F16 fabric, is a prime example of what is going on.)

According to the box counters at IDC, Ethernet switch sales in the datacenter and across the campus rose by 12.7 percent in the fourth quarter of 2018, to $7.82 billion, and total ports shipped in the period were up by 17.5 percent to 155.5 million. Companies still need routers to connect their datacenters to each other and to the Internet, so this market has not died yet. (We are kidding. It won’t die.) Router sales rose by 15.6 percent in the quarter to $4.6 billion. Routers continue to be big ticket items that companies buy begrudgingly, and it is no wonder that hyperscalers and cloud builders have been pushing switch ASIC and network operating system makers to embed more routing functions into their switchery. For the full year, Ethernet switch revenues were up 9.2 percent to $28.2 billion, and the aggregate switch ports pushed rose by 14.9 percent to 572.5 million ports.

The publicly available data from IDC does not break datacenter and campus switching out separately from each other, but we can create a proxy of sorts for the datacenter by adding up sales of switches that have 25 Gb/sec and higher downlink ports and taking a chunk of the 10 Gb/sec switching and putting some of it into the datacenter pile. Just for a thought experiment, we allocated half of 10 Gb/sec switch revenues and half the ports to the datacenters, and came to a result of $3.39 billion in datacenter Ethernet switch sales in the fourth quarter of 2018, up 30 percent, and a total of 18 million ports sold, up 53.6 percent. For the full year, this datacenter Ethernet switching proxy accounted for $11.6 billion in revenues, about 41 percent of sales, and 60.8 million ports, up 53.6 percent.

Here’s a summary table of how the different Ethernet speeds did during the fourth quarter, based on IDC data:

The number of ports shipped at 100 Gb/sec continues to grow as hyperscalers, cloud builders, large enterprises, and some HPC centers are upgrading to higher bandwidth and equally importantly higher radix switches to build better clusters. The cost of a 100 Gb/sec port on a switch has come way down, and you can thank the hyperscalers for that as they compelled the industry to move to 25 Gb/sec signaling faster than the IEEE and the vendors wanted to, and with the advent of PAM4 encoding we are getting what is effectively 50 Gb/sec signaling, which is allowing for the cost of 100 Gb/sec chips to come way down and therefore the cost per port is following that curve.

Four years ago, when 100 Gb/sec started to trickle into datacenters, often for network backbones linking datacenters but sometimes for network fabrics linking machines, a port cost about $3,000 a pop. If you do the math on the market data from IDC, it has dropped by more than a factor of 10X, averaging $288 per port. If you use a quad cable splitter, as some datacenters do from their top of racks, then the cost of a 25 Gb/sec port is around $72 a pop. At these prices, it is no wonder that 100 Gb/sec port shipments were almost triple year-on-year in the fourth quarter, to 3.8 million units, and revenues were up 58.5 percent to $1.1 billion. Sales of switches with real 25 Gb/sec and 50 Gb/sec ports, which IDC lumps together into one category, were up by 2.75X to $342 million, pushing 3 million ports, up by the same amount. The cost per port on these switches, assuming an even split of 25 Gb/sec and 50 Gb/sec switches, is around $115.

You might be thinking that the era of 40 Gb/sec Ethernet is over, but revenues for these switches were up 29 percent to $798 million in the fourth quarter and port counts rose by 54.5 percent to 2 million units. The average port cost here is just under $400 and the cost per gigabit of bandwidth is three times that of the average 100 Gb/sec port. The only explanation for continuing 40 Gb/sec switch sales is that companies want to deploy gear that they have already certified for their networks. (Perhaps there is something else going on here; we are the first to admit that we don’t know.) Sales of 10 Gb/sec Ethernet switches are still dominant in the datacenter and have some share in the campus networks, with $2.32 in revenues. But sales are down by 1.6 percent in the fourth quarter even as port counts rose by 14.6 percent year on year to 18.4 million units. The average cost per port is flattening out here, at around $125 per port. The cost per gigabit of bandwidth is a third again as high as that of 40 Gb/sec switches and is more than four times that of 100 Gb/sec switches.

In other words, it is way more costly to move slow, on a per gigabit basis, than it is to move fast. It all comes down to capital outlay, which companies try to minimize. But at some point, with some workloads, it doesn’t pay to scrimp to save. Time is money, too.

So who is making all of this money in Ethernet switching?

As usual, Cisco Systems is leading the pack, according to IDC, with just a tad under $4.2 billion in sales, up 13.2 percent and giving the networking juggernaut 53.2 percent share of the overall Ethernet market. We suspect rather strongly that Cisco’s share of datacenter Ethernet switching is way lower than this and that it has a very strong command of campus switching.

Huawei Technologies is the number two switch maker by revenues, having supplanted Juniper Networks three years ago; the Chinese company sold $759 million in Ethernet switching gear in the fourth quarter, up 6 percent and accounting for 9.7 percent share. Arista Networks, the upstart switch maker started by serial entrepreneurs and partners David Cheriton and Andy Bechtolsheim (both of Stanford University fame as well), has risen to be the number three vendor in the fourth quarter, with $508 million in revenues, up 23.6 percent and giving it 6.5 percent share of the market. Hewlett Packard Enterprise was knocked out of position three by Arista, with sales up a market beating 20.2 percent to $477 million. Juniper rounded out the top five, with $206 million in sales, off 2 percent and with only 2.6 percent of the market. Other vendors – and there is a healthy number of them, using all kinds of ASICs these days, accounted for $1.71 billion in sales, up 12.7 percent, as a group growing the same speed as the market at large.