Ethernet Switch Fabric Cards

I was working with a buddy today about ethernet switch fabric cards in the nexus 7000 and not doing a very good job at explaining it, so I figured I would draw some pictures. The conversation was around how adding more fabric cards increases the bandwidth. I think some, myself included growing up on Catalyst 6500’s got accustomed to a single stage architecture which in the case of the ASR 9000 and similarly the  Sup 7200 and newer, have the switch fabric living on the route processor (RP), routing engine (RE juniper) or supervisor (sup Cat 6500). Chip, ASIC, fabric hardware developers are tackling the same problems in silicon design that we as network engineers/architects tackle  in macro engineering network design. Its all modularity the building blocks of scale and abstraction.

I was comparing the Nexus 7000 to the ASR 9000. The ASR is single stage The Nexus 7000 is a typical three stage Clos fabric (figure 2) meaning it interconnects three separate crossbar (figure 1) fabrics together.

Figure 1. Crossbar Fabric

Figure 2. Three Stage Clos Fabric

The interesting thing is as we find better ways to try and improve and cut coherent network architectures often times we do not need to look any further than the actual silicon under the hood.

Figure 3. Spine Leaf Data Center Design.

I do not have time to do this post much justice there are dozens of models such as Banyan, Batcher, Benes and so on and the interactions buffering, memory logic, arbitration, queuing all being orchestrated in a multiplexed fashion at light speed that is hard to wrap ones head around. The basic understanding of whats under the hood can be applied in scaling how we put these things together in a pile and plug them into one another.

I will be working on some material soon, as I try and guess/understand upcoming changes in our hardware on the horizon and in all reality commoditization is already happening. In some cases I think we are trying to tackle software problems in hardware. Thanks goodness Moore’s law is theoretically hanging on by a thread which maybe get is to an inflection point soon. Hell that is highly debatable but what isn’t.

A fantastic fabric post by Ferro can be found here

One of my favorite books ever. I recommend to everyone except probably my mom is: Network Processors: Architecture, Programming, and Implementation (Systems on Silicon)