OpenFlow: Proactive vs Reactive
As more people begin entering the Software Defined Networking conversation, there is still only one wire protocol that has a reasonably good chance at becoming the de-facto open SDN southbound messaging standard, OpenFlow. There is quite a bit of debate around whether or not OpenFlow can scale. Most claims that OpenFlow does not scale are probably grounded in misconception. That is not to say we do not have a long way to go, but spending the next 5-10 years debating a hardware abstraction layer would be counterproductive in the face of unmanageable networks, with exponential growth occurring annually. As forwarding abstractions in hardware continue to become more flexible, the scale debate should not be whether OpenFlow can scale as a messaging protocol, but be focused on when and if to use reactive flow based rules as opposed to proactive instantiation.
Recap of Why OpenFlow is Important
Today we have switches that operate at L2, routers that operate at L3 and firewalls that operate at L4. Some of those functions are blended together in purpose-built hardware such L3 switches. An access-list is a common function that uses TCAM (ternary content addressable memory) to have the flexibility to match on various layers of a packets L2-L4 headers. That TCAM memory has the ability to match (0), not match(1) (the binary is counterintuitive there) or not care (wildcard). Each field or tuple is searched in parallel all in one clock cycle. So, very quickly and at “line rate” in hardware.
Figure 1. TCAM matching
The simplicity of OpenFlow is it was merely a mechanism to exploit TCAM tables in existing vendor switches in order instantiate whatever flows a programmer wants to push. Now with OpenFlow, one can tell a switch to match on any field and apply an action. For network operators and engineers this is policy based routing, on steroids. We can decouple logic and push a set of flow policy from off board the switch, through a TCP socket, using a structured messaging protocol to describe the policy and flow rules to put in TCAM.
OpenFlow Proactive vs. Reactive
When using OpenFlow to populate TCAM in switches there are essentially three modes of operation:
- Reactive flow instantiation – When a new flow comes into the switch, the OpenFlow agent SW on the switch, does a lookup in the flow tables, either in a search ASIC if in hardware or a software flow table in the case of a vSwitch. If no match for the flow is found, the switch creates an OFP packet-in packet and fires it off to the controller for instructions. Reactive mode reacts to traffic, consults the OpenFlow controller and creates a rule in the flow table based on the instruction. The problem with reactive is today’s hardware has laughable amounts of CPU in it. Think iPhone 5 or PowerPC circa 2001 amount. That little guy then must do a DMA copy to replicate and encapsulate the packet. HW switches outside of NPUs or general purpose CPU vSwitches don’t do this over a few thousand packet per second. Most in the hundred and some in the 10’s from some rather large expensive chassis.
- Proactive flow instantiation – Rather than reacting to a packet, an OpenFlow controller could populate the flow tables ahead of time for all traffic matches that could come into the switch. Think of a typical routing table today. You have longest prefix matching that will match the most granular route to a destination prefix in a prefix tree lookup. By pre-defining all of your flows and actions ahead of time in the switches flow tables, the packet-in event never occurs. The result is all packets are forwarded at line rate, if the flow table is in TCAM, by merely doing a flow lookup in the switches flow tables. That is the same hardware that populates its forwarding tables today from “routing by rumor” in todays routing protocols and “flood and spray” layer2 learning standards. Proactive OpenFlow flow tables eliminates any latency induced by consulting a controller on every flow.
- Hybrid flow instantiation – A combination of both would allow for flexibility of reactive for particular sets a granular traffic control that while still preserving low-latency forwarding for the rest of your traffic. Low-latency financial markets operate in the nano seconds, that would not be a good place for reactive forwarding today but granular security in an enterprise would possibly be reasonable if the policy was important.
Summary
There are several proactive flow based products with proprietary protocols that have been around for a while. Hearing arguments that one scales, while OpenFlow doesn’t is misleading. Rules in TCAM perform the exact same regardless of the method of instantiation. The majority of early OpenFlow deployments (at least ones I am designing) are going to be almost all proactive combined with granular selections of traffic for specific use cases being reactive. This is not to say there will never be a place for reactive forwarding with solid performance, but I think that will come in the form of embedded systems and more hardware distribution at the edge. All networks are not low-latency trading floors and Hadoop racks in data centers. I cannot tell the difference when I am surfing the Internet on a fully centralized wireless controller, to when I am wired in at 1Gb. The human brain does not register microseconds.
As the disruption that SDN represents to the way we have always done networking continues, those who prefer todays primitives over programmatic abstraction will make claims that SDN, OpenFlow and flow based forwarding doesn’t scale. Ask them if the proactive access-lists in their networking hardware TCAM today scale, or if their proactive QOS policies implemented in their networking gears TCAM operate at line rate. OpenFlow is not grounded in reactive flow policy, but a protocol to exploit TCAM tables for programmatic L2-L4 forwarding flexibility, nothing more. The value will be in abstraction layers, services and applications.
Additional Resources
OpenFlow: Design Considerations Series
Part 2 OpenFlow: Course vs. Fine Flows
Part 3 OpenFlow: Hybrid Deployments
Part 4 OpenFlow: SDN Looking Ahead (Pending)
For additional insight into how the forwarding abstractions are being adjusted to be more than just TCAM take a listen to this podcast with Curt Beckmann of the ONF discuss his work as the chairman of the forwarding abstraction working group (FAWG).
- Packet Pushers: PQ Show 20 –Open Network Foundation – FAWG Update
Thanks for stopping by!
IMO the real scalability issue is the number of entries. Some controllers create one entry per host and some create 1,000 entries per host. How big is your TCAM?
Heya Wes, TCAM is a problem. I think longer term more isnt the answer either if more flexibility to flow table abstraction can be worked out by the vendors w/ v1.0+. More room for differentiation too. TCAMs sizes are all over the map for sure. from 800 to someone like NEC with external TCAM at 96k.
I am hearing of flow tables in some of the traditional chassis being around 100k this year at minimum. Broadcom and Intel will drive a lot of it with whats coming out of their foundries IMO.
Thanks for the comment!
-Brent
My point was really more about fine-grained vs. coarse-grained flows because no matter how big your TCAM is you still don’t want to waste it with fine-grained flows that you don’t need. But some controllers will do that.
Brent,
While I totally agree with you, please do keep in mind that:
A) some of the more popular open-source OF controllers use reactive mode (ex: Floodlight);
B) per-flow session state in the network core (or in TCAM entries) will never scale (ex: Intserv);
C) number of TCAM entries in typical ToR switch is severely limited. Large TCAM, wire-speed forwarding, low-price: you get at most two out of three (usually you don’t even get two).
D) TCAM update rate in current hardware OF switches is ridiculously low, further limiting the usability of reactive or hybrid models.
Until the OF aficionados admit there are lessons to be learned from the last 30+ years of networking failures (including numerous scalability failures, many of them caused by per-session state in network core), we won’t be moving anywhere.
Fortunately at least some of the thought leaders figured out MPLS designers were not overly stupid 😉 (see http://blog.ioshints.info/2013/01/edge-and-core-openflow-and-why-mpls-is.html)
Best,
Ivan
Thanks Ivan, really enjoyed your MPLS/OF post. I started to comment but it was the middle of the night and I realized it was oncoherant and deleted it lol.
Really appreaciate the comments, you know I rely on you to keep me grounded in the reality plane. You and Greg are the angel and devil on each shoulder, not sure which is which lol.
A) I think they load flood and learn forwarding modules in reactive so folks can download it and just get started with no work involved. I got a controller from a commcerial vendor the other day and it loaded empty and I was like hrmph, you mean I have to confiugure this :)?
B)I agree per-flow is well beyond scale, heck its only been the past few years we arent panicking over running multiple I1 copies and blowing up our FIBs. Some still are blowing up their FIBs on older gear just on multiple 450k Internet routes. Introducing L2-L4 matching wildcard or not would be nutty.
That said working on a follow up to this post digging into course vs fine grain flows. Big flow matching for steering or whatever based on a prefix or tranport field.
C) Most the flow table sizes I am seeing are capping around 1,500 to 3,000 on TOR. The Trident+/2 are all about the same and ofcourse shared with the regular forwarding pipeline. Cost, heat, silicon space is going to be a dead end with TCAM at some point. Really curius to see what the FAWG recommends, Curt will tell us if we cant figure it out any other way :/
D) I probably am dreaming on reactive anytime soon other than embedded systems, but then we have hundreds of controllers. Is that as atraucious as it sounds?
You know what is still not being scrathced is a meaningful way of classification at the edge. We have always had buckets, even if it is a FEC/LSP, but there is still no new ways of sorting what goes in those buckets. I guess the analytics are still a ways off, but reactive even if a tiny but peeled would be neat for classification. I think if you had an embedded SOC on the edge for DPI/reactive punting there is cool use cases, probably corner but cool.
I like OF for path stichting in the core but not much else today, but that is a pretty huge use case. How SPs manage their TE policies is beyond me, if done well. Trying to integrate anything L4 into MPLS via flow filtering and route servers sounds like a kludgy overcomplicated mess imo. If we go beyond L2-L3 I think its time to put the old girl down.
If anyone missed Ivan’s post on hyper-V its a must read!
http://blog.ioshints.info/2012/12/hyper-v-network-virtualization-wnvnvgre.html
You inspired me to track down a MS call. Thats a lot of inspiration btw pal!
All large purchases are always intimidating, especially if you
are uninformed about the industry. One of the scariest purchases is buying cars.
Many people fear they are getting ripped off and you surely don’t want that. Avoid buying a lemon by looking through these great tips and tricks regarding car purchases.
Hi there! Quick question that’s completely off topic. Do you know
how to make your site mobile friendly? My site looks weird when browsing from my iphone.
I’m trying to find a theme or plugin that might be able to resolve this problem.
If you have any recommendations, please share. Appreciate it!