Setting Up Overlays on Open vSwitch

Setting Up Overlays on Open vSwitch

Posted Mar 6 2014 by Brent Salisbury in  with 9 Comments

Network Overlay

Setting Up Overlays on Open vSwitch

Most “SDN” solutions involve overlays or at the least HW overlay gateways/ToR of some type. Some sell overlays terminating in hardware, others sell overlays terminating in the server. The encaps include standards like GRE, VXLAN and soon to be Geneve (Generic Network Virtualization Encapsulation: basically the good parts of the other encaps evolved). While none of these overlay networks should ever be setup by hand like we provision networks currently today, if you are trying to get a good understanding of OVS and overlays for proofing something before you code it this may be helpful. Before we do any coding to automate provisioning, it helps to work out any potential blocking ahead of time. OVS is the reference SDN datapath from developers who I regard as the best in the business. If you invest time into evolving your network, OVS is the place to start.

Rather then type out a bunch of details, I did a detailed drawing of how things can look w/ example addresses. This isn’t necessarily a good beginner how-to for OpenFlow, Open vSwitch etc, but if you are looking to understand how to decouple the physical constraints of current generation networks, OVS and overlays are the only rational architecture that is actually implementable in the brownfield data center today. These are things I think you need to understand cold if you are developing or operating (or both) overlay solutions. For architecture, understanding the picture and not the implementation details is probably helpful if dealing with OpenStack in particular. A decent OpenStack integration will abstract the details but understanding them is going to be important to those working on cloud infrstructures.

Setting Up Overlays on Open vSwitch Tutorial

This is for three Hypervisors running nested Hypervisors. Basically take a box, install KVM and boot three hosts on it. Then grab a small linux distribution like Cirros. Libvirt is handy for this but too low level to mix into this overview. Google for it, lots of documentation out there for it.

The topology will look something like the following:

SDN Overlays

I have different subnets to emphasize the flexibility overlays present. You don’t need to add physical eth0 interfaces into a bridge for this. your TEP only needs to use any address on the Linux host that can reach the destination tunnel endpoint. Adding Ethernet NICs to the bridge is necessary if you are setting up a gateway host to redistribute overlay traffic into another non-overlay network.

I highly recommend only doing the configuration portion if you are new to flow forwarding and OpenFlow unless you learn through troubleshooting and are willing to work through issues which is awesome and a great way to learn. Google is your friend, I know it is mine. If you do configuration (OVSDB) only change key=flow to key=10 or some number. key=flow means the OpenFlow v1.3 flowmod will set the tun_id using the OXM extension. set_tunnel_id is as advanced as OpenFlow gets on the tunnel side which is a downside and why we are currently implementing NXM extensions in ODL. This is another good example why many of us feel there needs to be a separate software OF spec for those of us who desire to move faster then the silicon foundries. This is a totally proactive setup, meaning it can have or not have a controller. You can use (or script) client commands to implement the scenario which is actually what the OpenStack OVS plugin does. Alternatively you can use open source controllers such as OpenDaylight and RYU or commercial controller from Vmware (NSX).

The example will setup a full mesh of tunnels, attach a VM to br0 and all VMs on that segment should ping. I recommend Open vSwitch 2.0 or higher to ensure feature availability w/vxlan etc. Also if you are new to Open vSwitch or Linux networking I recommend just installing and tinkering around with merely adding bridges, ports and how hypervisors integrate into OVS, as tunnels will be a bit confusing to start with.

http://networkstatic.net/install-open-vswitch-networking-red-hat-fedora-20/ →

Verify your flows are installed and counters are incrementing if you have a guest VM attached to a port. An example from an OpenStack deloyment I had running locally in a VM while writing this is:

Here are some helpful aliases/commands that I use for client commands.

To setup overlays using OpenStack, OpenDaylight, Open vSwitch, OpenFlow, OVSDB, lol thats a lot of Opens, see the following:

http://networkstatic.net/opendaylight-openstack-integration-devstack-fedora-20/ →

For more regarding OVS and flow forwarding using OpenFlow and/or OpenFlow/Nicira extensions take a look at Ben Pfaff’s latest draft of OpenFlow/OVS fields in Open vSwitch. This is what in my opinion OpenFlow should look like in order to facilitate overlays with open standards in the Data Center. Don’t confuse overlays with hypervisor networking only. By far the majority of SDN data center implementations use overlays regardless of whether the TEP terminates on the soft vSwitch or the ToR physical switch. Cisco ACI for example setup TEPs in their leaf HW nodes using some sort of signaling which I have no idea about beyond overlays and a few bits in the VXLAN header to be keyed on for application classification.
I am a developer on the OpenDaylight projejct and we managed to get basic overlay functionality implemented but we have certainly hit the wall on adding functionality without using NXM extensions. Long story shot, if the OpenFlow specification is to stay relevant (which Dan Pitt and company work hard to do so) it needs to address these functions in the spec rather then in OF-Config only. For example, IPv4 Tunnel Src/Dest is a very obvious one. The current ONF answer would be use OF-Config, in OpenDaylight we solved it using the awesomeness of the OVSDB protocol (RFC 7047) from Pfaff and Davie.

For for those in networking digging under the covers of SDN or developers implementing SDN solutions, I think its important to weigh in on this topic. Some incumbents would be fine with an outcome of SDN to be so fragmented that it resembles SNMP MIBs with little overall impact/disruption. I for one will use whatever protocol required as long as it is open and more important supported by the data plane. It would be nice if it is an agreed upon “standard” but that doesnt mean I need a committee of people not engaged with community. A community defacto can also be a standard and often a much more meaningful standard at that. Take a look at the document, Ben has added a ton of details that have only come from experience of being on the forefront of SDN from his leadership and development in the Open vSwitch project.

OpenFlow and Open vSwitch Fields 2nd Draft via benpfaff.orf →

Cya!

Tags: , , , Categories: 

About the Author

Brent SalisburyI have over 20 years of experience wearing various hats from, network engineer, architect, ops and software engineer. More at Brent's LinkedInView all posts by Brent Salisbury →

  1. VicVic03-07-2014


    Brent,

    Thanks for the post regarding this setup. I read it and it’s Greek to me, where do I go to learn more about the basics of this? Thanks!

    • David MahlerDavid Mahler03-11-2014


      Vic – openvswitch.org should help. You can also subscribe/read the mailing list there. For the flow entries Brent is showing http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities/ovs-ofctl.8 For commands related to adding/configuring OVS bridges http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities/ovs-vsctl.8. There is also a tutorial under documentation at openvswitch.org

      Brent – great post. The quote below was an ‘aha’ moment for me on some confusion I had about where TEPs need to be.

      “You don’t need to add physical eth0 interfaces into a bridge for this. your TEP only needs to use any address on the Linux host that can reach the destination tunnel endpoint.”

      • Brent SalisburyBrent Salisbury03-11-2014


        Hi David, That is a great point! The tunnels are stateless which is also handy very much like something else we know, good old MPLS 🙂 When I originally started working w/OVS it was prior to OpenStack so I was more in the traditional mindset which involved tying it to the “normal” network to VLANs etc. In that case adding a NIC iface like eth0 (ovs-vsctl add-port br0 eth0) made sense. If you look at br-ex in an OpenStack environment, that is all it is doing, br-int w/ flowmods facing the overlay and br-ex w/ a normal OF rule to flood and learn.

        There are also hidden flows with a high priority for in-band OF controller / OVSDB manager connectivity. Ben Pfaff talks about this in the FAQ which he puts a bunch of time in documenting for the community. Great read if you get a second, http://goo.gl/EcwKAx

        Here is the output w/ hidden flows from an OpenStack controller node:

        $ sudo ovs-appctl bridge/dump-flows br-int
        duration=71391s, priority=180008, n_packets=0, n_bytes=0, priority=180008,tcp,nw_src=172.16.86.1,tp_src=6640,actions=NORMAL
        duration=71391s, priority=180008, n_packets=0, n_bytes=0, priority=180008,tcp,nw_src=172.16.86.1,tp_src=6633,actions=NORMAL
        duration=71391s, priority=180000, n_packets=0, n_bytes=0, priority=180000,udp,in_port=LOCAL,dl_src=72:d8:e9:6e:5c:43,tp_src=68,tp_dst=67,actions=NORMAL
        duration=71391s, priority=32768, n_packets=0, n_bytes=0, in_port=1,dl_src=fa:16:3e:97:75:49,actions=set_field:0x1->tun_id,goto_table:10
        duration=71391s, priority=32768, n_packets=0, n_bytes=0, in_port=1,dl_src=fa:16:3e:ea:82:de,actions=set_field:0x2->tun_id,goto_table:10
        duration=71391s, priority=180006, n_packets=0, n_bytes=0, priority=180006,arp,arp_spa=172.16.86.1,arp_op=1,actions=NORMAL
        duration=71391s, priority=8192, n_packets=0, n_bytes=0, priority=8192,in_port=1,actions=drop
        duration=2654s, priority=32768, n_packets=0, n_bytes=0, dl_type=0x88cc,actions=CONTROLLER:56
        duration=71391s, priority=180002, n_packets=0, n_bytes=0, priority=180002,arp,dl_src=72:d8:e9:6e:5c:43,arp_op=1,actions=NORMAL
        duration=71391s, priority=180004, n_packets=0, n_bytes=0, priority=180004,arp,dl_src=00:50:56:f0:56:cd,arp_op=1,actions=NORMAL
        duration=71391s, priority=180001, n_packets=0, n_bytes=0, priority=180001,arp,dl_dst=72:d8:e9:6e:5c:43,arp_op=2,actions=NORMAL
        duration=71391s, priority=180003, n_packets=0, n_bytes=0, priority=180003,arp,dl_dst=00:50:56:f0:56:cd,arp_op=2,actions=NORMAL
        duration=71391s, priority=180005, n_packets=0, n_bytes=0, priority=180005,arp,arp_tpa=172.16.86.1,arp_op=2,actions=NORMAL
        duration=71391s, priority=180007, n_packets=0, n_bytes=0, priority=180007,tcp,nw_dst=172.16.86.1,tp_dst=6640,actions=NORMAL
        duration=71391s, priority=180007, n_packets=0, n_bytes=0, priority=180007,tcp,nw_dst=172.16.86.1,tp_dst=6633,actions=NORMAL
        table_id=10, duration=71391s, priority=8192, n_packets=0, n_bytes=0, priority=8192,tun_id=0x1,actions=goto_table:20
        table_id=10, duration=71391s, priority=8192, n_packets=0, n_bytes=0, priority=8192,tun_id=0x2,actions=goto_table:20
        table_id=20, duration=71391s, priority=8192, n_packets=0, n_bytes=0, priority=8192,tun_id=0x1,actions=drop
        table_id=20, duration=71391s, priority=8192, n_packets=0, n_bytes=0, priority=8192,tun_id=0x2,actions=drop
        table_id=20, duration=71391s, priority=16384, n_packets=0, n_bytes=0, priority=16384,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=output:1
        table_id=20, duration=71391s, priority=16384, n_packets=0, n_bytes=0, priority=16384,tun_id=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=output:1
        table_id=20, duration=71391s, priority=32768, n_packets=0, n_bytes=0, tun_id=0x2,dl_dst=fa:16:3e:ea:82:de,actions=output:1
        table_id=20, duration=71391s, priority=32768, n_packets=0, n_bytes=0, tun_id=0x1,dl_dst=fa:16:3e:97:75:49,actions=output:1
        table_id=254, duration=71457s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x3,actions=drop
        table_id=254, duration=71457s, priority=0, n_packets=17, n_bytes=2082, priority=0,reg0=0x1,actions=controller(reason=no_match)
        table_id=254, duration=71457s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x2,actions=drop

        Thanks for the comments and assistance. You made a great breakthrough! That was one when I finally realized the overlay had the same Aha moment!

        Respect,
        -Brent

    • Brent SalisburyBrent Salisbury03-16-2014


      Hi Vic, I would start out just getting a solid understanding of Hypervisors and switching traffic for VMs. Just starting out with Virtual Box ( https://www.virtualbox.org/wiki/Downloads ) and installing Fedora and Open vSwitch will teach you a ton about Linux and OVS which is the perfect place to start in my opinion. Your networking background will shine when working on these Linux tools when compared to the systems admin/engineer brethren.

      Cheers and thanks for the feedback!
      -Brent

  2. Dave LenrowDave Lenrow03-12-2014


    Oh yeah. Well I’ll setup ClosedStack, ClosedNightlight, ClosedpSwitch, ClosedFlow, and CVSDB, and kick your butt with a lot of closed stuff. Kidding of course. Great stuff Brent.

    • Brent SalisburyBrent Salisbury03-16-2014


      Haha, thanks Dave!! You the man! Added a bit to the end of the post, curious on your thoughts bro.

  3. Giang NguyenGiang Nguyen03-12-2014


    Hello Brent, Thank you for your interesting post

    I can not see anything from the last sentence included reference links. Please fix it

    Best regards