Open vSwitch GRE Tunnel Configuration

[fancy_header3 variation=”orange”]Open vSwitch GRE Tunnel Configuration[/fancy_header3]

[colored_box title=”Open vSwitch Updated Tutorial” variation=”silver”]-I updated this September 20, 2012. Also adding this PDF from a instruction I did over the summer at an Internet2 conference. Download the lab here OpenFlow and Open vSwitch GRE Tunnel Configuration Tutorial.
-Another lab building from source with video and pretty vetted is  Configuring VXLan and GRE Tunnels on OpenvSwitch[/colored_box]

[fancy_header3 variation=”orange”]Open vSwitch GRE Tunnel Screencast[/fancy_header3]

[youtube url=”http://www.youtube.com/watch?v=VMP4Q2XtFjw” width=”635″ height=”360″ hd=”1″]

We continue to destroy our nice clean data centers to facilitate poor application architectures that do not scale in a layer 3 fashion. We continue to come up with new technologies using old architectures to turn our networks into flat data centers under the guise of “cloud mobility”. Ok sorry, I got that out of my system.

[fancy_header3 variation=”orange”]Open vSwitch and Data Center Overlays[/fancy_header3]

Open vSwitch is a virtual switch or virtual Ethernet bridge (VEB) becoming quite popular and being backed by many vendors as the alternative to the virtual switches in some of the productized hypervisors like Hyper-V, VMware etc. Overlays are not new concepts, but are being used in new use cases. It is an early mechanism to uncouple the virtual from the physical. What remains to be seen, is whether this is a bridge until hardware becomes generalized or the new standard.

The OVS developers is still ironing out the control portion of OVS, with that will come the scale of L3 addressing and multipathing to each Tunnel End Point (TEP). The current code supports the GRE key lookup and forwarding while NVGRE and VXlan have at least taken a stab at the control plane, unfortunately through multicast for broadcast flooding L Next-gen ASICs will and Ethernet controllers will open the door to more plates of spaghetti.

Figure 1.  Generic look at where the headers are being modified.

I am still on the fence on where a TEP should reside, on a physical top of rack end of row network device or reside in the hypervisor. The networking guy in me would prefer that to be an S-tag in a pseudowire, but I can’t help but lean towards anything in software, moving forward to provide programmatic scale. We are on the path of the programmatic abstraction to being migrated into our network operating systems, but that is a big ship that will presumably take a while to steer.

Enough gibber on to an L2GRE configuration on an Open vSwitch. I did this because I was surprised to not see a GRE cookbook on the Open vSwitch cookbook page. http://openvswitch.org/support/config-cookbooks/

BTW why the kitchen cooking trend these days? What happened to the good old how-to? Not cloudy enough, that needed abstraction also. Do we change RTFM to RTFC? I blame the Opscode Chef Openstack installer.

[fancy_header3 variation=”orange”]Open vSwitch Installation Tutorials[/fancy_header3]

  • First install Open vSwitch on a couple of nodes. The following links will help you get started:

gre openvswitch

Figure 2. Here is the topology of the configuration.

Open vSwitch 1 configuration

Open vSwitch 2 configuration

Hypervisor 2 Configuration:

  • Add KVM tap VIF script

/etc/ovs-ifup and /etc/ovs-ifdown

#Paste in the following with br1 = equalling the bridge you made in ovs-vsctl. Since we want the VM on the bridge that will get tunneled we are not using br0.
Create this script for provisioning.

Then one for ifdown to de-provision.

  • Now spin up your VM. I use the following to boot Ubuntu off CD. The VM instance will tap br1 (bridge1) on the 10.1.1.0/24 network.

[fancy_header3 variation=”orange”]Attach OpenvSwitch to a Floodlight openFlow Controller[/fancy_header3]

Hypervisor 1

Hypervisor 2

Some commands to view the configuration and ports:

ubuntu-12:brent# ovs-appctl fdb/show bra (generate traffic ie ping as table age out after 300 seconds)
 port  VLAN  MAC                Age
    9     0  12:74:52:cc:cc:10    0 (VM 1)
   12     0  12:34:52:cc:cc:10    0 (VM 2)

openvswitch vlan

Figure 2. Now we have flattened out these two endpoints whether in the same data center or across the city/state that can be used for a live workload migration (e.g. vMotion).

openvswitch gre tunnels

Figure 3. There is some scale to where this can go, for either provisioning elastic compute nodes or migrating, live migrations or just fixing broken applications.

29 comments


  1. Hi, Brent. Which version of Open vSiwtch did you use? Once i add a gre tunnel between two hosts, one of the host will crash and i have to reboot the machine. Do you have any idea what is going on with my ovs? The ovs i am using are 1.5.0 on a Debian Squeeze host and 1.4.0 on a Ubuntu 12.04 host. Both hosts have only one nic.


    1. Hi Lanbo,

      The version was ovs_version: “1.4.0+build0” from the repos.

      Weird it all crashed. Can you paste some output and I can maybe help with it.

      When I get a second to breath this week I was planning on revisiting a couple of tunneling aspects in OVS. I will update a new post on that if that helps.

      Thanks,
      -Brent


    2. So Sorry, lanbo, missed your message. I am betting it is a loop there. Maybe make sure you are adding a physical nic to the right bridge and not both or else a bridging loop spins up. Just answering in case anyone else has the problem. Sorry again for missing it.


  2. I just try to follow the steps you posted under ubuntu12.04 and ovs ovs_version: “1.4.0+build0”
    but it gets error.
    root@c10:~# ovs-vsctl add-port br0 gre0 –set interface gre0 type=gre options:remote_ip=10.50.8.9
    ovs-vsctl: Port does not contain a column whose name matches “–set”

    also if it is a must to add gre module to the linux kernel,and my kernel is
    root@c10:~# uname -r
    3.2.0-26-generic
    root@c10:~# lsmod | grep gre
    root@c10:~#

    Thank you in advance, you timely reply is welcomed.


    1. Hi, Try using the install from packages here. http://networkstatic.net/2012/06/openflow-openvswitch-lab/

      Once you install this should work.

      Build GRE Tunnel
      On host ovs1 add:
      ovs-vsctl add-port br1 gre1 — set interface gre1 type=gre options:remote_ip=172.31.246.23
      On host ovs2 add:
      ovs-vsctl add-port br1 gre1 — set interface gre1 type=gre options:remote_ip=172.31.246.21

      Just vetted it last night. Let me know if it doesnt work from packages.

      Thanks!
      -Brent


  3. Hi,
    If I have 3 datacenters, the VMs on the datacenters are able to communicate with eachother via GRE tunnel, what should I do?


  5. Hi Bob, Absolutely. A GRE can be built over a routed network. You could build a tunnel between data centers even if they were only connected by the Internet. Take a look at Capwap also. It is essesntially mac-in-gre.


    1. Thanks for your reply, is there any existing tools to setup GRE tunnel over a routed network? The only way is to use Capwap?


  6. Hey Brent, Is it me, or are your ifup/down scripts both ifup scripts? They appear to be a copy/paste of the ifup version.


  7. sorry to nitpick, but again, is the switch2 gre remote_ip correct? it also appeasrs to be a copy/paste booboo (and in the image above, both hypervisors .155 – is that intentional?)


    1. Hi christopher, Ugh, I went through and saw more errors. Thanks for pointing those out. I redid the post, updated the picture, added a lab I put together over the summer and added a FloodLight controller. Take a peek if interested.
      Also wrote a python script to install all this stuff to make it quicker and easier for those who arent experienced w/Linux. I will get it posted in a bit.
      Thanks!
      -brent


  8. Hi brent,
    I don’t understand how br2 can have connectivity. Since eth0 is connected to br1, and eth0 is the outside interface, eth0 is not connected to br2. I don’t understand the island connected term you use to say that br2 is connected to eth0. What does it mean, how do you configure that?

    Thanks,
    Roger


  9. You have not installed any flow in any of the bridges. I wonder how this is working ? The vm-island bridge + gre bridge both need to have NORMAL(L2 learning switch) openflow action is order for this to work. Am i missing on something?Correct me if i am wrong.. Is you controller sending those flows?


  11. Hey Brent,

    I have a quick question for you regarding OpenStack Folsom and GRE tunnels. I used this as an install guide https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/stable/GRE/OpenStack_Folsom_Install_Guide_WebVersion.rst
    I am trying to figure out how spanning tree works when gre tunnels connect 3 OVS’s together in a loop. For example lets say I ping a VM that lives on a different compute node. It looks like the ARP goes over each GRE tunnel I have configured on my bridge on the node where my vm lives. But for whatever reason the ARP isn’t looping around and around.

    So here’s the setup.

    3 nodes.
    Each running ovs on them.
    Each has a gre tunnel to eachothers OVS. (So there is a loop in place via gre tunnels)
    Howcome ARPs don’t loop around and around over GRE tunnels?


  12. I wonder why two OVSs above are connected to two separate Floodlight controllers.
    How can I connect two OVSs to one Floodlight controller?

    Thanks,
    David


  13. Hi David, I think I cleaned a couple of things that had changed the other day and typoed that. Appreciate you pointing it out. You definently are right that you would want to use the same island since state would be centralized there. Appreciate the comment.

    Respect,
    -Brent


  14. I want to create a full-mesh topology whith GRE. Each physical node has gre tunnels to other nodes. When I tested this whith 5 nodes my VMs don’t respond to ping. However in star topology it works fine.
    My question now, does OVS support mesh-topology based on multiple gre tunnels or GRE it self couldn’t work in mesh topology ?


  15. How does GRE traffic work after arrive its remote destination?. It looks like that there are no connection between br1 and br2 at the same host. I could understand how traffic from br2 at host A go through br1 at host B. But I couldn’t understand traffic finds way to br2 at host B from br1 at host B(gre destination).


  16. Thank you Brent for the wealth of information and easy to follow labs! I’ve recently become really interested in sdn and 9 out of 10 times when I consult google with questions it directs me here.. Your website has become my starting point in this learning journey and your examples, labs, and information is easy to follow and has really softened the hard learning curve. You are providing a great service to anyone trying to wrap their head around software defined networking and virtual networks! Thanks!!!!
    -Carmine


  18. I am getting the following error when I am creating a VM

    “/etc/ovs-ifup: could not launch network script
    kvm: -net tap,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown: Device ‘tap’ couldnt be initialized”

    I created both the scripts and made them executable. But still getting the same error.


  19. Hello Brent,

    I didnt understand how openvswitch works with gre tunnels.
    I have a controller node and a compute node both with single NIC.

    I have launched 2 VM’s,one on controller(which also runs nova-compute) and the other on compute node. I want to have them connected.

    These are my openswitch details

    sudo ovs-vsctl show
    8a44add1-8232-4a90-8638-5ef54073e05b
    Bridge br-tun
    Port br-tun
    Interface br-tun
    type: internal
    Port patch-int
    Interface patch-int
    type: patch
    options: {peer=patch-tun}
    Port “gre-2”
    Interface “gre-2″
    type: gre
    options: {in_key=flow, local_ip=”172.16.201.32″, out_key=flow, remote_ip=”172.16.201.38”}
    Bridge br-int
    Port “tapca334e0f-28”
    tag: 1
    Interface “tapca334e0f-28”
    Port “qr-a230168b-cb”
    tag: 1
    Interface “qr-a230168b-cb”
    type: internal
    Port patch-tun
    Interface patch-tun
    type: patch
    options: {peer=patch-int}
    Port “tap32a7ff69-c7”
    tag: 1
    Interface “tap32a7ff69-c7”
    type: internal
    Port br-int
    Interface br-int
    type: internal
    Bridge br-ex
    Port br-ex
    Interface br-ex
    type: internal
    Port “qg-1639690a-93”
    Interface “qg-1639690a-93”
    type: internal
    ovs_version: “1.4.6”

    controller IP is 172.16.201.32
    Compute node IP is 172.16.201.38

    brctl show(controller)
    bridge name bridge id STP enabled interfaces
    br-ex 0000.4e93a1828c40 no qg-1639690a-93
    br-int 0000.d681d06ea54b no qr-a230168b-cb
    tap32a7ff69-c7
    br-tun 0000.e6544857944d no
    br0 0000.1e958c251840 no
    virbr0 8000.000000000000 yes

    and openflows defined on the tunneling bridge are

    sudo ovs-ofctl dump-flows br-tun
    NXST_FLOW reply (xid=0x4):
    cookie=0x0, duration=21.02s, table=0, n_packets=0, n_bytes=0, tun_id=0x1 actions=mod_vlan_vid:1,output:1
    cookie=0x0, duration=21.015s, table=0, n_packets=0, n_bytes=0, in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL
    cookie=0x0, duration=21.009s, table=0, n_packets=0, n_bytes=0, actions=drop

    Whenever I launch a vm with the above opeflow rules loaded,openvswitch is crashing(vswitchd).Could you please help me in solving this.

    And also could you please help me in understanding the following qeuries

    how br-tun(tunneling bridge) forwads packets to the remote node with out any physical interface attached.

    how does the action “NORMAL” work (ex action: NORMAL in a openflow rule)

    Does the tunneling bridge needs a physical interface connectivity

    Please note that I’m not using a openflow controller

Comments are closed.