Open vSwitch GRE Tunnel Configuration

Open vSwitch GRE Tunnel Configuration

Open Vswitch Lab

Open vSwitch GRE Tunnel Configuration
Open vSwitch Updated Tutorial
-I updated this September 20, 2012. Also adding this PDF from a instruction I did over the summer at an Internet2 conference. Download the lab here OpenFlow and Open vSwitch GRE Tunnel Configuration Tutorial.
-Another lab building from source with video and pretty vetted is  Configuring VXLan and GRE Tunnels on OpenvSwitch

Open vSwitch GRE Tunnel Screencast

We continue to destroy our nice clean data centers to facilitate poor application architectures that do not scale in a layer 3 fashion. We continue to come up with new technologies using old architectures to turn our networks into flat data centers under the guise of “cloud mobility”. Ok sorry, I got that out of my system.

Open vSwitch and Data Center Overlays

Open vSwitch is a virtual switch or virtual Ethernet bridge (VEB) becoming quite popular and being backed by many vendors as the alternative to the virtual switches in some of the productized hypervisors like Hyper-V, VMware etc. Overlays are not new concepts, but are being used in new use cases. It is an early mechanism to uncouple the virtual from the physical. What remains to be seen, is whether this is a bridge until hardware becomes generalized or the new standard.

The OVS developers is still ironing out the control portion of OVS, with that will come the scale of L3 addressing and multipathing to each Tunnel End Point (TEP). The current code supports the GRE key lookup and forwarding while NVGRE and VXlan have at least taken a stab at the control plane, unfortunately through multicast for broadcast flooding L Next-gen ASICs will and Ethernet controllers will open the door to more plates of spaghetti.

Figure 1.  Generic look at where the headers are being modified.

I am still on the fence on where a TEP should reside, on a physical top of rack end of row network device or reside in the hypervisor. The networking guy in me would prefer that to be an S-tag in a pseudowire, but I can’t help but lean towards anything in software, moving forward to provide programmatic scale. We are on the path of the programmatic abstraction to being migrated into our network operating systems, but that is a big ship that will presumably take a while to steer.

Enough gibber on to an L2GRE configuration on an Open vSwitch. I did this because I was surprised to not see a GRE cookbook on the Open vSwitch cookbook page.

BTW why the kitchen cooking trend these days? What happened to the good old how-to? Not cloudy enough, that needed abstraction also. Do we change RTFM to RTFC? I blame the Opscode Chef Openstack installer.

Open vSwitch Installation Tutorials
  • First install Open vSwitch on a couple of nodes. The following links will help you get started:

gre openvswitch

Figure 2. Here is the topology of the configuration.

Open vSwitch 1 configuration

Open vSwitch 2 configuration

Hypervisor 2 Configuration:

  • Add KVM tap VIF script

/etc/ovs-ifup and /etc/ovs-ifdown

#Paste in the following with br1 = equalling the bridge you made in ovs-vsctl. Since we want the VM on the bridge that will get tunneled we are not using br0.
Create this script for provisioning.

Then one for ifdown to de-provision.

  • Now spin up your VM. I use the following to boot Ubuntu off CD. The VM instance will tap br1 (bridge1) on the network.

Attach OpenvSwitch to a Floodlight openFlow Controller

Hypervisor 1

Hypervisor 2

Some commands to view the configuration and ports:

ubuntu-12:brent# ovs-appctl fdb/show bra (generate traffic ie ping as table age out after 300 seconds)
 port  VLAN  MAC                Age
    9     0  12:74:52:cc:cc:10    0 (VM 1)
   12     0  12:34:52:cc:cc:10    0 (VM 2)

openvswitch vlan

Figure 2. Now we have flattened out these two endpoints whether in the same data center or across the city/state that can be used for a live workload migration (e.g. vMotion).

openvswitch gre tunnels

Figure 3. There is some scale to where this can go, for either provisioning elastic compute nodes or migrating, live migrations or just fixing broken applications.

  1. lanbolanbo06-20-2012

    Hi, Brent. Which version of Open vSiwtch did you use? Once i add a gre tunnel between two hosts, one of the host will crash and i have to reboot the machine. Do you have any idea what is going on with my ovs? The ovs i am using are 1.5.0 on a Debian Squeeze host and 1.4.0 on a Ubuntu 12.04 host. Both hosts have only one nic.

    • Brent SalisburyBrent Salisbury06-21-2012

      Hi Lanbo,

      The version was ovs_version: “1.4.0+build0” from the repos.

      Weird it all crashed. Can you paste some output and I can maybe help with it.

      When I get a second to breath this week I was planning on revisiting a couple of tunneling aspects in OVS. I will update a new post on that if that helps.


    • Brent SalisburyBrent Salisbury08-18-2012

      So Sorry, lanbo, missed your message. I am betting it is a loop there. Maybe make sure you are adding a physical nic to the right bridge and not both or else a bridging loop spins up. Just answering in case anyone else has the problem. Sorry again for missing it.

  2. changliweichangliwei06-25-2012

    I just try to follow the steps you posted under ubuntu12.04 and ovs ovs_version: “1.4.0+build0”
    but it gets error.
    root@c10:~# ovs-vsctl add-port br0 gre0 –set interface gre0 type=gre options:remote_ip=
    ovs-vsctl: Port does not contain a column whose name matches “–set”

    also if it is a must to add gre module to the linux kernel,and my kernel is
    root@c10:~# uname -r
    root@c10:~# lsmod | grep gre

    Thank you in advance, you timely reply is welcomed.

    • Brent SalisburyBrent Salisbury06-27-2012

      Hi, Try using the install from packages here.

      Once you install this should work.

      Build GRE Tunnel
      On host ovs1 add:
      ovs-vsctl add-port br1 gre1 — set interface gre1 type=gre options:remote_ip=
      On host ovs2 add:
      ovs-vsctl add-port br1 gre1 — set interface gre1 type=gre options:remote_ip=

      Just vetted it last night. Let me know if it doesnt work from packages.


  3. BobBob07-30-2012

    If I have 3 datacenters, the VMs on the datacenters are able to communicate with eachother via GRE tunnel, what should I do?

  4. BobBob07-30-2012

    Is there any way to configure multiple remote_ip for a host?
    Thanks a lot.

  5. Brent SalisburyBrent Salisbury07-30-2012

    Hi Bob, Absolutely. A GRE can be built over a routed network. You could build a tunnel between data centers even if they were only connected by the Internet. Take a look at Capwap also. It is essesntially mac-in-gre.

    • BobBob07-31-2012

      Thanks for your reply, is there any existing tools to setup GRE tunnel over a routed network? The only way is to use Capwap?

  6. christopherchristopher09-20-2012

    Hey Brent, Is it me, or are your ifup/down scripts both ifup scripts? They appear to be a copy/paste of the ifup version.

  7. christopherchristopher09-20-2012

    sorry to nitpick, but again, is the switch2 gre remote_ip correct? it also appeasrs to be a copy/paste booboo (and in the image above, both hypervisors .155 – is that intentional?)

    • Brent SalisburyBrent Salisbury09-21-2012

      Hi christopher, Ugh, I went through and saw more errors. Thanks for pointing those out. I redid the post, updated the picture, added a lab I put together over the summer and added a FloodLight controller. Take a peek if interested.
      Also wrote a python script to install all this stuff to make it quicker and easier for those who arent experienced w/Linux. I will get it posted in a bit.

  8. RogerRoger11-13-2012

    Hi brent,
    I don’t understand how br2 can have connectivity. Since eth0 is connected to br1, and eth0 is the outside interface, eth0 is not connected to br2. I don’t understand the island connected term you use to say that br2 is connected to eth0. What does it mean, how do you configure that?


  9. TalhaTalha01-17-2013

    You have not installed any flow in any of the bridges. I wonder how this is working ? The vm-island bridge + gre bridge both need to have NORMAL(L2 learning switch) openflow action is order for this to work. Am i missing on something?Correct me if i am wrong.. Is you controller sending those flows?

  10. TalhaTalha01-17-2013

    Same comment question hold for all of the posts that you have done. Can you clarify?

  11. DougDoug03-28-2013

    Hey Brent,

    I have a quick question for you regarding OpenStack Folsom and GRE tunnels. I used this as an install guide
    I am trying to figure out how spanning tree works when gre tunnels connect 3 OVS’s together in a loop. For example lets say I ping a VM that lives on a different compute node. It looks like the ARP goes over each GRE tunnel I have configured on my bridge on the node where my vm lives. But for whatever reason the ARP isn’t looping around and around.

    So here’s the setup.

    3 nodes.
    Each running ovs on them.
    Each has a gre tunnel to eachothers OVS. (So there is a loop in place via gre tunnels)
    Howcome ARPs don’t loop around and around over GRE tunnels?

  12. DavidCDavidC05-13-2013

    I wonder why two OVSs above are connected to two separate Floodlight controllers.
    How can I connect two OVSs to one Floodlight controller?


  13. Brent SalisburyBrent Salisbury05-13-2013

    Hi David, I think I cleaned a couple of things that had changed the other day and typoed that. Appreciate you pointing it out. You definently are right that you would want to use the same island since state would be centralized there. Appreciate the comment.


  14. AhmedAhmed06-06-2013

    I want to create a full-mesh topology whith GRE. Each physical node has gre tunnels to other nodes. When I tested this whith 5 nodes my VMs don’t respond to ping. However in star topology it works fine.
    My question now, does OVS support mesh-topology based on multiple gre tunnels or GRE it self couldn’t work in mesh topology ?

  15. Jinhwan HwangJinhwan Hwang07-18-2013

    How does GRE traffic work after arrive its remote destination?. It looks like that there are no connection between br1 and br2 at the same host. I could understand how traffic from br2 at host A go through br1 at host B. But I couldn’t understand traffic finds way to br2 at host B from br1 at host B(gre destination).

  16. Carmine BufanoCarmine Bufano08-09-2013

    Thank you Brent for the wealth of information and easy to follow labs! I’ve recently become really interested in sdn and 9 out of 10 times when I consult google with questions it directs me here.. Your website has become my starting point in this learning journey and your examples, labs, and information is easy to follow and has really softened the hard learning curve. You are providing a great service to anyone trying to wrap their head around software defined networking and virtual networks! Thanks!!!!

  17. AlexAlex11-26-2013

    Is it possible to have GRE tunnel on a single VM (Mininet) with two switches?

  18. HemanthHemanth02-09-2014

    I am getting the following error when I am creating a VM

    “/etc/ovs-ifup: could not launch network script
    kvm: -net tap,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown: Device ‘tap’ couldnt be initialized”

    I created both the scripts and made them executable. But still getting the same error.

  19. suryasurya02-21-2014

    Hello Brent,

    I didnt understand how openvswitch works with gre tunnels.
    I have a controller node and a compute node both with single NIC.

    I have launched 2 VM’s,one on controller(which also runs nova-compute) and the other on compute node. I want to have them connected.

    These are my openswitch details

    sudo ovs-vsctl show
    Bridge br-tun
    Port br-tun
    Interface br-tun
    type: internal
    Port patch-int
    Interface patch-int
    type: patch
    options: {peer=patch-tun}
    Port “gre-2”
    Interface “gre-2″
    type: gre
    options: {in_key=flow, local_ip=”″, out_key=flow, remote_ip=””}
    Bridge br-int
    Port “tapca334e0f-28”
    tag: 1
    Interface “tapca334e0f-28”
    Port “qr-a230168b-cb”
    tag: 1
    Interface “qr-a230168b-cb”
    type: internal
    Port patch-tun
    Interface patch-tun
    type: patch
    options: {peer=patch-int}
    Port “tap32a7ff69-c7”
    tag: 1
    Interface “tap32a7ff69-c7”
    type: internal
    Port br-int
    Interface br-int
    type: internal
    Bridge br-ex
    Port br-ex
    Interface br-ex
    type: internal
    Port “qg-1639690a-93”
    Interface “qg-1639690a-93”
    type: internal
    ovs_version: “1.4.6”

    controller IP is
    Compute node IP is

    brctl show(controller)
    bridge name bridge id STP enabled interfaces
    br-ex 0000.4e93a1828c40 no qg-1639690a-93
    br-int 0000.d681d06ea54b no qr-a230168b-cb
    br-tun 0000.e6544857944d no
    br0 0000.1e958c251840 no
    virbr0 8000.000000000000 yes

    and openflows defined on the tunneling bridge are

    sudo ovs-ofctl dump-flows br-tun
    NXST_FLOW reply (xid=0x4):
    cookie=0x0, duration=21.02s, table=0, n_packets=0, n_bytes=0, tun_id=0x1 actions=mod_vlan_vid:1,output:1
    cookie=0x0, duration=21.015s, table=0, n_packets=0, n_bytes=0, in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL
    cookie=0x0, duration=21.009s, table=0, n_packets=0, n_bytes=0, actions=drop

    Whenever I launch a vm with the above opeflow rules loaded,openvswitch is crashing(vswitchd).Could you please help me in solving this.

    And also could you please help me in understanding the following qeuries

    how br-tun(tunneling bridge) forwads packets to the remote node with out any physical interface attached.

    how does the action “NORMAL” work (ex action: NORMAL in a openflow rule)

    Does the tunneling bridge needs a physical interface connectivity

    Please note that I’m not using a openflow controller