OpenDaylight OpenStack Integration with DevStack on Fedora

OpenDaylight OpenStack Integration with DevStack on Fedora

The following is a walk through of the OVSDB project within OpenDaylight for OpenStack integration. There are a couple of bugs so it is not for the faint of heart. It is intended for those looking to get their development environment up and running. We will have videos and what not walking through the installation and code reviews of the implementation. It was developed by folks from various vendors and users in the community. Some are in companies that aren’t dedicating resources to the project but are doing it in their personal time. Both Madhu and I are maintaining this post to keep it as accurate as possible and up to date with our own individual notes as we move into the next phase of development for Helium. Madhu is going to do a separate Fedora 19 post on a new blog he is setting up which I am pumped about! Until then, keep an eye out for Fedora19 and Fedora20 specific instructions below.

Recordings of the Installation

The following are some late night / early morning recordings from Madhu and me. In case anyone gets stuck and wants to follow along at home. They are all done on our laptops so resources are a bit tight. For questions and issues please send them to the OVSDB Listserv so that one of us or other awesome folks in the community can assist since our Q/A bandwidth is focused there to try and build a good collective of knowledge to share. If you have further interest in the project, just assisting others in the Listserv, documenting and code contributions are all amazing and much appreciated and respected.

• OpenStack/OpenDaylight/OVSDB Installation Part 1 – Configuring VirtualBox and VM Fusion

• OpenStack/OpenDaylight/OVSDB Installation Part 2 – Stacking and Spinning up Multi-Node OpenStack w/the OpenDaylight Controller

• For more OpenDaylight / OVSDB videos and weekly Hangouts please see our YouTube channel:

OpenDaylight OVSDB Channel →

OVSDB Project Control and Management Logic

No standards were hurt in the making of this recording. The only southbound protocols we used in the OVSDB project OpenStack implementation were OpenFlow v1.3 and OVSDB. We chose not to use any extensions or the use of agents. Open vSwitch supported the necessary OpenFlow v1.3 and OVSDB functionality we required for this architecture. Those of us in the OVSDB project are pretty agnostic to southbound protocols as long as there is a healthy adoption so as not to waste our time and based on open standards such as OpenFlow v1.3, RFCs 7047 (Informational OVSDB RFC) and/or de facto drafts like draft-mahalingam-dutt-dcops-vxlan (VXLAN framing). We are keen to see NXM extension functionality upstream into the OpenFlow specification. OVS ARP responder is something we are beginning to work on proofing now. NXM and OXM extensions merging for ARP and Tunnel feature parity would make our design and coding lives easier. The overall architecture looks something like the following. I have hardware TEPs in the diagram. We have cycles to help hardware vendors implement the hardware_vtep database schema (assuming they prescribe to open operating systems):

The provider segmentation keys used in the encap (GRE key/VNI) is a hash of Network and Tenant ID since as long as we are subnet bound, networks will always need to support multi-tenant logical networks until we eradicate L2 all together. The design is flexible and as generic as possible to allow for any vendor to add differentiation on top of the base network virtualization. Of course, we have plenty to do between now and stability, so moving right along.

• A quick visual of the OVSDB Neutron implementation code flow itself and how it ties into the controller project and OpenStack is the excellent diagram Madhu did:

Configure the Fedora Images for your Environment

There are two options for images. Fedora 19 and Fedora 20. We tend to recommend F19 due to an issue with MariaDB and hostnames in the F20 VM. More on that when we edit the hostname in the tutorial. For assistance with getting the stack going please ping the OVSDB Listserv and check the archives for answers.

Download the pre-built image we made that contains OpenDaylight, DevStack installing Ice House OpenStack, Open vSwitch all on Fedora:

Fedora 19 based all-in-one VM:

or if you prefer, you can download Fedora 20 based all-in-one VM:

Clone this Virtual Machine image into two images. One is for the Control (This VM runs both the OpenStack Controller and OpenDaylight Controller) and the other for the Compute instance. If you use VM Fusion the vanilla image works as is with no need to change any adaptor settings. Use the ‘ip addr’ configuration output as a reference in the next section. I recommend using SSH to connect to the host rather then using the TTY interface.

Here are two screenshots with VirtualBox network adaptor examples. The first are the two networks you can create. vxboxnet0 is there by default. Create the 2nd network with the +w/a nic picture in the following example. Note: you have to manually fill in the DHCP server settings on the new network. Refer to the existing if unsure of the values to use. When complete the host OS should be able to reach the guest OS.

The second example is what the VirtualBox NIC setup can look like without have to deal with the NAT Network option in VirtualBox. VM Fusion has integrated hooks in to resolve the need for host only etc. NAT and Host only work fine with NAT so the host can reach your networks default gateway and get to the Inets as needed. With host only that is not the case but it is plenty to run the stack and integration.

Boot both guest VMs write down the four IP addresses from both NICs. You will primarily only use one of them other then a gateway or out of band SSH connectivity etc.

If you are using the Fedora 19 VM, then please use the following credentials to login:

If you are using the Fedora 20 VM, then please use :

Oops 🙂

In this example the configuration of the IP addrs are as follows:

Record the IP addresses of both of the hosts:

Controller IP addresses:

Compute IP addresses:

Go to the home directory of the user id odl:

Start the OVS Service (DevStack should start this svc but I have seen this not on occasion fwiw). This startup script can be loaded at startup of OVS to load at the OS init.

Configure the /etc/hosts file to reflect your controller and compute hostname mappings. While not necessarily required it can cause issues for Nova output.

Verify the OpenStack Controller /etc/hosts file. The only edit is adding the compute IP to hostname mapping. E.g. x.x.x.x fedora-odl-2

You will need to edit the compute nodes /etc/hosts from fedora-odl-1 to fedora-odl-2:

Then change the compute hostname from (compute only):

Then reboot the cloned Compute node for the change to take affect:

After the host restarts verify the hostnames like so:

Unfortunately, in the Fedora 20 VM, if you comment out “#127.0.0.1 localhost fedora-odl-1” you will blow up MySql. (Thanks for digging into that Vijay!) So avoid doing any changes to the host name locally resolving to 127.0.0.1. Mestery has a link on one of his blogs to this issue also. Net is, leave localhost. If you change it and try and revert back it will still get angry.

Start OpenDaylight Controller on the OpenStack Controller Node

Check that the configuration is set for OpenFlow v1.3 with the following to ensure that ovsdb.of.version=1.3 is uncommented:

If it is not uncommented, adjust the config.ini file to uncomment the line ovsdb.of.version=1.3
The file is located at /home/odl/opendaylight/configuration/config.ini

Or, simply paste the following:

Lastly, start the ODL controller w/ the following:

When the controller if finished loading here are some typical messages in the OSGI console:

You can verify the sockets/ports are bound with the following command. Ports 6633, 6640 and 6653 should all be bound and listening:

Configure the DevStack for the OpenStack Controller

Make sure all bridges are removed only if you have previously “stacked”

Once the OpenDaylight Controller is running, stack the OpenStack Controller:

If you are using the Fedora 19 VM :

For the Fedora 20 VM :

Edit the local.conf you just copied with the appropriate IPs. Replace all instances with brackets to the Daylight SDN controller, the OpenStack controller IP or the Openstack compute IP (Compute ethX address is only on the compute node). In this example I am using the addresses for eth1 on the guest VM. Again ensure you have IP reachability between controller and compute API services when troubleshooting issues.

In the local.conf you will see four lines that require the hardcoding of an IP address. You can always replace some of those w/ variables but we thought it important to help understand the DevStack configuration and Neutron REST call to the ODL OpenStack API implemented in org.opendaylight.controller.networkconfig.neutron.implementation.

The following is the OpenStack controller local.conf for this tutorial:

Verify the local.conf by greping for the IP prefix used:

Finally execute the stack.sh shell script:

You should see activity in your OSGI console as Neutron adds the default private and public networks like so:

You will see more activity as ODL programs the OVSDB server running on the OpenStack node.

Here is the state of Open vSwitch after the stack completes and prior to booting a VM instance. If you do not see the is_connected: true boolean after Manager (OVSDB) and Controller (OpenFlow) something has gone wrong and check that the controller/manager IPs are reachable and the ports are bound using the lsof command listed earlier:

Next up is stacking the compute node.

Configure the OpenStack Compute Node

The compute configuration steps are virtually identical to the controller other then the configurations and that it does not run the Daylight controller.

If you are using the Fedora 19 VM :

For the Fedora 20 VM :

Edit the local.conf you just copied with the appropriate IPs in the devstack directory on the compute host like the following example with your controller and compute host IPs:

Or check the conf file quickly by grepping it.

And now stack the compute host:

Once you get the stack working SNAPSHOT the image 🙂 If you break a fraction of the things that I do, when I touch them snapshots can be a handy timesaver. Also leaving DevStack “offline=true” and “reclone=no” except for when you need to pull a patch. Get to your functioning stack and it is rock solid and incredibly useful and necessary in my experience. I would say we merry few consumed by this project are laser focused on simplification, not dragging along every use case under the sun starting out. Lets do the basics right and well while we have the chance.

The state of OVS after the stack should be the following:

Next up, verify functionality.

Verifying OpenStack is Functioning

Verify the stack with the following on either host. First we will list that there are two KVM hypervisors registered with Nova. *Note openrc will populate the proper Keystone credentials for service client commands. These can be viewed using the export command from your shell:

Before we can boot VM instances, there is one more minor configuration difference between the Fed19 and Fed20 VM.

If you are using the Fedora 19 VM,

If you are using Fedora 20 VM,

Next lets boot a couple of VMs and verify the network overlay is created by ODL/OVSDB. There are lots of ways to boot a VM but what we tend to use is the following one liner. We are going to boot off the default private network that was setup by Devstack:

Boot a second node:

You can also force a host to boot to a particular hypervisor using the following (note: this requires an admin role which is implicitly granted to the admin user):

View the state of the VMs with the following:

You can also look directly at Libvirt using Virsh. This is handy for determining where a host is located:

From here lets make sure we can ping the endpoints. For this I just use the shell by grabbing a namespace for qdhcp or qrouter. This provides an L3 source to ping the VMs. These will only exist on the controller or wherever you are running those services in your cloud:

Verify the OF13 flowmods short for “flow modifications”, or also referred to as “flow rules”, “flow tables”, “forwarding tables” or whatever you think it should be called.

You can also define new networks with encaps of VXLAN or GRE along with specifying the segmentation ID. In this case GRE:

And then boot those instances using those networks:

Create Multi Network Types, Gre and VXLAN

Suppose we want to create some hosts in an overlay using the VXLAN encap with specified segmentation IDs (VNIs):

Next take a look at the networks which were just created.

Now boot the VMS

You can pull up the Horizon UI and take a look at the nodes you have spun up by pointing your web browser at the controller IP (port 80).

Now lets ping one of the hosts we just created to verify it is functional:

Now create three new Neutron networks using the GRE encapsulation. Depending on your VM memory you can always run the risk of blowing up your VM with too many guest VMs.

Here is what the OVS config looks like. Remember, since the tunnel ID is being set using the OpenFlow OXM metadata field to to set the logical port OFPXMT_OFB_TUNNEL_ID implemented in OpenFlow v1.3.

Neutron mappings from the Neutron client output:

Next take a look at the Open vSwitch configuration. Worthy of note is the tunnel IPv4 src/dest endpoints are defined using OVSDB but the Tunnel ID is set using the flowmod in OpenFlow using key=flow. This tells OVSDB to look for the tunnel ID in the flowmod. There is also a similar concept for IPv4 tunnel source/destination using Nicira extensions with NXM_NX_TUN_IPV4_SRC and NXM_NX_TUN_IPV4_DST that was implemented in OVS 2.0. The NXM code points are referenced in the OF v1.3 specification but it seems pretty nascent wether the ONF is looking to handle tunnel operations with OF-Config or via flowmods such as the NXM references. The NXM code points are defined the ODL openflowjava project that implements the library model for OFv1.3 and would just need to be plumbed through the MD-SAL convertor.

And then the OF v1.3 flowmods:

For more on TEPs please see a nice document authored by Ben Pfaff who needs no introduction, that can be found here.

Next take a look at the flowmods. We simply break the pipeline into three tables, a classifier, egress and ingress. Over the next 6 months we will be adding services into pipeline for a much more complete implementation. We are looking for user contributions in the roadmap and even better, pushing code upstream as the project continues to grow.

Lastly if you want to force availability zones from say the “demo” UID. You can add the admin role to different UIDs using the following Keystone client calls.

Unstack and Cleanup DevStack

Getting used to this setup will take trial and error. Use the following to teardown the stack and reset the state of the VM to pre-stack.

running unstack.sh will kill the stack. It doesn’t hurt to look at the OVS config and make sure all bridges have been deleted:

A slightly lame cleanup, but handy nonetheless, is to run a few commands to ensure the stack was effectively torn down. Paste the following to create a shell script called ./reallyunstack.sh. Madhu and I always laugh about this one but it can help level set the platform for debugging:

Thats all for now. I am on my way back from the OpenDaylight Summit and have to catch a redeye. My cycles are a wreck at the moment for keeping up with comments so if you have any issues that require attention please shoot them to the ovsdb-dev listserv at OpenDaylight or even better jumping on the irc channel located at #opendaylight-ovsdb on irc.freenode.net. We are driving hard over the next few months for the next release so please be patient with bugs and lack of cycles at times. Cloud infrastructures are complex, troubleshooting is the best way to learn the various frameworks. Hopefully more will get to hacking and join the community to help grow the project and one another. Concrete is what matters now. Plenty of time for talk later.

Thanks much to the relentless work from the key guys on the coding and integration effort:

• Madhu Venugopal (Red Hat)@MadhuVenugopal
• Kyle Mestery @mestery (Cisco)
• (Ryan Moats (IBM) if he would ever get on Twitterz ehem :_)

Thanks to all the folks who contributed and certainly give a follow to some of the guys on Twitters that either directly or indirectly (out of time to list them all) in making this happen like (Cisco) @kernelcdub (Red Hat) @dave_tucker (HP) @edwarnicke (Cisco), Arvind (Cisco) @FlorianOtel (HP) @ashw7n @evanzeller @alagalah (Nexusis), Bachman, Meo, Hague, DMM, Anees, Dixon, phrobb and of course thanks to the creators of OVS @mcasado and @Ben_Pfaff and the rest of the OVS team which without we would not be able to do a project such as this.

Thanks for stopping by!