OpenStack Folsom Quantum Devstack Installation Tutorial

OpenStack Folsom Quantum Devstack Installation Tutorial

DevStack Openstack Lab

OpenStack Folsom Quantum Devstack Installation Tutorial and Screencast

This is a quick guide that includes a diagram of a working reference architecture for installation of the OpenStack Folsom release using which includes the Quantum networking component using the DevStack installation bash script.

With the OpenStack Essex release, I was writing installers with Python for Linux bridging and the Quantum plugin. So far the OpenStack Folsom install is more complex due to the added options mainly from the Quantum networking plugin. A big feature of Quantum out of the box using this build with OpenvSwitch is it adds features such as layer2 multi-tenancy for path isolation with GRE tunnels and/or Vlan IDs.

This tutorial will install the packages as listed under Folsom below. Trying to keep the Python installer up to date is too much maintenance for me so switching to DevStack for people who get paid to do it. While installing by hand is good for initial builds, this release mainly due to the complexity of Quantum is trickier than Essex in my opinion. I think that’s some ironic commentary on SDN for those who track developments and try and separate reality, fear and myth on that front.

OpenStack Releases

Figure 1. OpenStack Releases (Note Quantum is still in development and not stable until ‘Grizzly’ Q2Y2013.

openstack-contributions-folsom[/crayon]
Figure 2. Vendor contributions and how they have differed from Essex to Folsom.

OpenStack Folsom Installation Pre-Requisites

I am using Ubuntu 12.04. Never hurts to patch and update but not required. Then we will download Devstack project from GitHub. Since the hooks for OpenvSwitch are coupled with KVM you will need physical hardware that supports hardware virtualization. Nested support is road-mapped and may already be integrated with HEAT in OpenStack for hypervisor hooks into the public cloud.

Check for hardware virtualization on Intel (Intel-VT) with:

Check for hardware virtualizationon AMD (AMD-V) with:

You should see VMX or SVM highlighted in red from the commands above if HW virtualization is supported.

Download and install git. Then clone the devstack repo on github.

Here is what my network NIC configurations look like in /etc/network/interfaces. We want the port to be up in promiscuous mode. Think of Snort or OpenVPN interface configuration :)

Above is the /etc/network/interfaces file that determines the “ifconfig” output. Other handy binaries ethtool “apt-get install ethtool” to determine if you have link to a NIC, e.g. “ethtool eth0″ would return a Boolean on link detected: yes|no and tcpdump is installed by default, “tcpdump host 172.16.1.100 -i eth0″ would promiscuously listen for the host specified on interface eth0.

DevStack Localrc File

Add the following in the devstack directory and name it “localrc”. It is the same directory that contains the “stack.sh” shell script. The localrc file contains all of your build parameters to pass to the script. Finding the right combination took way too many hours but here is what is used in the screencast at the bottom of the post.

Be sure to read the script and understand what other parameters are available. This will create unique Vlan IDs between the host and Hypervisor. If for example you wanted to use GRE tunnels instead you would add the line “ENABLE_TENANT_TUNNELS=True” to the configuration. This is the key to the install. If you have two NICs on a host this should work for you.

If you downloaded devstack git source a a few days or even hours before you execute it you may consider updating the installer. Assuming you allowed devstack to install into the default /opt/stack/ directory, running this will update the latest code merge.

Run ./stack.sh to Install OpenStack Folsom

With your localrc file in the same directory run ./stack.sh. Once the installation is done run the following to pull-down and import the Amazon EC2 Ubuntu Cloud Image. I had a terrible time with what gets pulled down with the script and you will notice at the end I actually get errors on tokens from Glance but everything still works fine EXCEPT for the Cirros image. You can always test if your image is working by looking at it in the Dashboard through VNC. Nova will report a host up even if it is sitting their with a broken image trying to pxi boot.


Your shell environmentals need to have similar values to the following. This is service authentication. Merely pasting it into your bash shell or adding it to localrc for persistency will give you CLI permissions.


folsom-glance-openstack[/crayon]
Figure 3. Check the VM if you are troubleshooting. Nova still reports it up even if it is in this state.

Glance Errors
If you get “You must provide a username via either –os-username or env[OS_USERNAME]” Verify you added proper environmental variables to match what is in localrc and then try another CLI command like “glance index”.

export SERVICE_TOKEN=openstack
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=openstack
export OS_AUTH_URL=http://localhost:5000/v2.0/
export SERVICE_ENDPOINT=http://localhost:35357/v2.0

The Glance error looks something like this. It’s like the SERVICE_TOKEN variable isnt getting passed even though it is defined in keystone somewhere. Did not spend any time on it since I want Ubuntu Precise Cloud Image. You can build windows images with this tutorial.

Before you boot a host you need to do one thing that is not automated. You have to add your fixed_range or back-end NIC to the br-int OpenvSwitch bridge. You are taking that physical interface eth1 and binding it to br-int. Think of it as a locally significant Vlan with no tags being inserted (Probably worst explanation ever). That physical host still does not have any addresses on it.

Folsom Quantum L3-agent

The L3-agent will take care of arp-proxying the hosts. The eth1 host is up promiscuously. That is how if two hosts in the same tenant/bridge/subnet would always show the same arp entry for neighboring VMs. Example, VM Host1 at 192.168.1.10/24 and VM Host2 at 192.168.1.11/24 and a gateway of 192.168.1.1. Pinging 192.168.1.1 from Host1 would give you the same mac address in the arp entry of the gateway as you would see if your pinged Host2. This is because all traffic is funneled through the L3-agent. The datapaths (Layer2 path) is kept isolated by two mechanisms, one being Vlan tagging from the VM to the Physical host and the other option being GRE tunnels from a VM to Physical. This was not possible prior to the OpenStack Folsom release OpenStack Essex. Essex used default Linux bridging.

Folsom leverages advanced vSwitch features in the case OpenvSwitch in conjunction with IPTables filtering and DNSMASQ to deliver self provisioning L2/L3/L4 header for policy application from flow based forwarding. Elegant solution and more on networking in the next section with some drawings.

Note that when you spawn VM from the DevStack install as shown in the video. I use the “demo” group and in the networking tab select “private”.

 


Video 1. Quick walk through of post script operations.

OpenStack Quantum Networking

openstack.folsom.isolation[/crayon]
Figure 4. Data Path (Layer 2) isolation is achieved between hypervisor and VM host with Vlans a GRE tunnels. Those encapsulations break out at the hypervisor or could be picked up by emerging SDN/Data Center orchestration networking.

The concepts are pretty cool. Our future is apparently going to be reliant on DNSMASQ, IPTables or plug-ins like NVP to build state and forwarding policy, see Cisco VSG and Nexus 1k, its all the same concepts, forwarding instantiation. Remember that Quantum won’t be released as stable until Grizzle in spring of 2013. There are quite a few pieces that are not baked into Horizon (Dashboard) since Horizon development froze for Folsom release. Take a look at the blueprint if interested in seeing upcoming features. Floating addresses is not supported via Horizon so you need to add those via the API or CLI.

openstack.tenant.vlans[/crayon]
Figure 5. Here are where some of the DevStack parameters fit into the topology and how OpenStack networking from Hypervisor down looks.

route.table.openstack.quantum
Figure 6. Routing table of the physical host. br-ex is rather confusing at first. Think of it as a NAT bridge between frontend and backend. br-ex will be attached to the public bridge eth0 (public facing) while VMs reside on br-int bridge in OVS.
netns.quantum.exec

Figure 7. Quantum Namespaces are the path isolation components. Getting familiar with ‘ip netns exec’ commands will be part of a deep dive.

Uninstall or Reinstall the Devstack OpenStack Environment

Depending on if I ran into problems or not I will often delete installed packages and directories to get as vanilla an install as I can get. Since this will be physical hosts the snapshot image isn’t available. Something along the following will delete some main packages and delete some directories that will get reinstalled as soon as you run the ./stack.sh script again. Remember to just unload the Devstack script and processes like quantum, nova, glance etc run ./unstack.sh. That will not stop dependencies like your hypervisor or vSwtich which in this case is KVM/libvirt-bin and OpenvSwitch.

 

Potential DevStack Folsom Errors to Troubleshoot

Horizon Error: Template error Dashboard with slug “nova” is not registered. Just delete /opt/stack/horizon/ and pull it done again and the error will clean up. I didn’t troubleshoot Django to try and see why since just rebuilding the directory fixes it up.

About the Author

Brent SalisburyI have over 15 years of experience wearing various hats from, network engineer, architect, devops and software engineer. I currently have the pleasure of working at the company that develops my favorite software I have ever used, Docker. My comments here are my personal thoughts and opinions. More at Brent's BioView all posts by Brent Salisbury →

Guest
Kanisht
1 year 7 months ago


Hi brent,
thanks for this great article, it was quite helpful . i was trying to add a new compute node to this architecture (in a new system) but am unable to do it(got stuck with the ip problem.. cant connect ). it would be great if you could suggest me a way. the compute node installation which i am following is https://openstack-folsom-install-guide.readthedocs.org/en/latest/#id1

thanks

Guest
1 year 8 months ago


It’s a pity you don’t have a donate button! I’d without a doubt donate to this superb blog!
I guess for now i’ll settle for bookmarking and adding your
RSS feed to my Google account. I look forward tto new updates and will talk about this site with my Facebook group.
Talk soon!

Guest
1 year 9 months ago


Hi,

How to uninstall openstack..?…

regards

vikas rao…

Guest
Anna
2 years 3 months ago


when installing was done, I checked route -n in host machine , I didn’t see any route for fixed (172.24.17.0/24) and floating (172.31.246.128/25) ip, for fixed ip seems OK but what a bout L3 (floating network ). So, I could not ping external and internal network. When I uploaded one instance from horizon,it showed fixed ip is assigned and even I could allocate floating ip but when I checked the log file of VM,

” cloudsetup: failed to read iid from metadata. tried 30
WARN: /etc/rc3.d/S45cloudsetup failed
Starting dropbear sshd: OK
===== cloudfinal: system completely up in 51.81 seconds ====
wget: can’t connect to remote host (169.254.169.254): Network is unreachable
wget: can’t connect to remote host (169.254.169.254): Network is unreachable
wget: can’t connect to remote host (169.254.169.254): Network is unreachable
instanceid:
publicipv4:
localipv4 :
wget: can’t connect to remote host (169.254.169.254): Network is unreachable
clouduserdata: failed to read instance id
WARN: /etc/rc3.d/S99clouduserdata failed ”

and when I loged in to VM, it could not recieve ip address,
Can anyone help me?

Guest
Kevin
2 years 4 months ago


Do you have any experience playing with Ubuntu Cloud via Canonical? It is OpenStack Folsom deployed through Juju whatever that is. I have installed DevStack 3 times, each time was a success. I really LOVE using it, especially how it expands into “screen” and rejoining the stack and rejoining the screen after reboots. I even did some manual qemu-kvm to build a Windows 8 instance right in X windows on Ubuntu on my devstack server. But now I am going to build another, and wonder if I should go to the official Ubuntu Cloud “Canonical” version or keep doing it the way I’m comfortable – Ubuntu 12.04LTS + devstack via github.. thoughts???

Guest
Sandeep
2 years 5 months ago


I have installed Openstack Essex – multinode on Ubuntu 12.04 LTS ; The multinode is working fine with all smilies :-) with the restart of services –

service nova-compute restart.

1. But the problem over here is we cannot able to ping / SSH to the VM that are created at [@Node1] – error no route to host 22. where as the VM’s created at [@cnt] are able to ping / ssh. we have tried changing the nova.conf and Bridge settings and nothing has gone in resolving the prob.

Network Conf: [Node – Nova Compute]

auto lo
iface lo inet loopback

auto br100
iface br100 inet static
bridge_ports eth1
bridge_stp off
bridge_maxwait 0
bridge_fd 0
============================
auto eth0
iface eth0 inet static
address 172.16.2.5
netmask 255.255.255.0
gateway 172.16.2.1
dns-nameservers 14.140.144.66

auto eth1
iface eth1 inet static
address 192.168.22.1
netmask 255.255.255.0
============================

Can you help me out any changes or modifications in my settings / config;

Thanks,
Sandeep.

Guest
N
2 years 6 months ago


Hi,
I followed you steps and I have some issues when i try to start an instance I get on the log :
TypeError: can’t compare datetime.datetime to NoneType
ansd also
Instance already created

deleting the old instances and recreating the db doesn’t make things better here and also the error happens at the first step of creation before networking and so
this is what makes me believe it’s related to the scheduler , ntp or rabbitmq maybe …
If you can share any thought on that it would help me a lot.
Thanx

Guest
Sam
2 years 7 months ago


Did you install devstack on a virtual machine or on a physical node?

If you did could you give more feedback on your actual network configuration, such as what kind of devices are used? Bridged, NAT, host-only etc.

Also could you describe your physical network situation? That would help me a lot when understanding how to configure folsom, as it seems dependent on your physical network configuration?