OpenStack Essex and Quantum Installation using OpenvSwitch from Scratch

OpenStack Essex and Quantum Installation using OpenvSwitch from Scratch

---

Here is a walkthrough for the Quantum Network Manager Plugin for OpenStack Essex with a Quantum Installation with OpenvSwitch. It is still in dev so expect weird things and stability issues. For a guy with a networking bread and butter this is the extremely exciting piece of OpenStack for me. I will do a followup and try and breakdown the OpenvSwitch components but the data path is using the same call into the kernel as of 3.2 as Linux bridging is so the performance is in the kernel. It has a control plane component in the way of an OpenFlow controller which executes in user land which has incredible potential. Keystone is clunky still here in that you still have to run random scripts people open source, right your own or what would probably be best is figure out what its doing under the hood, but I don’t have time for it and its still arcane for a next gen UI. Here is a nice guide from Stackops I got some of the Quantum configs from here.

• I recommend getting a working install OpenStack Essex core components as a precursor to this, only because of the added module complexity of mixing in Quantum and OVS @
• OpenStack Essex Installation and Configuration Screencast from Scratch (Part 1)

Get to know that conf file. It is the heart of the architecture for today.
To find and replace the address in the file with yours you can get lazy with

sed//g :  $sed -i ‘s/128.163.188.8/192.168.1.208/g’ /etc/nova/nova.conf

Updated and verified July 12, 2012.

Figure 1. Two NICs on a host. Note* If you have one NIC you can use a vNIC i.e. eth0 & eth0:1

Prerequisites

The KVM requires an x86 machine with either Intel VT or AMD w/AMD-V support. Anything fairly new will have that support in the processor. There are a few older HW builds that support hardware assisted virtualization by enabling it in the bios. Pretty much Googling your machine for hardware virtualization will let you know. Qemu can be run on non VT HW but the machines will probably get brutalized by a few host VMs. When you are setting up the vSwitch either have an out of band or be on it physically. Be careful when you are adding multiple interfaces to bridges since you can spin up a bridging loop pretty quickly unless you have STP on. I recommend a test/dev network or mom’s basement network. If not BPDUguard is your friend 🙂 This is done on a fresh install of 64-bit Ubuntu 12.04 (Precise).
$apt-get purge network-manager

/*Updates
$apt-get update
$apt-get dist-upgrade

/*If not FQDN
$nano /etc/hosts
127.0.0.1 localhost
127.0.1.1 openstack1
$apt-get install ntp

/*Edit the NTP config to survive an Internet/Network outage:
$nano /etc/ntp.conf
server ntp.ubuntu.com iburst
server 127.127.1.0
fudge 127.127.1.0 stratum 10
/*Restart the service
$service ntp restart

/*More packages please suh
$apt-get install tgt
$apt-get install open-iscsi open-iscsi-utils

/*Restart Networking
$/etc/init.d/networking restart

/*More packages please suh
$apt-get install rabbitmq-server memcached python-memcache
$apt-get install kvm libvirt-bin
$apt-get install -y mysql-server python-mysqldb

/* Log into MySQL
$mysql -u root -p
/*Create the Nova database with a password ‘password’,

/****The quote mark is formatted and won’t copy and paste. Type the database entries****

Here is a text file you can copy/paste from mysql-quatum.txt

/*Edit my.cnf
$nano /etc/mysql/my.cnf

/*from
bind-address = 127.0.0.1
/*to
bind-address = 0.0.0.0

/*Restart mysql
$/etc/init.d/mysql restart

KeyStone Identity Management

/*More packages
$apt-get install keystone python-keystone python-keystoneclient

$rm /var/lib/keystone/keystone.db

/*Edit keystone.conf
$nano /etc/keystone/keystone.conf
/*Replace or comment this:
#connection = sqlite:////var/lib/keystone/keystone.db
/*With this:
connection = mysql://keystone:password@localhost:3306/keystone

/* Change ‘admin_token’ FROM

#bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
admin_token = ADMIN

/*TO

#bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
admin_token = password

/*Change the catalog values FROM:

[catalog]
driver = keystone.catalog.backends.sql.Catalog

TO:
[catalog]
# driver = keystone.catalog.backends.sql.Catalog

driver = keystone.catalog.backends.templated.TemplatedCatalog
template_file = /etc/keystone/default_catalog.templates

/*Restart Keystone
$service keystone restart
/*Synch Keystone to MySQL.
$ keystone-manage db_sync
/*One more restart to avoid handlers error
$service keystone restart

/*If you get something like  “Access denied for user keystone” make sure your Admin = is the root MySql user and also double check your formatting of quotes on a paste into MySql and privileges.

/*  This part is rather ridiculous. Hopefully the Folsom release is much more user friendly than this. Either scripting your own or looking for how someone else has kludged together to populate this is a bit silly for the next gen could ecosystem… Anyways here are some people but the time into which at the end of the day you may just want to figure out what its doing and just paste it in. Edit keystone_data.sh to have the proper password and token don’t copy and paste the parentheses. WordPress mangles them. Download the initial Devstack based Keystone script here 

ADMIN_PASSWORD=${ADMIN_PASSWORD:-password}
SERVICE_PASSWORD=${SERVICE_PASSWORD:-$ADMIN_PASSWORD}
export SERVICE_TOKEN=”password”
export SERVICE_ENDPOINT=”http://localhost:35357/v2.0″
SERVICE_TENANT_NAME=${SERVICE_TENANT_NAME:-service}

/*Save and run the keytone.sh script.
$chmod +x keystone.sh
$./keystone.sh

/* If you get this error “Unable to communicate with identity service: [Errno 111] Connection refused. No handlers could be found for logger “keystoneclient.client”-” make sure you restarted the keystone service after your db sync.

/*This should return a ‘0’ for success.
$echo $?

/* Restart Keystone
$/etc/init.d/keystone restart

/* Go ahead and add these to the end of /root/.bashrc

export SERVICE_TOKEN=password
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_AUTH_URL=http://localhost:5000/v2.0/
export SERVICE_ENDPOINT=http://localhost:35357/v2.0

/*Then refresh the shell. You can also use source novarc but keep it simple to start.
$bash

Glance Image Management

/*Install Glance packages
$apt-get install glance glance-api glance-client glance-common glance-registry python-glance

$rm /var/lib/glance/glance.sqlite

Edit the following files. Make sure you touch each with privilege changes and/or flavors as listed.

/etc/glance/glance-registry-paste.ini (Password Change)
/etc/glance/glance-api-paste.ini (Password Change)
/etc/glance/glance-registry.conf (sql connection to Mysql and flavor at the bottom)
/etc/glance/glance-api.conf (flavor at the bottom)

/*Change the values to match below at the bottom of glance-registry-paste.ini & glance-api-paste.ini
$nano /etc/glance/glance-registry-paste.ini
$nano $nano /etc/glance/glance-api-paste.ini
admin_tenant_name = admin
admin_user = admin
admin_password = password

/*Change the values to match below at the bottom of glance-api-paste.ini
$nano /etc/glance/glance-api-paste.ini & /etc/glance/glance-api-paste.ini
admin_tenant_name = admin
admin_user = admin
admin_password = password

/*Adjust mysql privileges
$nano /etc/glance/glance-registry.conf

/*Replace or comment the sqlite:
#connection = sqlite
sql_connection = mysql://glance:password@localhost/glance

/*and add this to the bottom of the file glance-registry.conf file
[paste_deploy]
flavor = keystone
/*Add flavors to /etc/glance/glance-api.conf at the bottom
$nano /etc/glance/glance-api.conf
[paste_deploy]
flavor = keystone

/* Restart for funzys
$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart

/* Version control fix for 12.04
$glance-manage version_control 0
/*If you get errors on the following db_sync verify your variable by typing ‘export’.
$glance-manage db_sync

/* Again for good measure
$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart

/* 11.10 image
wget http://uec-images.ubuntu.com/releases/11.10/release/ubuntu-11.10-server-cloudimg-amd64-disk1.img

/* 12.04 image
wget https://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart
or
$/etc/init.d/glance-registry restart && /etc/init.d/glance-api restart

/*List Images
$glance index (Should be empty)
/* Add images to Glance

$glance add name=”Ubuntu 11.10″ is_public=true container_format=ovf disk_format=qcow2 < ubuntu-11.10-server-cloudimg-amd64-disk1.img

/*Uploading image ‘Ubuntu 11.10’
=======================================================================[100%] 167.139407M/s, ETA 0h 0m 0s
Added new image with ID: 3a2404f6-0a4f-44db-b1cd-a81401dd6015

*/

$glance add name=”Ubuntu 12.04″ is_public=true container_format=ovf disk_format=qcow2 < precise-server-cloudimg-amd64-disk1.img

Uploading image ‘Ubuntu 12.04’
=======================================================================[100%] 143.583705M/s, ETA 0h 0m 0s
Added new image with ID: 92ca562e-bb18-425a-94ac-eb292fbcabec
*/
/* If your image is well above 220Mb you have the wrong image not a Ostack compatible one

$glance index

Nova Installation

$apt-get install nova-api nova-cert nova-common nova-compute nova-compute-kvm nova-doc nova-network nova-objectstore nova-scheduler nova-vncproxy nova-volume
$apt-get install nova-consoleauth novnc python-nova python-novaclient novnc

/*Add to /etc/libvirt/qemu.conf at the top.
/*If you copy/paste make sure the quotes are not formatted from browser

clear_emulator_capabilities = 0
user = “root”
group = “root”
cgroup_device_acl = [
“/dev/null”, “/dev/full”, “/dev/zero”,
“/dev/random”, “/dev/urandom”,
“/dev/ptmx”, “/dev/kvm”, “/dev/kqemu”,
“/dev/rtc”, “/dev/hpet”, “/dev/net/tun”,
]

/* Next edit /etc/nova/api-paste.ini and change the password

admin_tenant_name = admin
admin_user = admin
admin_password = password

/* Fix Permissions (if still needed in your release)
chown -R nova:nova /etc/nova

/*sync with mysql
$nova-manage db sync

/* UP/Down Startup Script

for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” stop; done
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” start; done

$chmod +x <ex. restart-nova (filename)>
/* Edit /etc/nano.conf Pull files from comments below or file at the top of post and replace.

Install Quantum Network Plugin

More details here http://openvswitch.org/openstack/documentation/

$apt-get install -y quantum-server quantum-plugin-openvswitch
$apt-get install -y openvswitch-datapath-source
$apt-get install module-assistant

/*Add OVS DP module
$module-assistant auto-install openvswitch-datapath

/*Change the default plugin for Quantum in /etc/quantum/plugins.ini

[PLUGIN]
provider = quantum.plugins.openvswitch.ovs_quantum_plugin.OVSQuantumPlugin

Install OpenvSwitch

$apt-get install -y openvswitch-switch quantum-plugin-openvswitch-agent

/*Load the ovs module:
$modprobe openvswitch_mod

/*Add to the module startup file:
$echo “openvswitch_mod”>>/etc/modules

/*Edit /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini and change the SQL
connection FROM:
sql_connection = sqlite://
TO
sql_connection = mysql://ovs_quantum:password@128.163.188.8:3306/ovs_quantum

Create “quantum-agent.sh” to the startup directory /etc/init.d./
$nano /etc/init.d/quantum-agent.sh
/*Agent Script quantum-agent.sh paste the following into the file you created in init.d

#!/bin/bash
quantum-openvswitch-agent /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

/*Make executable
$chmod +x /etc/init.d/quantum-agent.sh

/*Add to startup directories:
$update-rc.d quantum-agent.sh defaults

/*Restart
$/etc/init.d/quantum-server start

/* Create an OVS “integration” bridge, to which all VMs will connect:
$ovs-vsctl add-br br-int

/*If you want to tie your machine to a physical interface which I am in this.
/*That adds our connector bridge to an interface.
/*Warning- do not add two physical interfaces unless they are bonded or you may be explaining a bridging loops to your VP the next morning. I highly recommend something like BPDUguard on your interfaces connecting you OpenStack nodes to protect the network and cranky network guys.
$ ovs-vsctl add-port br-int eth0

/*Start with the ‘&’ to run in bg.
$/etc/init.d/quantum-agent.sh &

/*One more restart for good measure

for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” stop; done
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” start; done

/*Check Services
$nova-manage service list
4nova-manage network list

/*If you have problems the rabbit-qm scheduler seems to die at some points on me. Resolve with:

$/etc/init.d/rabbitmq-server restart

Then check services again $nova-manage service list

/*Should see something along these lines:

$ ps -ea | grep nov
11176 ? 00:00:03 nova-cert
14282 ? 00:00:00 nova-network
14293 ? 00:00:00 nova-compute
14303 ? 00:00:01 nova-api
14313 ? 00:00:00 nova-objectstor
14323 ? 00:00:00 nova-scheduler
14353 ? 00:00:00 nova-xvpvncprox

$ps –ea | grep libvirt
pre>2665 ? 00:00:00 libvirtd

/*Good

$nova-manage service list
Binary Host Zone Status State Updated_At
nova-scheduler openstack-dev nova enabled 🙂 2012-05-10 06:39:29
nova-compute openstack-dev nova enabled 🙂 2012-05-10 06:39:09
nova-network openstack-dev nova enabled 🙂 2012-05-10 06:39:29

/*Bad most of the time. Compute can come in and out of that state and recover.
/*I have never seen network recover, it is normally down hard and time to start scouring the logs.

$nova-manage service list
Binary Host Zone Status State Updated_At
nova-scheduler openstack-dev nova enabled 🙂 2012-05-10 07:26:29
nova-compute openstack-dev nova enabled XXX 2012-05-10 07:19:20
nova-network openstack-dev nova enabled XXX 2012-05-10 07:21:06

/*Add your vNIC “Flat Network” as defined in nova.conf Here is an example config.

/*Ensure your hostname is resolvable to the value in nova.conf for ‘–quantum_connection_host=’

/* TRACE nova QuantumIOException: Unable to connect to server. Got error: [Errno -2] Name or service not known (Indicates a bad hostname in nova.conf for ‘–quantum_connection_host=’

/*Note the difference from the nova-network creation with Quantum Manager. Dont try and copy and paste this or else you get unicode errors “TRACE nova UnicodeEncodeError: ‘decimal’ codec can’t encode character”
$ nova-manage network create –label=public –fixed_range_v4=172.31.252.0/24

/*To delete files you need to use the ID found in ‘nova-manage network list’
$nova-manage network delete –uuid 6ec716d0-1b22-497e-be6c-2017b268af1d <–ID

/*(Optional) Create a floating pool. Can be any address you want to assign. Essentially NAT.
/*Quantum Manager uses the same ‘nova-manage floating’ command.
$nova-manage floating create –ip_range=128.163.188.240/28

/*Floating Commands (Optional)
$nova-manage floating list
$nova-manage floating create 128.163.188.248/29

/*Virtual Machines from CLI-Skip if only interested in dashboard.
/*Make sure your nova db has been synced if you skip:

/*If not synced or sure if synced, sync.
$nova-manage db sync

/*Iptables is e.g. ‘ nova secgroup-add-rule ‘ is not supported in OpenvSwitch v1.4

/*Create security keys
$ nova keypair-add ssh_key > ssh_key.pem

/*Set Permissions
$chmod 0600 ssh_key.pem
$ssh -i ssh_key.pem ubuntu@172.31.248.2
/*e.g.
$ssh -i ssh_key.pem ubuntu@

/*One more reboot for good measure

for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” stop; done
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” start; done

/*List your images
$glance index
/*List or edit your flavors
$nova flavor-list

/*If things get squirrely on you restart everything.
/etc/init.d/quantum-server restart
./restart-nova.sh <scripted>

/*Boot a VM from CLI or skip to the Dashboard frontend. Hyphens get mangled here so copy paste won’t work.

/*Ubuntu 11.10
$nova boot – -flavor 1 – -image 3a2404f6-0a4f-44db-b1cd-a81401dd6015 – -key_name ssh_key dem11.10
/* Ubuntu 12.04
$nova boot – -flavor 1 – -image 92ca562e-bb18-425a-94ac-eb292fbcabec – -key_name ssh_key demo12.04

/*Windows 2008 R2
$nova boot – -flavor 1 – -image 13fdbdb7-0800-4c70-9b6a-73656ca62432 – -key_name ssh_key demoWin2k8

Quicky on building a windows image
/*Create the raw logical disk
kvm-img create -f raw windowsserver.img 20G
/*Boot windows iso along with
kvm -m 1024 -cdrom win2k8_dvd.iso -drive file=windowsserver.img,if=virtio,boot=on -fda virtio-win-1.1.16.vfd -boot d -nographic -vnc :0

/*Here is the command used for this image using the virtio iso.
kvm -smp 2 -m 2048 -no-reboot -boot order=d -drive file=win2k8r2_dc.img,if=virtio,boot=off -drive file=windows2k8_dvd.iso,media=cdrom,boot=on -drive file=virtio-win-0.1-22.iso,media=cdrom,boot=off -net nic,model=virtio -nographic -vnc :1

/*Watch your machine boot or error ☺
$nova show demo1

/*View all VMs in your Project (Project you have creds in ENV for only)
$nova list

Dashboard

/*Install packages
$apt-get install libapache2-mod-wsgi openstack-dashboard

/*Restart Apache
$restart apache
$service apache2 restart

/*Connect to the Dashboard web page.
http://localhost or http://x.x.x.x

Getting to know OVS

Some quick commands to start leveraging a true vSwitch as opposed to Linux bridging. OpenvSwitch is good stuff.
ovs-vsctl list sflow
ovs-ofctl dump-flows
ovs-dpctl dump-flows br0
ovs-vsctl list-br
ovs-vsctl list-ports
ovs-vsctl show
ovs-dpctl show
ovs-ofctl dump-tables br-int
ovs-appctl fdb/show br-int

 

 

 

 

 

 

Figure 2. Example of the Data Path flow dump from ‘ovs-dpctl show br-int’.

Networking command outputs.

/*Notice there are not any bridges defined in /etc/network/interfaces. I was guessing the brcompat_mod.ko kernel was building that but that module isn’t loaded. Must either be loaded into openvswitch_mod.ko or calling brctl. You used to need to unload bridge.ko but I had not used OpenvSwitch since v1.2.

~# brctl show
bridge name bridge id STP enabled interfaces
br-int 0000.5c260a5ac8b2 no eth0
gw-0393c785-7f
tap0d144643-96
tapb8d11bf7-94
tape2d5bcc1-9c
virbr0 8000.000000000000 yes

/*View the OpenFlow Controller and classifier tables.

ovs-ofctl dump-tables br-int
OFPST_TABLE reply (xid=0x1): 255 tables
0: classifier: wild=0x3fffff, max=1000000, active=10
lookup=764491, matched=757280
1: table1 : wild=0x3fffff, max=1000000, active=0
lookup=0, matched=0
2: table2 : wild=0x3fffff, max=1000000, active=0
lookup=0, matched=0

/Show the forwarding tables

$ovs-appctl fdb/show br-int
port VLAN MAC Age
9 2 fa:16:3e:33:4f:e8 37
3 2 fa:16:3e:6f:c0:8b 37
1 0 00:1d:09:aa:71:16 28
1 0 00:e0:1e:5d:86:6f 23
1 0 00:9c:02:b1:ff:c0 19
1 0 00:23:69:62:26:09 18
1 0 00:9c:02:b2:76:c0 17
10 2 fa:16:3e:5f:22:22 15
11 2 fa:16:3e:48:21:4f 4
1 0 00:9c:02:b2:76:e8 3
0 0 5c:26:0a:5a:c8:b2 2
1 0 10:40:f3:94:e0:82 2
1 0 00:26:b9:c0:c8:b0 1