Here is a walkthrough for the Quantum Network Manager Plugin for OpenStack. It is still in dev so expect weird things and stability issues. For a guy with a networking bread and butter this is the extremely exciting piece of OpenStack for me. I will do a followup and try and breakdown the OpenvSwitch components but the data path is using the same call into the kernel as of 3.2 as Linux bridging is so the performance is in the kernel. It has a control plane component in the way of an OpenFlow controller which executes in user land which has incredible potential. This is Nicira’s pony. I used some of the StackOps guide and scripts for this, great work everyone is doing here.
- I recommend getting a working install OpenStack Essex core components as a precursor to this, only because of the added module complexity of mixing in Quantum and OVS @
- OpenStack Essex Installation and Configuration Screencast from Scratch (Part 1)
- All of the config files and scripts are @
- https://github.com/nerdalert/Openstack-Essex-install-config-files
Figure 1. Two NICs on a host. Note* If you have one NIC you can use a vNIC i.e. eth0 & eth0:1
System Dependency Preparations
The Host looks like this.
$apt-get update
/*If not FQDN
$nano /etc/hosts
127.0.0.1 localhost 127.0.1.1 openstack1
$apt-get install ntp
/*Edit the NTP config to survive an Internet/Network outage:
$nano /etc/ntp.conf
server ntp.ubuntu.com iburst server 127.127.1.0 fudge 127.127.1.0 stratum 10
/*Restart the service
$service ntp restart
/*More packages
$apt-get install tgt
$apt-get install open-iscsi open-iscsi-utils
$apt-get install bridge-utils
/*Restart Networking
$/etc/init.d/networking restart
$apt-get install rabbitmq-server memcached python-memcache
$apt-get install kvm libvirt-bin
$apt-get install -y mysql-server python-mysqldb
/* Log into MySQL
$mysql -u root -p
/*Create the Nova database with a password ‘password’ CREATE DATABASE nova; GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'password'; CREATE DATABASE glance; GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY 'password'; CREATE DATABASE keystone; GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'password'; CREATE DATABASE ovs_quantum; GRANT ALL PRIVILEGES ON ovs_quantum.* TO 'ovs_quantum'@'%' IDENTIFIED BY 'password'; FLUSH PRIVILEGES;
/*Edit my.cnf
$nano /etc/mysql/my.cnf
/*from bind-address = 127.0.0.1 /*to bind-address = 0.0.0.0
/*Restart mysql
$/etc/init.d/mysql restart
KeyStone Identity Management
/*More packages
$apt-get install keystone python-keystone python-keystoneclient
/*Edit keystone.conf
$nano /etc/keystone/keystone.conf
/*Replace or comment this: connection = sqlite:////var/lib/keystone/keystone.db /*With this: connection = mysql://keystone:password@128.163.188.8:3306/keystone /* Change ‘admin_token’ FROM #bind_host = 0.0.0.0 public_port = 5000 admin_port = 35357 admin_token = ADMIN /*TO #bind_host = 0.0.0.0 public_port = 5000 admin_port = 35357 admin_token = password
/*Change the catalog values FROM:
[catalog] driver = keystone.catalog.backends.sql.Catalog TO: [catalog] # driver = keystone.catalog.backends.sql.Catalog driver = keystone.catalog.backends.templated.TemplatedCatalog template_file = /etc/keystone/default_catalog.templates
/*Restart Keystone
$service keystone restart
/*Synch Keystone to MySQL.
$ keystone-manage db_sync
/* Edit keystone_data.sh to have the proper password and token
ADMIN_PASSWORD=${ADMIN_PASSWORD:-password}
SERVICE_PASSWORD=${SERVICE_PASSWORD:-$ADMIN_PASSWORD}
export SERVICE_TOKEN="password"
export SERVICE_ENDPOINT="http://localhost:35357/v2.0"
SERVICE_TENANT_NAME=${SERVICE_TENANT_NAME:-service}
/*Save and run the keytone.sh script.
$chmod +x keystone.sh
$./keystone.sh
/*This should return a ‘0’ for success.
$echo $?
/* Restart Keystone
$/etc/init.d/keystone restart
/*Install Glance packages
$apt-get install glance glance-api glance-client glance-common glance-registry python-glance
/* Go ahead and add these to the end of /root/.bashrc
export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=password export OS_AUTH_URL="http://localhost:5000/v2.0/" export OS_AUTH_KEY=password
/*Then refresh the shell. You can also use source novarc but keep it simple to start.
$bash
Glance Image Management
/etc/glance/glance-registry-paste.ini (privs)
/etc/glance/glance-api-paste.ini (privs)
/etc/glance/glance-registry.conf (sql conn and flavor)
/etc/glance/glance-api.conf (flavor)
/*Change the values to match below at the bottom of glance-registry-paste.ini
$nano /etc/glance/glance-registry-paste.ini
admin_tenant_name = admin admin_user = admin admin_password = password /*Change the values to match below at the bottom of glance-api-paste.ini $nano /etc/glance/glance-api-paste.ini admin_tenant_name = admin admin_user = admin admin_password = password
$nano /etc/glance/glance-registry.conf
/*Adjust mysql with privileges
$nano /etc/glance/glance-registry.conf
sql_connection = mysql://glancedbadmin:password@128.163.188.8/glance /*and add this to the bottom of the file [paste_deploy] flavor = keystone
/*Add flavors to /etc/glance/glance-api.conf at the bottom
$nano /etc/glance/glance-api.conf
[paste_deploy] flavor = keystone
/* Restart for funzys
$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart
/* Version control fix for 12.04
$glance-manage version_control 0
/*If you get errors on the following db_sync verify your variable by typing ‘export’.
$glance-manage db_sync
/* Again for good measure
$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart
/* 11.10 image
wget http://uec-images.ubuntu.com/releases/11.10/release/ubuntu-11.10-server-cloudimg-amd64-disk1.img
/* 12.04 image
wget https://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart
/*List Images
$glance index (Should be empty)
/* Add images to Glance $glance add name="Ubuntu 11.10" is_public=true container_format=ovf disk_format=qcow2 < ubuntu-11.10-server-cloudimg-amd64-disk1.img /*Uploading image 'Ubuntu 11.10' ==========================================================================================[100%] 167.139407M/s, ETA 0h 0m 0s Added new image with ID: 3a2404f6-0a4f-44db-b1cd-a81401dd6015 */ $glance add name="Ubuntu 12.04" is_public=true container_format=ovf disk_format=qcow2 < precise-server-cloudimg-amd64-disk1.img Uploading image 'Ubuntu 12.04' ==========================================================================================[100%] 143.583705M/s, ETA 0h 0m 0s Added new image with ID: 92ca562e-bb18-425a-94ac-eb292fbcabec */
/* If your image is well above 220Mb you have the wrong image and not a Ostack compatible one.
$glance index
Nova Installation
$apt-get install nova-api nova-cert nova-common nova-compute nova-compute-kvm nova-doc nova-network nova-objectstore nova-scheduler nova-vncproxy nova-volume
$apt-get install nova-consoleauth novnc python-nova python-novaclient novnc
/*Add to /etc/libvirt/qemu.conf at the top.
/*If you copy/paste make sure the quotes are not formatted from browser
clear_emulator_capabilities = 0 user = "root" group = "root" cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc", "/dev/hpet", "/dev/net/tun", ]
/* Edit /etc/nova/api-paste.ini and add
admin_tenant_name = admin admin_user = admin admin_password = password
/* Fix Permissions (if still needed in your release)
chown -R nova:nova /etc/nova
/*sync with mysql
$nova-manage db sync
/* UP/Down Startup Script
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service "$a" stop; done for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service "$a" start; done $chmod +x
/* Edit /etc/nano.conf Pull files from Github and replace.
Install Quantum Network Plugin
$apt-get install -y quantum-server quantum-plugin-openvswitch
$apt-get install -y openvswitch-datapath-source
$apt-get install module-assistant
/*Add OVS DP module
$module-assistant auto-install openvswitch-datapath
/*Change the default plugin for Quantum in /etc/quantum/plugins.ini
[PLUGIN] provider = quantum.plugins.openvswitch.ovs_quantum_plugin.OVSQuantumPlugin
Install OpenvSwitch
$apt-get install -y openvswitch-switch quantum-plugin-openvswitch-agent
/*Load the ovs module:
$modprobe openvswitch_mod
/*Add to the module startup file:
$echo “openvswitch_mod”>>/etc/modules
/*Edit /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini and change the SQL
connection FROM: sql_connection = sqlite:// TO sql_connection = mysql://ovs_quantum:password@128.163.188.8:3306/ovs_quantum
Copy “quantum-agent.sh” to /etc/init.d. directory download-script here
/*Make executable
$chmod +x /etc/init.d/quantum-agent.sh
/*Add to startup directories:
$update-rc.d quantum-agent.sh defaults
/*Restart
$/etc/init.d/quantum-server start
/* Create an OVS “integration” bridge, to which all VMs will connect:
$ovs-vsctl add-br br-int
/*If you want to tie your machine to a physical interface which I am in this.
/*That adds our connector bridge to an interface.
/*Warning- do not add two physical interfaces unless they are bonded or you may be explaining a bridging loops to your VP the next morning. I highly recommend something like BPDUguard on your interfaces connecting you OpenStack nodes to protect the network and cranky network guys.
$ ovs-vsctl add-port br-int eth0
/*Start with the ‘&’ to run in bg.
$/etc/init.d/quantum-agent.sh &
/*One more reboot for good measure
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service "$a" stop; done for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service "$a" start; done
/*Check Services
$nova-manage service list
4nova-manage network list
/*Should see something along these lines:
$ ps -ea | grep nov 11176 ? 00:00:03 nova-cert 14282 ? 00:00:00 nova-network 14293 ? 00:00:00 nova-compute 14303 ? 00:00:01 nova-api 14313 ? 00:00:00 nova-objectstor 14323 ? 00:00:00 nova-scheduler 14353 ? 00:00:00 nova-xvpvncprox $ps –ea | grep libvirt pre>2665 ? 00:00:00 libvirtd /*Good $nova-manage service list Binary Host Zone Status State Updated_At nova-scheduler openstack-dev nova enabled2012-05-10 06:39:29 nova-compute openstack-dev nova enabled
/*Add your vNIC “Flat Network” as defined in nova.conf
/*Note the difference from the nova-network creation with Quantum Manager.
$ nova-manage network create –label=public –fixed_range_v4=172.31.252.0/24
/*To delete files you need to use the ID found in ‘nova-manage network list’
$nova-manage network delete –uuid 6ec716d0-1b22-497e-be6c-2017b268af1d <–ID
/*(Optional) Create a floating pool. Can be any address you want to assign. Essentially NAT.
/*Quantum Manager uses the same ‘nova-manage floating’ command.
$nova-manage floating create –ip_range=128.163.188.240/28
/*Floating Commands (Optional)
$nova-manage floating list
$nova-manage floating create 128.163.188.248/29
/*Virtual Machines from CLI-Skip if only interested in dashboard.
/*Make sure your nova db has been synced if you skip:
/*If not synced or sure if synced, sync.
$nova-manage db sync
/*Iptables is e.g. ‘ nova secgroup-add-rule ‘ is not supported in OpenvSwitch v1.4
/*Create security keys
$ nova keypair-add ssh_key > ssh_key.pem
/*Set Permissions
$chmod 0600 ssh_key.pem
$ssh -i ssh_key.pem ubuntu@172.31.248.2
/*e.g.
$ssh -i ssh_key.pem ubuntu@
/*One more reboot for good measure
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service "$a" stop; done for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service "$a" start; done
/*List your images
$glance index
/*List or edit your flavors
$nova flavor-list
/*If things get squirrely on you restart everything.
/etc/init.d/quantum-server restart
./restart-nova.sh <scripted>
/*Boot a VM from CLI or skip to the Dashboard frontend.
/*Ubuntu 11.10
$nova boot –flavor 1 –image 3a2404f6-0a4f-44db-b1cd-a81401dd6015 –key_name ssh_key dem11.10
/* Ubuntu 12.04
$nova boot –flavor 1 –image 92ca562e-bb18-425a-94ac-eb292fbcabec –key_name ssh_key demo12.04
/*Windows 2008 R2
$nova boot –flavor 1 –image 13fdbdb7-0800-4c70-9b6a-73656ca62432 –key_name ssh_key demoWin2k8
Quicky on building a windows image /*Create the raw logical disk kvm-img create -f raw windowsserver.img 20G /*Boot windows iso along with kvm -m 1024 -cdrom win2k8_dvd.iso -drive file=windowsserver.img,if=virtio,boot=on -fda virtio-win-1.1.16.vfd -boot d -nographic -vnc :0 /*Here is the command used for this image using the virtio iso. kvm -smp 2 -m 2048 -no-reboot -boot order=d -drive file=win2k8r2_dc.img,if=virtio,boot=off -drive file=windows2k8_dvd.iso,media=cdrom,boot=on -drive file=virtio-win-0.1-22.iso,media=cdrom,boot=off -net nic,model=virtio -nographic -vnc :1
/*Watch your machine boot or error ☺
$nova show demo1
/*View all VMs in your Project (Project you have creds in ENV for only)
$nova list
Some quick commands to start leveraging a true vSwitch as opposed to Linux bridging.
ovs-vsctl list sflow ovs-ofctl dump-flows ovs-dpctl dump-flows br0 ovs-vsctl list-br ovs-vsctl list-ports ovs-vsctl show ovs-dpctl show
ovs-ofctl dump-tables br-int
ovs-appctl fdb/show br-int
Figure 2. Example of the Data Path flow dump from ‘ovs-dpctl show br-int’.
Networking command outputs.
/*Notice there are not any bridges defined in /etc/network/interfaces. I was guessing the brcompat_mod.ko kernel was building that but that module isn’t loaded. Must either be loaded into openvswitch_mod.ko or calling brctl. You used to need to unload bridge.ko but I had not used OpenvSwitch since v1.2.
$brctl show
bridge name bridge id STP enabled interfaces br-int 0000.782bcb16b22b no eth0 gw-2f0f9de6-a6 tap027e8414-f6 tap1cac52ac-9f tap2089130c-f6 tap222dd07b-c0 tap2b4b3ce5-10 tap3118ba42-8a tap3be9a323-08 tap46b69b9a-9d tap50320595-90 tap50a61e01-f1 tap534f2d0b-e0 tap535fc0be-15 tap57d53138-87 tap5e2d64ee-cb tap60eff348-6c tap8b77a5eb-a1 tap92315342-89 tap9a5b3f2a-34 tap9ffbdcf6-9e tapa7666617-ae tapb30fc838-f5 tapbde52bbf-34 tapc2d57ab0-b8 tapd11d6b5a-d3 tape010beb0-83 tape93935a1-4d tape9757942-ad tapf1fdbff3-3f tapf3fea029-f9 tapfd620308-16 virbr0 8000.000000000000 yes
/*View the OpenFlow Controller and classifier tables.
$ovs-ofctl dump-tables br-int
OFPST_TABLE reply (xid=0×1): 255 tables 0: classifier: wild=0x3fffff, max=1000000, active=6 lookup=192391, matched=192391
/Show the forwarding tables
$ovs-appctl fdb/show br-int
port VLAN MAC Age 36 1 fa:16:3e:32:fc:72 47 29 1 fa:16:3e:0c:ae:f6 41 33 1 fa:16:3e:46:2b:7d 41 18 1 fa:16:3e:64:fa:1e 39 26 1 fa:16:3e:71:71:ee 35 34 1 fa:16:3e:4b:21:43 33 28 1 fa:16:3e:2d:fb:59 32 15 1 fa:16:3e:10:da:95 32 37 0 00:12:80:b9:a4:d5 31 32 1 fa:16:3e:7c:9f:76 31 17 1 fa:16:3e:63:e3:53 29 5 1 fa:16:3e:16:95:dc 26 20 1 fa:16:3e:46:21:93 25 7 1 fa:16:3e:20:49:9c 19 19 1 fa:16:3e:65:53:39 19 31 1 fa:16:3e:54:e5:af 17 6 1 fa:16:3e:6a:64:da 13 23 1 fa:16:3e:30:78:77 13 25 1 fa:16:3e:11:8a:41 12 14 1 fa:16:3e:49:ac:70 12 35 1 fa:16:3e:61:5c:20 12 24 1 fa:16:3e:08:e5:27 10 21 1 fa:16:3e:25:f2:f4 8 22 1 fa:16:3e:57:bb:aa 4 8 1 fa:16:3e:19:88:48 3 30 1 fa:16:3e:22:fa:28 3 27 1 fa:16:3e:70:33:a4 1 37 0 00:00:0c:07:ac:01 1 16 1 fa:16:3e:39:14:67 0 3 1 fa:16:3e:2c:db:db 0
Dashboard
/*Install packages
$apt-get install libapache2-mod-wsgi openstack-dashboard
/*Restart Apache
$restart apache
$service apache2 restart
/*Connect to the Dashboard web page.
http://localhost or http://x.x.x.x
