OpenStack Essex and Quantum Installation using OpenvSwitch from Scratch

OpenStack Essex and Quantum Installation using OpenvSwitch from Scratch

openstack-logo

OpenStack Essex and Quantum Installation using OpenvSwitch from Scratch
Update: This is a very dated post, check out the Grizzly installation
 My latest installation document for Folsom can be found at the post OpenStack Folsom Quantum Devstack Installation Tutorial and Screencast. It’s too tough to try and keep up with debugging installers myself so I am just using DevStack in that tutorial. Thanks!

Here is a walkthrough for the Quantum Network Manager Plugin for OpenStack Essex with a Quantum Installation with OpenvSwitch. It is still in dev so expect weird things and stability issues. For a guy with a networking bread and butter this is the extremely exciting piece of OpenStack for me. I will do a followup and try and breakdown the OpenvSwitch components but the data path is using the same call into the kernel as of 3.2 as Linux bridging is so the performance is in the kernel. It has a control plane component in the way of an OpenFlow controller which executes in user land which has incredible potential. Keystone is clunky still here in that you still have to run random scripts people open source, right your own or what would probably be best is figure out what its doing under the hood, but I don’t have time for it and its still arcane for a next gen UI. Here is a nice guide from Stackops I got some of the Quantum configs from here.

  • I recommend getting a working install OpenStack Essex core components as a precursor to this, only because of the added module complexity of mixing in Quantum and OVS @
  • OpenStack Essex Installation and Configuration Screencast from Scratch (Part 1)

Get to know that conf file. It is the heart of the architecture for today.
To find and replace the address in the file with yours you can get lazy with

sed//g :  $sed -i ‘s/128.163.188.8/192.168.1.208/g’ /etc/nova/nova.conf

Updated and verified July 12, 2012.

Figure 1. Two NICs on a host. Note* If you have one NIC you can use a vNIC i.e. eth0 & eth0:1

Prerequisites

The KVM requires an x86 machine with either Intel VT or AMD w/AMD-V support. Anything fairly new will have that support in the processor. There are a few older HW builds that support hardware assisted virtualization by enabling it in the bios. Pretty much Googling your machine for hardware virtualization will let you know. Qemu can be run on non VT HW but the machines will probably get brutalized by a few host VMs. When you are setting up the vSwitch either have an out of band or be on it physically. Be careful when you are adding multiple interfaces to bridges since you can spin up a bridging loop pretty quickly unless you have STP on. I recommend a test/dev network or mom’s basement network. If not BPDUguard is your friend :) This is done on a fresh install of 64-bit Ubuntu 12.04 (Precise).
$apt-get purge network-manager

/*Updates
$apt-get update
$apt-get dist-upgrade

/*If not FQDN
$nano /etc/hosts
127.0.0.1 localhost
127.0.1.1 openstack1
$apt-get install ntp

/*Edit the NTP config to survive an Internet/Network outage:
$nano /etc/ntp.conf
server ntp.ubuntu.com iburst
server 127.127.1.0
fudge 127.127.1.0 stratum 10
/*Restart the service
$service ntp restart

/*More packages please suh
$apt-get install tgt
$apt-get install open-iscsi open-iscsi-utils

/*Restart Networking
$/etc/init.d/networking restart

/*More packages please suh
$apt-get install rabbitmq-server memcached python-memcache
$apt-get install kvm libvirt-bin
$apt-get install -y mysql-server python-mysqldb

/* Log into MySQL
$mysql -u root -p
/*Create the Nova database with a password ‘password’,

/****The quote mark is formatted and won’t copy and paste. Type the database entries****

Here is a text file you can copy/paste from mysql-quatum.txt

mysql -u root -popenstack -e 'CREATE DATABASE nova;'
mysql -u root -popenstack -e "GRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'openstack';"
mysql -u root -popenstack -e "GRANT ALL ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'openstack';"
mysql -u root -popenstack -e 'CREATE DATABASE keystone;'
mysql -u root -popenstack -e "GRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'openstack';"
mysql -u root -popenstack -e "GRANT ALL ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'openstack';"
mysql -u root -popenstack -e 'CREATE DATABASE glance;'
mysql -u root -popenstack -e "GRANT ALL ON glance.* TO 'glance'@'%' IDENTIFIED BY 'openstack';"
mysql -u root -popenstack -e "GRANT ALL ON glance.* TO 'glance'@'localhost' IDENTIFIED BY 'openstack';"
mysql -u root -popenstack -e 'CREATE DATABASE ovs_quantum;'
mysql -u root -popenstack -e "GRANT ALL ON glance.* TO 'ovs_quantum'@'%' IDENTIFIED BY 'openstack';"
mysql -u root -popenstack -e "GRANT ALL ON glance.* TO 'ovs_quantum'@'localhost' IDENTIFIED BY 'openstack';"

/*Edit my.cnf
$nano /etc/mysql/my.cnf

/*from
bind-address = 127.0.0.1
/*to
bind-address = 0.0.0.0

/*Restart mysql
$/etc/init.d/mysql restart

KeyStone Identity Management

/*More packages
$apt-get install keystone python-keystone python-keystoneclient

$rm /var/lib/keystone/keystone.db

/*Edit keystone.conf
$nano /etc/keystone/keystone.conf
/*Replace or comment this:
#connection = sqlite:////var/lib/keystone/keystone.db
/*With this:
connection = mysql://keystone:password@localhost:3306/keystone

/* Change ‘admin_token’ FROM

#bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
admin_token = ADMIN

/*TO

#bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
admin_token = password

/*Change the catalog values FROM:

[catalog]
driver = keystone.catalog.backends.sql.Catalog

TO:
[catalog]
# driver = keystone.catalog.backends.sql.Catalog

driver = keystone.catalog.backends.templated.TemplatedCatalog
template_file = /etc/keystone/default_catalog.templates

/*Restart Keystone
$service keystone restart
/*Synch Keystone to MySQL.
$ keystone-manage db_sync
/*One more restart to avoid handlers error
$service keystone restart

/*If you get something like  “Access denied for user keystone” make sure your Admin = is the root MySql user and also double check your formatting of quotes on a paste into MySql and privileges.

/*  This part is rather ridiculous. Hopefully the Folsom release is much more user friendly than this. Either scripting your own or looking for how someone else has kludged together to populate this is a bit silly for the next gen could ecosystem… Anyways here are some people but the time into which at the end of the day you may just want to figure out what its doing and just paste it in. Edit keystone_data.sh to have the proper password and token don’t copy and paste the parentheses. WordPress mangles them. Download the initial Devstack based Keystone script here 

ADMIN_PASSWORD=${ADMIN_PASSWORD:-password}
SERVICE_PASSWORD=${SERVICE_PASSWORD:-$ADMIN_PASSWORD}
export SERVICE_TOKEN=”password”
export SERVICE_ENDPOINT=”http://localhost:35357/v2.0″
SERVICE_TENANT_NAME=${SERVICE_TENANT_NAME:-service}

/*Save and run the keytone.sh script.
$chmod +x keystone.sh
$./keystone.sh

/* If you get this error “Unable to communicate with identity service: [Errno 111] Connection refused. No handlers could be found for logger “keystoneclient.client”-” make sure you restarted the keystone service after your db sync.

/*This should return a ‘0’ for success.
$echo $?

/* Restart Keystone
$/etc/init.d/keystone restart

/* Go ahead and add these to the end of /root/.bashrc

export SERVICE_TOKEN=password
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_AUTH_URL=http://localhost:5000/v2.0/
export SERVICE_ENDPOINT=http://localhost:35357/v2.0

/*Then refresh the shell. You can also use source novarc but keep it simple to start.
$bash

Glance Image Management

/*Install Glance packages
$apt-get install glance glance-api glance-client glance-common glance-registry python-glance

$rm /var/lib/glance/glance.sqlite

Edit the following files. Make sure you touch each with privilege changes and/or flavors as listed.

/etc/glance/glance-registry-paste.ini (Password Change)
/etc/glance/glance-api-paste.ini (Password Change)
/etc/glance/glance-registry.conf (sql connection to Mysql and flavor at the bottom)
/etc/glance/glance-api.conf (flavor at the bottom)

/*Change the values to match below at the bottom of glance-registry-paste.ini & glance-api-paste.ini
$nano /etc/glance/glance-registry-paste.ini
$nano $nano /etc/glance/glance-api-paste.ini
admin_tenant_name = admin
admin_user = admin
admin_password = password

/*Change the values to match below at the bottom of glance-api-paste.ini
$nano /etc/glance/glance-api-paste.ini & /etc/glance/glance-api-paste.ini
admin_tenant_name = admin
admin_user = admin
admin_password = password

/*Adjust mysql privileges
$nano /etc/glance/glance-registry.conf

/*Replace or comment the sqlite:
#connection = sqlite
sql_connection = mysql://glance:password@localhost/glance

/*and add this to the bottom of the file glance-registry.conf file
[paste_deploy]
flavor = keystone
/*Add flavors to /etc/glance/glance-api.conf at the bottom
$nano /etc/glance/glance-api.conf
[paste_deploy]
flavor = keystone

/* Restart for funzys
$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart

/* Version control fix for 12.04
$glance-manage version_control 0
/*If you get errors on the following db_sync verify your variable by typing ‘export’.
$glance-manage db_sync

/* Again for good measure
$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart

/* 11.10 image
wget http://uec-images.ubuntu.com/releases/11.10/release/ubuntu-11.10-server-cloudimg-amd64-disk1.img

/* 12.04 image
wget https://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

$/etc/init.d/glance-api restart
$/etc/init.d/glance-registry restart
or
$/etc/init.d/glance-registry restart && /etc/init.d/glance-api restart

/*List Images
$glance index (Should be empty)
/* Add images to Glance

$glance add name=”Ubuntu 11.10″ is_public=true container_format=ovf disk_format=qcow2 < ubuntu-11.10-server-cloudimg-amd64-disk1.img

/*Uploading image ‘Ubuntu 11.10′
=======================================================================[100%] 167.139407M/s, ETA 0h 0m 0s
Added new image with ID: 3a2404f6-0a4f-44db-b1cd-a81401dd6015

*/

$glance add name=”Ubuntu 12.04″ is_public=true container_format=ovf disk_format=qcow2 < precise-server-cloudimg-amd64-disk1.img

Uploading image ‘Ubuntu 12.04′
=======================================================================[100%] 143.583705M/s, ETA 0h 0m 0s
Added new image with ID: 92ca562e-bb18-425a-94ac-eb292fbcabec
*/
/* If your image is well above 220Mb you have the wrong image not a Ostack compatible one

$glance index

Nova Installation

$apt-get install nova-api nova-cert nova-common nova-compute nova-compute-kvm nova-doc nova-network nova-objectstore nova-scheduler nova-vncproxy nova-volume
$apt-get install nova-consoleauth novnc python-nova python-novaclient novnc

/*Add to /etc/libvirt/qemu.conf at the top.
/*If you copy/paste make sure the quotes are not formatted from browser

clear_emulator_capabilities = 0
user = “root”
group = “root”
cgroup_device_acl = [
"/dev/null", "/dev/full", "/dev/zero",
"/dev/random", "/dev/urandom",
"/dev/ptmx", "/dev/kvm", "/dev/kqemu",
"/dev/rtc", "/dev/hpet", "/dev/net/tun",
]

/* Next edit /etc/nova/api-paste.ini and change the password

admin_tenant_name = admin
admin_user = admin
admin_password = password

/* Fix Permissions (if still needed in your release)
chown -R nova:nova /etc/nova

/*sync with mysql
$nova-manage db sync

/* UP/Down Startup Script

for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” stop; done
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” start; done

$chmod +x <ex. restart-nova (filename)>
/* Edit /etc/nano.conf Pull files from comments below or file at the top of post and replace.

Install Quantum Network Plugin

More details here http://openvswitch.org/openstack/documentation/

$apt-get install -y quantum-server quantum-plugin-openvswitch
$apt-get install -y openvswitch-datapath-source
$apt-get install module-assistant

/*Add OVS DP module
$module-assistant auto-install openvswitch-datapath

/*Change the default plugin for Quantum in /etc/quantum/plugins.ini

[PLUGIN]
provider = quantum.plugins.openvswitch.ovs_quantum_plugin.OVSQuantumPlugin

Install OpenvSwitch

$apt-get install -y openvswitch-switch quantum-plugin-openvswitch-agent

/*Load the ovs module:
$modprobe openvswitch_mod

/*Add to the module startup file:
$echo “openvswitch_mod”>>/etc/modules

/*Edit /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini and change the SQL
connection FROM:
sql_connection = sqlite://
TO
sql_connection = mysql://ovs_quantum:password@128.163.188.8:3306/ovs_quantum

Create “quantum-agent.sh” to the startup directory /etc/init.d./
$nano /etc/init.d/quantum-agent.sh
/*Agent Script quantum-agent.sh paste the following into the file you created in init.d

#!/bin/bash
quantum-openvswitch-agent /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

/*Make executable
$chmod +x /etc/init.d/quantum-agent.sh

/*Add to startup directories:
$update-rc.d quantum-agent.sh defaults

/*Restart
$/etc/init.d/quantum-server start

/* Create an OVS “integration” bridge, to which all VMs will connect:
$ovs-vsctl add-br br-int

/*If you want to tie your machine to a physical interface which I am in this.
/*That adds our connector bridge to an interface.
/*Warning- do not add two physical interfaces unless they are bonded or you may be explaining a bridging loops to your VP the next morning. I highly recommend something like BPDUguard on your interfaces connecting you OpenStack nodes to protect the network and cranky network guys.
$ ovs-vsctl add-port br-int eth0

/*Start with the ‘&’ to run in bg.
$/etc/init.d/quantum-agent.sh &

/*One more restart for good measure

for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” stop; done
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” start; done

/*Check Services
$nova-manage service list
4nova-manage network list

/*If you have problems the rabbit-qm scheduler seems to die at some points on me. Resolve with:

$/etc/init.d/rabbitmq-server restart

Then check services again $nova-manage service list

/*Should see something along these lines:

$ ps -ea | grep nov
11176 ? 00:00:03 nova-cert
14282 ? 00:00:00 nova-network
14293 ? 00:00:00 nova-compute
14303 ? 00:00:01 nova-api
14313 ? 00:00:00 nova-objectstor
14323 ? 00:00:00 nova-scheduler
14353 ? 00:00:00 nova-xvpvncprox

$ps –ea | grep libvirt
pre>2665 ? 00:00:00 libvirtd

/*Good

$nova-manage service list
Binary Host Zone Status State Updated_At
nova-scheduler openstack-dev nova enabled :-) 2012-05-10 06:39:29
nova-compute openstack-dev nova enabled :-) 2012-05-10 06:39:09
nova-network openstack-dev nova enabled :-) 2012-05-10 06:39:29

/*Bad most of the time. Compute can come in and out of that state and recover.
/*I have never seen network recover, it is normally down hard and time to start scouring the logs.

$nova-manage service list
Binary Host Zone Status State Updated_At
nova-scheduler openstack-dev nova enabled :-) 2012-05-10 07:26:29
nova-compute openstack-dev nova enabled XXX 2012-05-10 07:19:20
nova-network openstack-dev nova enabled XXX 2012-05-10 07:21:06

/*Add your vNIC “Flat Network” as defined in nova.conf Here is an example config.

/*Ensure your hostname is resolvable to the value in nova.conf for ‘–quantum_connection_host=’

/* TRACE nova QuantumIOException: Unable to connect to server. Got error: [Errno -2] Name or service not known (Indicates a bad hostname in nova.conf for ‘–quantum_connection_host=’

/*Note the difference from the nova-network creation with Quantum Manager. Dont try and copy and paste this or else you get unicode errors “TRACE nova UnicodeEncodeError: ‘decimal’ codec can’t encode character”
$ nova-manage network create –label=public –fixed_range_v4=172.31.252.0/24

/*To delete files you need to use the ID found in ‘nova-manage network list’
$nova-manage network delete –uuid 6ec716d0-1b22-497e-be6c-2017b268af1d <–ID

/*(Optional) Create a floating pool. Can be any address you want to assign. Essentially NAT.
/*Quantum Manager uses the same ‘nova-manage floating’ command.
$nova-manage floating create –ip_range=128.163.188.240/28

/*Floating Commands (Optional)
$nova-manage floating list
$nova-manage floating create 128.163.188.248/29

/*Virtual Machines from CLI-Skip if only interested in dashboard.
/*Make sure your nova db has been synced if you skip:

/*If not synced or sure if synced, sync.
$nova-manage db sync

/*Iptables is e.g. ‘ nova secgroup-add-rule ‘ is not supported in OpenvSwitch v1.4

/*Create security keys
$ nova keypair-add ssh_key > ssh_key.pem

/*Set Permissions
$chmod 0600 ssh_key.pem
$ssh -i ssh_key.pem ubuntu@172.31.248.2
/*e.g.
$ssh -i ssh_key.pem ubuntu@

/*One more reboot for good measure

for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” stop; done
for a in libvirt-bin nova-network nova-compute nova-api nova-objectstore nova-scheduler nova-volume nova-vncproxy; do service “$a” start; done

/*List your images
$glance index
/*List or edit your flavors
$nova flavor-list

/*If things get squirrely on you restart everything.
/etc/init.d/quantum-server restart
./restart-nova.sh <scripted>

/*Boot a VM from CLI or skip to the Dashboard frontend. Hyphens get mangled here so copy paste won’t work.

/*Ubuntu 11.10
$nova boot – -flavor 1 – -image 3a2404f6-0a4f-44db-b1cd-a81401dd6015 – -key_name ssh_key dem11.10
/* Ubuntu 12.04
$nova boot – -flavor 1 – -image 92ca562e-bb18-425a-94ac-eb292fbcabec – -key_name ssh_key demo12.04

/*Windows 2008 R2
$nova boot – -flavor 1 – -image 13fdbdb7-0800-4c70-9b6a-73656ca62432 – -key_name ssh_key demoWin2k8

Quicky on building a windows image
/*Create the raw logical disk
kvm-img create -f raw windowsserver.img 20G
/*Boot windows iso along with
kvm -m 1024 -cdrom win2k8_dvd.iso -drive file=windowsserver.img,if=virtio,boot=on -fda virtio-win-1.1.16.vfd -boot d -nographic -vnc :0

/*Here is the command used for this image using the virtio iso.
kvm -smp 2 -m 2048 -no-reboot -boot order=d -drive file=win2k8r2_dc.img,if=virtio,boot=off -drive file=windows2k8_dvd.iso,media=cdrom,boot=on -drive file=virtio-win-0.1-22.iso,media=cdrom,boot=off -net nic,model=virtio -nographic -vnc :1

/*Watch your machine boot or error ☺
$nova show demo1

/*View all VMs in your Project (Project you have creds in ENV for only)
$nova list

Dashboard

/*Install packages
$apt-get install libapache2-mod-wsgi openstack-dashboard

/*Restart Apache
$restart apache
$service apache2 restart

/*Connect to the Dashboard web page.
http://localhost or http://x.x.x.x

Getting to know OVS

Some quick commands to start leveraging a true vSwitch as opposed to Linux bridging. OpenvSwitch is good stuff.
ovs-vsctl list sflow
ovs-ofctl dump-flows
ovs-dpctl dump-flows br0
ovs-vsctl list-br
ovs-vsctl list-ports
ovs-vsctl show
ovs-dpctl show
ovs-ofctl dump-tables br-int
ovs-appctl fdb/show br-int


 

 

 

 

 

 

Figure 2. Example of the Data Path flow dump from ‘ovs-dpctl show br-int’.

Networking command outputs.

/*Notice there are not any bridges defined in /etc/network/interfaces. I was guessing the brcompat_mod.ko kernel was building that but that module isn’t loaded. Must either be loaded into openvswitch_mod.ko or calling brctl. You used to need to unload bridge.ko but I had not used OpenvSwitch since v1.2.

~# brctl show
bridge name bridge id STP enabled interfaces
br-int 0000.5c260a5ac8b2 no eth0
gw-0393c785-7f
tap0d144643-96
tapb8d11bf7-94
tape2d5bcc1-9c
virbr0 8000.000000000000 yes

/*View the OpenFlow Controller and classifier tables.

ovs-ofctl dump-tables br-int
OFPST_TABLE reply (xid=0×1): 255 tables
0: classifier: wild=0x3fffff, max=1000000, active=10
lookup=764491, matched=757280
1: table1 : wild=0x3fffff, max=1000000, active=0
lookup=0, matched=0
2: table2 : wild=0x3fffff, max=1000000, active=0
lookup=0, matched=0

/Show the forwarding tables

$ovs-appctl fdb/show br-int
port VLAN MAC Age
9 2 fa:16:3e:33:4f:e8 37
3 2 fa:16:3e:6f:c0:8b 37
1 0 00:1d:09:aa:71:16 28
1 0 00:e0:1e:5d:86:6f 23
1 0 00:9c:02:b1:ff:c0 19
1 0 00:23:69:62:26:09 18
1 0 00:9c:02:b2:76:c0 17
10 2 fa:16:3e:5f:22:22 15
11 2 fa:16:3e:48:21:4f 4
1 0 00:9c:02:b2:76:e8 3
0 0 5c:26:0a:5a:c8:b2 2
1 0 10:40:f3:94:e0:82 2
1 0 00:26:b9:c0:c8:b0 1

  1. Emilien MacchiEmilien Macchi05-16-2012


    Can you make working Quantum with OVS in multi compute node ?

    • Brent SalisburyBrent Salisbury05-17-2012


      Sure can. Wendlandt with Nicira looks to have put it on the master branch a couple weeks ago. Single master/central whatever node with Quantum agents on other nodes. Curious if that punts the datapath to the userland OF controller on the central node for DP lookup?

  2. Emilien MacchiEmilien Macchi05-22-2012


    You don’t use Quantum according to https://github.com/nerdalert/Openstack-Essex-install-config-files

    You should replace :
    –network_manager=nova.network.manager.FlatDHCPManager

    By ;
    –network_manager=nova.network.quantum.manager.QuantumManager
    –linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
    –quantum_use_dhcp=True
    –quantum_connection_host=X.X.X.X

    • Brent SalisburyBrent Salisbury05-23-2012


      Good catch Emilien, I had the wrong conf file up there. I posted a temp until I get a breather to fix the git.

      # Rabbit-MQ #
      –rabbit_host=128.163.188.8

      # SGBD with MySQL #
      –sql_connection=mysql://nova:password@128.163.188.8/nova

      # API #
      –cc_host=128.163.188.8
      –auth_strategy=keystone
      –s3_host=128.163.188.8
      –ec2_host=128.163.188.8
      –nova_url=http://128.163.188.8:8774/v1.1/
      –ec2_url=http://128.163.188.8:8773/services/Cloud
      –keystone_ec2_url=http://128.163.188.8:5000/v2.0/ec2tokens
      –api_paste_config=/etc/nova/api-paste.ini
      –allow_admin_api=true
      –use_deprecated_auth=false

      # Nova-Schedule #
      –scheduler_driver=nova.scheduler.simple.SimpleScheduler

      # Nova-Network with Quantum integration #
      –network_manager=nova.network.quantum.manager.QuantumManager
      –linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
      –quantum_use_dhcp=True
      –quantum_connection_host=openstack1
      –public_interface=eth5
      –flat_interface=eth0
      –flat_network_bridge=br0
      –floating_range=128.163.188.248/29
      –network_size=500
      –flat_injected=False
      –force_dhcp_release
      –dhcpbridge_flagfile=/etc/nova/nova.conf
      –dhcpbridge=/usr/bin/nova-dhcpbridge
      –routing_source_ip=128.163.188.8

      # Nova-Compute with KVM Hypervisor #
      –libvirt_type=kvm
      –libvirt_use_virtio_for_bridges=true
      –start_guests_on_host_boot=true
      –resume_guests_state_on_host_boot=true
      –connection_type=libvirt
      –libvirt_ovs_bridge=br-int
      –libvirt_vif_type=ethernet
      –libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver

      # Nova-Volume #
      –iscsi_ip_prefix=172.31.252
      –iscsi_helper=tgtadm

      # Glance #
      –glance_api_servers=128.163.188.8:9292
      –image_service=nova.image.glance.GlanceImageService

      # VNC #
      –novnc_enable=true
      –novncproxy_base_url=http://128.163.188.8:6080/vnc_auto.html
      –vncserver_proxyclient_address=127.0.0.1
      –vncserver_listen=127.0.0.1

      # Misc #
      –logdir=/var/log/nova
      –state_path=/var/lib/nova
      –lock_path=/var/lock/nova
      –root_helper=sudo nova-rootwrap
      –verbose
      –max_cores=400

  3. nesrine85nesrine8506-06-2012


    hi , i hope thet you’re fine !
    i have a problem with quantum Network:
    so when i make: nova list
    +————————————–+——–+——–+———-+
    | ID | Name | Status | Networks |
    +————————————–+——–+——–+———-+
    | 5b276bb3-eadd-47b7-a8a3-c81e0025eefe | essex2 | ERROR | |
    +————————————–+——–+——–+———-+
    i have yet a network with any instance i do no why
    either quantum it look worked well:

    brctl show
    bridge name bridge id STP enabled interfaces
    br-int 0000.00144fe26e5b no eth1
    br100 8000.000000000000 no
    virbr0 8000.000000000000 yes

    ovs-vsctl show
    13e44f2b-49b7-4fb2-8dd1-cb98355bd00f
    Bridge br-int
    Port br-int
    Interface br-int
    type: internal
    Port “eth1″
    Interface “eth1″
    ovs_version: “1.4.0+build0″

    ovs-dpctl show
    system@br-int:
    lookups: hit:27654 missed:4109 lost:0
    flows: 0
    port 0: br-int (internal)
    port 3: eth1

    nova-manage network list:

    3 172.31.252.0/24 None 172.31.252.2 8.8.4.4 None None None d08e4d31-fefd-44c9-910e-5ccd7b96ca12

    Thanks in advance!

  4. nesrine85nesrine8506-06-2012


    ssh -i ssh_key.pem ubuntu@172.31.248.2

    ssh: connect to host 172.31.248.2 port 22: No route to host

    • Brent SalisburyBrent Salisbury06-07-2012


      Hiya, Well, I dont see any taps (vNIC) being built in in your ‘ovs-vsctl show’ output.

      I built a lab tonight and tested to make sure no updates broke the how-to and everything seems to be working.

      What does ‘nova-manage services list’ output?

      Below are what some of my outputs are. More info the better. I see where you are ssh’ing to a host in your 2nd post. Did you get a VM up and working? Paste some more info and maybe we can work through it. I am traveling the next couple of days so I may be slow on replies.

      Cheers.

      root@brent-Latitude-E6520:openstack# nova list
      +————————————–+——–+——–+——————–+
      | ID | Name | Status | Networks |
      +————————————–+——–+——–+——————–+
      | d2da80f8-d2b7-4ea4-b740-9fa3442fbee5 | demo11 | ACTIVE | public=192.168.2.2 |
      +————————————–+——–+——–+——————–+
      root@brent-Latitude-E6520:openstack# root@brent-Latitude-E6520:openstack# ovs-vsctl show

      root@brent-Latitude-E6520:openstack# ovs-vsctl show
      9957a38a-834c-47aa-80a5-c7e78d66b6c7
      Bridge br-int
      Port “eth0″
      Interface “eth0″
      Port “gw-0393c785-7f”
      tag: 2
      Interface “gw-0393c785-7f”
      type: internal
      Port “tapcdb4b6ba-15″
      tag: 2
      Interface “tapcdb4b6ba-15″
      Port br-int
      Interface br-int
      type: internal
      ovs_version: “1.4.0+build0″
      root@brent-Latitude-E6520:openstack# nova show demo11
      +————————————-+———————————————————-+
      | Property | Value |
      +————————————-+———————————————————-+
      | OS-DCF:diskConfig | MANUAL |
      | OS-EXT-SRV-ATTR:host | brent-Latitude-E6520 |
      | OS-EXT-SRV-ATTR:hypervisor_hostname | None |
      | OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
      | OS-EXT-STS:power_state | 1 |
      | OS-EXT-STS:task_state | None |
      | OS-EXT-STS:vm_state | active |
      | accessIPv4 | |
      | accessIPv6 | |
      | config_drive | |
      | created | 2012-06-07T06:00:30Z |
      | flavor | m1.tiny |
      | hostId | 2b24fc83c889a95313ef35b5e40b8f2db76dd757658b3b87b37593dd |
      | id | d2da80f8-d2b7-4ea4-b740-9fa3442fbee5 |
      | image | Ubuntu 11.10 |
      | key_name | ssh_key |
      | metadata | {} |
      | name | demo11 |
      | progress | 0 |
      | public network | 192.168.2.2 |
      | status | ACTIVE |
      | tenant_id | e8a558710c434835a87e79796741bd8e |
      | updated | 2012-06-07T06:01:02Z |
      | user_id | f3c15140018e47d1beef6bb768ffebe7 |
      +————————————-+———————————————————-+
      root@brent-Latitude-E6520:openstack# ping 192.168.2.2
      PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
      64 bytes from 192.168.2.2: icmp_req=1 ttl=64 time=1.18 ms
      64 bytes from 192.168.2.2: icmp_req=2 ttl=64 time=0.652 ms
      ^C
      — 192.168.2.2 ping statistics —
      2 packets transmitted, 2 received, 0% packet loss, time 1001ms
      rtt min/avg/max/mdev = 0.652/0.916/1.181/0.266 ms
      root@brent-Latitude-E6520:openstack# route -n
      Kernel IP routing table
      Destination Gateway Genmask Flags Metric Ref Use Iface
      0.0.0.0 192.168.1.1 0.0.0.0 UG 100 0 0 br-int
      169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth1
      192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br-int
      192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 gw-0393c785-7f
      192.168.2.0 0.0.0.0 255.255.255.0 U 2 0 0 eth1
      192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
      root@brent-Latitude-E6520:openstack# arp -a
      ? (192.168.2.2) at fa:16:3e:19:e4:ce [ether] on gw-0393c785-7f
      ? (192.168.1.1) at 00:23:69:62:26:09 [ether] on br-int

  5. ninanina06-21-2012


    Hi,
    I followed your tutorial but I think I missed something.

    nova-manage service list returns:
    nova-cert Server1 nova enabled :-)
    nova-consoleauth Server1 nova enabled :-)
    nova-scheduler Server1 nova enabled :-)
    nova-volume Server1 nova enabled :-)
    nova-compute Server1 nova enabled XXX
    nova-network Server1 nova enabled :-)

    when i check the log I get :
    libvirtError: Cannot recv data: Connection reset by peer

    and this is really annoying I tried to get arround this by setting my user in the libvirt group or increasing the max connection number but it didn’t work either.

    Do you have any idea about this error?

    • Brent SalisburyBrent Salisbury06-27-2012


      Hi Nina, Sorry I missed your question earlier this week. I think you are right it looks like a libvirt problem. Maybe check the /var/log/libvirt/libirt.log(off top of my head) and paste it. Also irc.freenode.net #openstack for real time help. Bummer, the issues are a pain but once you work through them it is nice and stable.

  6. marcelmarcel06-26-2012


    Nice Tutorial. Works for me.
    For german peoples use this one

    https://blog.jahnke.com.de/?p=48

    with best regards
    Marcel

    • Brent SalisburyBrent Salisbury06-27-2012


      Thanks for the link marcel

    • marcelmarcel06-27-2012


      I had problems after a system reboot.
      The error message was: could not configure /dev/net/tun (tapeb68de6b-8b): Operation not permitted
      -netdev tap,ifname=tapeb68de6b-8b,script=,id=hostnet0: Device ‘tap’ could not be initialized
      After a long search I found that apparmor does not set the correct permissions.
      Unfortunately I don’t found the right settings for the permissions.
      A solution that worked for me was that I finally uninstalled apparmor.

      Do you have maybe a another solution?

      With best regards
      Marcel

      • Brent SalisburyBrent Salisbury07-03-2012


        Hi Marcel, I am not sure. I don’t think I had run into that or more likely had the problem but didn’t realize it lol. Cheers!

  7. NesrineNesrine07-03-2012


    hi ! when i reboot my cloud ! i have a problem to terminate a last instance! what i can do !
    thanks in advance

  8. nesrinenesrine07-06-2012


    hi brent! please can you have a goog tuto for volume manage for nova, beacause h have a problem with.

    when i boot an instance et i attach it with a volume (with dashboard)
    the vnc console display the message: no bootable device!
    i use ubnutu desktop!
    thx

  9. NesrineNesrine07-10-2012


    hi brent!
    thank you for all!
    so i have an issue concerning the Volume:
    i create a LVM volumes-nova (pvcreate, vg create)

    when i attach a volume to the instance i have
    , no bootable device (image ubuntu OVF format)
    and my volume have this specification :

    — Volume group —
    VG Name nova-volumes
    System ID
    Format lvm2
    Metadata Areas 1
    Metadata Sequence No 2
    VG Access read/write
    VG Status resizable
    MAX LV 0
    Cur LV 1
    Open LV 1
    Max PV 0
    Cur PV 1
    Act PV 1
    VG Size 136,81 GiB
    PE Size 4,00 MiB
    Total PE 35024
    Alloc PE / Size 2560 / 10,00 GiB
    Free PE / Size 32464 / 126,81 GiB
    VG UUID 5XwX6E-ASE0-Kqyf-EZgV-8IcB-IDF0-vTYbge

    Any help please , i don’t understand that?
    Best Regards Nesrine Bahroun

  10. daviddavid07-18-2012


    Hi, I am trying to setup openstack nova on two different VMs, one acting as the controller the other acting as compute node. I am completely new to openstack, so since you are using just one server it is completely difficult for me to understand which installation goes where.so please can you help with which installations should I make on the controller and which shoudl I make on the compute.

  11. NesrineNesrine07-20-2012


    Hi david this tuto is good and also you can try this https://github.com/EmilienM/doc-openstack
    good luck !

  12. MarcelMarcel08-08-2012


    Hy Brent,
    i have a new Problem.
    The VMs not get any IP address after a restart of the server. Further Quantum or OVS doesn’t create the bridge or a tap device.

    Any ideas?

    Best regards from germany
    Marcel

  13. AntonioAntonio10-26-2012


    Hi, i have followed your installation steps for my 2-nodes OpenStack Essex installation.
    The problem is that when I start the openvswitch agent, I get this exception
    python /usr/lib/python2.6/site-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini -v

    DEBUG:__main__:## running command: ovs-ofctl del-flows br-int
    DEBUG:__main__:## running command: ovs-ofctl add-flow br-int priority=1,actions=normal
    INFO:__main__:Connecting to database “ovs_quantum” on 192.168.80.163
    DEBUG:__main__:## running command: ovs-vsctl –timeout=2 list-ports br-int
    DEBUG:__main__:## running command: ovs-vsctl –timeout=2 get Interface eth2 external_ids
    DEBUG:__main__:## running command: ovs-vsctl –timeout=2 get Interface eth2 ofport
    DEBUG:__main__:## running command: ovs-vsctl –timeout=2 get Interface gw-78f673bd-20 external_ids
    DEBUG:__main__:## running command: ovs-vsctl –timeout=2 get Interface gw-78f673bd-20 ofport
    DEBUG:__main__:## running command: ovs-vsctl –timeout=2 set Port gw-78f673bd-20 tag=4095
    DEBUG:__main__:## running command: ovs-ofctl add-flow br-int priority=2,in_port=2,actions=drop
    INFO:__main__:Unable to commit to database! Exception: commit
    Traceback (most recent call last):
    File “/usr/lib/python2.6/site-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py”, line 744, in
    main()
    File “/usr/lib/python2.6/site-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py”, line 739, in main
    plugin.daemon_loop(db_connection_url)
    File “/usr/lib/python2.6/site-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py”, line 329, in daemon_loop
    db.rollback()
    File “/usr/lib/python2.6/site-packages/sqlalchemy/ext/sqlsoup.py”, line 558, in __getattr__
    return self.entity(attr)
    File “/usr/lib/python2.6/site-packages/sqlalchemy/ext/sqlsoup.py”, line 547, in entity
    table = Table(attr, self._metadata, autoload=True, schema=schema or self.schema)
    File “/usr/lib/python2.6/site-packages/sqlalchemy/schema.py”, line 113, in __call__
    return type.__call__(self, name, metadata, *args, **kwargs)
    File “/usr/lib/python2.6/site-packages/sqlalchemy/schema.py”, line 241, in __init__
    _bind_or_error(metadata).reflecttable(self, include_columns=include_columns)
    File “/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py”, line 1265, in reflecttable
    self.dialect.reflecttable(conn, table, include_columns)
    File “/usr/lib/python2.6/site-packages/sqlalchemy/databases/mysql.py”, line 1664, in reflecttable
    sql = self._show_create_table(connection, table, charset)
    File “/usr/lib/python2.6/site-packages/sqlalchemy/databases/mysql.py”, line 1835, in _show_create_table
    raise exc.NoSuchTableError(full_name)
    sqlalchemy.exc.NoSuchTableError: rollback
    The database ovs_quantum does exist, isn’t empty and is accessable by the second node. Have you got any clue about it?

  14. ramram12-08-2012


    Hi Brent thanks for ur tutorial can u please tell me the network details for openstack setup because i am using college Lan for this i have two systems with each one having two NIC’s and i won’t get college ip address more than one fro each system so i want to create one public network and one private network manually using two modems(AUSES Wireless router) from last one month iam struggling with openstack installation can u please help me.Please….

  15. Abhinav KanaviAbhinav Kanavi02-06-2013


    I am getting the following error while adding image in Glance:
    Error communicating with /v1/images/detail?limit=10: [Errno 113] No route to host

    Can you please help me with this??

    • Brent SalisburyBrent Salisbury02-25-2013


      Apologies for slow reply, in case others are having an issue, “route -n” should give you the host routing table. I recommend starting with Devstack and working your way backwards when it comes to Folsom and Quantum. The complexity is pretty high. Also for starting nova-network only is probably easiest to start with before the mind warp of quantum.

      Respect,
      -Brent

  16. CassieCassie02-24-2013


    You actually make it seem so easy with your presentation but
    I find this matter to be really something which I think I would never understand.
    It seems too complex and extremely broad for me. I am looking
    forward for your next post, I’ll try to get the hang of it!

  17. Brent SalisburyBrent Salisbury02-25-2013


    Thanks Cassie, keep it up!

    Respect,
    -Brent


  18. Hello, i think that i saw you visited my web site thus i came to “return the
    favor”.I am attempting to find things to improve
    my website!I suppose its ok to use a few of your ideas!
    !


  19. 靴の先端が内側に寄っていること、趾のつけ根の所で靴の幅がきつ


  20. その後、府の都市計画審議会などを経て秋に正式決定される区画整理を予定しているのは

  21. othmaneothmane07-08-2013


    Hi all,
    thinks brent for this tutorial, i just have a question about openstack and openflow. Actually i already have an openflow network composed of some OVS bridges, VMs and Floodlight controller, i want to integrate Nova compute and quantum to my setting in order to add/delete VM. while creating instance with nova, is it possible to specify witch bridge the VM will be connected to? when i read about nova, i saw that all VMs created are connected to the same default bridge “br100″, is that mean that we can not set a VM network configuration while adding it?

    i also saw that VM network atachement point (bridge and port DPID) is sent by Quantum to Nova, is this done automatically or can i specifie wich bridge the new instance should be connected to??? maybe i m missing or missundertstanding something…..
    thinks a lot for your help


  22. fantastic issues altogether, you just wwon a neww reader.
    What may you suggest in regards to your put uup that
    you simply made a few days ago? Anny positive?