OpenStack Multi-Node DevStack Nova Network Tutorial

OpenStack Multi-Node DevStack Nova Network Tutorial

Devops Sm

OpenStack Multi-Node DevStack Nova Network Tutorial

DevStack is a scripted OpenStack installer  maintained for developers to prototype and debug. It is also handy for operators to proof concepts and test new services and configurations. This is a good starter for folks new to OpenStack to help understand how to install and operate OpenStack. Some folks have been asking about multi-node DevStack deployment using nova network, so I thought this tutorial might help.

Another difference in this tutorial. is using nova-network instead of the Quantum network plugin. The Quantum plugin is very flexible and offers much richer networking feature, by implementing vSwitches, rather then Linux bridging. With flexibility, comes complexity.The reality if most people looking to do test deployments, proof of concepts or richer KVM/XEN hypervisor management do not really need the added complexity. Nova-network is slated for depreciation at some point, but I am not sure that is the best plan of action for typical until the installation becomes easier for folks in my humble opinion. Most enterprises don’t have “DevOps” programmers, and I think OpenStack brings some good value and expenditure savings to the enterprise. The complexity of the advanced networking may turn some folks off to the project in smaller organizations. Then again smaller organizations aren’t the ones running OpenStack and that may be attractive to vendors looking to find revenue from service offerings. Anywho, thats my two cents, let’s get on to the tutorial.

OpenStack Flat Network Manager Topology
OpenStack Flat Network Manager Topology

Figure 1. Flat Network Manager Topology

Configuring DevStack Installation

DevStack is not a persistent installation, meaning it will be erased when you run ./unstack.sh. This works fine on a single machine running in VMfusion since it supports nested virtualization.

% git clone https://github.com/openstack-dev/devstack.git
% cd devstack

Create a file called localrc for the controller and compute node and fill it in with the localrc configurations below. Change the IP addresses to match your addressing. Put the localrc file in the root of the DevStack folder.

 % ./stack.sh

After that is complete, if you do not have any errors you should have a functioning controller and working compute node(s). The compute nodes run the guest VMs. The compute node also does the NAT and security functionality. That can be configuring in the web GUI or from CLI. Your Floating Range will be your publicly reachable network. The fixed range is for the guest VM to talk to the compute node for resources. Since we are using one NIC in each machine, all of the addresses on the compute node are bound to the br100 bridge that is in turn bound to eth0 as secondary addresses. Everything is sharing the same broadcast domain on br100. It is fairly confusing for many without a strong networking background, albeit much simpler than Quantum services. The more you dig in and explore, the clearer it will inevitably become.

When installing note the path of the logs. This is helpful for reverse engineering the installation and troubleshooting services while stack.sh is running. Logs are not dumped to /var/log/nova* like a regular installation.

OpenStack Controller localrc

Place this in a file named localrc in the root of the devstack directory you clone. That installs into the /opt/ directory be default.

## Controller Host ##
 HOST_IP=192.168.1.10
 MULTI_HOST=1
## Network nova-network ##
 FLAT_INTERFACE=eth0
 FIXED_RANGE=172.24.17.0/24
 FIXED_NETWORK_SIZE=254
 FLOATING_RANGE=192.168.1.128/25
## Leaving Default Services Enabled ##
 DISABLED_SERVICES=quantum
## Logs ##
 LOGFILE=/opt/stack/logs/stack.sh.log
 VERBOSE=True
 LOG_COLOR=False
 SCREEN_LOGDIR=/opt/stack/logs

OpenStack Controller Node nova.conf

nova.conf is located in /etc/nova/nova.conf

[DEFAULT]
 verbose=True
 auth_strategy=keystone
 allow_resize_to_same_host=True
 api_paste_config=/etc/nova/api-paste.ini
 rootwrap_config=/etc/nova/rootwrap.conf
 compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
 dhcpbridge_flagfile=/etc/nova/nova.conf
 force_dhcp_release=True
 fixed_range=172.24.17.0/24
 default_floating_pool=nova
 s3_host=192.168.1.10
 s3_port=3333
 osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
 my_ip=192.168.1.11
 sql_connection=mysql://root:password@192.168.1.10/nova?charset=utf8
 libvirt_type=kvm
 libvirt_cpu_mode=none
 instance_name_template=instance-%08x
 enabled_apis=ec2,osapi_compute,metadata
 state_path=/opt/stack/data/nova
 lock_path=/opt/stack/data/nova
 instances_path=/opt/stack/data/nova/instances
 multi_host=True
 send_arp_for_ha=True
 logging_context_format_string=%(asctime)s %(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s] %(instance)s%(message)s
 network_manager=nova.network.manager.FlatDHCPManager
 public_interface=br100
 vlan_interface=eth0
 flat_network_bridge=br100
 flat_interface=eth0
 novncproxy_base_url=http://192.168.1.10:6080/vnc_auto.html
 xvpvncproxy_base_url=http://192.168.1.10:6081/console
 vncserver_listen=127.0.0.1
 vncserver_proxyclient_address=127.0.0.1
 ec2_dmz_host=192.168.1.10
 rabbit_host=192.168.1.10
 rabbit_password=password
 glance_api_servers=192.168.1.10:9292
 compute_driver=libvirt.LibvirtDriver
 firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver

OpenStack Compute localrc

Place this in a file named localrc in the root of the devstack directory you clone. That installs into the /opt/ directory be default.

## Compute Host ##
 #SERVICE_HOST_NAME=controller
 SERVICE_HOST=192.168.1.10
 HOST_IP=192.168.1.11
 MULTI_HOST=1
## Network nova-network ##
 FLAT_INTERFACE=eth0
 FIXED_RANGE=172.24.17.0/24
 FIXED_NETWORK_SIZE=254
 FLOATING_RANGE=192.168.1.128/25
## Compute Node Services ##
 ENABLED_SERVICES=n-cpu,n-net,n-api,n-vol
## API URIs ##
 Q_HOST=$SERVICE_HOST
 MYSQL_HOST=$SERVICE_HOST
 RABBIT_HOST=$SERVICE_HOST
 GLANCE_HOSTPORT=$SERVICE_HOST:9292
 KEYSTONE_AUTH_HOST=$SERVICE_HOST
 KEYSTONE_SERVICE_HOST=$SERVICE_HOST
## Auth ##
 ADMIN_PASSWORD=password
 MYSQL_PASSWORD=password
 RABBIT_PASSWORD=password
 SERVICE_PASSWORD=password
 SERVICE_TOKEN=password
## Logs ##
 LOGFILE=/opt/stack/logs/stack.sh.log
 VERBOSE=True
 LOG_COLOR=False
 SCREEN_LOGDIR=/opt/stack/logs

OpenStack Compute Node nova.conf

nova.conf is located in /etc/nova/nova.conf

[DEFAULT]
 verbose=True
 auth_strategy=keystone
 allow_resize_to_same_host=True
 api_paste_config=/etc/nova/api-paste.ini
 rootwrap_config=/etc/nova/rootwrap.conf
 compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
 dhcpbridge_flagfile=/etc/nova/nova.conf
 force_dhcp_release=True
 fixed_range=172.24.17.0/24
 default_floating_pool=nova
 s3_host=192.168.1.10
 s3_port=3333
 osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
 my_ip=192.168.1.11
 sql_connection=mysql://root:password@192.168.1.10/nova?charset=utf8
 libvirt_type=kvm
 libvirt_cpu_mode=none
 instance_name_template=instance-%08x
 enabled_apis=ec2,osapi_compute,metadata
 state_path=/opt/stack/data/nova
 lock_path=/opt/stack/data/nova
 instances_path=/opt/stack/data/nova/instances
 multi_host=True
 send_arp_for_ha=True
 logging_context_format_string=%(asctime)s %(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s] %(instance)s%(message)s
 network_manager=nova.network.manager.FlatDHCPManager
 public_interface=br100
 vlan_interface=eth0
 flat_network_bridge=br100
 flat_interface=eth0
 novncproxy_base_url=http://192.168.1.10:6080/vnc_auto.html
 xvpvncproxy_base_url=http://192.168.1.10:6081/console
 vncserver_listen=127.0.0.1
 vncserver_proxyclient_address=127.0.0.1
 ec2_dmz_host=192.168.1.10
 rabbit_host=192.168.1.10
 rabbit_password=password
 glance_api_servers=192.168.1.10:9292
 compute_driver=libvirt.LibvirtDriver
 firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver

SSH Keys and Booting a Guest VM
nova keypair-add ssh_key > ssh_key.pem > ~/ssh_key.pem
chmod 0600 ~/ssh_key.pem

OpenStack Glance Guest VM Images

DevStack will download CirrOS but in case it doesn’t or you want to install another image like the Ubuntu Cloud image you can do the follwing.

Download a pre-built OpenStack ready image:

#Download Ubuntu 12.04 (The default login for this image is "ubuntu")
 % wget https://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
 #or:
 #Download CirrOS 0.3 (The default login is "cirros")
 % wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img

Import the downloaded image into Glance

% glance add name=Ubuntu-12.04 is_public=true container_format=ovf \
 disk_format=qcow2 < precise-server-cloudimg-amd64-disk1.img
% glance add name=CirrOS-0.3.0 is_public=true container_format=ovf \
 disk_format=qcow2 < cirros-0.3.0-x86_64-disk.img
#Verify image imports with % glance index

Verifying OpenStack Services

If you get errors stating you are not authorized or don’t have permsissions to use the client tools, verify your environmental variables are set by typing export. Manually add ENVs with the following.

export SERVICE_TOKEN=openstack
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=openstack
export OS_AUTH_URL=http://localhost:5000/v2.0/
export SERVICE_ENDPOINT=http://localhost:35357/v2.0
OpenStack nova-manage

Figure 2. Viewing processes running in DevStack with ‘nova-manage service list’ ‘: -)’ means the process is functioning. ‘XXX’ means the process has died.

OpenStack Processes

 Figure 3. View running procs in Linux with ‘ps auxw’.

Nova-Network Configuration

DevStack will build your network as specified in the localrc. The manual operations of that are as follows:

#Create the private address space where guest VMs will reside.
 % nova-manage network create private 172.24.17.0/24 1 254
#Create the floating address pool and associate it to an object named Nova for something meaningful to tenants.
 % nova-manage floating create 192.168.1.128/25 --pool=nova

Since we are only using one NIC in this interface, we will have both the Flat Interface (Public) and the Fixed Range (Privates for VMs) bound to br0100.

% ip addr show
br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
 link/ether 00:0c:29:65:fd:82 brd ff:ff:ff:ff:ff:ff
 inet 172.24.17.3/24 brd 172.24.17.255 scope global br100
 inet 192.168.1.11/24 brd 192.168.1.255 scope global br100
 inet 192.168.1.129/32 scope global br100
 inet6 fe80::d4ee:3cff:fef2:556d/64 scope link
 valid_lft forever preferred_lft forever
OpenStack Processes

Figure 4. View the Linux routing tables ‘route -n’.

To view the Linux bridge configuration use the brctl command. If for some reason DevStack doesn’t create the bridge br100, you can define it yourself with ‘brctl add-br br100′ and ‘brctl addif br100 eth0′. Remember, eth0 is nested in br100. That becomes your Layer3 interface.

% brctl show
 bridge   name         bridge id STP enabled interfaces
 br100 8000.000c2965fd82 no eth0
 vnet0
OpenStack Brctl

Figure 5. Linux bridge output from ‘brctl’

10 Quick OpenStack Operational Notes
  1. At anytime you can check to make sure the services show smiley faces :-) and not XXX with ‘nova-manage service list’.
  2. To manually boot an image, running ‘glance index’ and copy the ID of the image that was loaded in the script.
  3. nova boot –flavor 1 –image <insert image ID here> –key_name ssh_key demohost
  4. Run ‘nova list’ to check the progress and look for any errors.
  5. After a couple of minutes the instance will be booted, make sure you can ping it.
  6. SSH to the ubuntu instance by running “ssh -i ~/ssh-key.pem ubuntu@<ip in nova list>” The key was created by the script.
  7. Check out the web page with http://localhost or http://<public address used>
  8. Under the “Admin” project in Dashboard (the webpage) go to security and access and allocate a floating address to your project.
  9. Once allocated to the project, allocate the address to your VM you started.
  10. Running ./unstack will unload DevStack. Running ./stack.sh will rebuild the stack.

Additional DevStack Resources

Thanks for stopping by.


About the Author

Brent SalisburyBrent Salisbury works as a Network Architect, CCIE #11972. He blogs at NetworkStatic.net with a focus on disruptive technologies, that have a focus on operational efficiencies. Brent can be reached on Twitter @NetworkStatic.View all posts by Brent Salisbury →

  1. vinayusvinayus07-27-2013


    i installed openstack using devstack tutorial….. i uploaded ubuntu 12.04 image too….. but while i create an instance it spawns for a while and ends up being error… i saw log in “syslog” which stated ” failed to load names from /opt/stack/data/nova/networks/nova-br100.hosts: No such file or directory” please help.

    • WoodyWoody09-07-2013


      You should post your localrc for a diagnostic start point.

  2. KashyapKashyap03-11-2014


    HI,
    i am getting following error:

    [root@nova1 devstack]# chmod u+x tools/create-stack-user.sh
    [root@nova1 devstack]# tools/create-stack-user.sh
    Giving stack user passwordless sudo privileges
    [root@nova1 devstack]# chown -R stack:stack /root/Desktop/devstack/*
    [root@nova1 devstack]# su stack
    [stack@nova1 devstack]$ ./stack.sh
    ./stack.sh: line 41: /root/Desktop/devstack/functions: Permission denied
    ./stack.sh: line 44: /root/Desktop/devstack/lib/config: Permission denied
    ./stack.sh: line 49: GetDistro: command not found
    rm: cannot remove `/root/Desktop/devstack/.localrc.auto’: Permission denied
    ./stack.sh: line 98: log_error: command not found
    ./stack.sh: line 100: /root/Desktop/devstack/stackrc: Permission denied
    ./stack.sh: line 107: export_proxy_variables: command not found
    ./stack.sh: line 125: log_error: command not found
    ./stack.sh: line 131: log_error: command not found
    ./stack.sh: line 135: /root/Desktop/devstack/lib/database: Permission denied
    ./stack.sh: line 136: /root/Desktop/devstack/lib/rpc_backend: Permission denied
    ./stack.sh: line 141: disable_negated_services: command not found
    WARNING: this script has not been tested on
    ./stack.sh: line 148: die: command not found
    ./stack.sh: line 154: check_rpc_backend: command not found
    ./stack.sh: line 166: trueorfalse: command not found
    ./stack.sh: line 186: is_package_installed: command not found
    ./stack.sh: line 186: install_package: command not found
    sudo: >>> /etc/sudoers.d/50_stack_sh: syntax error near line 1 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 2 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 1 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 2 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 1 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 2 <<<
    sudo: parse error in /etc/sudoers.d/50_stack_sh near line 1
    sudo: no valid sudoers sources found, quitting
    sudo: unable to initialize policy plugin
    ./stack.sh: line 263: safe_chown: command not found
    ./stack.sh: line 272: trueorfalse: command not found
    ./stack.sh: line 277: trueorfalse: command not found
    ./stack.sh: line 280: trueorfalse: command not found
    ./stack.sh: line 290: get_default_host_ip: command not found
    ./stack.sh: line 292: die: command not found
    ./stack.sh: line 302: trueorfalse: command not found
    ./stack.sh: line 310: trueorfalse: command not found
    ./stack.sh: line 324: /root/Desktop/devstack/lib/apache: Permission denied
    ./stack.sh: line 327: /root/Desktop/devstack/lib/tls: Permission denied
    ./stack.sh: line 330: /root/Desktop/devstack/lib/infra: Permission denied
    ./stack.sh: line 331: /root/Desktop/devstack/lib/oslo: Permission denied
    ./stack.sh: line 332: /root/Desktop/devstack/lib/stackforge: Permission denied
    ./stack.sh: line 333: /root/Desktop/devstack/lib/horizon: Permission denied
    ./stack.sh: line 334: /root/Desktop/devstack/lib/keystone: Permission denied
    ./stack.sh: line 335: /root/Desktop/devstack/lib/glance: Permission denied
    ./stack.sh: line 336: /root/Desktop/devstack/lib/nova: Permission denied
    ./stack.sh: line 337: /root/Desktop/devstack/lib/cinder: Permission denied
    ./stack.sh: line 338: /root/Desktop/devstack/lib/swift: Permission denied
    ./stack.sh: line 339: /root/Desktop/devstack/lib/ceilometer: Permission denied
    ./stack.sh: line 340: /root/Desktop/devstack/lib/heat: Permission denied
    ./stack.sh: line 341: /root/Desktop/devstack/lib/neutron: Permission denied
    ./stack.sh: line 342: /root/Desktop/devstack/lib/baremetal: Permission denied
    ./stack.sh: line 343: /root/Desktop/devstack/lib/ldap: Permission denied
    ./stack.sh: line 420: initialize_database_backends: command not found
    No database enabled
    ./stack.sh: line 426: is_service_enabled: command not found
    ./stack.sh: line 434: is_service_enabled: command not found
    ./stack.sh: line 460: is_service_enabled: command not found

  3. KashyapKashyap03-25-2014


    Hi,

    After installation when we do “nova-manage service list” we get 2 nova-compute service running, one on controller node and other on compute node. If we are having multi-node architecture then controller should not be running nova-compute service. So, how do we fix this and still able to create VM?
    I tried doing with “nova-manage service disable –host=controller –service=nova-compute” , but after that I am not able to create VM.

    Thanking You,
    Kashyap