OpenStack Multi-Node DevStack Nova Network Tutorial

OpenStack Multi-Node DevStack Nova Network Tutorial

Devops Sm

OpenStack Multi-Node DevStack Nova Network Tutorial

DevStack is a scripted OpenStack installer  maintained for developers to prototype and debug. It is also handy for operators to proof concepts and test new services and configurations. This is a good starter for folks new to OpenStack to help understand how to install and operate OpenStack. Some folks have been asking about multi-node DevStack deployment using nova network, so I thought this tutorial might help.

Another difference in this tutorial. is using nova-network instead of the Quantum network plugin. The Quantum plugin is very flexible and offers much richer networking feature, by implementing vSwitches, rather then Linux bridging. With flexibility, comes complexity.The reality if most people looking to do test deployments, proof of concepts or richer KVM/XEN hypervisor management do not really need the added complexity. Nova-network is slated for depreciation at some point, but I am not sure that is the best plan of action for typical until the installation becomes easier for folks in my humble opinion. Most enterprises don’t have “DevOps” programmers, and I think OpenStack brings some good value and expenditure savings to the enterprise. The complexity of the advanced networking may turn some folks off to the project in smaller organizations. Then again smaller organizations aren’t the ones running OpenStack and that may be attractive to vendors looking to find revenue from service offerings. Anywho, thats my two cents, let’s get on to the tutorial.

OpenStack Flat Network Manager Topology
OpenStack Flat Network Manager Topology

Figure 1. Flat Network Manager Topology

Configuring DevStack Installation

DevStack is not a persistent installation, meaning it will be erased when you run ./ This works fine on a single machine running in VMfusion since it supports nested virtualization.

Create a file called localrc for the controller and compute node and fill it in with the localrc configurations below. Change the IP addresses to match your addressing. Put the localrc file in the root of the DevStack folder.

After that is complete, if you do not have any errors you should have a functioning controller and working compute node(s). The compute nodes run the guest VMs. The compute node also does the NAT and security functionality. That can be configuring in the web GUI or from CLI. Your Floating Range will be your publicly reachable network. The fixed range is for the guest VM to talk to the compute node for resources. Since we are using one NIC in each machine, all of the addresses on the compute node are bound to the br100 bridge that is in turn bound to eth0 as secondary addresses. Everything is sharing the same broadcast domain on br100. It is fairly confusing for many without a strong networking background, albeit much simpler than Quantum services. The more you dig in and explore, the clearer it will inevitably become.

When installing note the path of the logs. This is helpful for reverse engineering the installation and troubleshooting services while is running. Logs are not dumped to /var/log/nova* like a regular installation.

OpenStack Controller localrc

Place this in a file named localrc in the root of the devstack directory you clone. That installs into the /opt/ directory be default.

OpenStack Controller Node nova.conf

nova.conf is located in /etc/nova/nova.conf

OpenStack Compute localrc

Place this in a file named localrc in the root of the devstack directory you clone. That installs into the /opt/ directory be default.

OpenStack Compute Node nova.conf

nova.conf is located in /etc/nova/nova.conf

SSH Keys and Booting a Guest VM

OpenStack Glance Guest VM Images

DevStack will download CirrOS but in case it doesn’t or you want to install another image like the Ubuntu Cloud image you can do the follwing.

Download a pre-built OpenStack ready image:

Import the downloaded image into Glance

Verifying OpenStack Services

If you get errors stating you are not authorized or don’t have permsissions to use the client tools, verify your environmental variables are set by typing export. Manually add ENVs with the following.

OpenStack nova-manage

Figure 2. Viewing processes running in DevStack with ‘nova-manage service list’ ‘: -)’ means the process is functioning. ‘XXX’ means the process has died.

OpenStack Processes

 Figure 3. View running procs in Linux with ‘ps auxw’.

Nova-Network Configuration

DevStack will build your network as specified in the localrc. The manual operations of that are as follows:

Since we are only using one NIC in this interface, we will have both the Flat Interface (Public) and the Fixed Range (Privates for VMs) bound to br0100.

OpenStack Processes

Figure 4. View the Linux routing tables ‘route -n’.

To view the Linux bridge configuration use the brctl command. If for some reason DevStack doesn’t create the bridge br100, you can define it yourself with ‘brctl add-br br100’ and ‘brctl addif br100 eth0’. Remember, eth0 is nested in br100. That becomes your Layer3 interface.

OpenStack Brctl

Figure 5. Linux bridge output from ‘brctl’

10 Quick OpenStack Operational Notes
  1. At anytime you can check to make sure the services show smiley faces 🙂 and not XXX with ‘nova-manage service list’.
  2. To manually boot an image, running ‘glance index’ and copy the ID of the image that was loaded in the script.
  3. nova boot –flavor 1 –image <insert image ID here> –key_name ssh_key demohost
  4. Run ‘nova list’ to check the progress and look for any errors.
  5. After a couple of minutes the instance will be booted, make sure you can ping it.
  6. SSH to the ubuntu instance by running “ssh -i ~/ssh-key.pem ubuntu@<ip in nova list>” The key was created by the script.
  7. Check out the web page with http://localhost or http://<public address used>
  8. Under the “Admin” project in Dashboard (the webpage) go to security and access and allocate a floating address to your project.
  9. Once allocated to the project, allocate the address to your VM you started.
  10. Running ./unstack will unload DevStack. Running ./ will rebuild the stack.

Additional DevStack Resources

Thanks for stopping by.

About the Author

Brent SalisburyI have over 15 years of experience wearing various hats from, network engineer, architect, devops and software engineer. I currently have the pleasure of working at the company that develops my favorite software I have ever used, Docker. My comments here are my personal thoughts and opinions. More at Brent's BioView all posts by Brent Salisbury →

  1. vinayusvinayus07-27-2013

    i installed openstack using devstack tutorial….. i uploaded ubuntu 12.04 image too….. but while i create an instance it spawns for a while and ends up being error… i saw log in “syslog” which stated ” failed to load names from /opt/stack/data/nova/networks/nova-br100.hosts: No such file or directory” please help.

    • WoodyWoody09-07-2013

      You should post your localrc for a diagnostic start point.

  2. KashyapKashyap03-11-2014

    i am getting following error:

    [root@nova1 devstack]# chmod u+x tools/
    [root@nova1 devstack]# tools/
    Giving stack user passwordless sudo privileges
    [root@nova1 devstack]# chown -R stack:stack /root/Desktop/devstack/*
    [root@nova1 devstack]# su stack
    [stack@nova1 devstack]$ ./
    ./ line 41: /root/Desktop/devstack/functions: Permission denied
    ./ line 44: /root/Desktop/devstack/lib/config: Permission denied
    ./ line 49: GetDistro: command not found
    rm: cannot remove `/root/Desktop/devstack/’: Permission denied
    ./ line 98: log_error: command not found
    ./ line 100: /root/Desktop/devstack/stackrc: Permission denied
    ./ line 107: export_proxy_variables: command not found
    ./ line 125: log_error: command not found
    ./ line 131: log_error: command not found
    ./ line 135: /root/Desktop/devstack/lib/database: Permission denied
    ./ line 136: /root/Desktop/devstack/lib/rpc_backend: Permission denied
    ./ line 141: disable_negated_services: command not found
    WARNING: this script has not been tested on
    ./ line 148: die: command not found
    ./ line 154: check_rpc_backend: command not found
    ./ line 166: trueorfalse: command not found
    ./ line 186: is_package_installed: command not found
    ./ line 186: install_package: command not found
    sudo: >>> /etc/sudoers.d/50_stack_sh: syntax error near line 1 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 2 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 1 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 2 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 1 <<>> /etc/sudoers.d/50_stack_sh: syntax error near line 2 <<<
    sudo: parse error in /etc/sudoers.d/50_stack_sh near line 1
    sudo: no valid sudoers sources found, quitting
    sudo: unable to initialize policy plugin
    ./ line 263: safe_chown: command not found
    ./ line 272: trueorfalse: command not found
    ./ line 277: trueorfalse: command not found
    ./ line 280: trueorfalse: command not found
    ./ line 290: get_default_host_ip: command not found
    ./ line 292: die: command not found
    ./ line 302: trueorfalse: command not found
    ./ line 310: trueorfalse: command not found
    ./ line 324: /root/Desktop/devstack/lib/apache: Permission denied
    ./ line 327: /root/Desktop/devstack/lib/tls: Permission denied
    ./ line 330: /root/Desktop/devstack/lib/infra: Permission denied
    ./ line 331: /root/Desktop/devstack/lib/oslo: Permission denied
    ./ line 332: /root/Desktop/devstack/lib/stackforge: Permission denied
    ./ line 333: /root/Desktop/devstack/lib/horizon: Permission denied
    ./ line 334: /root/Desktop/devstack/lib/keystone: Permission denied
    ./ line 335: /root/Desktop/devstack/lib/glance: Permission denied
    ./ line 336: /root/Desktop/devstack/lib/nova: Permission denied
    ./ line 337: /root/Desktop/devstack/lib/cinder: Permission denied
    ./ line 338: /root/Desktop/devstack/lib/swift: Permission denied
    ./ line 339: /root/Desktop/devstack/lib/ceilometer: Permission denied
    ./ line 340: /root/Desktop/devstack/lib/heat: Permission denied
    ./ line 341: /root/Desktop/devstack/lib/neutron: Permission denied
    ./ line 342: /root/Desktop/devstack/lib/baremetal: Permission denied
    ./ line 343: /root/Desktop/devstack/lib/ldap: Permission denied
    ./ line 420: initialize_database_backends: command not found
    No database enabled
    ./ line 426: is_service_enabled: command not found
    ./ line 434: is_service_enabled: command not found
    ./ line 460: is_service_enabled: command not found

  3. KashyapKashyap03-25-2014


    After installation when we do “nova-manage service list” we get 2 nova-compute service running, one on controller node and other on compute node. If we are having multi-node architecture then controller should not be running nova-compute service. So, how do we fix this and still able to create VM?
    I tried doing with “nova-manage service disable –host=controller –service=nova-compute” , but after that I am not able to create VM.

    Thanking You,