Rally “verify” as the control plane for Gabbi, Tempest & in-tree functional tests

It goes without saying that making OpenStack easy testable is crucial for future of OpenStack adoption. Let’s see how OpenStack testing process can be improved by Rally functional testing control plane.

OpenStack architecture pros & cons in nutshell

OpenStack has micro-services architecture. There are bunch of projects, each project has bunch of services, and these services are collaborating together to provide IaaS and PaaS. Micro services approach is not the silver bullet architecture that resolves all issues.

Benefits of Micro Services approach

  1. Isolation. Every part of system: VM, Volumes, Images, Object Storage is separated project, with separated API. So even if implementation is bad it can be rewritten without affecting other parts of system.
  2. Scale. Projects are developed separately, this means separated teams (with their experts and leads) are working on separated projects.

Issues of micro services approach

  1. Common functionality. If you would like to add new API method to all services, or to new
  2. Deployment configuration and management. You need to use separated projects that will just install and manage it. Like Fuel, RDO, JuJu.
  3. CI/CD. Testing requires very smart CI/CD that can pick proper versions of every project, configure all projects properly, start all services and then run tests.
  4. Testing. Every project requires test that brings big issues. Those issues and how to mitigate them is the goal of this blogpost.

Why it is hard to test OpenStack?

Let’s assume that we have pre-production OpenStack cloud, and we would like to verify that it works. Before OpenStack end users (DevOps, Ops) had only Tempest. And they had few headache:

  1. Automation of Tempest installation
  2. Automation of Tempest.conf generation. It has over 200 options that are changed over time.
  3. Parsing subunit stream
  4. Storing and managing results
  5. Integration of 1-4 points to local usage or CI/CD

All these tasks were complicated but not *impossible*. Everybody was more or less happy with such situation. What changed? 

Functional in-tree tests or let’s make testing even harder!

Functional in-tree tests resolves part of issues:

  1. Tempest wasn’t able to scale enough, to handle raising amount of functionality and projects.
  2. Single patch contains: Code + Unit + Functional tests


  1. Running all tests against your cloud becomes pain:
    1. Clone all projects
    2. Install testing requirements for all projects
    3. Configure N specific for each project test.conf files
    4. Run N times tox/testr/nose …
    5. Parse and collect all results
    6. Pain pain pain
  2. Testing N environment from one place, will require changing N conf files each time.

Rally “verify” control plane brings simplicity in testing.

What we have?

As you maybe know Rally is not just a tool that has own framework for testing. It as well automates usage of Tempest. Rally has special command “rally verify” that does next thing:

  1. installation  (it installs tempest in venv)
  2. configuration
  3. running
  4. parsing results
  5. storing results in DB
  6. Work with results

In other words it fully automates work with Tempest. For more details take a look at: https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/

What we should have?

The process of running tempest tests and functional in tree is similar. Download project, configure it, run tests with testr/nose, parse and store results.

Changing “rally verify” to accept project name that will configure project tests and run them against selected cloud will fully automate and unify process of OpenStack testing.

Proposed changes are something like this:


# 1. Check is installed swift for active rally deployment.
# IF NO:
#   Downloads from default (our specified place) swift
#   Switch to master or specified tag
#   Installs in venv swift
#   Configure swift functional test config for active deployment
# 2. Run swift functional test
# 3. Parse subunit output and store to Rally DB (for future work)
rally verify swift start   

# List all swift verification runs 
rally verify swift list

# Shows results relate to one specific run
rally verify swift show UUID

# Compare results of two runs
rally verify swift compare UUID1 UUID2 


Generalization of “rally verify” and making it pluggable control plane for Tempest and in-tree functional tests, from one side allows to avoid huge duplication of efforts in OpenStack community, from another it makes process of testing OpenStack much simpler, unified and tested. Making testing of OpenStack product simple is crucial for it’s quality and better adoption in enterprise world.

P.S. If you are interested: join us on #openstack-rally

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>