The Hello Bar is a simple notification bar that engages users and communicates a call to action.

OpenStack Development with virtualenvwrapper

Posted on 24 Jan 2011 in openstack

Coming from the ruby/ruby on rails world, i’ve been a bit lost when it comes to the python development process used in the openstack project. One of the biggest hurdles has been the usage of virtualenv in the workflow. Basically, virtualenv lets you create a stable configuration of python libraries (eggs) much like freezing gems in your rails application. The pitfalls here is that you need to integrate it’s usage into your development flow (activate/deactivate environments), it can take some time to recreate environments if you use a lot of eggs (like nova does) and it seems pretty fragile (it lives in repo and takes some chicanery to avoid duplicating in each bzr branch).

However, I recently found virtualenvwrapper which has erased much of this heartache:

virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

Basically, it centralizes the location of your virtual environments and gives you easy commands to manipulate them: mkvirtaulenv creates one, rmvirtaulenv deletes one and workon activates one. It also allows you to create per environment, user customizable hooks that automatically fire on all the operations. I have created a simple one that automagically changes my directory to my nova trunk directory when I activate my nova virtual environment.

With all that said, here is how I hack on OpenStack Nova on Ubuntu 10.10 (illustrated with nova bzr revno 604). First, install the pre-reqs for getting the code and your basic python environment:

$ sudo apt-get install python-dev swig libssl-dev python-pip bzr
This loads a ton of dependencies packages. Next, pull in our virtualenv eggs: 
$ sudo pip install virtualenv 
$ sudo pip install virtualenvwrapper
Now, setup virtualenvwrapper for your account: 
$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bashrc 
$ source /usr/local/bin/virtualenvwrapper.sh 
$ export WORKON_HOME=~/.virtualenv
With that done, we can start pulling down the source code. I’ve put my nova code branches in ~/src/nova/ : 
$ mkdir -p src 
$ cd src 
$ bzr init-repo nova 
$ cd nova/ 
$ bzr branch lp:nova trunk 
$ cd trunk/
With that in place, let’s create our virtual environment and pull in our python requirements. This is where virtualenvwrapper shines - just one command and we have a portable virtual environment and another to fill it up for all of our branches: 
$ mkvirtualenv nova 
New python executable in nova/bin/python Installing setuptools............done. virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/predeactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/postdeactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/preactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/postactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/get_env_details
(nova)nova@openstack:~/src/nova/trunk$ 
(nova)nova@openstack:~/src/nova/trunk$ pip install -r tools/pip-requires
Now that we got everything good to go, let’s run our tests to make sure everything is good (use -N option, otherwise the script wants to wrap everything with plain virtualenv commands): 
(nova)nova@openstack:~/src/nova/trunk$ ./run_tests.sh -N 
AdminAPITest 
test_admin_disabled ok 
test_admin_enabled ok 
APITest 
test_exceptions_are_converted_to_faults ok
…. SNIP …
Perfect. When you are done, just deactivate: 
(nova)nova@openstack:~/src/nova/trunk$ deactivate
While that makes everything nice and neat, the big win is when you decide to move your repo or create a new branch. To branch, just create the directory (which can be anywhere now), branch your code from trunk and activate your virtual environment (suing the workon command): 
nova@openstack:~/src$ cd patch 
nova@openstack:~/src/patch$ bzr branch ../nova/trunk/ lp708221 
Branched 604 revision(s). 
nova@openstack:~/src/patch$ cd lp708221/ 
nova@openstack:~/src/patch/lp708221$ workon nova 
(nova)nova@openstack:~/src/patch/lp708221$ ./run_tests.sh -N 
AdminAPITest 
test_admin_disabled ok 
test_admin_enabled ok
.... snip ...
If you need to recreate your virtualenv (say after a big change in tools/pip-requires), you can just remove it and then recreate it 
$ rmvirtualenv nova 
$ mkvirtualenv nova2 
(nova2)nova@openstack:~/src$ cd nova/trunk 
(nova2)nova@openstack:~/src/nova/trunk$ pip install -r tools/pip-requires
All in all, virtualenvwrapper has made my hacking flow much better. Once I get a chance, I’ll document this same setup in Fedora and see if we can’t change some of the nova support scripts to better utilize virtualenvwrapper.