Ken Pepple

OpenStack Folsom Architecture

2012-09-25T00:00:00-07:00

As the Folsom release of OpenStack is due to be released this week, I've taken the time to update my "Intro to OpenStack Architecture 101" for the official documentation. It merged into the repos yesterday and below is an expanded version of it.

OpenStack Components

There are currently seven core components of OpenStack: Compute, Object Storage, Identity, Dashboard, Block Storage, Network and Image Service. Let's look at each in turn:

Object Store (codenamed "Swift") allows you to store or retrieve files (but not mount directories like a fileserver). Several companies provide commercial storage services based on Swift. These include KT, Rackspace (from which Swift originated) and Internap. Swift is also used internally at many large companies to store their data.
Image (codenamed "Glance") provides a catalog and repository for virtual disk images. These disk images are mostly commonly used in OpenStack Compute. While this service is technically optional, any cloud of size will require it.
Compute (codenamed "Nova") provides virtual servers upon demand. Rackspace and HP provide commercial compute services built on Nova and it is used internally at companies like Mercado Libre and NASA (where it originated).
Dashboard (codenamed "Horizon") provides a modular web-based user interface for all the OpenStack services. With this web GUI, you can perform most operations on your cloud like launching an instance, assigning IP addresses and setting access controls.
Identity (codenamed "Keystone") provides authentication and authorization for all the OpenStack services. It also provides a service catalog of services within a particular OpenStack cloud.
Network (codenamed "Quantum") provides "network connectivity as a service" between interface devices managed by other OpenStack services (most likely Nova). The service works by allowing users to create their own networks and then attach interfaces to them. Quantum has a pluggable architecture to support many popular networking vendors and technologies. Quantum is new in the Folsom release.
Block Storage (codenamed "Cinder") provides persistent block storage to guest VMs. This project was born from code originally in Nova (the nova-volume service described below). Please note that this is block storage (or volumes) not filesystems like NFS or CIFS share. Cinder is new for the Folsom release.

In addition to these core projects, there are also a number of "incubation" projects that are being considered for future inclusion in the OpenStack core.

Comparison Against AWS

Although all of the OpenStack services are made to stand on their own, many users want some form of AWS compatibility for tools, APIs or just familiarity:

Nova is conceptually similar to EC2. In fact, there is are a variety of ways to provide EC2 API compatibility to it.
Swift is conceptually similar to S3. It implements a subset of the S3 API in WSGI middleware.
Glance provides many of the same features as Amazon's AMI catalog.
Cinder provides block services similar to EBS.

Conceptual Architecture

The OpenStack project as a whole is designed to "deliver(ing) a massively scalable cloud operating system." To achieve this, each of the constituent services are designed to work together to provide a complete Infrastructure as a Service (IaaS). This integration is facilitated through public application programming interfaces (APIs) that each service offers (and in turn can consume). While these APIs allow each of the services to use another service, it also allows an implementer to switch out any service as long as they maintain the API. These are (mostly) the same APIs that are available to end users of the cloud.

Conceptually, you can picture the relationships between the services as so:

Dashboard ("Horizon") provides a web front end to the other OpenStack services
Compute ("Nova") stores and retrieves virtual disks ("images") and associated metadata in Image ("Glance")
Network ("Quantum") provides virtual networking for Compute.
Block Storage ("Cinder") provides storage volumes for Compute.
Image ("Glance") can store the actual virtual disk files in the Object Store("Swift")
All the services authenticate with Identity ("Keystone")

This is a stylized and simplified view of the architecture, assuming that the implementer is using all of the services together in the most common configuration. It also only shows the "operator" side of the cloud -- it does not picture how consumers of the cloud may actually use it. For example, many users will access object storage heavily (and directly).

Logical Architecture

As you can imagine, the logical architecture is far more complicated than the conceptual architecture shown above. As with any service-oriented architecture, diagrams quickly become "messy" trying to illustrate all the possible combinations of service communications. The diagram below, illustrates the most common architecture of an OpenStack-based cloud. However, as OpenStack supports a wide variety of technologies, it does not represent the only architecture possible.

This picture is consistent with the conceptual architecture above in that:

End users can interact through a common web interface (Horizon) or directly to each service through their API
All services authenticate through a common source (facilitated through Keystone)
Individual services interact with each other through their public APIs (except where privileged administrator commands are necessary)

In the sections below, we'll delve into the architecture for each of the services.

Dashboard

Horizon is a modular Django web application that provides an end user and administrator interface to OpenStack services.

As with most web applications, the architecture is fairly simple:

Horizon is usually deployed via mod_wsgi in Apache. The code itself is separated into a reusable python module with most of the logic (interactions with various OpenStack APIs) and presentation (to make it easily customizable for different sites).
A database (configurable as to which one). As it relies mostly on the other services for data, it stores very little data of its own.

From a network architecture point of view, this service will need to be customer accessible as well as be able to talk to each service's public APIs. If you wish to use the administrator functionality (i.e. for other services), it will also need connectivity to their Admin API endpoints (which should not be customer accessible).

Compute

Nova is the most complicated and distributed component of OpenStack. A large number of processes cooperate to turn end user API requests into running virtual machines. Below is a list of these processes and their functions:

nova-api accepts and responds to end user compute API calls. It supports OpenStack Compute API, Amazon's EC2 API and a special Admin API (for privileged users to perform administrative actions). It also initiates most of the orchestration activities (such as running an instance) as well as enforces some policy (mostly quota checks).
The nova-compute process is primarily a worker daemon that creates and terminates virtual machine instances via hypervisor's APIs (XenAPI for XenServer/XCP, libvirt for KVM or QEMU, VMwareAPI for VMware, etc.). The process by which it does so is fairly complex but the basics are simple: accept actions from the queue and then perform a series of system commands (like launching a KVM instance) to carry them out while updating state in the database.
nova-volume manages the creation, attaching and detaching of persistent volumes to compute instances (similar functionality to Amazon’s Elastic Block Storage). It can use volumes from a variety of providers such as iSCSI or Rados Block Device in Ceph. A new OpenStack projects, Cinder, will eventually replace nova-volume functionality. In the Folsom release, nova-volume and the Block Storage service will have similar functionality.
The nova-network worker daemon is very similar to nova-compute and nova-volume. It accepts networking tasks from the queue and then performs tasks to manipulate the network (such as setting up bridging interfaces or changing iptables rules). This functionality is being migrated to Quantum, a separate OpenStack service. In the Folsom release, much of the functionality will be duplicated between nova-network and Quantum.
The nova-schedule process is conceptually the simplest piece of code in OpenStack Nova: take a virtual machine instance request from the queue and determines where it should run (specifically, which compute server host it should run on).
The queue provides a central hub for passing messages between daemons. This is usually implemented with RabbitMQ today, but could be any AMPQ message queue (such as Apache Qpid). New to the Folsom release is support for Zero MQ (note: I've only included this so that Eric Windisch won't be hounding me mercilessly about it's omission).
The SQL database stores most of the build-time and run-time state for a cloud infrastructure. This includes the instance types that are available for use, instances in use, networks available and projects. Theoretically, OpenStack Nova can support any database supported by SQL-Alchemy but the only databases currently being widely used are sqlite3 (only appropriate for test and development work), MySQL and PostgreSQL.
Nova also provides console services to allow end users to access their virtual instance's console through a proxy. This involves several daemons (nova-console, nova-vncproxy and nova-consoleauth).

Nova interacts with many other OpenStack services: Keystone for authentication, Glance for images and Horizon for web interface. The Glance interactions are central. The API process can upload and query Glance while nova-compute will download images for use in launching images.

Object Store

The swift architecture is very distributed to prevent any single point of failure as well as to scale horizontally. It includes the following components:

Proxy server (swift-proxy-server) accepts incoming requests via the OpenStack Object API or just raw HTTP. It accepts files to upload, modifications to metadata or container creation. In addition, it will also serve files or container listing to web browsers. The proxy server may utilize an optional cache (usually deployed with memcache) to improve performance.
Account servers manage accounts defined with the object storage service.
Container servers manage a mapping of containers (i.e folders) within the object store service.
Object servers manage actual objects (i.e. files) on the storage nodes.

There are also a number of periodic process which run to perform housekeeping tasks on the large data store. The most important of these is the replication services, which ensures consistency and availability through the cluster. Other periodic processes include auditors, updaters and reapers.

The object store can also serve static web pages and objects via HTTP. In fact, the diagrams in this blog post are being served out of Rackspace Cloud's Swift service.

Authentication is handled through configurable WSGI middleware (which will usually be Keystone).

Image Store

The Glance architecture has stayed relatively stable since the Cactus release. The biggest architectural change has been the addition of authentication, which was added in the Diablo release. Just as a quick reminder, Glance has four main parts to it:

glance-api accepts Image API calls for image discovery, image retrieval and image storage.
glance-registry stores, processes and retrieves metadata about images (size, type, etc.).
A database to store the image metadata. Like Nova, you can choose your database depending on your preference (but most people use MySQL or SQlite).
A storage repository for the actual image files. In the diagram above, Swift is shown as the image repository, but this is configurable. In addition to Swift, Glance supports normal filesystems, RADOS block devices, Amazon S3 and HTTP. Be aware that some of these choices are limited to read-only usage.

There are also a number of periodic process which run on Glance to support caching. The most important of these is the replication services, which ensures consistency and availability through the cluster. Other periodic processes include auditors, updaters and reapers.

As you can see from the diagram in the Conceptual Architecture section, Glance serves a central role to the overall IaaS picture. It accepts API requests for images (or image metadata) from end users or Nova components and can store its disk files in the object storage service, Swift.

Identity

Keystone provides a single point of integration for OpenStack policy, catalog, token and authentication.

keystone handles API requests as well as providing configurable catalog, policy, token and identity services.
Each Keystone function has a pluggable backend which allows different ways to use the particular service. Most support standard backends like LDAP or SQL, as well as Key Value Stores (KVS).

Most people will use this as a point of customization for their current authentication services.

Network

Quantum provides "network connectivity as a service" between interface devices managed by other OpenStack services (most likely Nova). The service works by allowing users to create their own networks and then attach interfaces to them. Like many of the OpenStack services, Quantum is highly configurable due to it's plug-in architecture. These plug-ins accommodate different networking equipment and software. As such, the architecture and deployment can vary dramatically. In the above architecture, a simple Linux networking plug-in is shown.

quantum-server accepts API requests and then routes them to the appropriate quantum plugin for action.
Quantum plugins and agents perform the actual actions such as plugging and unplugging ports, creating networks or subnets and IP addressing. These plugins and agents differ depending on the vendor and technologies used in the particular cloud. Quantum ships with plugins and agents for: Cisco virtual and physical switches, Nicira NVP product, NEC OpenFlow products, Open vSwitch, Linux bridging and the Ryu Network Operating System. Midokua also provides a plug-in for Quantum integration. The common agents are L3 (layer 3), DHCP (dynamic host IP addressing) and the specific plug-in agent.
Most Quantum installations will also make use of a messaging queue to route information between the quantum-server and various agents as well as a database to store networking state for particular plugins.

Quantum will interact mainly with Nova, where it will provide networks and connectivity for its instances.

Block Storage

Cinder separates out the persistent block storage functionality that was previously part of Openstack Compute (in the form of nova-volume) into it's own service. The OpenStack Block Storage API allows for manipulation of volumes, volume types (similar to compute flavors) and volume snapshots.

cinder-api accepts API requests and routes them to cinder-volume for action.
cinder-volume acts upon the requests by reading or writing to the Cinder database to maintain state, interacting with other processes (like cinder-scheduler) through a message queue and directly upon block storage providing hardware or software. It can interact with a variety of storage providers through a driver architecture. Currently, there are drivers for IBM, SolidFire, NetApp, Nexenta, Zadara, linux iSCSI and other storage providers.
Much like nova-scheduler, the cinder-scheduler daemon picks the optimal block storage provider node to create the volume on.
Cinder deployments will also make use of a messaging queue to route information between the cinder processes as well as a database to store volume state.

Like Quantum, Cinder will mainly interact with Nova, providing volumes for its instances.

Future Projects

A number of projects are in the work for future versions of OpenStack. They include:

Ceilometer is a metering project. The project offers metering information and the ability to code more ways to know what has happened on an OpenStack cloud. While it provides metering, it is not a billing project. A full billing solution requires metering, rating, and billing. Metering lets you know what actions have taken place, rating enables pricing and line items, and billing gathers the line items to create a bill to send to the consumer and collect payment. Ceilometer is available as a preview.
Heat provides a REST API to orchestrate multiple cloud applications implementing standards such as AWS CloudFormation.

Revisiting OpenStack Architecture: Essex Edition

2012-02-21T00:00:00-08:00

While it is still seven weeks until OpenStack "Essex" (2012.1) officially is released, release candidates are just around the corner. With this in mind, I thought it would be a good chance to revisit my earlier blog post on OpenStack Compute ("Nova") architecture. This time around, instead of detailing the architecture of just a single service, I'll look at all the pieces of the OpenStack project working together.

To level-set everyone's understanding, let's briefly review the OpenStack project components and history. Founded in 2010 by Rackspace and NASA, the project has released four versions and is set to release the fifth ("Essex" or 2012.1) in April. Originally, it consisted of a trio of "core" services:

Object Store ("Swift") provides object storage. It allows you to store or retrieve files (but not mount directories like a fileserver). Several companies provide commercial storage services based on Swift. These include KT, Rackspace (from which Swift originated) and my company Internap. In fact, the images for this blog post are being served via the Internap Swift implementation.
Image ("Glance") provides a catalog and repository for virtual disk images. These disk images are mostly commonly used in OpenStack Compute. While this service is technically optional, any cloud of size will require it.
Compute ("Nova") provides virtual servers upon demand. Similar to Amazon's EC2 service, it also provides volume services analogous to Elastic Block Services (EBS). Internap provide a commercial compute service built on Nova and it is used internally at Mercado Libre and NASA (where it originated).

The upcoming release promotes two new projects to "core" project status:

Dashboard ("Horizon") provides a modular web-based user interface for all the OpenStack services.
Identity ("Keystone") provides authentication and authorization for all the OpenStack services. It also provides a service catalog of services within a particular deployment.

These new projects provide additional infrastructure to support the original three projects.

Conceptual Architecture

Conceptually, you can picture the relationships between the services as so:

Horizon provides a web front end to the other OpenStack services
Nova stores and retrieves virtual disks ("images") and associated metadata in Glance
Glance can store the actual virtual disk files in Swift
All the services (will eventually) authenticate with Keystone

This is a stylized and simplified view of the architecture, assuming that the implementer is using all of the services together in the most common configuration. It also only shows the "operator" side of the cloud -- it does not picture how consumers of the cloud may actually use it. For example, many compute users will use object storage heavily (and directly).

Logical Architecture

As you can imagine, the actual logical architecture is far more complicated than the conceptual architecture shown above. As with any service-oriented architecture, diagrams quickly become "messy" trying to illustrate all the possible combinations of service communications. In the diagram below, I illustrate what I believe will be the most common, "integrated" architecture of an OpenStack-based cloud.

This picture is consistent with the description above in that:

End users can interact through a common web interface (Horizon) or directly to each service through their API
All services authenticate through a common source (facilitated through Keystone)
Individual services interact with each other through their public APIs (except where privileged administrator commands are necessary)

In the sections below, we'll delve into the architecture for each of the services.

Dashboard

Horizon is a modular Django web application that provides an end user and administrator interface to OpenStack services.

As with most web applications, the architecture is fairly simple:

Horizon is usually deployed via mod_wsgi in Apache. The code itself is seperated into a reusable python module with most of the logic (interactions with various OpenStack APIs) and presentation (to make it easily customizable for different sites).
A database (configurable to which one). As it relies mostly on the other services for data, it stores very little data of it's own.

From a network architecture point of view, this service will need to customer accessible as well as be able to talk to each services public APIs. If you wish to use the administrator functionality (i.e. for other services), it will also need connectivity to their Admin API endpoints (which should be non-customer accessible).

Compute

Not much has really changed with Nova's architecture. They have added a few new helper services for EC2 compatibility and console services.

nova-api accepts and responds to end user compute and volume API calls. It supports OpenStack API, Amazon's EC2 API and a special Admin API (for privileged users to perform administrative actions). It also initiates most of the orchestration activities (such as running an instance) as well as enforces some policy (mostly quota checks). In the Essex release, nova-api has been modularized, allowing for implementers to run only specific APIs.
The nova-compute process is primarily a worker daemon that creates and terminates virtual machine instances via hypervisor's APIs (XenAPI for XenServer/XCP, libvirt for KVM or QEMU, VMwareAPI for VMware, etc.). The process by which it does so is fairly complex but the basics are simple: accept actions from the queue and then perform a series of system commands (like launching a KVM instance) to carry them out while updating state in the database.
nova-volume manages the creation, attaching and detaching of persistent volumes to compute instances (similar functionality to Amazon’s Elastic Block Storage). It can use volumes from a variety of providers such as iSCSI or Rados Block Device in Ceph.
The nova-network worker daemon is very similar to nova-compute and nova-volume. It accepts networking tasks from the queue and then performs tasks to manipulate the network (such as setting up bridging interfaces or changing iptables rules).
The nova-schedule process is conceptually the simplest piece of code in OpenStack Nova: take a virtual machine instance request from the queue and determines where it should run (specifically, which compute server host it should run on).
The queue provides a central hub for passing messages between daemons. This is usually implemented with RabbitMQ today, but could be any AMPQ message queue (such as Apache Qpid).
The SQL database stores most of the build-time and run-time state for a cloud infrastructure. This includes the instance types that are available for use, instances in use, networks available and projects. Theoretically, OpenStack Nova can support any database supported by SQL-Alchemy but the only databases currently being widely used are sqlite3 (only appropriate for test and development work), MySQL and PostgreSQL.

During the last two releases, Nova has augmented it's console services. Console services allow end users to access their virtual instance's console through a proxy. This involves a pair of new daemons (nova-console and nova-consoleauth).

Nova interacts with all of the usual suspects: Keystone for authentication, Glance for images and Horizon for web interface. The Glance interacts is interesting, though. The API process can upload and query Glance while nova-compute will download images for use in launching images.

Object Store

The swift architecture is very distributed to prevent any single point of failure as well as to scale horizontally. It includes the following components:

Proxy server accepts incoming requests via the OpenStack Object API or just raw HTTP. It accepts files to upload, modifications to metadata or container creation. In addition, it will also serve files or container listing to web browsers. The proxy server may utilize an optional cache (usually deployed with memcache) to improve performance.
Account servers manage accounts defined with the object storage service.
Container servers manage a mapping of containers (i.e folders) within the object store service.
Object servers manage actual objects (i.e. files) on the storage nodes.
There are also a number of periodic process which run to perform housekeeping tasks on the large data store. The most important of these is the replication services, which ensures consistency and availability through the cluster. Other periodic processes include auditors, updaters and reapers.

Authentication is handled through configurable WSGI middleware (which will usually be Keystone).

Image Store

glance-api accepts Image API calls for image discovery, image retrieval and image storage
glance-registry stores, processes and retrieves metadata about images (size, type, etc.)
A database to store the image metadata. Like Nova, you can choose your database depending on your preference (but most people use MySQL or SQlite).
A storage repository for the actual image files. In the diagram, I have shown the most likely configuration (using Swift as the image repository), but this is configurable. In addition to Swift, Glance supports normal filesystems, RADOS block devices, Amazon S3 and HTTP. Be aware that some of these choices are limited to read-only usage.

As you can see from the diagram, Glance serves a central role to the overall IaaS picture. It accepts API requests for images (or image metadata) from end users or Nova components and can store it's disk files in

Identity

Keystone provides a single point of integration for OpenStack policy, catalog, token and authentication.

Keystone handles API requests as well as providing configurable catalog, policy, token and identity services.
Each keystone function has a pluggable backend which allows different ways to use the particular service. Most support standard backends like LDAP or SQL, as well as Key Value Stores (KVS).

Most people will use this as a point of customization for their current authentication services.

Future Projects

This completes the tour of the OpenStack Essex architecture. However, OpenStack will not be stopping here - the following OpenStack release ("Folsom") will welcome another core service to the fold:

Network (Quantum) provides "network connectivity as a service" between interface devices managed by other Openstack services (most likely Nova). The service works by allowing users to create their own networks and then attach interfaces to them.

Although the release schedule for Folsom is not yet set (probably Fall 2012), I won't wait six months to update the picture for this.

OpenStack Has A Mascot: The "Tenrec" Book

2011-08-05T00:00:00-07:00

You may have noticed that I have not posted anything in the last two months. And as you have probably guessed, it was because I was busy working on something else.

That "something else" was writing "Deploying OpenStack" for O'Reilly Media.

The book is intended to provide an introduction to the OpenStack project (Glance, Swift and Nova) "Cactus" release, an architectural overview of each component and some best practices for their deployment. In particular, we wanted to go beyond the documentation to arm readers with information to make smart decisions for their installations. However, we also went step by step to describe the installation process using normal packages as well as the StackOps distribution. Here are the chapter listings:

The OpenStack Project
Understanding Swift
Understanding Glance
Understanding Nova
Obtaining Nova
Planning Nova Deployment
Installing Nova (via packages and StackOps)
Using Nova
Administering Nova

However, this book is not "OpenStack: The Definitive Guide" or any such tome:

It is a relatively short 80 pages.
It is also very heavily weighted towards Nova. That is not to say that we don't cover Swift or Glance, just not to the same level of detail. For example, we don't cover Swift installation, while we devote an entire chapter to Nova's installation.
It only documents a single node installation in two controlled ways. It doesn't cover every hardware variation or deployment scenario.

Most importantly, OpenStack now has an mascot. The humble tailless tenrec is an African mammal similar to hedgehogs (I prefer to believe that the cover picture is a Greater Hedgehog Tenrec). Please commence creating plushy toys.

If you are interested in this book, I would highly recommend getting the Ebook version. It is cheaper, it comes in almost any format (ePub, Mobi, PDF) and it will be updated. Yes, updated. You will see additional content come out for it, as it is written. For anyone that has been following the OpenStack community, you will realize that the code changes quickly.

I also wanted to send big thanks to my technical reviewers:

Josh Kearney (jk0)
Diego Parrilla (CEO of StackOps)
Brian Pepple (former chairman of the Fedora Engineering Steering Committee)
Su-hyung "Nick" Lee

Their expertise made the book immeasurably better.

Oh yeah, you can buy it here.

Finding Your OpenStack Nova version

2011-04-30T00:00:00-07:00

I just got back from the OpenStack Developers Summit and while I am still trying to compile my thoughts on the event, I thought I'd dash off this little tidbit:

A slight annoyance in administering (and troubleshooting) OpenStack Nova is identifying your installed version. This information is actually readily (i.e. programmatically) available within the code (and logged at the beginning of most logfiles), but we hadn't exposed it to administrators on the command line until Nova trunk revision #1036. With this change, you can now simply type nova-manage version list to find which OpenStack Nova version is installed. For example:

$ nova-manage version list
2011.3-dev (2011.3-workspace:tarmac-20110428165803-elcz2wp2syfzvxm8)

As you can probably guess, this machine is installed with a post-Cactus / pre-Diablo version of Nova ("2011.3-dev") from the ppa:trunk packages ("tarmac-20110428165803"). Final released versions will report non-dev version numbers like 2011.1 (Bexar), 2011.2 (Cactus) or 2011.3 (upcoming Diablo). Releases from Launchpad's ppa/trunk or ppa/release repositories will report "tarmac" in the string.

For those who aren't following Nova trunk (cutting edge builds) or wanting to understand this deeper: In nova/version.py there are two methods, version_string() and version_string_with_vcs(). You can see their use here in the python shell:

$ nova-manage shell python
Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) 
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from nova import version
>>> version.version_string()
'2011.3-dev'
>>> version.version_string_with_vcs()
u'2011.3-workspace:tarmac-20110428165803-elcz2wp2syfzvxm8'
>>>

version_string() pulls info straight from constants in the source code while version_string_with_vcs() is generated at package creation time in the setup.py file (which basically writes the nova/vcsversion.py file with bzr version-info --python output). Unless you have run setup.py (such as when you are developing a local branch), you'll just see 2011.3-LOCALBRANCH:LOCALREVISION as your vcsversion string.

I am working on adding more diagnostic, operational support features for Diablo (service versions, flag state, etc.) and will post as they merge into OpenStack Nova trunk.

OpenStack Nova Architecture

2011-04-22T00:00:00-07:00

NOTE: I've updated and expanded this blog post for the Folsom release. Click here to read the updated version.

One of the common refrains I hear from people getting started with OpenStack is the lack of good introductory architectural overviews of the project. I was confronted by the same problem when I first started with the project - it was easy to get the low level code and API documentation but it was very difficult to find a “lay of the land”-type overview. Now that Cactus (OpenStack’s third version) has been released, I thought I’d take advantage of the lull in development to write up a quick architectural overview from my point of view. Since OpenStack is a fairly broad topic, I’ll break my thoughts into several posts. Today’s post will deal with OpenStack Nova’s (compute cloud) high level architecture.

Before we dive into the conceptual and logic architecture, let’s take a second to explain the OpenStack project:

OpenStack is a collection of open source technologies delivering a massively scalable cloud operating system.

You can think of it as software to power your own Infrastructure as a Service (IaaS) offering like Amazon Web Services. It currently encompasses three main projects:

Swift which provides object/blob storage. This is roughly analogous to Rackspace Cloud Files (from which it is derived) or Amazon S3.
Glance which provides discovery, storage and retrieval of virtual machine images for OpenStack Nova.
Nova which provides virtual servers upon demand. This is similar to Rackspaces Cloud Servers or Amazon EC2.

While these three projects provide the core of the cloud infrastructure, OpenStack is open and evolving — there will be more projects (there are already related projects for web interfaces and a queue service). With that brief introduction, let’s delve into a conceptual architecture and then examine how OpenStack Nova could map to it.

Cloud Provider Conceptual Architecture

Imagine that we are going to build our own IaaS cloud and offer it to customers. To achieve this, we would need to provide several high level features:

Allow application owners to register for our cloud services, view their usage and see their bill (basic customer relations management functionality)
Allow Developers/DevOps folks to create and store custom images for their applications (basic build-time functionality)
Allow DevOps/Developers to launch, monitor and terminate instances (basic run-time functionality)
Allow the Cloud Operator to configure and operate the cloud infrastructure

While there are certainly many, many other features that we would need to offer (especially if we were to follow are more complete industry framework like eTOM), these four get to the very heart of providing IaaS. Now assuming that you agree with these four top level features, you might put together a conceptual architecture that looks something like this:

In this model, I’ve imagined four sets of users (developers, devops, owners and operators) that need to interact with the cloud and then separated out the functionality needed for each. From there, I’ve followed a pretty common tiered approach to the architecture (presentation, logic and resources) with two orthogonal areas (integration and management). Let’s explore each a little further:

As with presentation layers in more typical application architectures, components here interact with users to accept and present information. In this layer, you will find web portals to provide graphical interfaces for non-developers and API endpoints for developers. For more advanced architectures, you might find load balancing, console proxies, security and naming services present here also.
The logic tier would provide the intelligence and control functionality for our cloud. This tier would house orchestration (workflow for complex tasks), scheduling (determining mapping of jobs to resources), policy (quotas and such) , image registry (metadata about instance images), logging (events and metering).
There will need to integration functions within the architecture. It is assumed that most service providers will already have a customer identity and billing systems. Any cloud architecture would need to integrate with these systems.
As with any complex environment, we will need a management tier to operate the environment. This should include an API to access the cloud administration features as well as some forms of monitoring. It is likely that the monitoring functionality will take the form of integration into an existing tool. While I’ve highlighted monitoring and an admin API for our fictional provider, in a more complete architecture you would see a vast array of operational support functions like provisioning and configuration management.
Finally, since this is a compute cloud, we will need actual compute, network and storage resources to provide to our customers. This tier provides these services, whether they be servers, network switches, network attached storage or other resources.

With this model in place, let’s shift gears and look at OpenStack Nova’s logical architecture.

OpenStack Nova Logical Architecture

Now that we’ve looked at a proposed conceptual architecture, let’s see how OpenStack Nova is logically architected. Since Cactus is the newest release, I will concentrate there (which means if you are viewing this after around July 2011, this will be out of date). There are several logical components of OpenStack Nova architecture but the majority of these components are custom written python daemons of two varieties:

WSGI applications to receive and mediate API calls (nova-api, glance-api, etc.)
Worker daemons to carry out orchestration tasks (nova-compute, nova-network, nova-schedule, etc.)

However, there are two essential pieces of the logical architecture are neither custom written nor Python based: the messaging queue and the database. These two components facilitate the asynchronous orchestration of complex tasks through message passing and information sharing. Putting this all together we get a picture like this:

This complicated, but not overly informative, diagram as it can be summed up in three sentences:

End users (DevOps, Developers and even other OpenStack components) talk to nova-api to interface with OpenStack Nova
OpenStack Nova daemons exchange info through the queue (actions) and database (information) to carry out API requests
OpenStack Glance is basically a completely separate infrastructure which OpenStack Nova interfaces through the Glance API

Now that we see the overview of the processes and their interactions, let’s take a closer look at each component.

The nova-api daemon is the heart of the OpenStack Nova. You may see it illustrated on many pictures of OpenStack Nova as API and “Cloud Controller”. While this is partly true, cloud controller is really just a class (specifically the CloudController in trunk/nova/api/ec2/cloud.py) within the nova-api daemon. It provides an endpoint for all API queries (either OpenStack API or EC2 API), initiates most of the orchestration activities (such as running an instance) and also enforces some policy (mostly quota checks).
The nova-schedule process is conceptually the simplest piece of code in OpenStack Nova: take a virtual machine instance request from the queue and determines where it should run (specifically, which compute server host it should run on). In practice however, I am sure this will grow to be the most complex as it needs to factor in current state of the entire cloud infrastructure and apply complicated algorithm to ensure efficient usage. To that end, nova-schedule implements a pluggable architecture that let’s you choose (or write) your own algorithm for scheduling. Currently, there are several to choose from (simple, chance, etc) and it is a area of hot development for the future releases of OpenStack Nova.
The nova-compute process is primarily a worker daemon that creates and terminates virtual machine instances. The process by which it does so is fairly complex (see this blog post by Laurence Luce for the gritty details) but the basics are simple: accept actions from the queue and then perform a series of system commands (like launching a KVM instance) to carry them out while updating state in the database.
As you can gather by the name, nova-volume manages the creation, attaching and detaching of persistent volumes to compute instances (similar functionality to Amazon’s Elastic Block Storage). It can use volumes from a variety of providers such as iSCSI or AoE.
The nova-network worker daemon is very similar to nova-compute and nova-volume. It accepts networking tasks from the queue and then performs tasks to manipulate the network (such as setting up bridging interfaces or changing iptables rules).
The queue provides a central hub for passing messages between daemons. This is currently implemented with RabbitMQ today, but theoretically could be any AMPQ message queue supported by the python ampqlib.
The SQL database stores most of the build-time and run-time state for a cloud infrastructure. This includes the instance types that are available for use, instances in use, networks available and projects. Theoretically, OpenStack Nova can support any database supported by SQL-Alchemy but the only databases currently being widely used are sqlite3 (only appropriate for test and development work), MySQL and PostgreSQL.
OpenStack Glance is a separate project from OpenStack Nova, but as shown above, complimentary. While it is an optional part of the overall compute architecture, I can’t imagine that most OpenStack Nova installation will not be using it (or a complimentary product). There are three pieces to Glance: glance-api, glance-registry and the image store. As you can probably guess, glance-api accepts API calls, much like nova-api, and the actual image blobs are placed in the image store. The glance-registry stores and retrieves metadata about images. The image store can be a number of different object stores, including OpenStack Swift.
Finally, another optional project that we will need for our fictional service provider is an user dashboard. I have picked the OpenStack Dashboard here, but there are also several other web front ends available for OpenStack Nova. The OpenStack Dashboard provides a web interface into OpenStack Nova to give application developers and devops staff similar functionality to the API. It is currently implemented as a Django web application.

This logical architecture represents just one way to architect OpenStack Nova. With it’s pluggable architecture, we could easily swap out OpenStack Glance with another image service or use another dashboard. In the coming releases of OpenStack, expect to see more modularization of the code especially in the network and volume areas.

Nova Conceptual Mapping

Now that we’ve seen a conceptual architecture for a fictional cloud provider and examined the logical architecture of OpenStack Nova, it is fairly easy to map the OpenStack components to the conceptual areas to see what we are lacking:

As you can see from the illustration, I’ve overlaid logical components of OpenStack Nova, Glance and Dashboard to denote functional coverage. For each of the overlays, I’ve added the name of the logical component within the project that provides the functionality. While all of these judgements are highly subjective, you can see that we have a majority coverage of the functional areas with a few notable exceptions:

The largest gap in our functional coverage is logging and billing. At the moment, OpenStack Nova doesn’t have a billing component that can mediate logging events, rate the logs and create/present bills. That being said, most service providers will already have one (or many) of these so the focus is really on the logging and integration with billing. This could be remedied in a variety of ways: augmentations of the code (which should happen in the next release “Diablo”), integration with commercial products or services (perhaps Zuora) or custom log parsing.
Identity is also a point which will likely need to be augmented. Unless we are running a stock LDAP for our identity system, we will need to integrate our solution with OpenStack Nova. Having said that, this is true of almost all cloud solutions.
The customer portal will also be an integration point. While OpenStack Nova provides a user dashboard (to see running instance, launch new instances, etc.), it doesn’t provide an interface to allow application owners to signup for service, track their bills and lodge trouble tickets. Again, this is probably something that it is already in place at our imaginary service provider.
Ideally, the Admin API would replicate all functionality that we’d be able to do via the command line interface (which in this case is mostly the exposed through the nova-manage command). This will get better in the Diablo release with the Admin API work.
Cloud monitoring and operations will be an important area of focus for our service provider. A key to any good operations approach is good tooling. While OpenStack Nova provides nova-instancemonitor, which tracks compute node utilization, we’re really going to need a number of third party tools for monitoring.
Policy is an extremely important area but very provider specific. Everything from quotas (which are supported) to quality of service (QoS) to privacy controls can fall under this. I’ve given OpenStack Nova partial coverage here, but that might vary depending on the intricacies of the providers needs. For the record, OpenStack Nova Cactus provides quotas for instances (number and cores used), volumes (size and number), floating IP addresses and metadata.
Scheduling within OpenStack Nova is fairly rudimentary for larger installations today. The pluggable scheduler supports chance (random host assignment), simple (least loaded) and zone (random nodes within an availability zone). As within most areas on this list, this will be greatly augmented in Diablo. In development are distributed schedulers and schedulers that understand heterogeneous hosts (for support of GPUs and differing CPU architectures).

As you can see, OpenStack Nova provides a fair basis for our mythical service provider, as long as we are willing to do some integration here and there. In my next post, I’ll dive deeper into OpenStack Nova with a discussion on deployment architecture choices.

Auto Blogging with Rake & Rails

2011-03-26T00:00:00-07:00

I have a long running blog that I struggle to keep current. The blog centers on mountain bike racing scene here in Northern California, so the content needs to stay timely as there are races every weekend all year long. Unfortunately, I don't have a lot of free time to devote to it. Long story short, I wanted to automatically create a post every Thursday morning to let people know what races were being held that weekend. You can see an example of what I wanted at NorCal MTB Racing blog.

The basic idea was to create a cron job that pulls a list of races (with a link for details) from my rails application's database, plot the events on map, add some text and then post it to my blog on Blogger. Most of this is pretty standard rails stuff, but I did need some help for the map and posting.

For the maps, I used Google Static Maps. This allows you to embed a Google Maps image on your webpage just like a normal image. All you need to do it feed it the right parameters. For example, this HTML code:

<img alt="NorCal MTB Race Event Map" src="http://maps.google.com/maps/api/staticmap?size=600x200&amp;maptype=roadmap&amp;sensor=false&amp;markers=color:blue|label:1|38.7067,-122.903&amp;markers=color:blue|label:2|37.7756,-122.438&amp;markers=color:blue|label:4|38.028,-121.885" />

Produces this map:

To post to Blogger, I used the blogger gem (installation instructions). There are other ruby gems out there that interface with Blogger, but this was by far the most straight forward. To use the gem, you need username, password and blog ID. I found my blog ID through an answer in the blogger support forum:

Got to your dashboard and click Settings for the blog you want the ID of. In the address bar you should then see something like this:

http://www.blogger.com/blog-options-basic.g?blogID=9101099271220909428

The long number at the end of the URL is your blog's ID number.

I decided that this would be best implemented as a rake task, so I could full access to all the rails application environment (especially the database connection). Here is the complete rake task:

namespace :blogger do
  desc "post weekend update to blogger"
  task :weekend_update  => :environment do
    require 'blogger'
    
    # preferences for blogger login
    username = 'sample@gmail.com' # change this to your username
    password = 'xxx129304xxxx' # change this to your password
    blog_id = '123456' # change this to your blogid

    # my entry preferences
    days_ahead = 5 # how many days of events ?
    blog_entry = "For those of you making your weekend plans, 
                  here is quick rundown of this weekend\'s MTB racing schedule 
                  for NorCal:\n\n"
    
    # google static map preferences 
    base_url = "http://maps.google.com/maps/api/staticmap?"
    map_params = ["size=400x400", "maptype=roadmap", "sensor=false"]

    # find all events in the mtbcalendar.com db for this weekend
    @events = Event.active.find_tagged_with(
        "norcal", 
        :conditions => ["start_date >= CURRENT_DATE AND start_date < ?", 
                        days_ahead.days.from_now], 
        :order => "start_date asc")

    # loop through events adding place markers to the google static map URL
    x = 0
    @events.each do |marker|
      x += 1
      map_params << "markers=color:blue|label:#{x}|#{marker.lat},#{marker.lng}"
      blog_entry += "#{x}. [#{marker.name}](http://mtbcalendar.com/events/#{marker.id})
                    (#{marker.start_date.to_s(:compact)})\n"
    end # marker loop

    # add the google static map to post body
    map_pic_url = base_url + map_params.join('&')
    blog_entry += "\n![NorCal MTB Race Event Map](" + map_pic_url + ")"

    # create title for the post
    blog_title = "NorCal MTB Racing This Weekend: 
                  #{Date.today.to_s(:short)} - 
                  #{days_ahead.days.from_now.to_date.to_s(:short)}, 
                  #{Date.today.year}"

    # login into blogger and post this draft
    account     = Blogger::Account.new(username,password)
    new_post    = Blogger::Post.new(:title      => blog_title, 
                                    :content    => blog_entry,
                                    :formatter  => :rdiscount,
                                    :draft      => true)
    account.post(blog_id,new_post)

  end # task    
end # namespace

Although I did this with a rake task, you could also make it just a regular ruby script. However, then you would need to handle all of the database login and queries yourself.

Custom iPhone Notifications with Prowl

2011-03-21T00:00:00-07:00

On of my favorite iPhone feature is push notifications. However, only a few applications use these well (ESPN ScoreCenter, AP mobile, Twitter, etc.) and I've always wished I could write my own without creating a full blown iPhone application. Enter a simple application called Prowl. Prowl is a Growl client for iOS that pushes notifications to your iPhone, iPod touch or iPad with a simple API.

To send yourself custom push notifications, you'll first need to buy Prowl for your iPhone (around USD$2.99). Once you've downloaded and installed it on your iPhone, you need to generate an API key. With that completed, you are all set to receive Prowl alerts.

Some desktop/server apps work with it out of the box (a semi-full list is at http://www.prowlapp.com/apps.php). Others like znc, flexget or Nagios require a plugin. To create you own custom alert, you'll need to write some scripting code. I prefer using Python or Ruby for this, but there are plenty of libraries available for other languages.

Prowl API

The Prowl API is very simple:

add (POST) which sends notifications to your iPhone
verify (GET) which checks the validity of your API key

Just recently the API was updated to version 1.2 which added two new methods - retrieve/token (GET) and retrieve/apikey (GET) - which appeal mostly to website operators. They facilitate giving people your API key in a secure manner.

While the API is very simple, and you could certainly use it from the command line with a tool like curl, most people will want to use a library for their favorite programming language. I use two of these libraries, Prowly for Ruby and ProwlPy for Python.

Prowly for Ruby

The Prowly gem was written by Rafael Magana to make the Prowl API a bit more Ruby-friendly. It is dead simple to use. First, install the gem:

$ sudo gem install prowly

Then create a simple script like the one below:

require 'prowly'

# your api key from https://www.prowlapp.com/api_settings.php
my_api_key = "1234567890123456789012345678901234567890"

Prowly.notify do |n|
    n.apikey = my_api_key 
    n.priority = Prowly::Notification::Priority::MODERATE
    n.application = "Prowly"
    n.event = "Notification"
    n.description = "Your server is under attack!!!"
    n.url = "http://www.myserver.com" # requires API 1.2 support
end

That is basically all there is to it. Of course, you'll want your script to actually do something (check a stock, parse a RSS feed, etc.), but sending the prowl is really that simple. Prowly also includes a command line script to send notifications without any scripting.

Right now, the current gem has not been updated to use version 1.2 of the API. I've put in a pull request to get my upgrades merged in but in the meantime you can clone the updated one from my gtihub repository at https://github.com/slashk/prowly.

ProwlPy for Python

A corresponding Python library is called ProwlPy and was written by Jacob Burch and Olivier Hervieu. It is very similar to the Prowly. To use it, install the python library (which doesn't appear to be on PyPi):

$ git clone https://github.com/jacobb/prowlpy.git
... snip ...
$ cd prowlpy
$ python setup.py install

Then just create a simple script like the ruby one above:

import prowlpy

# your api key from https://www.prowlapp.com/api_settings.php
my_api_key = '1234567890123456789012345678901234567890' # your api key
p = prowlpy.Prowl(my_api_key)
try:
    p.add('ProwlPy', 
          'Notification', 
          'Your server is under attack',
          1, 
          None, 
          "http://www.myserver.com")
except Exception,msg:
    print msg

I also upgraded this library to version 1.2 of the API and my changes have already been merged into the main repository at https://github.com/slashk/prowly.

Creating Your Own Prowl Notifiers

Once you've grokked the basics of using the Prowl API, it's time to actually put it to use. To do that, you'll need to create your own script (with whatever logic you need), have it execute at common intervals (usually via cron) and then just wait for prowl notifications to roll in.

To get you started, I've created a repository of scripts that I use over at https://github.com/slashk/prowl-scripts. There you'll see a few scripts to notify you of:

transmission bit torrent download completions
daily and monthly Google AdSense balances
woot! deals

I'll be adding more as I clean up my other scripts (weather, stock prices, steep and cheap, etc).

Configurable Instance Types For OpenStack Nova

2011-03-03T00:00:00-08:00

Over the past three weeks, new Nova core developer Josh Kearney (congrats @jk0) and I have been working on adding runtime configuration of instance types (read the full specification) to the OpenStack Nova compute service. Instance types (or "flavors" as Rackspace calls them) are resources granted to virtual machines ("instances") in the Nova cloud. In more specific terms, this is the size of the instance (vCPUs, RAM, Storage, etc.) that you will be launching. You may recognize these by the names "m1.large" or "m1.tiny" in AWS EC2 parlance. The Rackspace's API (PDF) calls these "flavors" and they tend to have names like "256 MB Server" and such.

In the released version of OpenStack Nova (Bexar), instance types are defined statically in the nova/instance_types.py file (which corresponds to the initial Amazon EC2 instance types) before run-time. This change, which will be part of the upcoming version of Nova (Cactus), instance types can be created by the Nova administrator during run-time through the nova-manage command. If you are following the Nova trunk (or just impatient), you can find this merged in at revision number 757.

Basic Management

Instance types / flavor are managed through the nova-manage binary with the instance_type command and an appropriate subcommand. As we are using the nova-manage command, we need to be on the Nova database server to configure instance types. Right now, instance type manipulation isn't exposed through the APIs nor the adminclient.

Note that you can also use the flavor command as a synonym for instance_types for any of these examples.

To see all currently active instance types, use the list subcommand:

$ nova-manage instance_type list
m1.medium: Memory: 4096MB, VCPUS: 2, Storage: 40GB, FlavorID: 3, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.large: Memory: 8192MB, VCPUS: 4, Storage: 80GB, FlavorID: 4, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.tiny: Memory: 512MB, VCPUS: 1, Storage: 0GB, FlavorID: 1, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.xlarge: Memory: 16384MB, VCPUS: 8, Storage: 160GB, FlavorID: 5, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.small: Memory: 2048MB, VCPUS: 1, Storage: 20GB, FlavorID: 2, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB

Again, and just for emphasis, you could just have easily used the flavor subcommand to get the exact same output:

$ nova-manage flavor list
m1.medium: Memory: 4096MB, VCPUS: 2, Storage: 40GB, FlavorID: 3, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.large: Memory: 8192MB, VCPUS: 4, Storage: 80GB, FlavorID: 4, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.tiny: Memory: 512MB, VCPUS: 1, Storage: 0GB, FlavorID: 1, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.xlarge: Memory: 16384MB, VCPUS: 8, Storage: 160GB, FlavorID: 5, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.small: Memory: 2048MB, VCPUS: 1, Storage: 20GB, FlavorID: 2, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB

To create an instance type, use the create subcommand with the following positional arguments:

memory (expressed in megabytes)
vcpu(s) (integer)
local storage (expressed in gigabytes)
flavorid (unique integer)
swap space (expressed in megabytes, defaults to zero, optional)
RXTX quotas (expressed in gigabytes, defaults to zero, optional)
RXTX cap (expressed in gigabytes, defaults to zero, optional)

The following example creates an instance type named "m1.xxlarge":

$ nova-manage instance_type create m1.xxlarge 32768 16 320 0 0 0
m1.xxlarge created

To delete an instance type, use the delete subcommand and specify the name:

$ nova-manage instance_type delete m1.xxlarge
m1.xxlarge deleted

Please note that the delete command only marks the instance type as inactive in the database; it does not actually remove the instance type. This is done to preserve the instance type definition for long running instances (which may not terminate for months or years). If you are sure that you want to delete this instance type from the database, pass the --purge flag after the name:

$ nova-manage instance_type delete m1.xxlarge --purge
m1.xxlarge purged

Be careful with deleting instance types, you might need this information later. Unless you truly need to prune the size of your instance_types table you are much safer to just delete the instance type.

Implementation

This feature is implemented with an additional database table (instance_types), which has rows for each instance type/flavor. The nova-manage command allows you to manipulate this table. At installation time (actually when the nova-manage db sync command is run), the database table is seeded with instance type values for the five default types: m1.small, m1.tiny, m1.medium, m1.large and m1.xlarge.

OpenStack Development with virtualenvwrapper

2011-01-24T00:00:00-08:00

Coming from the ruby/ruby on rails world, i’ve been a bit lost when it comes to the python development process used in the openstack project. One of the biggest hurdles has been the usage of virtualenv in the workflow. Basically, virtualenv lets you create a stable configuration of python libraries (eggs) much like freezing gems in your rails application. The pitfalls here is that you need to integrate it’s usage into your development flow (activate/deactivate environments), it can take some time to recreate environments if you use a lot of eggs (like nova does) and it seems pretty fragile (it lives in repo and takes some chicanery to avoid duplicating in each bzr branch).

However, I recently found virtualenvwrapper which has erased much of this heartache:

virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

Basically, it centralizes the location of your virtual environments and gives you easy commands to manipulate them: mkvirtaulenv creates one, rmvirtaulenv deletes one and workon activates one. It also allows you to create per environment, user customizable hooks that automatically fire on all the operations. I have created a simple one that automagically changes my directory to my nova trunk directory when I activate my nova virtual environment.

With all that said, here is how I hack on OpenStack Nova on Ubuntu 10.10 (illustrated with nova bzr revno 604). First, install the pre-reqs for getting the code and your basic python environment:

$ sudo apt-get install python-dev swig libssl-dev python-pip bzr

This loads a ton of dependencies packages. Next, pull in our virtualenv eggs:

$ sudo pip install virtualenv 
$ sudo pip install virtualenvwrapper

Now, setup virtualenvwrapper for your account:

$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bashrc 
$ source /usr/local/bin/virtualenvwrapper.sh 
$ export WORKON_HOME=~/.virtualenv

With that done, we can start pulling down the source code. I’ve put my nova code branches in ~/src/nova/ :

$ mkdir -p src 
$ cd src 
$ bzr init-repo nova 
$ cd nova/ 
$ bzr branch lp:nova trunk 
$ cd trunk/

With that in place, let’s create our virtual environment and pull in our python requirements. This is where virtualenvwrapper shines - just one command and we have a portable virtual environment and another to fill it up for all of our branches:

$ mkvirtualenv nova 
New python executable in nova/bin/python Installing setuptools............done. virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/predeactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/postdeactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/preactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/postactivate virtualenvwrapper.user_scripts creating /home/nova/.virtualenv/nova/bin/get_env_details
(nova)nova@openstack:~/src/nova/trunk$ 
(nova)nova@openstack:~/src/nova/trunk$ pip install -r tools/pip-requires

Now that we got everything good to go, let’s run our tests to make sure everything is good (use -N option, otherwise the script wants to wrap everything with plain virtualenv commands):

(nova)nova@openstack:~/src/nova/trunk$ ./run_tests.sh -N 
AdminAPITest 
test_admin_disabled ok 
test_admin_enabled ok 
APITest 
test_exceptions_are_converted_to_faults ok
…. SNIP …

Perfect. When you are done, just deactivate:

(nova)nova@openstack:~/src/nova/trunk$ deactivate

While that makes everything nice and neat, the big win is when you decide to move your repo or create a new branch. To branch, just create the directory (which can be anywhere now), branch your code from trunk and activate your virtual environment (suing the workon command):

nova@openstack:~/src$ cd patch 
nova@openstack:~/src/patch$ bzr branch ../nova/trunk/ lp708221 
Branched 604 revision(s). 
nova@openstack:~/src/patch$ cd lp708221/ 
nova@openstack:~/src/patch/lp708221$ workon nova 
(nova)nova@openstack:~/src/patch/lp708221$ ./run_tests.sh -N 
AdminAPITest 
test_admin_disabled ok 
test_admin_enabled ok
.... snip ...

If you need to recreate your virtualenv (say after a big change in tools/pip-requires), you can just remove it and then recreate it

$ rmvirtualenv nova 
$ mkvirtualenv nova2 
(nova2)nova@openstack:~/src$ cd nova/trunk 
(nova2)nova@openstack:~/src/nova/trunk$ pip install -r tools/pip-requires

All in all, virtualenvwrapper has made my hacking flow much better. Once I get a chance, I’ll document this same setup in Fedora and see if we can’t change some of the nova support scripts to better utilize virtualenvwrapper.

Speaking at CommunityOne 2009

2009-05-30T00:00:00-07:00

Over the past few weeks, Scott Mattoon, John Stanford and I have been documenting infrastructure patterns to help application developers architect for cloud computing environments as part of a larger program around enabling cloud computing for our customers. It's been an interesting project, as cloud computing is both very new, but based on some very old concepts.

On Monday, we will be presenting our work in progress at CommunityOne West in San Francisco's Moscone Center. Around the same time, we hope to move our work into the public view on wikis.sun.com for comments and contributions. I'll post something here as soon as it happens.

Our talk will highlight three sets of pattern domains: provisioning, monitoring and resource administration. You can expect to hear talk of motivations and implementations, with sequence diagrams, architecture pictures and code snippets for illustration.

If you are attending the conference (you can still sign up for free here), our presentations will be:

June 1, 2009 from 2:40 to 3:30pm PDT
Practical Cloud Computing Patterns
Gateway 104, Moscone Center, San Francisco, CA

If you can't be in San Francisco for the conference, I believe you can watch our presentation on a live webcast: Check here for details. We are "2:40 - 3:30 pm | Practical Cloud Computing Patterns (Gateway 104)" under the Cloud Platforms topic.

Open Group Beijing Kickoff

2009-05-26T00:00:00-07:00

I had a great opportunity to revisit China, part of my old region (and now part of my new global responsibility), as part of The Open Group's kickoff last week:

The Open Group, which recently established a franchise in China with Kingdee, is a vendor-neutral and technology-neutral consortium, which drives the creation of Boundaryless Information Flow™ that will enable access to integrated information within and between enterprises based on open standards and global interoperability.

That's Allen Brown, President and CEO of the Open Group, giving the opening talk.

I was there to speak about cloud computing, its challenges and opportunities to enterprise architects. Kingdee and TOG put on a great conference, which was well attended. Maybe I'll go back in October when they have their practitioners conference in Hong Kong.

Learn more about the The Open Group Conference - Beijing at their website.