Difference between pages "Zope HOWTO" and "OpenStack Architecture"

(Difference between pages)
 
 
Line 1: Line 1:
This page documents how to use Zope with Funtoo Experimental, which currently has good Zope support thanks to [[Progress Overlay Python]] integration.
+
This page exists to document [http://www.openstack.org OpenStack] configuration.
  
== About Zope ==
+
Note that the current approach is to use devstack, which is not a good way to learn OpenStack. So much of this document will be about doing a devstack-like configuration for Funtoo.
  
Zope is an Open Source application server framework written in Python. It has an interesting history which you should familiarize yourself with before starting Zope development, as it contains several interesting twists and turns.
+
This document will split OpenStack configuration into each architectural component, describing configuration steps for each component separately.
  
=== Zope History ===
+
== SQL Database ==
  
{{fancynote| This HOWTO targets Zope 2.13, which includes Five. It is typically the version you should be using for new Zope projects.}}
+
A number of OpenStack services use a SQL back-end for storing various bits of data.
  
* There are two versions of Zope: Zope 2 and Zope 3. One might assume that Zope 3 is the version that people should use for new software development projects by default, but this is not the case. Most Zope-based projects continue to use Zope 2. Zope 3 was an attempt to redesign Zope 2 from scratch, and is completely different from Zope 2, but it was not adopted by the community.
+
While DevStack uses MySQL for its SQL deployment, multiple database back-ends are actually supported thanks to [http://sqlalchemy.org SQLAlchemy] being used behind the scenes, which is a re-targetable Python database API. Thus, it should be possible to use Postgres, etc, by simply using different connection strings. A list of SQLAlchemy connection types can be found on [http://docs.sqlalchemy.org/en/latest/core/engines.html this SQLAlchemy documentation page.]
  
* There is also something called [http://codespeak.net/z3/five/ Five] (named because it is "2 + 3") that backports many of the new features of Zope 3 into the Zope 2 framework. Several projects will use Zope 2 plus Five in order to use some of the newer features in Zope. Five was merged into mainline Zope 2 in early 2010, and first appeared in Zope 2.8.
+
Using a single root database user account for all services is not a good policy for production deployment. Ideally, each service should have its own restricted user account with only the ability to access its own database.
  
* You can learn more about the history of Zope 2, 3 and Five in the [http://svn.zope.org/Zope/trunk/src/Products/Five/README.txt?view=markup Five README].
+
Let's look at how each service is configured in regards to SQL:
  
* To make things even more interesting, work on [http://docs.zope.org/zope2/releases/4.0/ Zope 4] is underway, and it will be based on 2.13 rather than 3.x. It includes a number of [http://docs.zope.org/zope2/releases/4.0/CHANGES.html#restructuring incompatible changes] with prior versions.
+
=== nova ===  
=== Zope Resources ===
+
  
Now that you understand what version of Zope you should be targeting (2.13), we can point you towards the correct documentation :)
+
Here's how to set up a MySQL database back-end for nova and tell nova to initialize its database tables:
  
; '''[http://docs.zope.org/zope2/zope2book/ The Zope 2 Book]'''
 
: This book provides a general introduction to Zope concepts and ZMI. It is a good place to start, but doesn't provide a direct introduction to Zope development. It's recommended that you skim through this book to familiarize yourself with Zope. It generally does not assume much prior knowledge about Web development or Python.
 
; '''[http://docs.zope.org/zope2/zdgbook/ Zope Developer's Guide]'''
 
: This guide will give you a better introduction to Zope development. It assumes you already know Python. Skip chapters 1 and 2 and start in [http://docs.zope.org/zope2/zdgbook/ComponentsAndInterfaces.html chapter 3], which covers components and interfaces. [http://docs.zope.org/zope2/zdgbook/Products.html Chapter 5] covers the creation of your first product.
 
; '''[http://codespeak.net/z3/five/manual.html The Five Manual]'''
 
: We're not done yet. There is a bunch of stuff in Zope 2.13 that is not in the official documentation. Namely, the stuff in Five.
 
; '''[http://docs.zope.org/ztkpackages.html ZTK Documentation]'''
 
: ZTK 
 
; '''ZCA'''
 
: [http://www.muthukadan.net/docs/zca.html A Comprehensive Guide to Zope Component Architecture] offers a good introduction to the programming concepts of ZCA. We also have a new page on [[Zope Component Architecture]] which will help you to understand the big picture of ZCA and why it is useful. ZCML ("Z-camel") is a part of ZCA and  was introduced in Zope 3, so typically you will find ZCML documented within Zope 3 documentation and book.
 
; '''Content Components'''
 
: Views and Viewlets: [http://docs.zope.org/zope.viewlet/index.html This tutorial on viewlets] also contains some viewlet-related ZCML examples near the end. The "Content Component way" of developing in Zope seems to be a Zope 3 thing and tied to ZCML. Chapter 13+ of Stephan Richter's ''Zope 3 Developer's Handbook'' (book) seems to cover this quite well. You will probably also want to check out Philipp Weitershausen's ''Web Component Development with Zope 3'' (book).
 
; '''[http://wiki.zope.org/zope2/Zope2Wiki Zope 2 Wiki]'''
 
: Main wiki page for all things related to Zope 2.
 
; '''[http://docs.zope.org docs.zope.org]'''
 
: This is the main site for Zope documentation.
 
 
== First Steps ==
 
 
First, you will need to emerge {{Package|net-zope/zope}}:
 
 
<console>
 
<console>
###i## emerge zope
+
mysql> ##i##create database nova character set latin1;
 +
Query OK, 1 row affected (0.02 sec)
 +
 
 +
mysql> ##i##grant all privileges on nova.* to nova@localhost identified by 'foobar';
 +
Query OK, 0 rows affected (0.00 sec)
 
</console>
 
</console>
  
Zope is now installed.
+
Now set the following connection string in <tt>/etc/nova/nova.conf</tt>:
  
== Project Skeleton ==
+
<pre>
 +
--sql_connection=mysql://nova:foobar@localhost/nova
 +
</pre>
  
{{fancynote| Zope should be run by a regular user account, not as the root user.}}
+
Note the use of the latin1 character set when we created the tables in MySQL. This is so the following command will not cause an error due to the default UTF-8 character set creating indexes that are too big for MySQL to handle:
  
The first step in using Zope is to ensure that you are using a regular user account. As a regular user, create a new directory called <tt>zope_test</tt>:
 
 
<console>
 
<console>
$##i## cd
+
# ##i##nova-manage db sync
$##i## mkdir zope_test
+
2012-03-02 21:31:14 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.migration' from '/usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/migration.pyc'> from (pid=17779) __get_b
 +
ackend /usr/lib64/python2.7/site-packages/nova/utils.py:602
 
</console>
 
</console>
  
Now, enter the directory, and create an "instance", which is a set of files and directories that are used to contain a Zope project:
+
After running the command above, you should now have all the relevant database tables created:
<console>
+
$##i## cd zope_test
+
$##i## /usr/lib/zope-2.13/bin/mkzopeinstance
+
</console>
+
  
You will see the following output and will be prompted to answer a few questions:
 
 
<console>
 
<console>
Please choose a directory in which you'd like to install
+
xdev var # ##i##mysql -u root -p nova
Zope "instance home" files such as database files, configuration
+
Enter password:
files, etc.
+
Reading table information for completion of table and column names
 +
You can turn off this feature to get a quicker startup with -A
  
Directory: instance
+
Welcome to the MySQL monitor.  Commands end with ; or \g.
Please choose a username and password for the initial user.
+
Your MySQL connection id is 16
These will be the credentials you use to initially manage
+
Server version: 5.1.61-log Gentoo Linux mysql-5.1.61
your new Zope instance.
+
  
Username: admin
+
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Password: ****
+
Verify password: ****
+
</console>
+
  
Now, we will start our Zope instance:
+
Oracle is a registered trademark of Oracle Corporation and/or its
<console>
+
affiliates. Other names may be trademarks of their respective
$##i## cd instance
+
owners.
$##i## bin/runzope
+
 
 +
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 +
 
 +
mysql> ##i##show tables;
 +
+-------------------------------------+
 +
| Tables_in_nova                      |
 +
+-------------------------------------+
 +
| agent_builds                        |
 +
| aggregate_hosts                    |
 +
| aggregate_metadata                  |
 +
| aggregates                          |
 +
| auth_tokens                        |
 +
| block_device_mapping                |
 +
| bw_usage_cache                      |
 +
| certificates                        |
 +
| compute_nodes                      |
 +
| console_pools                      |
 +
...
 
</console>
 
</console>
  
Now that Zope is functional, you can go to the <tt>localhost:8080/manage</tt> URL in your web browser: you will be prompted to log in. Enter the username and password you specified. You are now logged in to the ZMI (Zope Management Interface.)
+
You have now validated that nova is connecting to your MySQL database correctly.
  
You can stop your application by pressing Control-C. In the future, you can start and stop your Zope instance using the following commands:
+
=== glance ===
  
<console>
+
From glance.openstack.org:
$##i## zopectl start
+
$##i## zopectl stop
+
</console>
+
  
{{fancynote| <tt>zopectl start</tt> will cause your instance to run in the background rather than consuming a shell console.}}
+
<blockquote>The Glance project provides services for discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image.</blockquote>
  
== First Project ==
+
Glance typically uses a MySQL database called <tt>glance</tt>, although the name is configurable in the connection string.
  
We will create a single, very primitive Zope package, consisting of an Interface for a TODO class, and a TODO class.
+
SQL connection settings might be stored in a glance configuration file located at <tt>/opt/stack/glance/etc/glance-registry.conf</tt>. In the devstack installation process, <tt>/opt/stack/glance</tt> contains a git checkout of the glance software.
  
Create the following files and directories relative to your project root:
+
The SQL connection configuration string might look something like this:
  
* Create the directory <tt>lib/python/example</tt>.
+
<pre>
* Create the file <tt>lib/python/example/__init__.py</tt> by typing <tt>touch lib/python/example/__init__.py</tt>.
+
sql_connection = mysql://glance:yourpassword@192.168.206.130/glance
* Create these files:
+
</pre>
  
=== <tt>example-configure.zcml</tt> ===
+
More info on glance configuration is available [http://docs.openstack.org/diablo/openstack-compute/install/content/glance-registry-conf-file.html here].
  
This file registers the <tt>example</tt> directory you created in <tt>lib/python</tt> as a ''package'', so that it is seen by Zope. Edit <code>/etc/package-includes/example-configure.zcml</code>:
+
=== keystone ===
 +
 
 +
Keystone, the OpenStack identity service, also uses SQL. <tt>etc/keystone.conf</tt> keystone install/git repo directory is used to store the SQL configuration:
  
 
<pre>
 
<pre>
<include package="example" />
+
sql_connection = %SQL_CONN%
 
</pre>
 
</pre>
  
=== <tt>interfaces.py</tt> ===
+
As everything else, the SQL connection string uses SQLAlchemy syntax.
  
The following file defines the <tt>ITODO</tt> interface, and also uses some Zope Schema functions to define what kind of data we expect to store in objects that implement <tt>ITODO</tt>. Edit <code>/lib/python/example/interfaces.py</code> with your favorite text editor:
+
=== Quantum and Open VSwitch ===
  
<syntaxhighlight lang="python">
+
<blockquote>Quantum is an incubated OpenStack project to provide "network connectivity as a service" between interface devices (e.g., vNICs) managed by other Openstack services (e.g., nova).</blockquote>
from zope.interface import Interface
+
from zope.schema import List, Text, TextLine, Int
+
  
class ITODO(Interface):
+
[http://openvswitch.org Open VSwitch] is described as:
    name = TextLine(title=u'Name', required=True)
+
    todo = List(title=u"TODO Items", required=True, value_type=TextLine(title=u'TODO'))
+
    daysleft = Int(title=u'Days left to complete', required=True)
+
    description = Text(title=u'Description', required=True)
+
</syntaxhighlight>
+
  
=== <tt>TODO.py</tt> ===
+
<blockquote>Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch or Cisco's Nexus 1000V. </blockquote>
  
Now, we define <tt>TODO</tt> to be a ''persistent'' object, meaning it can be stored in the ZODB. We specify that it implements our previously-defined <tt>ITODO</tt> interface, and provide reasonable defaults for all values when we create a new TODO object. Edit <code>/lib/python/example/TODO.py<code> using your favorite text editor:
+
There is an [http://openvswitch.org/openstack/documentation/ Open VSwitch Plug-in for OpenStack Quantum] which can be set up by DevStack. This plug-in uses SQL storage. The SQLAlchemy connection string is stored in (relative to git/install root) <tt>etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini</tt>. Similarly to everything but nova, the SQL connection string is stored in <tt>sql_connection =</tt> format.
<syntaxhighlight lang="python">
+
from persistent import Persistent
+
from zope.interface import implements
+
from example.interfaces import ITODO
+
  
class TODO(Persistent):
+
If the plugin is enabled, the following settings are added to nova.conf:
    implements(ITODO)
+
    name = u''
+
    todo = []
+
    daysleft = 0
+
    description = u''
+
</syntaxhighlight>
+
  
=== <tt>configure.zcml</tt> ===
+
<pre>
 +
--libvirt_vif_type=ethernet
 +
--libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
 +
--linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
 +
--quantum_use_dhcp
 +
</pre>
  
Create the <tt>/lib/python/example/configure.zcml</tt> configuration file:
+
=== Melange ===
<syntaxhighlight lang="xml">
+
<configure xmlns="http://namespaces.zope.org/zope"
+
    xmlns:five="http://namespaces.zope.org/five"
+
    xmlns:browser="http://namespaces.zope.org/browser">
+
</configure>
+
</syntaxhighlight>
+
  
== Debug Mode ==
+
From the Melange site:
  
We can test our first project by entering debug mode:
+
<blockquote>
<console>
+
Melange is intended to provide network information services for use across OpenStack services. The initial focus will be on IP address management (IPAM) and address discovery (DHCP/dnsmasq) functionality. Melange is intended to be a standalone service with it's own API. However, the initial use case will be to decouple existing IP management and VM address discovery from Nova and support the existing Nova networking capabilities.
$##i## bin/zopectl debug
+
</blockquote>
Starting debugger (the name "app" is bound to the top-level Zope object)
+
 
</console>
+
Melange also uses a <tt>sql_connection =</tt> string stored in its <tt>etc/melange/melange.conf</tt> (relative to install/git root).
 +
 
 +
== RabbitMQ ==
 +
 
 +
RabbitMQ  is a reliable messaging framework used by OpenStack. Currently, it looks like only nova uses it. Nova is configured to connect to rabbitmq by setting the following lines in <tt>/etc/nova/nova.conf</tt>:
 +
 
 +
<pre>
 +
--rabbit_host=$RABBIT_HOST
 +
--rabbit_password=$RABBIT_PASSWORD
 +
</pre>
 +
 
 +
Rabbit's password is configured using the following command, as root:
  
Now, let's try creating a new TODO object and writing it out to a ZODB database:
 
 
<console>
 
<console>
>>> from ZODB import FileStorage, DB
+
# ##i##rabbitmqctl change_password guest $RABBIT_PASSWORD
>>> storage = FileStorage.FileStorage('mydatabase.fs')
+
>>> db = DB(storage)
+
>>> connection = db.open()
+
>>> import transaction
+
>>> root = connection.root()
+
>>> from example.TODO import TODO
+
>>> a = TODO
+
>>> a.name = u'My TODOs'
+
>>> a.TODOS = [ u'Do Laundry', u'Wash Dishes' ]
+
>>> a.daysleft = 1
+
>>> a.description = u'Things I need to do today.'
+
>>> root[u'today'] = a
+
>>> transaction.commit()
+
 
</console>
 
</console>
  
[[Category:HOWTO]]
+
I am not yet completely sure how RabbitMQ fits into the OpenStack architecture. It may be that the supporting services expect it to be running locally, and that Nova compute nodes need to hook into a Nova instance, which would typically be running remotely. (Thus the ability for DevStack to target a remote RabbitMQ host.)
 +
 
 +
== Virtualization Technology ==
 +
 
 +
DevStack defaults to configuring OpenStack to use libvirt with [[KVM]], and will fall back to basic [[QEMU]] support if the <tt>kvm</tt> kernel module is not available. It also has support for using libvirt with [[LXC]], in addition to using [[Xen]] Server directly (bypassing libvirt.)
 +
 
 +
 
 +
[[Category:Virtualization]]
 +
[[Category:OpenStack]]

Latest revision as of 09:18, December 28, 2014

This page exists to document OpenStack configuration.

Note that the current approach is to use devstack, which is not a good way to learn OpenStack. So much of this document will be about doing a devstack-like configuration for Funtoo.

This document will split OpenStack configuration into each architectural component, describing configuration steps for each component separately.

SQL Database

A number of OpenStack services use a SQL back-end for storing various bits of data.

While DevStack uses MySQL for its SQL deployment, multiple database back-ends are actually supported thanks to SQLAlchemy being used behind the scenes, which is a re-targetable Python database API. Thus, it should be possible to use Postgres, etc, by simply using different connection strings. A list of SQLAlchemy connection types can be found on this SQLAlchemy documentation page.

Using a single root database user account for all services is not a good policy for production deployment. Ideally, each service should have its own restricted user account with only the ability to access its own database.

Let's look at how each service is configured in regards to SQL:

nova

Here's how to set up a MySQL database back-end for nova and tell nova to initialize its database tables:

mysql> create database nova character set latin1;
Query OK, 1 row affected (0.02 sec)

mysql> grant all privileges on nova.* to nova@localhost identified by 'foobar';
Query OK, 0 rows affected (0.00 sec)

Now set the following connection string in /etc/nova/nova.conf:

--sql_connection=mysql://nova:foobar@localhost/nova

Note the use of the latin1 character set when we created the tables in MySQL. This is so the following command will not cause an error due to the default UTF-8 character set creating indexes that are too big for MySQL to handle:

# nova-manage db sync
2012-03-02 21:31:14 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.migration' from '/usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/migration.pyc'> from (pid=17779) __get_b
ackend /usr/lib64/python2.7/site-packages/nova/utils.py:602

After running the command above, you should now have all the relevant database tables created:

xdev var # mysql -u root -p nova
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 5.1.61-log Gentoo Linux mysql-5.1.61

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables;
+-------------------------------------+
| Tables_in_nova                      |
+-------------------------------------+
| agent_builds                        |
| aggregate_hosts                     |
| aggregate_metadata                  |
| aggregates                          |
| auth_tokens                         |
| block_device_mapping                |
| bw_usage_cache                      |
| certificates                        |
| compute_nodes                       |
| console_pools                       |
...

You have now validated that nova is connecting to your MySQL database correctly.

glance

From glance.openstack.org:

The Glance project provides services for discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image.

Glance typically uses a MySQL database called glance, although the name is configurable in the connection string.

SQL connection settings might be stored in a glance configuration file located at /opt/stack/glance/etc/glance-registry.conf. In the devstack installation process, /opt/stack/glance contains a git checkout of the glance software.

The SQL connection configuration string might look something like this:

sql_connection = mysql://glance:yourpassword@192.168.206.130/glance

More info on glance configuration is available here.

keystone

Keystone, the OpenStack identity service, also uses SQL. etc/keystone.conf keystone install/git repo directory is used to store the SQL configuration:

sql_connection = %SQL_CONN%

As everything else, the SQL connection string uses SQLAlchemy syntax.

Quantum and Open VSwitch

Quantum is an incubated OpenStack project to provide "network connectivity as a service" between interface devices (e.g., vNICs) managed by other Openstack services (e.g., nova).

Open VSwitch is described as:

Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch or Cisco's Nexus 1000V.

There is an Open VSwitch Plug-in for OpenStack Quantum which can be set up by DevStack. This plug-in uses SQL storage. The SQLAlchemy connection string is stored in (relative to git/install root) etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini. Similarly to everything but nova, the SQL connection string is stored in sql_connection = format.

If the plugin is enabled, the following settings are added to nova.conf:

 --libvirt_vif_type=ethernet
 --libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
 --linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
 --quantum_use_dhcp

Melange

From the Melange site:

Melange is intended to provide network information services for use across OpenStack services. The initial focus will be on IP address management (IPAM) and address discovery (DHCP/dnsmasq) functionality. Melange is intended to be a standalone service with it's own API. However, the initial use case will be to decouple existing IP management and VM address discovery from Nova and support the existing Nova networking capabilities.

Melange also uses a sql_connection = string stored in its etc/melange/melange.conf (relative to install/git root).

RabbitMQ

RabbitMQ is a reliable messaging framework used by OpenStack. Currently, it looks like only nova uses it. Nova is configured to connect to rabbitmq by setting the following lines in /etc/nova/nova.conf:

--rabbit_host=$RABBIT_HOST
--rabbit_password=$RABBIT_PASSWORD

Rabbit's password is configured using the following command, as root:

# rabbitmqctl change_password guest $RABBIT_PASSWORD

I am not yet completely sure how RabbitMQ fits into the OpenStack architecture. It may be that the supporting services expect it to be running locally, and that Nova compute nodes need to hook into a Nova instance, which would typically be running remotely. (Thus the ability for DevStack to target a remote RabbitMQ host.)

Virtualization Technology

DevStack defaults to configuring OpenStack to use libvirt with KVM, and will fall back to basic QEMU support if the kvm kernel module is not available. It also has support for using libvirt with LXC, in addition to using Xen Server directly (bypassing libvirt.)