Difference between pages "Zope HOWTO" and "OpenStack Architecture"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
 
Line 1: Line 1:
This page documents how to use Zope with Funtoo Experimental, which currently has good Zope support thanks to [[Progress Overlay Python]] integration.
This page exists to document [http://www.openstack.org OpenStack] configuration.


== About Zope ==
Note that the current approach is to use devstack, which is not a good way to learn OpenStack. So much of this document will be about doing a devstack-like configuration for Funtoo.


Zope is an Open Source application server framework written in Python. It has an interesting history which you should familiarize yourself with before starting Zope development, as it contains several interesting twists and turns.
This document will split OpenStack configuration into each architectural component, describing configuration steps for each component separately.


=== Zope History ===
== SQL Database ==


{{fancynote| This HOWTO targets Zope 2.13, which includes Five. It is typically the version you should be using for new Zope projects.}}
A number of OpenStack services use a SQL back-end for storing various bits of data.


* There are two versions of Zope: Zope 2 and Zope 3. One might assume that Zope 3 is the version that people should use for new software development projects by default, but this is not the case. Most Zope-based projects continue to use Zope 2. Zope 3 was an attempt to redesign Zope 2 from scratch, and is completely different from Zope 2, but it was not adopted by the community.
While DevStack uses MySQL for its SQL deployment, multiple database back-ends are actually supported thanks to [http://sqlalchemy.org SQLAlchemy] being used behind the scenes, which is a re-targetable Python database API. Thus, it should be possible to use Postgres, etc, by simply using different connection strings. A list of SQLAlchemy connection types can be found on [http://docs.sqlalchemy.org/en/latest/core/engines.html this SQLAlchemy documentation page.]


* There is also something called [http://codespeak.net/z3/five/ Five] (named because it is "2 + 3") that backports many of the new features of Zope 3 into the Zope 2 framework. Several projects will use Zope 2 plus Five in order to use some of the newer features in Zope. Five was merged into mainline Zope 2 in early 2010, and first appeared in Zope 2.8.
Using a single root database user account for all services is not a good policy for production deployment. Ideally, each service should have its own restricted user account with only the ability to access its own database.


* You can learn more about the history of Zope 2, 3 and Five in the [http://svn.zope.org/Zope/trunk/src/Products/Five/README.txt?view=markup Five README].
Let's look at how each service is configured in regards to SQL:


* To make things even more interesting, work on [http://docs.zope.org/zope2/releases/4.0/ Zope 4] is underway, and it will be based on 2.13 rather than 3.x. It includes a number of [http://docs.zope.org/zope2/releases/4.0/CHANGES.html#restructuring incompatible changes] with prior versions.
=== nova ===  
=== Zope Resources ===


Now that you understand what version of Zope you should be targeting (2.13), we can point you towards the correct documentation :)
Here's how to set up a MySQL database back-end for nova and tell nova to initialize its database tables:


; '''[http://docs.zope.org/zope2/zope2book/ The Zope 2 Book]'''
<console>
: This book provides a general introduction to Zope concepts and ZMI. It is a good place to start, but doesn't provide a direct introduction to Zope development. It's recommended that you skim through this book to familiarize yourself with Zope. It generally does not assume much prior knowledge about Web development or Python.
mysql> ##i##create database nova character set latin1;
; '''[http://docs.zope.org/zope2/zdgbook/ Zope Developer's Guide]'''
Query OK, 1 row affected (0.02 sec)
: This guide will give you a better introduction to Zope development. It assumes you already know Python. Skip chapters 1 and 2 and start in [http://docs.zope.org/zope2/zdgbook/ComponentsAndInterfaces.html chapter 3], which covers components and interfaces. [http://docs.zope.org/zope2/zdgbook/Products.html Chapter 5] covers the creation of your first product.
; '''[http://codespeak.net/z3/five/manual.html The Five Manual]'''
: We're not done yet. There is a bunch of stuff in Zope 2.13 that is not in the official documentation. Namely, the stuff in Five.
; '''[http://docs.zope.org/ztkpackages.html ZTK Documentation]'''
: ZTK 
; '''ZCA'''
: [http://www.muthukadan.net/docs/zca.html A Comprehensive Guide to Zope Component Architecture] offers a good introduction to the programming concepts of ZCA. We also have a new page on [[Zope Component Architecture]] which will help you to understand the big picture of ZCA and why it is useful. ZCML ("Z-camel") is a part of ZCA and  was introduced in Zope 3, so typically you will find ZCML documented within Zope 3 documentation and book.
; '''Content Components'''
: Views and Viewlets: [http://docs.zope.org/zope.viewlet/index.html This tutorial on viewlets] also contains some viewlet-related ZCML examples near the end. The "Content Component way" of developing in Zope seems to be a Zope 3 thing and tied to ZCML. Chapter 13+ of Stephan Richter's ''Zope 3 Developer's Handbook'' (book) seems to cover this quite well. You will probably also want to check out Philipp Weitershausen's ''Web Component Development with Zope 3'' (book).
; '''[http://wiki.zope.org/zope2/Zope2Wiki Zope 2 Wiki]'''
: Main wiki page for all things related to Zope 2.
; '''[http://docs.zope.org docs.zope.org]'''
: This is the main site for Zope documentation.
 
== First Steps ==


First, you will need to emerge {{Package|net-zope/zope}}:
mysql> ##i##grant all privileges on nova.* to nova@localhost identified by 'foobar';
<console>
Query OK, 0 rows affected (0.00 sec)
###i## emerge zope
</console>
</console>


Zope is now installed.
Now set the following connection string in <tt>/etc/nova/nova.conf</tt>:


== Project Skeleton ==
<pre>
--sql_connection=mysql://nova:foobar@localhost/nova
</pre>


{{fancynote| Zope should be run by a regular user account, not as the root user.}}
Note the use of the latin1 character set when we created the tables in MySQL. This is so the following command will not cause an error due to the default UTF-8 character set creating indexes that are too big for MySQL to handle:


The first step in using Zope is to ensure that you are using a regular user account. As a regular user, create a new directory called <tt>zope_test</tt>:
<console>
<console>
$##i## cd
# ##i##nova-manage db sync
$##i## mkdir zope_test
2012-03-02 21:31:14 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.migration' from '/usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/migration.pyc'> from (pid=17779) __get_b
ackend /usr/lib64/python2.7/site-packages/nova/utils.py:602
</console>
</console>


Now, enter the directory, and create an "instance", which is a set of files and directories that are used to contain a Zope project:
After running the command above, you should now have all the relevant database tables created:
 
<console>
<console>
$##i## cd zope_test
xdev var # ##i##mysql -u root -p nova
$##i## /usr/lib/zope-2.13/bin/mkzopeinstance
Enter password:
</console>
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 5.1.61-log Gentoo Linux mysql-5.1.61
 
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.


You will see the following output and will be prompted to answer a few questions:
Oracle is a registered trademark of Oracle Corporation and/or its
<console>
affiliates. Other names may be trademarks of their respective
Please choose a directory in which you'd like to install
owners.
Zope "instance home" files such as database files, configuration
files, etc.


Directory: instance
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
Please choose a username and password for the initial user.
These will be the credentials you use to initially manage
your new Zope instance.


Username: admin
mysql> ##i##show tables;
Password: ****
+-------------------------------------+
Verify password: ****
| Tables_in_nova                      |
+-------------------------------------+
| agent_builds                        |
| aggregate_hosts                    |
| aggregate_metadata                  |
| aggregates                          |
| auth_tokens                        |
| block_device_mapping                |
| bw_usage_cache                      |
| certificates                        |
| compute_nodes                      |
| console_pools                      |
...
</console>
</console>


Now, we will start our Zope instance:
You have now validated that nova is connecting to your MySQL database correctly.
<console>
$##i## cd instance
$##i## bin/runzope
</console>


Now that Zope is functional, you can go to the <tt>localhost:8080/manage</tt> URL in your web browser: you will be prompted to log in. Enter the username and password you specified. You are now logged in to the ZMI (Zope Management Interface.)
=== glance ===


You can stop your application by pressing Control-C. In the future, you can start and stop your Zope instance using the following commands:
From glance.openstack.org:


<console>
<blockquote>The Glance project provides services for discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image.</blockquote>
$##i## zopectl start
$##i## zopectl stop
</console>


{{fancynote| <tt>zopectl start</tt> will cause your instance to run in the background rather than consuming a shell console.}}
Glance typically uses a MySQL database called <tt>glance</tt>, although the name is configurable in the connection string.


== First Project ==
SQL connection settings might be stored in a glance configuration file located at <tt>/opt/stack/glance/etc/glance-registry.conf</tt>. In the devstack installation process, <tt>/opt/stack/glance</tt> contains a git checkout of the glance software.


We will create a single, very primitive Zope package, consisting of an Interface for a TODO class, and a TODO class.
The SQL connection configuration string might look something like this:


Create the following files and directories relative to your project root:
<pre>
sql_connection = mysql://glance:yourpassword@192.168.206.130/glance
</pre>


* Create the directory <tt>lib/python/example</tt>.
More info on glance configuration is available [http://docs.openstack.org/diablo/openstack-compute/install/content/glance-registry-conf-file.html here].
* Create the file <tt>lib/python/example/__init__.py</tt> by typing <tt>touch lib/python/example/__init__.py</tt>.
* Create these files:


=== <tt>example-configure.zcml</tt> ===
=== keystone ===


This file registers the <tt>example</tt> directory you created in <tt>lib/python</tt> as a ''package'', so that it is seen by Zope. Edit <code>/etc/package-includes/example-configure.zcml</code>:
Keystone, the OpenStack identity service, also uses SQL. <tt>etc/keystone.conf</tt> keystone install/git repo directory is used to store the SQL configuration:


<pre>
<pre>
<include package="example" />
sql_connection = %SQL_CONN%
</pre>
</pre>


=== <tt>interfaces.py</tt> ===
As everything else, the SQL connection string uses SQLAlchemy syntax.


The following file defines the <tt>ITODO</tt> interface, and also uses some Zope Schema functions to define what kind of data we expect to store in objects that implement <tt>ITODO</tt>. Edit <code>/lib/python/example/interfaces.py</code> with your favorite text editor:
=== Quantum and Open VSwitch ===


<syntaxhighlight lang="python">
<blockquote>Quantum is an incubated OpenStack project to provide "network connectivity as a service" between interface devices (e.g., vNICs) managed by other Openstack services (e.g., nova).</blockquote>
from zope.interface import Interface
from zope.schema import List, Text, TextLine, Int


class ITODO(Interface):
[http://openvswitch.org Open VSwitch] is described as:
    name = TextLine(title=u'Name', required=True)
    todo = List(title=u"TODO Items", required=True, value_type=TextLine(title=u'TODO'))
    daysleft = Int(title=u'Days left to complete', required=True)
    description = Text(title=u'Description', required=True)
</syntaxhighlight>


=== <tt>TODO.py</tt> ===
<blockquote>Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch or Cisco's Nexus 1000V. </blockquote>


Now, we define <tt>TODO</tt> to be a ''persistent'' object, meaning it can be stored in the ZODB. We specify that it implements our previously-defined <tt>ITODO</tt> interface, and provide reasonable defaults for all values when we create a new TODO object. Edit <code>/lib/python/example/TODO.py<code> using your favorite text editor:
There is an [http://openvswitch.org/openstack/documentation/ Open VSwitch Plug-in for OpenStack Quantum] which can be set up by DevStack. This plug-in uses SQL storage. The SQLAlchemy connection string is stored in (relative to git/install root) <tt>etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini</tt>. Similarly to everything but nova, the SQL connection string is stored in <tt>sql_connection =</tt> format.
<syntaxhighlight lang="python">
from persistent import Persistent
from zope.interface import implements
from example.interfaces import ITODO


class TODO(Persistent):
If the plugin is enabled, the following settings are added to nova.conf:
    implements(ITODO)
    name = u''
    todo = []
    daysleft = 0
    description = u''
</syntaxhighlight>


=== <tt>configure.zcml</tt> ===
<pre>
--libvirt_vif_type=ethernet
--libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
--linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
--quantum_use_dhcp
</pre>
 
=== Melange ===
 
From the Melange site:
 
<blockquote>
Melange is intended to provide network information services for use across OpenStack services. The initial focus will be on IP address management (IPAM) and address discovery (DHCP/dnsmasq) functionality. Melange is intended to be a standalone service with it's own API. However, the initial use case will be to decouple existing IP management and VM address discovery from Nova and support the existing Nova networking capabilities.
</blockquote>
 
Melange also uses a <tt>sql_connection =</tt> string stored in its <tt>etc/melange/melange.conf</tt> (relative to install/git root).


Create the <tt>/lib/python/example/configure.zcml</tt> configuration file:
== RabbitMQ ==
<syntaxhighlight lang="xml">
<configure xmlns="http://namespaces.zope.org/zope"
    xmlns:five="http://namespaces.zope.org/five"
    xmlns:browser="http://namespaces.zope.org/browser">
</configure>
</syntaxhighlight>


== Debug Mode ==
RabbitMQ  is a reliable messaging framework used by OpenStack. Currently, it looks like only nova uses it. Nova is configured to connect to rabbitmq by setting the following lines in <tt>/etc/nova/nova.conf</tt>:
 
<pre>
--rabbit_host=$RABBIT_HOST
--rabbit_password=$RABBIT_PASSWORD
</pre>
 
Rabbit's password is configured using the following command, as root:


We can test our first project by entering debug mode:
<console>
<console>
$##i## bin/zopectl debug
# ##i##rabbitmqctl change_password guest $RABBIT_PASSWORD
Starting debugger (the name "app" is bound to the top-level Zope object)
</console>
</console>


Now, let's try creating a new TODO object and writing it out to a ZODB database:
I am not yet completely sure how RabbitMQ fits into the OpenStack architecture. It may be that the supporting services expect it to be running locally, and that Nova compute nodes need to hook into a Nova instance, which would typically be running remotely. (Thus the ability for DevStack to target a remote RabbitMQ host.)
<console>
 
>>> from ZODB import FileStorage, DB
== Virtualization Technology ==
>>> storage = FileStorage.FileStorage('mydatabase.fs')
 
>>> db = DB(storage)
DevStack defaults to configuring OpenStack to use libvirt with [[KVM]], and will fall back to basic [[QEMU]] support if the <tt>kvm</tt> kernel module is not available. It also has support for using libvirt with [[LXC]], in addition to using [[Xen]] Server directly (bypassing libvirt.)
>>> connection = db.open()
 
>>> import transaction
>>> root = connection.root()
>>> from example.TODO import TODO
>>> a = TODO
>>> a.name = u'My TODOs'
>>> a.TODOS = [ u'Do Laundry', u'Wash Dishes' ]
>>> a.daysleft = 1
>>> a.description = u'Things I need to do today.'
>>> root[u'today'] = a
>>> transaction.commit()
</console>


[[Category:HOWTO]]
[[Category:Virtualization]]
[[Category:OpenStack]]

Latest revision as of 09:18, December 28, 2014

This page exists to document OpenStack configuration.

Note that the current approach is to use devstack, which is not a good way to learn OpenStack. So much of this document will be about doing a devstack-like configuration for Funtoo.

This document will split OpenStack configuration into each architectural component, describing configuration steps for each component separately.

SQL Database

A number of OpenStack services use a SQL back-end for storing various bits of data.

While DevStack uses MySQL for its SQL deployment, multiple database back-ends are actually supported thanks to SQLAlchemy being used behind the scenes, which is a re-targetable Python database API. Thus, it should be possible to use Postgres, etc, by simply using different connection strings. A list of SQLAlchemy connection types can be found on this SQLAlchemy documentation page.

Using a single root database user account for all services is not a good policy for production deployment. Ideally, each service should have its own restricted user account with only the ability to access its own database.

Let's look at how each service is configured in regards to SQL:

nova

Here's how to set up a MySQL database back-end for nova and tell nova to initialize its database tables: 

mysql> create database nova character set latin1;
Query OK, 1 row affected (0.02 sec)

mysql> grant all privileges on nova.* to nova@localhost identified by 'foobar';
Query OK, 0 rows affected (0.00 sec)

Now set the following connection string in /etc/nova/nova.conf: 

--sql_connection=mysql://nova:foobar@localhost/nova

Note the use of the latin1 character set when we created the tables in MySQL. This is so the following command will not cause an error due to the default UTF-8 character set creating indexes that are too big for MySQL to handle: 

root # nova-manage db sync
2012-03-02 21:31:14 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.migration' from '/usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/migration.pyc'> from (pid=17779) __get_b
ackend /usr/lib64/python2.7/site-packages/nova/utils.py:602

After running the command above, you should now have all the relevant database tables created: 

xdev var # mysql -u root -p nova
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 5.1.61-log Gentoo Linux mysql-5.1.61

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables;
+-------------------------------------+
| Tables_in_nova                      |
+-------------------------------------+
| agent_builds                        |
| aggregate_hosts                     |
| aggregate_metadata                  |
| aggregates                          |
| auth_tokens                         |
| block_device_mapping                |
| bw_usage_cache                      |
| certificates                        |
| compute_nodes                       |
| console_pools                       |
...

You have now validated that nova is connecting to your MySQL database correctly.

glance

From glance.openstack.org:

The Glance project provides services for discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image.

Glance typically uses a MySQL database called glance, although the name is configurable in the connection string.

SQL connection settings might be stored in a glance configuration file located at /opt/stack/glance/etc/glance-registry.conf. In the devstack installation process, /opt/stack/glance contains a git checkout of the glance software.

The SQL connection configuration string might look something like this: 

sql_connection = mysql://glance:yourpassword@192.168.206.130/glance

More info on glance configuration is available here.

keystone

Keystone, the OpenStack identity service, also uses SQL. etc/keystone.conf keystone install/git repo directory is used to store the SQL configuration: 

sql_connection = %SQL_CONN%

As everything else, the SQL connection string uses SQLAlchemy syntax.

Quantum and Open VSwitch

Quantum is an incubated OpenStack project to provide "network connectivity as a service" between interface devices (e.g., vNICs) managed by other Openstack services (e.g., nova).

Open VSwitch is described as:

Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch or Cisco's Nexus 1000V.

There is an Open VSwitch Plug-in for OpenStack Quantum which can be set up by DevStack. This plug-in uses SQL storage. The SQLAlchemy connection string is stored in (relative to git/install root) etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini. Similarly to everything but nova, the SQL connection string is stored in sql_connection = format.

If the plugin is enabled, the following settings are added to nova.conf: 

 --libvirt_vif_type=ethernet
 --libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
 --linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
 --quantum_use_dhcp

Melange

From the Melange site:

Melange is intended to provide network information services for use across OpenStack services. The initial focus will be on IP address management (IPAM) and address discovery (DHCP/dnsmasq) functionality. Melange is intended to be a standalone service with it's own API. However, the initial use case will be to decouple existing IP management and VM address discovery from Nova and support the existing Nova networking capabilities.

Melange also uses a sql_connection = string stored in its etc/melange/melange.conf (relative to install/git root).

RabbitMQ

RabbitMQ is a reliable messaging framework used by OpenStack. Currently, it looks like only nova uses it. Nova is configured to connect to rabbitmq by setting the following lines in /etc/nova/nova.conf: 

--rabbit_host=$RABBIT_HOST
--rabbit_password=$RABBIT_PASSWORD

Rabbit's password is configured using the following command, as root: 

root # rabbitmqctl change_password guest $RABBIT_PASSWORD

I am not yet completely sure how RabbitMQ fits into the OpenStack architecture. It may be that the supporting services expect it to be running locally, and that Nova compute nodes need to hook into a Nova instance, which would typically be running remotely. (Thus the ability for DevStack to target a remote RabbitMQ host.)

Virtualization Technology

DevStack defaults to configuring OpenStack to use libvirt with KVM, and will fall back to basic QEMU support if the kvm kernel module is not available. It also has support for using libvirt with LXC, in addition to using Xen Server directly (bypassing libvirt.)

Retrieved from "https://www.funtoo.org/index.php?title=Zope_HOWTO&oldid=7949"