The Grimoire Python action tree implementation

0.1.30 from main@grimoire.gna.org

Egil Möller

This document is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This document is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

Abstract

This document explains the rationales behind action trees and Grimoire and how to deploy Grimoire for system administration, extend it with new methods and interface against it from other systems.


Table of Contents

About
Action trees
The Grimoire action tree
Installation
Installation procedure - tgz
Requirements
Installation instructions
Installation procedure - rpm
Introduction
Accessing a tree from the command line UI
Accessing a tree from Python
Access control
The main tree
Introspection
The directory
Treevars
Configuration trees
The trees branch
Remote connections
The LDAP tree
LDAP specific directory entries
Listing LDAP entries
Access control
The SQL tree
The filesystem tree
The Printer tree
The login mini tree
Constructing trees
Constructing trees out of old trees
Access control
Writing new methods
Virtual trees
Referencing methods
Loading trees
Adding introspection to a tree
Implementing new directory services
Writing new UIs
Core library layout
Type system
Access control
Policies
Tree structure
Character encodings
Further help

About

This chapter introduces Grimoire in particular and the concept of action trees in general.

Action trees

System administration is traditionally done by one or a small group of (equally) trusted person(s). In larger organizations however, this group might not be big enought to fullfill all the administrative needs of the organization smoothly, and other people within the organization might not be trusted with full access to the system. Thus there is a need for delegation of specific administrative tasks or sets of tasks to other not fully trusted persons, e.g. group leaders or heads of departments.

Some people traditionally given (non computer) administative tasks, do not have the skill or knowledge to handle all the details of the tasks within their responsibilities, while other people should not be allowed to perform any but some combinations of tasks. Thus there is a need for easy developement of abstractions over existing tasks.

An action tree is a set of methods, tasks that can be performed, organized into a tree. Note that there are no objects (files, hosts, users) on which methods are performed; a method is a combination of what to do and with what, in one unit. This might sound counter intuitive but proves advantageous for the delegation of responsibilites. For example, to create a new user in the group hackers of the IT department one might use the method create.user.it.hackers.

Access control on an action tree is done on subtrees. As both task and object makes up the method, one can control access to both classes and subclasses of tasks and of objects. Access control can be as fine grained or coarse grained as needed for any particular user and method subtree.

To ease developement of abstractions over existing methods, a tree should both be exportable over an rpc channel and possible to cut, paste and merge with other trees as needed to form a tree suitable for a particular user.

The Grimoire action tree

Grimoire is a Python implementation of an action tree representing trees as trees of Python objects (and their member variables and methods). It has an access control system that can be coupled to an LDAP database where paths down the tree are used to specify subtrees that are either allowed or denied. Such paths are considered in an order that lets abilities coupled to different entries in LDAP override each other in a controllable manner.

Grimoire includes a base set of methods/actions for manipulation of LDAP entries for user accounts (both POSIX accounts and Samba accounts), groups, courier mail aliases and BIND DNS zone information, and for manipulation of the local file system on a Grimoire server (e.g. creation of home directories using templates). It also includes a Webware based web GUI and a command line UI to manipulate the tree.

Installation

This chapter describes how to install Grimoire, its components and any related software on a server or on a network of servers.

Installation procedure - tgz

Requirements

In order to install Grimoire the following other software packages are required:

  1. Python >= 2.3

Depending on what methods you want to install and use in your tree, some or all of the following software packages may need to be installed in addition to the one listed above:

  1. For translations to work:

    1. GNU Gettext

  2. To compile the documentation into useful formats:

    1. libxslt (xsltproc) >= 1.1
    2. Norman Walsh's XSL stylesheets for DocBook XML

  3. To connect to and share Grimoire trees over encrypted channels:

    1. M2Crypto >= 1.2
    2. OpenSSL >= 0.9.x (or rather, the version your version of M2Crypto requires)

  4. To use the LDAP Grimoire tree (trees.local.ldap):

    1. python-ldap >= 1.9.999
    2. libldap >= 2.1.23 (or rather, the version required by your version of python-ldap)

  5. To use the SQL Grimoire tree (trees.local.sql):

    1. PostgreSQL
    2. PyGreSQL >= 3.4
    3. postgresql-libs >= 7.3.4 (or rather, the version required by your version of PyGreSQL)

  6. To use the printer administration tree (trees.local.printers):

    1. CUPS

  7. To use the Client public terminal (user registration) tree:

    1. LaTeX
    2. dvips
    3. (GNU) sed
    4. lpd or CUPS

  8. To use the Web based user interface (Grimweb)

    1. Apache
    2. Webware >= 0.8
    3. FunFormKit >= 0.4.1

Installation instructions

  1. Untar the distribution tgz in /usr/lib/python2.3/site-packages. It will create a directory called 'Grimoire'.

  2. In the new directory, run the command Tools/compiletranslations.sh. This will compile the translation catalogues for various languages from .po text files to binary .mo files.

  3. Optionally, to compile the Python C extension modules, which will dramatically increase Grimoires speed, run the command Tools/compilecmodules.sh. You will need to have the Python header files installed in order to do this.

  4. Create /usr/bin/grimoire as a symbolic link to /usr/lib/python2.3/site-packages/Grimoire/root/clients/cli.py.

  5. Create a directory /etc/Grimoire and copy /usr/lib/python2.3/site-packages/Grimoire/Documentation/Scripts/Config.d into it.

  6. Modify the files in /etc/Grimoire/Config.d to match your system and what you intend to use your Grimoire server for.

  7. You may place additional configuration, in the same format as /etc/Grimoire/Config.d, in a per user ~/.Grimoire/Config.d or, if you have the $CHOICESPATH environment variable set, in any directory Grimoire/Config.d in any of the directories specified in that variable.

    Values from the different files override each other: files from directories earlier (to the left) in $CHOICESPATH override later ones which in turn override ~/.Grimoire/Config.d which in turn override the system global /etc/Grimoire/Config.d.

  8. If you want a Grimoire server (serving a Grimoire tree to other hosts) to be running on the machine

    1. If you are using SysV init (you are, if you are running RedHat, Mandrake, SuSE or Debian), create /etc/init.d/grimoire as a symbolic link to /usr/lib/python2.3/site-packages/Grimoire/Documentation/Scripts/grimoire.init.d and add symbolic links for starting and stopping it in the run level directories (/etc/rc?.d) you see fit. Alternatively, if your distribution supports it, the run level selection can be done using chkconfig --levels LEVELS grimoire on.

    2. For other init schemes (such as the BSD one), check out the grimoire.init.d script and copy the line that runs Grimoire with some arguments (and stdout and stderr redirected to /dev/null) into whatever init script is suitable on your system.

  9. If you want to use the Webware GUI

    1. Create a new Webware work directory

    2. In the workdir make a new context as a symbolic link to /usr/lib/python2.3/site-packages/ Grimoire/root/clients/html/funformkit/webware/_grimwebcontext.

    3. Make sure you have filled in the Grimoire expression for the Grimoire tree to present in the UI under ['tree'] in Config.d/parameters/_settings/clients/base.py.

  10. If you want to use the LDAP tree, run the script setup.sh in the Documentation/Scripts/Ldap directory on the machine intended to run the LDAP database server. Note that this script needs to be run with that directory as its current directory to find its helper files.

Installation procedure - rpm

  1. Install the package Grimoire and optionally Grimoire-SSL if you want to use encrypted network connections.

  2. Modify the files in /etc/Grimoire/Config.d to match your system and what you intend to use your Grimoire server for.

  3. You may place additional configuration (in the same format as /etc/Grimoire/Config.d) in a per user ~/.Grimoire/Config.d or, if you have the $CHOICESPATH environment variable set, in any directory Grimoire/Config.d in any of the directories specified in that variable.

    Values from the different files override each other: files from directories earlier (to the left) in $CHOICESPATH override later ones which in turn override ~/.Grimoire/Config.d which in turn override the system global /etc/Grimoire/Config.d.

  4. If you want a Grimoire server (serving a Grimoire tree to other hosts) to be running on the machine, use chkconfig grimoire on.

  5. If you want to use the Webware GUI

    1. Install the package Grimoire-Webware.

    2. Make sure you have filled in the Grimoire expression for the Grimoire tree to present in the UI under ['tree'] in Config.d/parameters/_settings/clients/base.py.

  6. If you want to use the LDAP tree

    1. Install the package Grimoire-LDAP.

    2. On the machine intended to run the LDAP database server

      1. Install the package Grimoire-LDAP-Server

      2. Set up an LDAP directory by running the script setup.sh in the /usr/share/doc/Grimoire/Scripts/Ldap directory. Note that this script needs to be run with that directory as its current directory to find its helper files.

Introduction

Grimoire is made up of four main parts; the trees of methods, the different user interfaces, the core library and a vast set of utility functions. This chapter gives an overview of those parts and describes various methods for accessing a Grimoire tree.

Accessing a tree from the command line UI

Grimoire does have a small language of its own, although its syntax and semantics are subsets of those of Python expressions (with one extension, but that is irrelevant for the normal user). Since expressions are used throughout Grimoire for various configuration tasks it is crucial to understand them. The command line client also uses such expressions and is thus suitable for testing and learning them.

The command line client grimoire accepts a "tree" expression and any number of "normal" expressions. The tree expression is evaluated in the main Grimoire tree and if it evaluates to a new Grimoire tree the remaining normal expressions are evaluated in that tree. If no "normal" expressions are given but a single extra argument being the empty string "", the command line client will present the user with a prompt at which normal commands can be written and evaluated directly.

All python data expressions are Grimoire expressions, that is numbers, strings, lists, tuples, mappings, True, False and None, and evaluates to themselves. In addition, the special symbol _ evaluates to the tree within which the expression is evaluated.

A branch/subtree or method may be selected from a tree using the same syntax as for object attribute access in python, that is by a path of method names separated by dots, e.g. _.foo.bar.fie means the branch fie on the branch bar on the branch foo on the tree in which the expression is evaluated. Method names may contain any character except for separators (parenthesises, braces, brackets, less/greater than, dot, comma, colon, citation marks, minus, plus and spaces) and must begin with a non digit. However, any of the disallowed characters may still be included if escaped with a backslash (backslash must itself be escaped with another backslash).

Given a method it might be applied to some arguments using the same syntax as used in python - a parenthesized list of values and/or name = value pairs. E.g. _.create.user.customers.gazonkware('fred', password='qwerty').

A complex example connecting to a remote tree, running a method on that tree that returns a tree, and starting a local Grimoire server in the background reexporting that tree.

_.trees.server.dirt(_.trees.remote.dirt.grimoireserver\.gazonkware\.com(
    ).trees.local.ldap('fred', 'qwerty'), 0)
    

Accessing a tree from Python

This chapter describes how to use Grimoire from within a Python program. You can skip this if you are only going to use Grimoire to administer your system, and will neither be extending it with new methods nor write specialized user interfaces to ease some task for non skilled users.

A method tree in Grimoire, is implemented as a python object (of the class Grimoire.Performer.Logical), with member variables which in turn are such python objects, and so on. Some of these objects represents Grimoire methods, and are callable.

For example, assume you have an instance of the LDAP Grimoire tree in the variable t, and that there is a user fred (under ou=hackers,ou=gazonkware,ou=customers,ou=people). Then

t.change.password.customers.gazonkware.hackers.fred('gazonk')
    

would change freds password to 'gazonk'.

Of course, if you changed passwords on people under customers.gazonkware often, you could assign t.change.password.customers.gazonkware to another variable, say f.

f = t.change.password.customers.gazonkware
f.hackers.fred('gazonk')
f.marketeers.fred('qwerty')
    

A complex example connecting to a remote tree, running a method on that tree that returns a tree, and starting a local Grimoire server in the background reexporting that tree.

Grimoire._.trees.server.dirt(
    getattr(Grimoire._.trees.remote.dirt,
            'grimoireserver.gazonkware.com'
           )().trees.local.ldap('fred', 'qwerty'),
    0)
    

As all types of names are allowed as method names, the class Grimoire.Performer.Logical fully overrides the member lookup functionality. Sometimes however, one must access some internal structure or method on the objects that builds up the tree. Actually, the tree is made up of objects of subclasses of the class Grimoire.Performer.Performer, of which there are two different views, one is the logical view as described above, and one is as a physical view, that allows access to internal (physical) methods and member variables, but not to (logical) tree methods. To change view of an object, simply pass it through Grimoire.Performer.Logical and Grimoire.Performer.Physical respectively.

An example:

print t.change.password.customers.gazonkware.hackers.fred('gazonk')
pt = Grimoire.Performer.Physical(t)
print pt._treeOp(['change', 'password', 'customers', 'gazonkware'],
                 'translate', ('sv', 'hackers'))
t2 = Grimoire.Performer.Logical(pt)
t2 == t
    

Access control

The access control on a Grimoire tree is based on path prefix matching. A path is matched by an access control rule if the path begins with the path in the rule. As a special case, if the path in the rule ends with the empty string, the rule will match that path with the empty string removed, and only that path.

More complex access rights are built up using a list of rules, each a pair of a "category" and a path, where the category is one of Allow, Deny, or Ignore. Such a list is called an ability. If a path is allowed by an ability is decided by walking down the list from top to bottom, and if a path in a rule matches the path to decide over, acting upon the category of the rule

  1. If the category of a matched rule is Allow, the path is allowed.

  2. If it is Deny, it is likewise denied.

  3. If it is Ignore however, the search continues down the list, but the next rule matching the path (regardless if it is an Allow, Deny, or Ignore rule), is ignored.

The list

Deny: create.user.gazonkware
Allow: create.user
Allow: create.host
    

will for example deny create.user.gazonkware.customers.fred, allow create.user.barware.customers.anna, deny create.domain.bar\.com and allow create.host.mycomputer.

The exact syntax used to specify such list varies between different places in Grimoire - lists in LDAP by need uses LDAP attributes for the category, and thus must use the ldif format as external syntax, whereas from within python a list of tuples of subclasses of a Category class and string lists representing paths is used.

The main tree

The main tree contains most of the Grimoire functionality, including methods that loads all the other trees. You should at least skim this chapter through to get an idea of what functionality there is.

The main tree is available in the Grimoire._ variable (and the tree expression in the command line UI is evaluated within the main tree):

>>> for leaf, path in Grimoire._.introspection.dir():
...     print leaf, string.join(path, '.')
...
1 introspection.about
1 introspection.translate
1 introspection.able
1 introspection.dir
1 introspection.methodOfExpression
1 introspection.eval
1 introspection.exist
1 introspection.params
1 directory.list
1 directory.new
1 directory.set
1 directory.get
1 directory.implementation.new.data
1 directory.implementation.get
1 directory.implementation.push.frame.data
1 directory.implementation.set
1 trees.test
1 trees.test.Error
1 trees.remote.ssl.dirt
1 trees.remote.dirt
1 trees.rpc.binding.dirt
1 trees.rpc.connect.ssl
1 trees.rpc.connect.ssl.error
1 trees.rpc.connect.tcp
1 trees.rpc.listen.ssl
1 trees.rpc.listen.tcp
1 trees.local.sql
1 trees.local.sql.error
1 trees.local.unprotected.filesystem
1 trees.local.filesystem
1 trees.local.ldap
1 trees.local.client
1 trees.local.unprotected.printers
1 trees.local.printers
1 trees.local.login
1 trees.server.treestore
1 trees.server.servertree
1 trees.server.socket.bound.dirt
1 trees.server.dirt
1 trees.server.ssl.dirt
   

If you replace each . by a / in the paths above, you can compare it with paths in the directory tree of python source files defining the methods, named root, inside the Grimoire directory.

Introspection

The introspection branch contains methods to query the tree and other methods for information such as which parameters a method takes, or what methods there are. For example, given a method trees.local.Ldap, a description and a parameter specification can be retrieved with

Grimoire._.introspection.params.trees.local.ldap()
    

The value introspection.params returns is a machine-readable one, but if you mangle it with unicode() and then print it, you will get a nice human description. For more descriptions on a specific method (e.g. introspection.dir one) than is given in this overview text, you may use introspection.params.

The value returned by introspection.params consists of several objects of different types wrapped up in each other. Outermost is an AnnotatedValue object - a pair of another object and a comment.

The functions Grimoire.Types.getValue and Grimoire.Types.getComment are available for extracting these two components. Note that these are not methods on the objects of that class, and can be applied to other objects too, in which case getValue will return its argument, and getComment will either return None or any value supplied as a second argument.

The object in the AnnotatedValue is of the type ParamsType, which is a specification of what parameters a method takes. It contains four elements: arglist, resargstype, reskwtype, and required.

arglist is a list of pairs of parameter names and types. The types may either be normal python type objects or classes, or AnnotatedValues of such objects.

resargstype and reskwtype specifies the types for extra arguments and extra keyword arguments, respectively. They may be either None, meaning no extra arguments are allowed, or a python type, class or AnnotatedValue of a type or class.

required is a numeric value, which specifies how many of the parameters specified in arglist, counted from the left, must be specified in a call.

One would also be able to list all methods with

>>> Grimoire._.introspection.dir()
[(1, ['introspection', 'about']), (1, ['introspection', 'translate']),
(1, ['introspection', 'able']), (1, ['introspection', 'dir']), (1,
['introspection', 'methodOfExpression']), (1, ['introspection',
'eval']), (1, ['introspection', 'exist']), (1, ['introspection',
'params']), (1, ['directory', 'list']), (1, ['directory', 'new']), (1,
['directory', 'set']), (1, ['directory', 'get']), (1, ['directory',
'implementation', 'new', 'data']), (1, ['directory', 'implementation',
'get']), (1, ['directory', 'implementation', 'push', 'frame',
'data']), (1, ['directory', 'implementation', 'set']), (1, ['trees',
'Test']), (1, ['trees', 'Test', 'error']), (1, ['trees', 'remote',
'ssl', 'dirt']), (1, ['trees', 'remote', 'dirt']), (1, ['trees',
'rpc', 'binding', 'dirt']), (1, ['trees', 'rpc', 'connect', 'ssl']),
(1, ['trees', 'rpc', 'connect', 'ssl', 'error']), (1, ['trees', 'rpc',
'connect', 'tcp']), (1, ['trees', 'rpc', 'listen', 'ssl']), (1,
['trees', 'rpc', 'listen', 'tcp']), (1, ['trees', 'local', 'sql']),
(1, ['trees', 'local', 'sql', 'error']), (1, ['trees', 'local',
'unprotected', 'filesystem']), (1, ['trees', 'local', 'filesystem']),
(1, ['trees', 'local', 'ldap']), (1, ['trees', 'local', 'client']),
(1, ['trees', 'local', 'unprotected', 'printers']), (1, ['trees',
'local', 'printers']), (1, ['trees', 'local', 'login']), (1, ['trees',
'server', 'treestore']), (1, ['trees', 'server', 'servertree']), (1,
['trees', 'server', 'socket', 'bound', 'dirt']), (1, ['trees',
'server', 'dirt']), (1, ['trees', 'server', 'ssl', 'dirt'])]
    

The result is a list of pairs, each a true boolean value and a path to a method. The dir method also takes an optional depth argument, indicating how far down in the tree to recurse during the search for methods. The default value is infinity. As a special case, the depth 0 may be used to test for existence of a method in the tree. If this argument is used to obscure branches deeper down in the tree, the listing will include pairs with a false boolean value and a path to the point under which the obscured branch resists.

The directory

The directory is similar in usage to GNOME:s gconf database and Windows' registry, but is slightly more general. You might want to skip this chapter until later if you have just started to learn Grimoire.

A directory is a database optimized for reading, that is, the opposite of an action tree (such as Grimoire) that is optimized for writing. To make things a bit confusing, Grimoire contains a directory, which is accessible using a tree methods in the action tree...

The directory in Grimoire is actually a meta directory that merges together any number of directories such as the configuration data for Grimoire (initially from files, but stored in RAM), LDAP, etc.

The directory consists of an inheritance tree, with nodes containing trees of data. A lookup will be performed on a "keypath" in a node in the inheritance tree specified by a path, and on all nodes up along that path until a node with data at the keypath. This model catches several common usages of directories - such as: looking up a default domain, add a mail account, and adding a user at a specific place in the LDAP tree.

>>> Grimoire.Utils.getpath(t.directory.get.ldap,
                           ['ou=people', 'ou=administrators', 'uid=redhog']
                          )(['cn=defaults', 'mailDomain'])
['jamtlinux.net']
    

The above searches for ['cn=defaults', 'mailDomain'] in

t.directory.get.ldap.ou=people.ou=administrators.uid=redhog
    

where it is not found,

t.directory.get.ldap.ou=people.ou=administrators
    

where it is found. It would then have continued if it wasn't found:

t.directory.get.ldap.ou=people
t.directory.get.ldap
    

The ldap directory will just append the keypath to the end of the path, and construct a DN out of all but the last item, and use the last one as attribute name.

>>> t.directory.get.parameters(['local', 'ldap', 'server'])
'ldap'
    

Note that the different directories need not be spread out over different subtrees under directory.get, but may overlap and override each other. For instance, you might set ['cn=defaults', 'mailDomain'] in ldap.ou=people.ou=administrators.uid=redhog in a config file, overriding the value from LDAP. Normally, the directory in RAM ("data") overrides other directories, but this is not required for all trees.

Treevars

If Grimoire finds a part of a method path that begins with a dollar sign ($), it will construct a second method path for the same method. The new method path is formed by expanding the name that began with a dollar sign, replacing it with a path from the directory (if one is found).

Grimoire searches for the keypath ['treevar', NAME], where NAME is the name searched for, in the tree treevar.METHODNAME, where METHODNAME is the prefix of the method path up to, but not including, the name searched for.

Note that the value in the directory should be a path (represented as a list of strings), not just one string, and the whole path will be inserted in place of the name.

Configuration trees

At start up, Grimoire loads several configuration files from /etc/Grimoire/Config.d etc.. These files are normal python files which manipulate the Grimoire directory.

To ease the mainaince and writing of such files, two special methods are defined within these when they are loaded - get and set, which calls Grimoire._.directory.get and Grimoire._.directory.set respectively, and which uses the path to the current file relative to the main configuration directory it is in (like/etc/Grimoire/Config.d) to construct prefixes to prepend to the path and keypath.

The parts of the file paths used for paths and key paths are split using a directory named _settings, and the file extension .py is removed from the final file name in the paths, e.g. a file path parameters/_settings/clients/html.py will result in paths being prefixed with ['parameters'] and key paths with ['clients', 'html']. That is, in that file a call to

set.currentMachine.currentUser(['static', 'url'],
                                'http://example.com/GrimWeb-static')
     

will result in a call

Grimoire._.directory.set.parameters.currentMachine.currentUser(
    ['clients', 'html', 'static', 'url'],
    'http://example.com/GrimWeb-static')
     

The trees branch

The trees branch contains a set of methods that all returns new trees - either specialized ones for different task such as LDAP manipulation, or remote ones retrieved from another host running a Grimoire server. Most of these trees also includes a linked-in "copy" of the main tree for convienence.

Remote connections

Connections to a remote tree is done by the method trees.remote.dirt.HOSTNAME[.PORT] (or trees.remote.ssl.dirt.HOSTNAME[.PORT]), which returns an object that represent the remote tree locally. It can be used exactly as any local tree. Calling a method on that object, will result in a remote call over the network to a method on the remote tree on the sever.


>>> unicode(Grimoire._.introspection.params.trees.remote.dirt())
u'Connects to a remote host, specified by a path (host, port), using
Grimoire over DIRT over tcp and returns the remote tree as a locally
accessible one: '

     

The output is a description of the method with the parameters listed last (after the colon). In this case there is none.

To use this method, you of course needs a remote tree to connect to. To publish a tree so that it is available remotely, any method under trees.server may be used (depending on what protocol you want to publish it over, e.g. DIRT or DIRT over SSL), e.g. trees.server.dirt:

>>> unicode(Grimoire._.introspection.params.trees.server.dirt())
u'Serves an object to the outside world using DIRT (optionally over
ssl): obj: Performer to serve: Grimoire.Performer.Performer, daemonic:
Whether to detach the server thread or not (if detached, the Python
process will not terminate until all serve threads has terminated):
Grimoire.Types.Derived.BooleanType'
     

Start a server in a new thread in the current process of serving the main Grimoire tree itself:


>>> unicode(Grimoire._.trees.server.dirt(Grimoire._))
u'dirt server for <Grimoire.Performer.Composer instance at 0x40a587cc>
on :8445: <Thread(dirt server for <Grimoire.Performer.Composer
instance at 0x40a587cc> on :8445, started daemon)>'

     

You can now connect to this tree from the same machine, and even the same process, just from another thread (note that currently, the SSL implementation used, OpenSSL in conjunction with M2Crypto, is not thread safe, and connecting over SSL to a tree served by another thread in the same process in the same manner will deadlock):

>>> t = Grimoire._.trees.remote.dirt.locahost()
     

You can see that t.introspection.dir gives exact the same output as Grimoire.tree.introspection.dir did above. This is because you published the complete main Grimoire tree (Grimoire._) using the server you launched above.

The LDAP tree

The LDAP tree contains methods for manipulating the LDAP data base. It uses LDAP based authentication and access control. The function local.ldap is used to log in to LDAP. It establishes a connection to the LDAP server, authenticates the user and returns a Grimoire tree.

>>> unicode(Grimoire._.introspection.params.trees.local.ldap())
u"Returns an LDAP manipulation tree for an LDAP server:
 userId: User name: Grimoire.Types.Derived.UsernameType,
 password: User password: nonempty Grimoire.Types.Derived.LosePasswordType
 (server: LDAP server name: <type 'str'>,
  realm: LDAP realm (base DN appended to all DNs): <type 'str'>,
  admindn: LDAP administrator DN (relative to realm)): <type 'str'>,
  adminpwd: Administrator password: nonempty
            Grimoire.Types.Derived.LosePasswordType)"
     

Note that the output is re-indented to make it more readable, at the moment, the real output is unfortunately not at all as readable, and that the optional arguments are really only optional if their corresponding values in the Grimoire config file are set (except for admindn, which is assumed to be cn=admin,REALM if not set).

LDAP specific directory entries

An LDAP tree has some extra directory entries, notably for accessing the LDAP connection object, the DN of the logged in user and objects and attributes in the LDAP database itself. All but the last category are stored in the normal "data" implementation.

'local', 'ldap', 'user', 'dn'

contains the DN of the current logged in user, with the LDAP REALM removed.

'local', 'ldap', 'user', 'conn'

contains an LDAP connection object, connected as the logged in user.

'local', 'ldap', 'admin', 'conn'

contains an LDAP connection object, connected ad the database administrator. Note that most operations on LDAP a user is allowed to perform, are made up by operations which each taken by itself, the user is not allowed to perfom. Thus, such operations must be performed in the name of the administrator.

Listing LDAP entries

The method list.ldapentries can be used to list ldap entries, for example to construct a result to return from _dir() on a SubMethod. To ease this specific usage, its first argument and any extra path (it is a SubMethod itself), share the same semantics as the arguments to _dir(), that is, the extra path is a prefix under which searches are perfomed, and depth governs how deep to search in the tree.

Note that since depth in LDAP search semantics, scope, can only be one of BASE (corresponding to depth = length(prefix)), ONE (corresponding to depth = length(prefix) + 1) and SUB (corresponding to depth = UnlimitedDepth), all depths > length(prefix) + 1 will result in a search with scope = SUB, throughout the entire LDAP subtree under the specified prefix.

Access control

The entries of the ability list for a Grimoire LDAP tree are gathered from several LDAP database entries. The entries are constructed from the grimoireAbilityAllow, grimoireAbilityDeny and grimoireAbilityIgnore attributes of the following LDAP database entries, and in this order:

  1. cn=security under the users own entry,

  2. cn=security under the home group and any surrounding group (e.g. groups having a suffix of the home group DN as DN), sorted on the length of the DN, with the longest ones first,

  3. and cn=security under each group the user is a member of (that is, the ones that has a memberUid attribute equal to the username of the user), sorted on the length of their DNs, with the longest ones first.

Each LDAP entry may have several grimoireAbilityAllow, grimoireAbilityDeny or grimoireAbilityIgnore attributes, each holding a single path. The paths are normal dot separated Grimoire paths, but with an extra leading dot. The attributes are sorted on path length, and each attribute is the used to construct one rule for the ability.

The SQL tree

FIXME: Write section

The filesystem tree

The filesystem tree provides several methods for manipulating the local filesystem, for example on a server hosting home directories or maildirs. It is protected by a username/password pair, set in the config file (on the server serving the tree), Config.d/parameters/_settings/local/filesystem.py, under ['login', 'username'] and ['login', 'password'], respectively. All filesystem methods include a prefix specified by the treevar fileservername.

FIXME: Write section

The Printer tree

FIXME: Write section

The login mini tree

The method trees.local.login might be used as a wrapper for any other method that takes at least a username and a password, such as the LDAP or SQL login methods, providing a nicer user interface when logging in. It takes two parameters, a description of the tree, and a login method.

Constructing trees

This chapter describes how to extend Grimoire with new methods or whole new trees. You will only need to read this if you plan to use Grimoire to administrate a new type of system or add new administrative tasks. In that case, you should also read the section called “Core library layout” below.

Constructing trees out of old trees

Grimoire trees can be cut into pieces, merged and rechaped in various ways. This chapter describes how to do this. You might want to skip this chapter until you actually have a need to forge existing trees.

A subtree of a tree can be cut out using a Gimoire.Performer.Cutter object. For example, given a tree in the variable t1,

t2 = Gimoire.Performer.Cutter(t1, ['create', 'user', 'fred'])
    

will assign the branch/subtree create.user.fred to the variable t2.

The same can be accomplished directly on a logical view of trees using a short-cut that uses just ordinary Python attribute access. The following code carries out the same operation as the code in the example above, but with a logical view of the tree in t1

t2 = t1.create.user.fred
    

The opposite, to prefix a tree with some extra path, is done using a Gimoire.Performer.Prefixer object. Such an object will represent a tree containing all methods of the tree within it, but with the same prefix prepended to the paths to each of them. For example, given a tree t with the methods password and language, and

t2 = Gimoire.Performer.Prefixer(['change, 'own'], t2)
    

the tree in t2 will have the two methods change.own.password and change.own.language.

Two or more trees can be combined to form a larger one using a Gimoire.Performer.Composer object. For example, assume you have the two trees x and y, x having the method create.user.gazonkware.fred and y having the methods change.own.password and change.defaults.people, and

z = Gimoire.Performer.Composer([x, y])
    

Then z would have the methods create.user.gazonkware.fred, change.own.password and change.defaults.people.

Access control

Access to methods of a tree can be restricted to just some subtrees, that is, arbitrary branches/subtrees can be removed, using an object of the class Gimoire.Performer.Restrictor. Given a tree t, and a function f of one argument, u = Gimoire.Performer.Restrictor(t, f) is a new tree, containing all branches of t for which f returns True, given the path of the branch.

There are a set of standard functions for access control included in Grimoire, described in the core library layout section.

Writing new methods

Objects of subclasses of Grimoire.Performer.Base are used to construct new trees. Each such object contains a set of methods implemented in the subclass as a class variable containing a subclass of Grimoire.Performer.Method. As this may sound a bit confusing, an example might be in place:

class MyTree(Grimoire.Performer.Base):
    class bite_fred(Grimoire.Performer.Method):
      ...
    class write_email_fred@example.com(Grimoire.Performer.Method):
      ...
    class restart_server_apache(Grimoire.Performer.Method):
      ...

t = MyTree()
t.restart.server.apache()
    

As seen above, the names of the class variables all begin with an underscore, which is not there in the method names, and each subsequent underscore is converted into a dot in the method names.

The Grimoire.Performer.Method subclass must implement some virtual methods (and may override some default ones too, if desirable), namely the one that is called when the Grimoire method it implements is called (_call(self, *arg, **kw)), and the one describing which parameters the former one takes (_params(self)):

class bite_fred(Grimoire.Performer.Method):
    def _call(self, hardnes = 0):
        return "Can't bite fred that hard! He won't like that..."

    def _params(self):
        # We only take one parameter, of type int, and there are no
        # required parameters (0).
        return Grimoire.Types.ParamsType.derive([('hardness', types.int)], 0)
    

Note that _params should always return an object of the class Grimoire.Types.ParamsType.

Virtual trees

It is often desirable to construct a virtual tree of methods - all implemented by the same code, and the names of them taken from one database (e.g. an LDAP tree or the directory structure on a file system). This can be done using a subclass of a special subclass of Grimoire.Performer.Method, Grimoire.Performer.SubMethod. Subclasses of this class must implement one more method (_dir), and the other methods has a slightly different API:

class bite(Grimoire.Performer.SubMethod):
    def _call(self, path, hardnes = 0):
        return "Can't bite " + string.join(path, '.') + \
               " that hard! He/She won't like that..."

    def _dir(self, path, depth):
        # You don't _need_ to care about depth, it is
	# only there for optimization.
        # However, path, you need to care for. Only items with that
        # path as a prefix should be returned, and the prefix removed
        # from them. For now, just fire an error if it is not the
        # empty path...
	if path != []:
	    raise Exception("Can't handle this yet...")
        # Each item should be a pair of the number 1, and a path.
        return [(1, ['Andersson', 'Anna']),
                (1, ['Johansson', 'Johan']),
                (1, ['Hacker', 'Random', 'J']),
                (1, ['Loser', 'Random', 'J'])]

    def _params(self, path):
        # You can have the different virtual methods take different
        # sets of parameters, then use path to check for which of them
        # the description is asked for.
        # We still only take one "real" parameter, of type int, and there are no
        # required parameters (0).
        return Grimoire.Types.ParamsType.derive([('hardness', types.int)], 0)
    

The difference in API is that both _call and _params takes an extra path argument, that will contain the "rest path" when a virtual method is called. Let's say the above SubMethod is in the Base object t, then calling t.bite.Andersson.Anna would result in a call to _call, with path set to ['Andersson', 'Anna'].

Referencing methods

The physical methods _getpath(root = Grimoire.Types.CurrentNode, levels = 0, path = []), _physicalRoot(), _physicalParent(), _physicalBase() and _callWithUnlockedTree(function, *arg, **kw) of a Grimoire tree object (Method, SubMethod, Base etc) can be used to refference other methods from within a method in various ways:

_getpath(root = Grimoire.Types.CurrentNode, levels = 0, path = [])
references another part of the logical tree.

Given a tree t, Grimoire.Performer.Physical(t)._getpath(path=['create', 'user', 'administrators']) is equivalent to Grimoire.Performer.Logical(t).create.user.administrators.

The root argument selects where the path is rooted. If it is the default, Grimoire.Types.CurrentNode, the path is rooted at the object on which _getpath was invoked. If it is Grimoire.Types.TreeRoot and the current tree object is actually part of a larger tree, the path is rooted at the root of that larger tree. For example given a tree t as above

t2 = Grimoire.Performer.Logical(t).create.user
t3 = Grimoire.Performer.Physical(t2)._getpath(
    Grimoire.Types.TreeRoot, 0, ['change', 'password'])
t4 = Grimoire.Performer.Logical(t).change.password
t3 == t4
        

There is one more value possible for root, Grimoire.Types.MethodBase. This value is only valid if the current tree object is a Grimoire.Performer.Method or Grimoire.Performer.SubMethod object within a Grimoire.Performer.Base object, and will reference the point in the logical tree where the Grimoire.Performer.Base object is rooted.

Lastly, the levels argument can be used to ascend any number of levels towards the root of the tree (It is of course not available when root is set to Grimoire.Types.TreeRoot) prior to descending along the given path argument.

_physicalParent()

returns the surrounding Grimoire object (Prefixer, Base, Composer, Restrictor, etc). Use with care.

_physicalBase()

returns the surrounding Base object of a Method or SubMethod. As Base creates several intermediary Prefixer objects on the fly, _physicalParent can not be used (easily) from within a Method or SubMethod to access the surrounding Base. Use with care.

_physicalRoot()

returns the outhermost surrounding Grimoire object (the grand grand grandparent :). Use with care.

_callWithUnlockedTree(function, *arg, **kw)

calls function(*arg, **kw) but with no access restrictions from the current user. Neither any subsecuent call done by function will have any such restrictions, until the return of function, even if it in turn calls _callWithUnlockedTree on some other function. The return value of _callWithUnlockedTree is that of function.

Loading trees

This chapter describes how to create whole new trees and how the different files making up a tree implementation are combined into the final tree. You might want to read this chapter if you have problem putting your method implementation in the right place, or if you want to create a whole new tree of methods.

Most Grimoire trees are implemented in several separate files in a directory structure that resembles the tree structure of the Grimoire tree once loaded.

Each .py file in the tree may contain a descendant of the Performer.Base class called Performer, which will be instantiated. All such instantiated classes will be joined with a Performer.Composer according to the rules below. In addition, .po files with translations may be scattered over the tree to be loaded into the resulting Grimoire tree:

  1. The Performer class in the file foo/bar/fie.py will be joined in the Composer with the prefix ['foo', 'bar']. Note that the filename is deliberately left out of that path.

  2. The Performer class in the file foo/bar/fie/__init__.py will be joined in the Composer with the prefix ['foo', 'bar']. Note that the last directory name (as well as the filename) is deliberately left out of that path.

  3. The load function will not descend into directories whose name has a leading underscore, except for the top level directory.

  4. Translations in the file foo/bar/fie/_Translation/$LANG/LC_MESSAGES/naja.po are loaded into the prefix used for the file or directory foo/bar/fie/naja.

To load such a tree, the method trees.local.load is used. It takes one argument, the module path to the tree to load as a string (like 'Grimoire.root.trees.local.ldap._Performes'). Usually, it should be called with a name relative to the current module, using the Python special variable __name__, like Grimoire._.trees.local.load(__name__ + '._performers').

The main tree is implemented this way and resides in the directory root. It is loaded using this mechanism when the Grimoire python module is imported, and put into the variable Grimoire._. It has been described under the section called “The main tree”.

Adding introspection to a tree

The class returned by trees.introspection implements a Grimoire tree that adds introspection to any tree it is combined with. That is, given a tree t1 and

t2 = Gimoire.Performer.Composer([
    ([], t1),
    ([], Grimoire._.trees.introspection())])
     

t2 would hold a set of extra methods to query the tree about the ones defined in t1.

The method trees.local.load.standardtree is a wrapper around trees.local.load which automatically adds introspection and directory services to the loaded tree, aswell as supports execution of init commands in the loaded tree. For more information, please use introspection.params.

Implementing new directory services

This chapter describes how to implement new back ends for the directory. If you skipped the chapter on the directory under the main tree above, you should skip this chapter as well.

The methods directory.get and directory.set only implements the recursion up along the directory path, and uses data storage back end methods under directory.implementation.get and directory.implementation.set respectively, to actually provide the data storage.

The mechanism is similar for get and set. When directory.get gets called with a path and keypath, it will construct a method path to call for each prefix of the path, by appending the path directory.implementation.get, the path prefix, and element consisting of the value of Grimoire.Types.pathSeparator, and the keypath. It will call those methods beginning with the longest prefix and continuing with shorter and shorter prefixes until the call succeds (does not throw an AttributeError, or there are no shorter prefixes of the path, in which case an AttributeError is thrown.

What this all means is that, to implement a directory data storage back end, you need to provide methods or sub methods under directory.implementation.get and/or directory.implementation.set, that takes the same arguments as directory.get and directory.set, except for the keypath which is mangled into the path. These methods does not have to do any recursion or other magic, only provide a value or accept a value to store. For a simple example, take a look in the file root/trees/local/ldap/_performers/directory/implementation/Ldap.py

Writing new UIs

A new user interface could easily be written as a normal python program that just calls the appropriate methods on a Grimoire tree object, and presents the results to the user in some way. However, there are several tasks that most UIs will have to perform. Such as rendering some sort of tree of methods and when the tree expands further, perform introspection.dir() calls. The methods under clients provides such services as normal python classes.

Core library layout

This chapter describes the core library functions. The core library is made up of the tree system (already pretty well covered), the type system, the access control system and helper functions. This chapter is only of interrest to those implementing new Grimoire methods or who uses Grimoire trees from within a Python program.

Type system

The type system is implemented in the module Grimoire.Types. It is split up over several files in the Types directory, all merged into one module. Conceptually, is is made up of two sets of types.

The first is solely extensions/restrictions of types in the python types module - integer ranges, lists containing values of a specific type, etc. These are used to specify the type required for an argument of a Grimoire method more in detail.

The second set provides text objects that can be translated and composed in various ways to adhere to whatever rendering device is available in a client software (teletype terminal, HTML renderer, Gtk+, etc). A special subset of the second set is a set of types that represent some real data type, such as an e-mail address, an URI or a path. These types have additional functionality to parse strings into such objects.

For more information on the type system, please read the doc strings of each type (using the python help function).

Access control

The module Grimoire.Types.Ability defines two access control functions (or rather, objects that can act as functions).

Objects of the Grimoire.Types.Ability.Simple class matches a path by prefix, and allows all paths that begins with that prefix (or if the "prefix" ends in an empty string, allows only the path equal to the prefix except that last empty string item).

More complex access rights can be built up from the Grimoire.Types.Ability.Simple ones using an object of the Grimoire.Types.Ability.List class. Such an object is initialized with a list of pairs of a subclass of Grimoire.Types.Ability.Category and a Grimoire.Types.Ability.Simple object. The Grimoire.Types.Ability.Category subclasses are Grimoire.Types.Ability.Allow, Grimoire.Types.Ability.Deny, or Grimoire.Types.Ability.Ignore.

The List class determines if a given path is allowed or not by walking down the list, as described in the introduction.

>>> a = Grimoire.Types.Ability.Allow
>>> d = Grimoire.Types.Ability.Deny
>>> l = Grimoire.Types.Ability.List([
 (d, ['create', 'user', 'gazonkware']),
 (a, ['create', 'user']),
 (a, ['create', 'host'])])
>>> l(['create', 'user', 'gazonkware', 'customers', 'fred'])
False
>>> l(['create', 'user', 'barware', 'customers', 'anna'])
True
>>> l(['create', 'domain', 'bar.com'])
False
>>> l(['create', 'host', 'mycomputer'])
True
    

Note that the Grimoire.Types.Ability.List class is initialized from a list of pairs of category and path, where path is either a simple list of strings, or a Grimoire.Types.Ability.Simple object, and category is either Grimoire.Types.Ability.Allow, Grimoire.Types.Ability.Deny or Grimoire.Types.Ability.Ignore, or 0 or False (same as Grimoire.Types.Ability.Deny) or 1 or True (same as Grimoire.Types.Ability.Allow).

Policies

This chapter describes Grimoire coding standards. When coding Grimoire methods (or changing the Grimoire library itself), please try to adhere to the guidelines specified in this section if there are no severe problems with doing so in your specific case.

Tree structure

The whole point of Grimoire is that the tree could be extended with any module you need to fulfill all your administration needs. The tree could look any way you like. However, to be easily and quickly understood by the user, some naming conventions are needed. Also, if you want your module to be a part of the Grimoire distribution, you will probably have to follow this naming policy. It's been created to:

  1. reduce the amount of information that the user has to go through to get to the right place. More than seven options in each step is not recommended by UI gurus.

  2. protect the user from having to choose between several slightly ambigous choises. For example, to configure a printer queue, do I select "manage", "change" or "modify"?

After writing a fair amount of modules, we have succeeded to isolate some management "verbs" that should cover most management tasks. If you for some "verb" invent a sub category, fill it in here so that everyone will be using the same sub categories. When doing so, please try to keep the tree orderly and minimal.

In the naming conventions below, sysnonyms listed within parenthesis are the verbs that should not be used.

The naming conventions for methods that users will call directly

list (Synonyms: show, view)
create (Synonyms: add)
ability (Internal. Create an ability)
delete (Synonyms: remove, kill)
ability (Internal. Delete an ability)
change (Synonyms: modify, manage)
ability
allow (Synonyms: grant)
deny
cancel (Synonyms: revoke, delete, remove)
allow (Synonyms: allowance)
deny (Synonyms: denial)
disable (Synonyms: stop, unload)
enable (Synonyms: start, load)
reset (Synonyms: restart, reload)

and the naming conventions for internal methods of all trees and for methods in the main tree

trees (Grimoire trees and tree functions)
local (Locally available trees)
remote (Connect to remote trees)
server (Serve trees to remote hosts)
introspection (Aquire information or operate on the tree itself)
dir (Synonyms: list, denumerate)
params (Synonyms: parameters, help, about)
eval (Synonyms: evaluate, interpret)
directory (Generalized configuration data access)
get (Synonyms: read, find)
set (Synonyms: write)
list (Synonyms: dir, find)
implementation (Data storage implementations)
get (Synonyms: read, find)
set (Synonyms: write)
client (User interfaces. Synonyms: ui)

Character encodings

Internally (Method path elements, method parameters)

Strings may be either Types.StringType or Unicode Types.UnicodeString or subclasses of one of these. In the case of Types.StringType, the content is allways US ASCII, never Latin-1 or somesuch.

DIRT (Grimoire RPC)

Everything is UTF-8 encoded.

Externally (Grimoire interfacing to other systems)

If possible, UTF-8 is used. With some systems it is possible to choose encoding when connecting. In those cases UTF-8 is allways used. With systems that has to be configured to use one encoding, the likelieness that other systems besides Grimoire are also talking to that system and the negative implications of forcing UTF-8 on all those must be considered, but if possible, UTF-8 is still be used. If it is highly impractical to enforce an encoding, the encoding is a configuration option, but with a default value of UTF-8.

UI

If it is possible to enforce an encoing without the user noticing, as in the case with web interfaces, UTF-8 is used. If the user environment provides an encoding configuration, as in the case with a UNIX text interface (locale provides this), the provided encoding is used.

Further help

Use

  1. introspection.params and introspection.dir to check out a tree.

  2. the python help() function to get more information on a sub module, class, object, method or function.