This manual is for GNU Conjure (version 0.1, September 2004).
Copyright © 2004, 2005 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with the Front-Cover Texts being “A GNU Manual,” and with the Back-Cover Texts as in (a) below. A copy of the license is included in the section entitled “GNU Free Documentation License.”
(a) The FSF's Back-Cover Text is: “You have freedom to copy and modify this GNU Manual, like GNU software. Copies published by the Free Software Foundation raise funds for GNU development.”
--- The Detailed Node Listing ---
Cleansing and housekeeping
This chapter provides a very broad overview of Conjure's goals, motivation and design.
GNU Conjure is a build tool in the spirit of the well known Make program. Using Conjure you will be able to specify the tasks conducting to the accomplishment of a series of goals. Usually, these goals will involve the creation of new files (e.g. a compiled program, or an info document) from existing resources.
The resources needed by a given goal are known as its dependencies, and can include local files (source code in any programming language, texinfo files, etc.), information contained in a relational database, or whatever other data source accessible by a local program. Most importantly, a goal can also depend on other goals.
Conjure's job is not directly creating the products of a goal: you must specify a procedure, possibly invoking external programs, in charge of creating the goal's final artifacts; Conjure will take care of invoking it whenever is needed. A goal will need rebuilding when its products are stale, i.e., when the goal's dependencies have changed since the last time it was (re) built. Many predefined procedures are already provided by Conjure libraries, offering common operations like invoking external compilers, document generation tools, local and remote filesystem access, etc. You can use them out of the box, or combine or extend them as you see fit in order to specify the building procedures for your goals.
The traditional way of assessing a product staleness is comparing the filesystem timestamps of products with those of its dependencies. If the latter have been modified after the former were created, the goal needs rebuilding (of course, this is only possible when both products and dependencies consist, ultimately, of files). While this is also the default behaviour of Conjure, you can provide arbitrary staleness predicates for testing whether goals need updating. For instance, you may prefer using MD5 checksums to timestamps. Or maybe one of your goals will be populating a database table from information contained in other tables; here, the staleness predicate would involve accessing and checking these tables, a task not easily expressible in terms of file modification times! Again, Conjure comes with a rich set of predefined functions that you can use to specify the most commonly used staleness predicates.
Make, in its various incarnations, is a widely used, time-sanctioned tool that plays a central role not only in software development, but also in Unix and GNU systems administration. Therefore, it's only fair to ask why should we develop a replacement and, while we are at it, why a Scheme-based one.
Despite being an excellent tool, Make has, from our point of view, some deficiencies that we intend to address in Conjure, namely
Some of the above problems may be circumvented with more or less awkward workarounds, but we think that the main issue here is that the Make “language” lacks abstraction power. It is difficult to express complex operations in a clean way, hiding away that complexity under an appropriate abstraction umbrella. Ideally, we should be able to write libraries of reusable functions easily, using a clean, readable and natural syntax, and yielding high-level build task specifications which, in addition, are modular and extensible.
These are the very issues that programming languages are supposed to solve. Thus, it seems only natural to think of converting makefiles into executable scripts in an adequate high-level language. This has in fact been recognised by recent build systems like, e.g., scons, which is based on Python. But we must keep in mind that, most of the time, we don't need a general purpose language when writing a build specification. What is really needed is a domain specific language suited to the task at hand, that is, a high-level mini-language with clear syntax that allows a natural expression of the chores of building tasks. A language in which you can directly speak of goals, targets, dependencies, build procedures, etc., etc., but that is, nonetheless, flexible and easily extensible.
Given these requirements, Scheme pops up as a perfect implementation language for Conjure. Its syntax is as simple as you can get, and the suitability of Lisp for DSL creation (thanks to its powerful abstraction mechanisms, like, e.g., macros) is unparallelled. Syntactic abstraction will allow the creation of a mini-language as close as possible to our problem domain, without sacrificing the extensibility of our build procedures: Conjure scripts are still Scheme programs, and you have the full power of the language under your belt when writing them.
In addition, Guile, our default underlying Scheme implementation, is the preferred extension language of the GNU project, so that we can count on its being present in virtually all installed GNU systems, a practical necessity when you intend to replace a universal tool like Make.
From a user's standpoint, Conjure can be viewed under two complementary perspectives. On the one hand, Conjure and its libraries provide a build specification mini-language, with lispy syntax. On the other hand, a Conjure build specification is actually a Scheme program and, as such, it puts under your fingertips the whole power of the Scheme programming language. You can think of it as build system being seamlessly extensible using Scheme.
One of the main design goals of Conjure is to allow a clear separation of the above perspectives. That is, you don't need to learn Scheme to use Conjure effectively. Most of the time, you will just use it as a domain-specific language (DSL) that lets you specify your building process in a purely declarative way. For instance, let's say that you want to build a program foo which depends on a C source file foo.c and a library libbar.a whose sources are bar.c and qux.c. You would write the following specification file:
(conj:c-program (name "foo") (depends "foo.c" "bar") (flags "-0" "-g")) (conj:c-library (name "bar") (depends "bar.c" "qux.c") (flags "-O2")) (conj:clean-goal)
which, as promised, is purely declarative. Once you become better acquainted with Conjure libraries, you will be able to write more complex specs, like, e.g.,
(conj:with-build-dir "./lib" (conj:c-library (name "bar") (depends "bar.c" "qux.c") (flags "-O"))) (conj:with-build-dir "./bin" (conj:c-program (name "foo") (depends "foo.c" "bar") (flags "-02" "-g")) (conj:clean-goal)
Here, and from the DSL point of view, we just used a new language
with-build-dir, that lets us specify the target
directory of our goal's products.
Under the hood, the above specs are just Scheme programs that are
executed by Conjure. For instance,
conj:with-build-dir is a
conj:clean-goal is actually a procedure. Both are
predefined in suitable libraries, but, other than that, nothing
differentiates them from any other Scheme procedure defined by you. In
fact, you can use any valid Scheme expression within a build file.
Moreover, Conjure provides an API for accessing its dependency
resolution engine, as well as utility libraries for writing your own
building procedures. You can even bundle your code up into an extension
library that other users can use simply as a set of new keywords for
writing their building specification.
This document describes how to use GNU Conjure. We begin with a tutorial introduction (see Getting started) by means of several simple (and not so simple) use cases. Afterwards, we move on to describing how to extend Conjure using the Scheme programming language (REF:FIXME), and provide a complete description of the builtin functionalities and libraries provided with the system (REF:FIXME).
This tutorial chapter, after introducing the basic concepts and nomenclature in our problem domain (see Conjure lingo), guides you through the creation of several simple Conjure build files. Our aim is to provide a hands-on view of how Conjure is used in a typical (though scaled down) project involving compilation and document generation tasks (see A sample project).
This section introduces the main terms and definitions used in the rest of the manual when talking about Conjure's build specifications.
Our objective while using Conjure will be the specification of a series
of build tasks. We call each one of these tasks a goal, and
identify it using a unique string identifier: the goal's name.
When the end product of a build task is a file, the name of the
corresponding goal may be simply the file's name (but we can specify
them independently, see below). Goals are declared in a plain text file
using the reserved word
conj:goal. We call the file containing
the list of a given project's goals its Conjure build specification
file, or simply its Conjure file for short.
So if we want to specify the building of, say, the file jokes.dvi, the Conjure file will contain a line of the form
(conj:goal "jokes.dvi" <build-spec>)
where <build-spec> is a series of Conjure directives specifying how jokes.dvi is created. Alternatively, we can specify separately the goal's name and its end product, using the keyworkds conj:name (or conj:names) and conj:product (or conj:products):
(conj:goal (conj:names "jokes" "make-jokes") (conj:product "jokes.dvi") <build-spec>)
In the above example,
make-jokes are the names
of a goal that has as end product the file jokes.dvi. End
products of a goal (also called artifacts), such as
jokes.dvi in our running example, will usually depend on other
files or goals, in the sense that, whenever any of them changes, the
dependent artifact must be rebuild. The list of the files and other
goals that a goal depends upon is the goal's dependencies. The
simpler way of specifying dependencies is via a parenthesised list
immediately following the name of the goal; e.g., if the TeX source
for jokes.dvi is jokes.tex and the latter includes the
file macros.tex, we would write:
(conj:goal "jokes.dvi" (conj:deps "jokes.tex" "macros.tex") <build-spec>)
As we will see, instead of explicitly listing the dependencies, you can use a function, a dependencies procedure, that computes them on the fly. Conjure provides some predefined dependencies procedures, and you can also write your own using Scheme.
Next, we need to tell Conjure how to (re)build the product(s) of a goal.
To that end, we specify a build procedure after the dependencies.
There are many ways of writing such a procedure, and they will be
described in all detail later. By way of example, the keyword
conj:bp:system lets you specify and arbitrary shell command to be
executed by Conjure; in our example we could have:
(conj:goal "jokes.dvi" (conj:deps "jokes.tex" "macros.tex") (conj:bp:system "latex jokes.tex"))
(As an aside, the last snippet shows how white space and newlines in
between arguments is not significant in Conjure files.) The code above
constitutes a complete goal specification that will run the command
latex on jokes.tex whenever the file jokes.dvi is
older that either jokes.tex, macros.tex or both. That is,
Conjure will use timestamp comparison to determine when the goal needs
goal admits also a fourth argument, the
staleness predicate. If provided, it must be a function returning
a boolean value, and it will be invoked by Conjure to determine whether
the load needs rebuilding, instead of comparing timestamps. Again, you
can provide your own staleness predicate or use any of the predefined
ones. For instance, conj:sp:md5 will create a staleness predicate
based on MD5 hashes of the dependencies instead of timestamps:
(conj:goal "jokes.dvi" (conj:deps "jokes.tex" "macros.tex") (conj:bp:system "latex jokes.tex") (conj:sp:md5))
With the basic concepts under our belt (see Conjure lingo), we are ready to write our first fully functional Conjure file. The archetypical use of any build system is the compilation of a C program from its source and, as such, a good starting point for us too.
Let's say we are developing a program called spell, which divines
the kind of a file using some inner magic of its own. spell is
written in C, and its
main function lives in the file
spell.c, which includes spell.h for basic declarations,
opts.h to use the argument parsing functions defined in
opts.c, and magic.h, the interface to the magic functions
in magic.c. In turn, the latter uses alchemy.h to
access the definitions in alchemy.c.
To build spell by hand, we would use the following incantations:
gcc -g -c opts.c gcc -g -c magic.c gcc -g -c incantations.c gcc -g -o spell spell.c opts.o magic.o alchemy.o
Every now and then, we'll want to get a clean slate:
rm -f opts.o magic.o alchemy.o spell
install the executable (after checking that it does not need rebuilding):
cp spell /usr/local/bin/
or make a release tarball of the source project:
mkdir -p spell-0.1 cp spell.h spell.c opts.h opts.c magic.h magic.c \ alchemy.h alchemy.c spell-0.1 tar cfz spell-0.1.tar.gz spell-0.1 rm -r spell-0.1
The aim of Conjure, as of any other build system, is to automate all these taks, and then more.
This section shows how to create a very bare-bones Conjure file to carry out the tasks described in the previous section. It will be an almost verbatim translation of the shell commands above, and, therefore, it will show very little of Conjure's advantages over other build tools. But don't despair: in subsequent sections we will greatly improve it, demonstrating in the process the (often unique) features that Conjure brings to the picture.
Let's begin with a simple Conjure file, named conjure.scm1. Since our main goal is building the spell executable, we'll specify it first:
(conj:goal "spell" (conj:deps "spell.h" "spell.c" "opts.h" "magic.h" "opts.o" "magic.o" "alchemy.o") (conj:bp:system "gcc -g -o spell spell.c opts.o magic.o alchemy.o"))
Here, we're recording the fact that spell is compiled from spell.c, uses the declarations in opts.h and magic.h, and is linked with the object files opts.o, magic.o and alchemy.o. These object files are also compilation products, so we'd better tell Conjure how to build them:
(conj:goal "opts.o" (conj:deps "opts.h" "opts.c") (conj:bp:system "gcc -g -c opts.c")) (conj:goal "magic.o" (conj:deps "magic.h" "alchemy.h" "magic.c") (conj:bp:system "gcc -g -c magic.c")) (conj:goal "alchemy.o" (conj:deps "alchemy.h" "alchemy.c") (conj:bp:system "gcc -g -c alchemy.c"))
Since each object file depends on its corresponding header, we can simplify our original spell goal, deleting the header files from its dependency list:
(conj:goal "spell" (conj:deps "spell.h" "spell.c" "opts.o" "magic.o" "alchemy.o") (conj:bp:system "gcc -g -o spell spell.c opts.o magic.o alchemy.o"))
In order to delete compilation byproducts, we can define a goal which does not produce any artifact and has no dependencies:
(conj:goal (conj:name "clean") (conj:deps) (conj:bp:system/no-error "rm spell opts.o magic.o alchemy.o"))
Note the use of conj:name to name the goal without any product
clean does not create any file) and the empty dependency
list, denoted by (conj:deps). We are also using
conj:bp:system/no-error, which creates a build procedure that invokes
the given system command but does not signal an error if it fails
rm will return an error code if any of its arguments does not
exist, but we won't consider a failure when invoking the goal).
Our next goal,
install, has also no products, but, unlike
clean, it depends on a previous target (
(conj:goal (conj:name "install") (conj:deps "spell") (conj:bp:system "cp spell /usr/local/bin/"))
Finally, the distribution tarball task will let us show the use of conj:bp:system/seq to specify a sequence of shell commands as the build procedure:
(conj:goal (conj:names "dist" "tarball") (conj:product "spell-0.1.tar.gz") (conj:bp:system/seq "mkdir -p spell-0.1" "cp spell.* opts.* magic.* alchemy.* spell-0.1" "tar cfz spell-0.1.tar.gz spell-0.1" "rm -r spell-0.1"))
where we have used conj:names (instead of conj:name) to give this goal more than one name. With all our goals defined, we can invoke Conjure by simply typing
where the optional argument goal is the name of the goal that we
want to build. If invoked without arguments, Conjure will build the
first defined goal. You can change this behaviour by explicitly defining
a default goal in the Conjure file using
Just for reference, this is how our conjure.scm file looks like:
;;; Conjure build file for spell ;;; default goal (conj:default-goal "spell") ;;; auxiliar libraries (conj:goal "opts.o" (conj:deps "opts.h" "opts.c") (conj:bp:system "gcc -g -c opts.c")) (conj:goal "magic.o" (conj:deps "magic.h" "alchemy.h" "magic.c") (conj:bp:system "gcc -g -c magic.c")) (conj:goal "alchemy.o" (conj:deps "alchemy.h" "alchemy.c") (conj:bp:system "gcc -g -c alchemy.c")) ;;; spell executable (conj:goal "spell" (conj:deps "spell.h" "spell.c" "opts.o" "magic.o" "alchemy.o") (conj:bp:system "gcc -g -o spell spell.c opts.o magic.o alchemy.o")) ;;; clean compilation byproducts (conj:goal (conj:name "clean") ; no products and (conj:deps) ; no dependencies (conj:bp:system/no-error "rm spell opts.o magic.o alchemy.o")) ;;; install spell (conj:goal (conj:name "install") (conj:deps "spell") (conj:bp:system "cp spell /usr/local/bin/")) ;;; create a distribution tarball (conj:goal (conj:names "dist" "tarball") (conj:product "spell-0.1.tar.gz") (conj:bp:system/seq "mkdir -p spell-0.1" "cp spell.* opts.* magic.* alchemy.* spell-0.1" "tar cfz spell-0.1.tar.gz spell-0.1" "rm -r spell-0.1"))
The listing above shows how to write comments in a Conjure file: they begin with a semicolon and end with a newline.
Admittedly, our first try at writing a Conjure file (see First iteration) was hardly an improvement over conventional makefiles, not to speak of elegance or efficiency. In this section, we will show how to take advantage of Conjure's features to write a better build specification. In particular, build files are actually Scheme programs, and we can use its abstraction capabilities as we see fit2.
One of the main problems of our build file is repetition: the same names appear explicity in multiple places. Of course, the cure is using variables. Variables can be defined using the Scheme special form define. For instance, let us begin by defining a variable to hold our project version number:
(define version "0.1")
The snippet above defines version to hold a string value. Strings
can be concatenated using
string-append, like this:
(define spell-dir (string-append "spell-" version)) (define tarball-name (string-append spell-dir ".tar.gz"))
With the above definitions, we can rewrite our
dist goal as
(conj:goal (conj:names "dist" "tarball") (conj:product tarball-name) (conj:bp:system/seq (string-append "mkdir" "-p" spell-dir) (string-append "cp" "spell.*" "opts.*" "magic.*" "alchemy.*" spell-dir) (string-append "tar" "cfz" tarball-name spell-dir) (string-append "rm" "-r" spell-dir)))
A second source of repetition in our original Conjure file is the list
of object files. In Scheme, we can create a list of objects using the
(define o-files (list "opts.o" "magic.o" "alchemy.o"))
which, since the list contains only constant values, can also be written as
(define o-files '("opts.o" "magic.o" "alchemy.o"))
To use our brand new variable o-files, we must mention that
conj:bp:system and friends accept, besides strings, a list of
strings specifying the name and arguments of the command to be executed.
Thus, we can rewrite
(conj:goal "spell" (conj:deps "spell.h" "spell.c" o-files) (conj:bp:system (append '("gcc" "-g" "-o" "spell" "spell.c") o-files))) (conj:goal (conj:name "clean") (conj:bp:system/no-error (append '("rm" "spell") o-files)))
You guessed right:
append takes any number of lists as arguments
and returns a new list created appending them; and
accepts also lists as arguments.
A more subtle kind of commonality is that of the definitions of the goals for creating the .o files. Clearly, they all share a common pattern. To begin with, their build procedure is essentially the same, and can be defined only once by means of a implicit build procedure definition. Let us see how.
The keyword conj:auto-build registers a build procedure to be used for creating product files matching a given regular expression from a source file whose name matches a second regular expression. The build procedure will be called with the product and source filenames as arguments. Defining a build procedure for our .o files is easy:
(define (build-o-file ofile cfile) (conj:system (string-append "gcc -g -c " cfile)))
and we can register it for files ending in .o with corresponding source ending in .c as:
(conj:auto-build "(.+)\\.o$" "\\1\\.c$" build-o-file)
or, if you do not want to mess with regular expressions, using conj:auto-build/ext, which simply takes the extensions of the product and target files as its two first arguments:
(conj:auto-build/ext ".o" ".c" build-o-file)
Now, we can rewrite our three goal definitions without needing to specify their build procedure:
(conj:goal "opts.o" (conj:deps "opts.h" "opts.c")) (conj:goal "magic.o" (conj:deps "magic.h" "alchemy.h" "magic.c")) (conj:goal "alchemy.o" (conj:deps "alchemy.h" "alchemy.c"))
We can go a step further by defining a procedure returning the list of dependencies of each object file. Lets call this new procedure object-deps. It will take as arguments the basename of the object file and and optional list of extra dependencies (to handle the magic.o case):
(define (object-deps basename . extra-deps) (conj:deps (string-append basename ".h") (string-append basename ".c") extra-deps))
Note the dot before extra-deps in the above definition: it marks the last argument as optional. With this new definition we can write our three goals as
(conj:goal "opts.o" (object-deps "opts")) (conj:goal "magic.o" (object-deps "magic" "alchemy.h")) (conj:goal "alchemy.o" (object-deps "alchemy"))
While we are at it, we can as well wrap the whole goal definition in a procedure:
(define (object-goal basename . extra-deps) (conj:goal (string-append basename ".o") (object-deps extra-deps)))
and simplify once more our goal definitions to:
(object-goal "opts") (object-goal "magic" "alchemy.h") (object-goal "alchemy")
In a final twist, Scheme savvy users would probably rewrite the above
definitions using the standard function
(for-each object-goal '("opts" "magic" "alchemy") '(() "alchemy.h" ()))
Summing up, the refactorings described so far yield this new Conjure file:
;; Conjure build file for 'spell ;;; shared variables (define version "0.1") (define o-files '("opts.o" "magic.o" "alchemy.o")) ;;; auxiliar procedures (define (build-o-file ofile cfile) (conj:system (string-append "gcc -g -c " cfile))) (define (object-deps basename . extra-deps) (conj:deps (string-append basename ".h") (string-append basename ".c") extra-deps)) (define (object-goal basename . extra-deps) (conj:goal (string-append basename ".o") (object-deps extra-deps))) ;;; default goal (conj:default-goal "spell") ;;; implicit goals (conj:auto-build/ext ".o" ".c" build-o-file) ;;; auxiliar libraries (for-each object-goal '("opts" "magic" "alchemy") '(nil "alchemy.h" nil)) ;;; spell executable (conj:goal "spell" (conj:deps "spell.h" "spell.c" o-files) (conj:bp:system (append '("gcc" "-g" "-o" "spell" "spell.c") o-files))) ;;; clean compilation byproducts (conj:goal (conj:name "clean") (conj:bp:system/no-error (append '("rm" "spell") o-files))) ;;; install spell (conj:goal (conj:name "install") (conj:deps "spell") (conj:bp:system "cp spell /usr/local/bin/")) ;;; create a distribution tarball (define spell-dir (string-append "spell-" version)) (define tarball-name (string-append spell-dir ".tar.gz")) (conj:goal (conj:names "dist" "tarball") (conj:product tarball-name) (conj:bp:system/seq '("mkdir" "-p" spell-dir) '("cp" "spell.*" "opts.*" "magic.*" "alchemy.*" spell-dir) '("tar" "cfz" tarball-name spell-dir) '("rm" "-r" spell-dir)))
In this section we will show more examples of how to use Scheme to customize our build files with new abstractions that can be subsequently used in other projects.
Let's modularize our project by creating a couple of C libraries, namely, libopt.a and libmagic.a, to further encapsulate the functionality offered by the interfaces defined in the headers opts.h and magic.h. In a Unix system, one would typically create a library with a sequence of commands along the lines of
ar c libf.a f1.o f2.o ranlib libf.a
where libf.a is a library containing the object files f1.o and f2.o. Clearly, we can abstract this library creation process using a Scheme procedure. Such a procedure could take two arguments: the library name and a list of object files. A simple implementation could be:
(define (c-library-goal name deps) (let ((lib-name (conj:file:ext "a" name)) (obj-files (conj:file:ext "o" deps))) (for-each obj-goal deps) (conj:goal (conj:product lib-name) (conj:deps obj-files) (conj:bp:system/seq (append '("ar" "c") obj-files) (list "ranlib" lib-name)))))
where we have used Conjure's utitity procedure conj:file:ext, which adds an extension to a file or file list. As for obj-goal, it will be a procedure taking one argument and producing a goal for the respective object file:
(define (obj-goal f) (let ((h-file (conj:file:ext "h" f)) (o-file (conj:file:ext "o" f)) (c-file (conj:file:ext "c" f)) (conj:goal (conj:product o-file) (conj:deps (list h-file c-file))))))
Here, we are relying on our previous implicit build procedure for object files (see Procedures). With these definitions in place, we can define our two libraries simply as:
(c-library-goal "opt" '("opt")) (c-library-goal "magic" '("magic" "alchemy"))
With libraries in place (see Defining C libraries), we can move on to defining a build procedure for C executables. This procedure will take as arguments the name of the executable main file, a list of other dependencies and a list of libraries for linking:
(define (c-exec-goal main deps libs) (let ((o-file (conj:file:ext "o" main)) (c-file (conj:file:ext "c" main)) (h-file (conj:file:ext "h" main))) (conj:goal (conj:product main) (conj:deps o-file h-file deps libs) (conj:bp (lambda () (conj:system (append (list "gcc" "-o" main c-file) libs)))))))
 This is the default file name looked up by Conjure when no other is specified. The .scm extension alludes to the fact that Conjure files actually contain Scheme code.
 In fact, the Conjure file is loaded into a running Scheme interpreter and, therefore, any valid Scheme code can appear in the build specification.
for-each takes as its first argument a
procedure followed by as many lists as arguments accepted by the given
procedure; it then applies the procedure to the first element of each
list, then to the second, and so on and so forth