muddle and its directories¶

About this chapter¶

muddle is three or four different things, entwined:

It is a command line tool, providing useful tools for managing build trees on Linux.
It is a Python package, muddled
It is a simple dependency tracking and analysis tool, with its state made evident via the file structure.
It is a binding of the above to the common tasks required for managing, building and preparing deployments of Linux software from common sources to multiple architectures and platforms.

In a perfect world (perhaps the eventual “muddle 3”) these would be better separated. In particular, it would be nice if it was obvious that muddle can be adapted for things other than Linux build trees.

This chapter is meant to be a longer and more discursive introduction to some of the basics of muddle, with a concentration on how the muddle build tree works as a directory tree, and with worked examples.

If you are just going to use muddle to check out and build an existing build tree, then this is probably much too much information, although the short section An overview of the directory structure may be useful.

If you are going to be developing new muddle builds, then this chapter should enable you to do so more effectively.

If you are going to be reading the muddle source code, or developing muddle itself, then it should explain much of the intent behind muddle’s workings.

Notation in this chapter¶

Directory names are consistently shown with a trailing /, as by ls -F. Since I’m talking about directories and files a lot, I think this makes it easier to tell which is which.
The “tree” listings of directory structures are produced with the command tree, normally as:
```
$ tree -aF -I .svn --noreport
```
i.e., showing files and directories that start with a dot, suffixing directory names with /, not showing .svn/ directories, and not reporting on the number of files and directories. In a few places the “stub” of a .svn/ directory (that is, just its name) may be shown - this should be obvious from context.
When referrring to the parts of a label (type:(domain)name{role}/tag) it is useful to keep the punctuation that indicates that part (so, type:, (domain), name (no punctuation!), {role} and /tag).

Remember that, in the current scheme of things, only package: labels use {role}, and “normal” builds do not use (domain).

An overview of the directory structure¶

Normal practice is to keep a muddle build in its own directory. This is not a requirement, but it helps keep it more obvious that it is a build.

The muddle directories are as follows:

`.muddle/`:	contains information about the build state, and signifies that this is a muddle build.
`src/`:	contains the build description and any other checkouts.
`obj/`:	generated by muddle to hold the results of building packages.
`install/`:	generated by muddle to hold the results of “installing” packages (this will be explained later on).
`deploy/`:	generated by muddle to hold the results of “deploying” things in `install/`.
`versions/`:	generated by muddle to hold the result of `muddle stamp version`.
`domains/`:	present if there are sub-domains, muddle will create this directory if it is needed.

A simple example build¶

The muddle repository has a variety of example build descriptions, of varying complexity.

Perhaps inevitably, not all of the examples are maintained as well as they should be. If you do find problems with any of them, please raise an issue about it on the muddle issues page.

We shall use the “cpio” example.

We start with a new, empty directory:

$ mkdir example
$ cd example

and then use the muddle init command:

$ muddle init svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio builds/01.py
> Make directory /home/tibs/sw/m3/example/.muddle
Initialised build tree in /home/tibs/sw/m3/example
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/builds builds
A    builds/01.py
Checked out revision 458.
> Make directory /home/tibs/sw/m3/example/.muddle/tags/checkout/builds
Done.

This leaves us with two new directories:

$ ls -AF
.muddle/  src/

Or, in more detail (with the .svn/ directory in src/builds/ shown as a “stub”):

.
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           `-- builds/
|               `-- checked_out
`-- src/
    `-- builds/
        |-- 01.py
        |-- 01.pyc
        `-- .svn/

Whilst the name and content of the src/builds directory might differ, this is the normal state of a muddle build tree after the init command.

The `.muddle` directory¶

The .muddle/ directory is, in many ways, the heart of the muddle build tree.

In the first place, it identifies a directory as the top-level of a muddle build. Indeed, the muddle command line tool looks in the current directory, and then upwards through the filesystem, to find a .muddle/ directory. If it cannot find one, then it decides that it is not in a muddle build tree, and behaves accordingly (depending on what action it was asked to do).

The .muddle/ directory always contains at least two files and one directory:

The `Description` file¶

$ cat .muddle/Description
builds/01.py

which should be recognisable from the init command. It tells muddle where its build description was checked out (in the src/ directory, since that is where checkouts go).

You should not normally need to edit this file, but if you do, the next time you run muddle, it will believe that the build description is in the new location.

The `RootRepository` file¶

$ cat .muddle/RootRepository
svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio

Again, this should be recognisable from the init command. This specifies the default repository, which muddle will assume is being used for checkouts unless the build description says otherwise.

You should not normally need to edit this file either, but again muddle will believe its contents after a change.

The `tags/` directory¶

The tags/ directory contains the current state of the build - it is essentially a database of “labels satisfied”, stored in the filesystem.

In the current state:

.muddle/tags
`-- checkout/
    `-- builds/
        `-- checked_out

we can see that the label checkout:builds/checked_out has been “built” (or has had its target satisfied). This corresponds to having checked out the builds checkout, which is indeed what has been done. We shall see later that it is possible to change the build state by deleting or “touch“ing tag files.

The checked_out file created by muddle does have content - it contains a timestamp:

$ cat .muddle/tags/checkout/builds/checked_out
2010-09-28 11:14:34

but this content is never actually used for anything, and if you do add a tag file, it is quite sufficient to use touch to create an empty file.

If we ask muddle what checkouts it knows about:

$ muddle query checkouts
builds
cpio_co

we see two checkouts. The absence of any tag files for the “cpio_co” checkout tells us that it hasn’t been checked out yet.

The `src/builds` directory¶

The other directory present is the src/ directory, which, so far, contains a single subdirectory for the build description.

`src/
 `-- builds/
     |-- 01.py
     |-- 01.pyc
     `-- .svn/

The mechanics of the build - the build description¶

The build description is one or more Python files, which the muddle command runs before doing anything else.

(A few muddle commands do not need a build description, obviously including init, but these are the exception.)

For various reasons, the main build description file is traditionally called src/builds/01.py. muddle automatically adds the directory containing the main build description (the file named in .muddle/Description) to the PYTHONPATH for the build description. Put more simply, the build description can import other Python files in the same directory, which makes it easy to split build descriptions if necessary.

This example build has a fairly simple build description, in src/builds/01.py:

#!  /usr/bin/env python
#
# An example of how to build a cpio archive as a
# deployment - e.g. for a Linux initrd.

import muddled
import muddled.pkgs.make
import muddled.deployments.cpio
import muddled.checkouts.simple

def describe_to(builder):
    # Checkout ..
    muddled.checkouts.simple.relative(builder, "cpio_co")
    muddled.pkgs.make.simple(builder, "pkg_cpio", "x86", "cpio_co")
    muddled.deployments.cpio.deploy(builder, "my_archive.cpio",
                                    {"x86": "/"},
                                    "cpio_dep", [ "x86" ])

    builder.add_default_role("x86")
    builder.by_default_deploy("cpio_dep")

# End file.

Briefly, muddle looks for a function called describe_to() in the build description, and runs it to define the build.

This particular example is quite old, and I’m not sure we’d write it quite like that any more. Regardless, it says the following things:

there is a checkout called “cpio_co”, which is checked out from the default repository (as defined in .muddle/RootRepository).
the package named pkg_cpio in role x86 is built from that checkout - this will, by default, be done using a Makefile in the checkout directory.
when deploying the final results of the build, a CPIO archive shall be created, with the role “x86” going at the root of the filesystem in that CPIO archive. The name of this deployment is “cpio_dep”.
the default package role to build is “x86”
the default thing to deploy is the “cpio_dep” deployment.

Some naming conventions¶

There are some strong naming conventions associated with muddle build trees:

The build description lives in a checkout called src/builds/
All packages build out-of-tree (that is, in the obj/ directory, which we shall describe later), so that the src/ directory only contains the checkout files as-checked-out.
If there is a checkout containing “helper” makefiles and other associated things, used to build checkouts in other directories, it should be called src/helpers/.

(For instance, if one is building kbus, and downloading it directly from its google code repository each time, then it is convenient to put the muddle-specific makefile in a different checkout, possibly as src/helpers/Makefile.kbus)

There was an earlier convention of using the name src/builders/ for such a directory, but this is visually confusing, and does not work well with tab-completion at the bash prompt.
Checkout names should reflect the name of the external package being built, and generally also the version. So, for instance screen-1.0.3.
Package names should be more general - if a package is built from screen-1.0.3, it would normally be named screen. This allows for a later change in the build to use (for instance) checkout screen-1.0.4 instead.
Deployment names should reflect the purpose of the deployment - for instance, firmware versus kernel, and so on. This is very much dependent on the purpose of the build itself.

Note

This particular build is not really following the naming convention for checkouts, packages and deployments as described above. This is because it is trying to make it very clear which name belongs to which label type (thus co_cpio, pkg_cpio and dep_cpio)

It is also moderately traditional to call the main Python file for a build description 01.py (or 02.py if it is version 2, and so on). This is not, however, a strong convention, it just makes the “main” file easy to spot.

Introspection of the dependency tree¶

muddle does have some ability to display the dependency tree that is generated from the build description, although it is not as user-friendly as we would like (and, ultimately, there should be graphical tools).

The simplest tool for dumping the dependency rules is the depends command (see muddle help depends for more details on what it does).

The following shows the dependency rules described directly by the build description above:

$ muddle depends user-short
-----
checkout:builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:builds/up_to_date[T] ]
checkout:builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:builds/changes_committed ]
checkout:builds/pulled <-VcsCheckoutBuilder-- [ checkout:builds/checked_out, checkout:builds/up_to_date[T] ]
checkout:builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:builds/checked_out ]
checkout:cpio_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:cpio_co/changes_committed ]
checkout:cpio_co/pulled <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out, checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out ]
deployment:cpio_dep/deployed <-CpioDeploymentBuilder-- [ package:*{x86}/postinstalled ]
package:pkg_cpio{x86}/built <-MakeBuilder-- [ package:pkg_cpio{x86}/configured ]
package:pkg_cpio{x86}/configured <-MakeBuilder-- [ package:pkg_cpio{x86}/preconfig ]
package:pkg_cpio{x86}/installed <-MakeBuilder-- [ package:pkg_cpio{x86}/built ]
package:pkg_cpio{x86}/postinstalled <-MakeBuilder-- [ package:pkg_cpio{x86}/installed ]
package:pkg_cpio{x86}/preconfig <-MakeBuilder-- [ checkout:cpio_co/checked_out ]
-----

Each line above shows three things:

a target label.
an arrow containing the name of the action to take to “satisfy” or “build” that label (nb: in the current trunk of muddle, this is not present).
a list of the labels that must be “satisfied” or “built” before that action can be performed.

It is sorted by target label, which unfortunately is not too much help, but one can see that, for instance, the deployment:cpio_dep/deployed label depends on package:*{x86}/postinstalled, where the * is a wildcard over all package names in the {x86} role.

A shorter and perhaps more helpful representation of that can be provided using the query deps command:

$ muddle query deps deployment:cpio_dep/deployed
Build order for deployment:cpio_dep/deployed ..
checkout:cpio_co/checked_out
package:pkg_cpio{x86}/preconfig
package:pkg_cpio{x86}/configured
package:pkg_cpio{x86}/built
package:pkg_cpio{x86}/installed
package:pkg_cpio{x86}/postinstalled
deployment:cpio_dep/deployed

This shows all of the labels that must be built before deployment:cpio_dep/deployed, and in what order they will be built.

Checking out the source code¶

Normally, after the init command, one would just use a “bare” muddle command to perform the rest of the checkouts, build them (as they are checked out), and do any other steps indicated by the build description. However, since I’m interested in the various stages of the build tree, I shall do all of these things one by one.

So we start by checking out the actual source for our build:

$ muddle checkout _all
> Building checkout:cpio_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/cpio_co cpio_co
A    cpio_co/hello_world.c
A    cpio_co/Makefile
Checked out revision 458.
> Make directory /home/tibs/sw/m3/example/.muddle/tags/checkout/cpio_co

After doing this, we’ve now gained a new tag in the .muddle/ directory:

`-- .muddle/
    |-- Description
    |-- RootRepository
    `-- tags/
        `-- checkout/
            |-- builds/
            |   `-- checked_out
            `-- cpio_co/
                `-- checked_out

indicating that we have successfully checked out the cpio_co checkout, and the source files for our missing checkout are now present in the src/ directory:

`-- src/
    |-- builds/
    |   |-- 01.py
    |   |-- 01.pyc
    |   `-- .svn/
    `-- cpio_co/
        |-- hello_world.c
        |-- Makefile
        `-- .svn/

(whilst I’m showing the .svn/ directories here, I’ve removed the listing of their contents).

The mechanics of the build - the makefile¶

This build only has a single checkout, which produces a single package. As such it also has a single makefile, src/cpio_co/Makefile:

# Makefile for cpio_co
#
# Just dumps some compiled C into the right place.

INSTALL=install

all:
        $(CC) -o $(MUDDLE_OBJ)/hello_world hello_world.c

install:
        if [ ! -d $(MUDDLE_INSTALL)/bin ]; then mkdir $(MUDDLE_INSTALL)/bin; fi
        $(INSTALL) -m 0755 $(MUDDLE_OBJ)/hello_world $(MUDDLE_INSTALL)/bin/hello_world
        $(INSTALL) -m 0644 hello_world.c $(MUDDLE_INSTALL)/hello_world.c

config:
        @echo Nothing to do

clean:
        rm -f $(MUDDLE_OBJ)/hello_world

distclean: clean
        @echo Distclean is just a clean

# end file.

We saw when querying the builds dependencies (in Introspection of the dependency tree) that various dependency rules had MakeBuilder as their action. This action knows what targets in a makefile to call in order to build a particular package: label tag.

The target label tags (the /xxx at the end of a label) correspond to makefile targets as follows:

Target tag Makefile target

/preconfig <none>

/configured config

/built all

/installed install

/postinstalled <none>

Target tag Makefile target

/clean clean

/distclean distclean

It can be useful to think of a package as “progressing” from /preconfig through to /postinstalled.

preconfig does not correspond to a make target, and there is no default action for it. Things that depend on a package existing (but nothing else) will depend on the /preconfig tagged package label.

There is no action defined by default for /preconfig. That doesn’t stop a particular build from defining one - for instance, ExpandingMakeBuilder is a MakeBuilder subclass which expands a .tgz file into source files for the later stages to compile and install (see muddled.pkgs.make).

Although there is no make target associated with /postinstalled, it is at this stage that pkg-config files are rewritten (if requested), and this is the tag that should be used when another package depends on something being “fully” built.

Building a package¶

Whilst it is possible for a package to be built from more than one checkout, this is relatively uncommon, and it is quite usual for a package to correspond directly to a checkout. As it does in this case.

If one tries to build a package that doesn’t exist, one will be told so:

$ muddle build cpio
Building package:cpio{x86}/postinstalled
There is no rule to build label package:cpio{x86}/postinstalled

It’s simple to find out what packages there are:

$ muddle query packages
pkg_cpio

So let’s build it:

$ muddle build pkg_cpio
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Make directory /home/tibs/sw/m3/example/obj/pkg_cpio/x86
> Make directory /home/tibs/sw/m3/example/install/x86
> Make directory /home/tibs/sw/m3/example/.muddle/tags/package/pkg_cpio
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

We now have two new directories:

$ ls -AFt
install/  obj/  src/  .muddle/

The src/ directory has not changed.

The .muddle/ directory has gained some new tags in .muddle/tags/package/pkg_cpio/, reflecting the new state of the build:

.muddle/
|-- Description
|-- RootRepository
`-- tags/
    |-- checkout/
    |   |-- builds/
    |   |   `-- checked_out
    |   `-- cpio_co/
    |       `-- checked_out
    `-- package/
        `-- pkg_cpio/
            |-- x86-built
            |-- x86-configured
            |-- x86-installed
            |-- x86-postinstalled
            `-- x86-preconfig

The obj/ directory contains the results of building packages. Thus we have a new directory called obj/pkg_cpio/x86.

obj/
`-- pkg_cpio/
    `-- x86/
        `-- hello_world*

Specifically, within obj/ there is a directory named after each package, within which there is a directory named after each role for that package. For convenience, the makefile can use the environment variable $(MUDDLE_OBJ) to refer to <root_dir>/obj/pkg_cpio/x86 (<root_dir> indicates the absolute path to the top-level build directory).

There would normally be more use of subdirectories such as $(MUDDLE_OBJ)/bin in a “real” build, and muddle provides some other environment variables to help with these.

The install/ contains the results of installing packages. Each {role} gets a separate directory in install/:

install/
`-- x86/
    |-- bin/
    |   `-- hello_world*
    `-- hello_world.c

Again, the makefile can use the environment variable $(MUDDLE_INSTALL) to refer to this role-specific directory.

As an aside, using the MUDDLE environment variables in this way means that if two packages (e.g., package:a{b}/* versus package:c{d}/*, or even package:a{b}/* versus package:a{d}/*) both use the same checkout, and thus the same makefile, the results of building the two packages will still end up in different, predictable directories.

It is possible to find out the entire environment muddle passes to a makefile with the makeenv query:

$ muddle query makeenv package:pkg_cpio{x86}/built
MUDDLE=/home/tibs/sw/m3/muddle3_labels/muddle/muddled/__main__.py
MUDDLE_INCLUDE_DIRS=
MUDDLE_INSTALL=/home/tibs/sw/m3/example/install/x86
MUDDLE_INSTRUCT=/home/tibs/sw/m3/muddle3_labels/muddle/muddled/__main__.py instruct pkg_cpio{x86}
MUDDLE_KIND=package
MUDDLE_LABEL=package:pkg_cpio{x86}/built
MUDDLE_LD_LIBRARY_PATH=
MUDDLE_LIB_DIRS=
MUDDLE_NAME=pkg_cpio
MUDDLE_OBJ=/home/tibs/sw/m3/example/obj/pkg_cpio/x86
MUDDLE_OBJ_INCLUDE=/home/tibs/sw/m3/example/obj/pkg_cpio/x86/include
MUDDLE_OBJ_LIB=/home/tibs/sw/m3/example/obj/pkg_cpio/x86/lib
MUDDLE_OBJ_OBJ=/home/tibs/sw/m3/example/obj/pkg_cpio/x86/obj
MUDDLE_PKGCONFIG_DIRS=
MUDDLE_PKGCONFIG_DIRS_AS_PATH=
MUDDLE_ROLE=x86
MUDDLE_ROOT=/home/tibs/sw/m3/example
MUDDLE_SRC=/home/tibs/sw/m3/example/src/cpio_co
MUDDLE_TAG=built
MUDDLE_TARGET_LOCATION=/
MUDDLE_UNINSTRUCT=/home/tibs/sw/m3/muddle3_labels/muddle/muddled/__main__.py instruct pkg_cpio{x86}

Builds specifying cross-compilation toolchains and other options may have much longer environments passed down.

Deployment¶

Deployment is the process of copying the files for the various roles from the install/ directories to the deploy/ directories.

Deployment is commonly the stage that prepares the “blob” that will be put onto the final device (if one is building an embedded device), and perhaps also aggregates the tools for doing so.

In this build, we are producing a CPIO archive.

(CPIO is a very old Unix archive format. For our purposes (“us” being muddle), its advantage is that it is possible to flag files within the archive with particular permission bits, to request creationg of device nodes, and other thing that would require superuser privileges if done “live” in the build tree.)

In this build we only have a single deployment target, so we don’t need to specify its name.

$ muddle deploy
Building deployment:cpio_dep/deployed
> Building deployment:cpio_dep/deployed
> Make directory /home/tibs/sw/m3/example/deploy/cpio_dep
Collecting package:*{x86}/*  for deployment to / ..
h = ---Roots---
/ -> [ / (fs /home/tibs/sw/m3/example/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c]
---Map---
/bin/hello_world -> [ /bin/hello_world (fs /home/tibs/sw/m3/example/install/x86/bin/hello_world) mode = 100755 uid = 7007 gid = 7007 kids = ]
/ -> [ / (fs /home/tibs/sw/m3/example/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c]
/bin -> [ /bin (fs /home/tibs/sw/m3/example/install/x86/bin) mode = 40755 uid = 7007 gid = 7007 kids = /bin/hello_world]
/hello_world.c -> [ /hello_world.c (fs /home/tibs/sw/m3/example/install/x86/hello_world.c) mode = 100644 uid = 7007 gid = 7007 kids = ]
---

base = /
Scanning instructions for role x86, domain None ..
> Writing /home/tibs/sw/m3/example/deploy/cpio_dep/my_archive.cpio ..
> Packing / ..
> Packing /bin ..
> Packing /bin/hello_world ..
> Packing /hello_world.c ..
> Packing TRAILER!!! ..
> Make directory /home/tibs/sw/m3/example/.muddle/tags/deployment/cpio_dep

That leaves us with a new deploy/ directory:

$ ls -AFt
deploy/  install/  obj/  src/  .muddle/

containing our CPIO archive:

deploy/
`-- cpio_dep/
    `-- my_archive.cpio

and the appropriate tag has been set in the .muddle/ directory tree:

.muddle/tags/deployment/
`-- cpio_dep/
    `-- deployed

Rebuilding things¶

muddle provides a convenience command to rebuild a package:

$ muddle rebuild pkg_cpio
Killing package:pkg_cpio{x86}/built
Clearing tags: package:pkg_cpio{x86}/built package:pkg_cpio{x86}/installed package:pkg_cpio{x86}/postinstalled package:pkg_cpio{x86}/built
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

As you can see, this “kills” (deletes) the tags saying that the package has been built, installed and postinstalled, and then rebuilds the /postinstalled tag. This does not, however, do anything about things that depend on this package.

It is frequently more useful to use the distrebuild command, which is a conflation of the distclean and build commands:

$ muddle distrebuild pkg_cpio
Building: package:pkg_cpio{x86}/distclean ..
> Building package:pkg_cpio{x86}/distclean[T]
> make  -f Makefile distclean
rm -f /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world
Distclean is just a clean
Killing: package:pkg_cpio{x86}/preconfig ..
Clearing tags: package:pkg_cpio{x86}/preconfig package:pkg_cpio{x86}/configured package:pkg_cpio{x86}/preconfig package:pkg_cpio{x86}/installed package:pkg_cpio{x86}/postinstalled package:pkg_cpio{x86}/built
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

The distclean also removes the tags for any packages that depended upon this package (not very obvious in this case where we only had one, of course).

(Beware - it unsets the state of other packages, but not the deployment.)

You can also, quite legally, do the same thing by direct manipulation of the tag files in the .muddle/ directories:

$ rm .muddle/tags/package/pkg_cpio/*
$ muddle build pkg_cpio
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

That is, muddle doesn’t care if you remove the tags directly, instead of via the command line tool.

You can even do things like:

$ rm .muddle/tags/checkout/builds/checked_out
$ muddle checkout _all
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/builds builds
Checked out revision 459.

This is quite deliberate. Whilst the muddle command line tool provides a variery of useful shorthand commands (such as “distclean”, “distrebuild” and “redeploy”), it is quite possible and even sensible to exert finer control over a build by deleting (or, indeed, “touch”ing) tag files in the .muddle directory.

Do what I say, not what I do¶

Please note that although I have been explicitly requesting indiviual builds, rebuilds, deployments and so on above, this is not necessarily the normal way of using muddle. The muddle command line tool is carefully designed so that it normally does “the right thing”, according to which directory you are in.

So, if one is at the top level of the build, and has checked out everything, it is more colloquial just to give the muddle command with no arguments than to be overly specific.

This is perhaps most easily shown by, well, showing it.

We can revert to the just checked out state quite easily, by just removing everything except the .muddle/ and src/ directories:

$ ls -AF
deploy/  install/  .muddle/  obj/  src/
$ rm -rf deploy/ install/ obj/
$ rm -rf .muddle/tags/package/ .muddle/tags/deployment/

Once we’re back to the stage just after init, we can just do:

$ muddle
Building deployment:cpio_dep/deployed
> Building package:pkg_cpio{x86}/preconfig
...
> Building package:pkg_cpio{x86}/configured
...
> Building package:pkg_cpio{x86}/built
...
> Building package:pkg_cpio{x86}/installed
...
> Building package:pkg_cpio{x86}/postinstalled
> Building deployment:cpio_dep/deployed
...
> Writing /home/tibs/sw/m3/example/deploy/cpio_dep/my_archive.cpio ..
...
> Make directory /home/tibs/sw/m3/example/.muddle/tags/deployment/cpio_dep

(I’ve shortened the log above, because by now it should be all too familiar.)

Contrariwise, if we clean it all up again:

$ rm -rf deploy/ install/ obj/
$ rm -rf .muddle/tags/package/ .muddle/tags/deployment/

and go into a particular checkout directory:

$ pushd src/cpio_co
$ muddle
Killing package:pkg_cpio{x86}/built
Clearing tags: package:pkg_cpio{x86}/built package:pkg_cpio{x86}/installed package:pkg_cpio{x86}/postinstalled package:pkg_cpio{x86}/built
Building package:pkg_cpio{x86}/*
> Building package:pkg_cpio{x86}/distclean[T]
> Make directory /home/tibs/sw/m3/example/obj/pkg_cpio/x86
> Make directory /home/tibs/sw/m3/example/install/x86
> make  -f Makefile distclean
rm -f /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world
Distclean is just a clean
> Building package:pkg_cpio{x86}/clean[T]
> make  -f Makefile clean
rm -f /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world
> Building package:pkg_cpio{x86}/preconfig
> Make directory /home/tibs/sw/m3/example/.muddle/tags/package/pkg_cpio
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled
$ popd

That is, muddle only rebuilds the packages that depend on this checkout. See muddle help build and its friends for more information on this.

People who like to cd around the filesystem a lot will probably find this intuitive. Those of us who prefer to sit at the top-level and work from there (not necessarily any more sensible, of course) are more likely just to do:

$ muddle distrebuild pkg_cpio

(yes, this has identical effect, and no, by now I’m not going to include the listing yet again).

Note

As an aside, if you want to preserve the source of a build, then (if there aren’t any domains) it is sufficient to do:

$ tar -zcvf build.tgz .muddle/Description .muddle/RootRepository .muddle/tags/checkout src

However, muddle provides an alternative that will work in all cases, including when there are subdomains:

  $ muddle distribute --with-vcs _source_release ../save_dir
  $ cd ..
  $ tar -zcvf build.tgz save_dir

(where ``save_dir`` should be replaced by something more informative).

Adding a new checkout/package¶

Let’s suppose we want to add a new package, based on a new checkout. There is more than one way to do this, but the following shows a sensible approach.

We shall create our new package by just copying one we’ve already got:

$ cp -a src/cpio_co src/fred

We must remember to remove the outdated repository information:

$ rm -rf src/fred/,svn/

And make this checkout do something different:

$ mv src/fred/hello_world.c src/fred/bye_world.c
$ sed -e s/hello_world/bye_world/g --in-place src/fred/Makefile
$ sed -e s/Hello/Byebye/g --in-place src/fred/bye_world.c

muddle has no knowledge of this checkout yet, so we need to add it to the build description. We can simply edit src/builds/01.py to add the lines:

muddled.checkouts.simple.relative(builder, "fred")

and:

muddled.pkgs.make.simple(builder, "fred", "x86", "fred")

(yes, that leaves us with a package and checkout with the same name, but that’s not a problem, and is closer to the more conventional usage).

That’s enough to give us:

$ muddle query checkouts
builds
cpio_co
fred

and:

$ muddle query deps deployment:cpio_dep/deployed
Build order for deployment:cpio_dep/deployed ..
checkout:fred/checked_out
checkout:cpio_co/checked_out
package:pkg_cpio{x86}/preconfig
package:fred{x86}/preconfig
package:pkg_cpio{x86}/configured
package:fred{x86}/configured
package:fred{x86}/built
package:pkg_cpio{x86}/built
package:fred{x86}/installed
package:pkg_cpio{x86}/installed
package:pkg_cpio{x86}/postinstalled
deployment:cpio_dep/deployed

However, if we were to try to build:

$ muddle
Building deployment:cpio_dep/deployed
> Building checkout:fred/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/fred fred
svn: URL 'http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/fred' doesn't exist
Can't build deployment:cpio_dep/deployed - Command 'svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/fred fred' execution failed - 1

This makes sense, because (a) we have not told muddle that the new checkout is present in its directory structure, and (b) we have not actually checked it in to the subversion repository it wants to look in.

Since we’ve not yet finished developing this new package, we don’t want to check it in to the repository yet (and, in particular in our case, we do not want to permanently add it to our example).

So we need to make muddle believe that it has been checked out.

As you might expect, there are two ways to do this. The one that you may guess is to manipulate the .muddle/` directory structure directly - so we could just do:

$ mkdir .muddle/tags/checkout/fred
$ touch .muddle/tags/checkout/fred/checked_out

However, that’s a bit clumsy to type, so muddle provides a convenient command for asserting a particular tag (just assuming you are comfortable with the label syntax):

$ muddle assert checkout:fred/checked_out
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/checkout/fred

and now the tag file is present:

$ ls .muddle/tags/checkout/fred
checked_out

and we can then do:

$ muddle
Building deployment:cpio_dep/deployed
> Building package:fred{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/obj/fred/x86
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/package/fred
> Building package:fred{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:fred{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/obj/fred/x86/bye_world bye_world.c
> Building package:fred{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example3/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example3/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example3/obj/fred/x86/bye_world /home/tibs/sw/m3/example3/install/x86/bin/bye_world
install -m 0644 bye_world.c /home/tibs/sw/m3/example3/install/x86/bye_world.c
> Building deployment:cpio_dep/deployed
> Make directory /home/tibs/sw/m3/example3/deploy/cpio_dep
Collecting package:*{x86}/*  for deployment to / ..
h = ---Roots---
/ -> [ / (fs /home/tibs/sw/m3/example3/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c /bye_world.c]
---Map---
/bin/bye_world -> [ /bin/bye_world (fs /home/tibs/sw/m3/example3/install/x86/bin/bye_world) mode = 100755 uid = 7007 gid = 7007 kids = ]
/bin -> [ /bin (fs /home/tibs/sw/m3/example3/install/x86/bin) mode = 40755 uid = 7007 gid = 7007 kids = /bin/bye_world /bin/hello_world]
/hello_world.c -> [ /hello_world.c (fs /home/tibs/sw/m3/example3/install/x86/hello_world.c) mode = 100644 uid = 7007 gid = 7007 kids = ]
/bin/hello_world -> [ /bin/hello_world (fs /home/tibs/sw/m3/example3/install/x86/bin/hello_world) mode = 100755 uid = 7007 gid = 7007 kids = ]
/bye_world.c -> [ /bye_world.c (fs /home/tibs/sw/m3/example3/install/x86/bye_world.c) mode = 100644 uid = 7007 gid = 7007 kids = ]
/ -> [ / (fs /home/tibs/sw/m3/example3/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c /bye_world.c]
---

base = /
Scanning instructions for role x86, domain None ..
> Writing /home/tibs/sw/m3/example3/deploy/cpio_dep/my_archive.cpio ..
> Packing / ..
> Packing /bin ..
> Packing /bin/bye_world ..
> Packing /bin/hello_world ..
> Packing /hello_world.c ..
> Packing /bye_world.c ..
> Packing TRAILER!!! ..
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/deployment/cpio_dep

This just leaves actually adding the new package to revision control somewhere - in this case, the build description is saying (implicitly) that this package is stored in the same subversion repository as the rest of the example, and so one would need to add it there by the normal means (which is a topic for another time).

(The fact that this is much easier to do for distributed revision control systems like bazaar, mercurial or git is in itself a good reason for using them!)

Version stamps¶

Version stamps allow one to produce a simple text file (actually, an INI file) representing the current state of a build. This can be useful for various purposes, but its main intent is to allow saving a build state so that it can be accurately recreated at a later date (for instance, so one can rebuild an earlier release to customers).

The muddle stamp and unstamp commands have documentation via muddle help stamp and help unstamp.

However, it is worth giving a quick example of creating a version stamp:

$ muddle stamp version
Finding all checkouts... found 2
Processing Svn checkout 'builds'
Processing Svn checkout 'cpio_co'
Creating directory /home/tibs/sw/m3/example/versions
Writing to /home/tibs/sw/m3/example/versions/_temporary.stamp
Wrote revision data to /home/tibs/sw/m3/example/versions/_temporary.stamp
File has SHA1 hash 189085d413217cb1865868b5d9916085d22e6f50
Renaming /home/tibs/sw/m3/example/versions/_temporary.stamp to /home/tibs/sw/m3/example/versions/01.stamp

As indicated above, we now have a new versions/ directory:

$ ls -AFt
versions/  deploy/  install/  obj/  src/  .muddle/

containing the stamp file:

versions
`-- 01.stamp

Modern build descriptions are encouraged to specify a build name (this is as simple as adding a line of the form:

builder.build_name = 'ExampleBuild'

to the Python code). If this is given, then the version stamp file will use that build name as its filename, otherwise it defaults to the filename of the main build description file (01 in this case).

The versions/ directory is suitable for putting into a version control system, as one might expect, and future versions of muddle are likely to provide more support for handling this (see issue 117 for some ideas on this).

The contents of a stamp file is deliberately human readable:

[ROOT]
description = builds/01.py
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio

[CHECKOUT builds]
name = builds
relative = builds
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
revision = 458

[CHECKOUT cpio_co]
name = cpio_co
relative = cpio_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
revision = 458

As normal. muddle will not stop you writing or editing stamp files. This may conceivably be useful.

Domains¶

Domains are part of advanced muddle use, and are probably not relevant to most builds. Regardless, it is probably worth reading at least What domains are for from this section, as useful background, and so that you can recognise when they might be applicable.

The way domains work follows naturally from the rest of muddle, but the implementation is still being tidied up - hence the muddle3_label branch.

What domains are for¶

Domains allow one to include one build description inside another, in much the same way as a Python module can import another.

A simple example might be when building a system with WebKit and X11 support. We might have two builds, constructed as follows:

def UI_Build:
   import WebKit_build
       import X11_build

and:

def Login_build:
    import X11_build

Alternatively, we might have some high-level software for handling internet television, and want to build two systems on different architectures. In this case, our contrasting builds might be:

def IPTV_stack:
    import ARM_based_stack

and:

def IPTV_stack:
    import MIPS_based_stack

How domains work¶

A build includes another build (a subdomain) using the include_domain() function. This takes the same arguments as the muddle init command (that is, a repository specification and a build description therein), plus a name for the domain.

The following sequence of actions is then performed:

Create a directory called domains/<name>/, where <name> is the name of the new domain.
cd into that new directory, and perform (the equivalent of) a muddle init command therein, using the repository specification and build description given.
Read in this new build, giving a stand-alone build tree datastructure.
Find all of the labels in the new build tree datastructures, and change them to add the domain name. So, for instance, checkout:fred/checked_out would become checkout:(<name>)fred/checked_out.
Incorporate this amended build tree datastructure into the original (“top level”) build.
Create a .muddle/am_subdomain file in the domain. This allows muddle to distinguish the actual (final) top-level build from any subdomains within it, which is a useful optimisation.

Note that the build description for a domain does not itself know that it is not the top-level of a muddle build. Indeed, this is an important property of domains - it means that any build description can potentially be included in any other.

For most purposes, the subdomain works just as if it were a “normal” top-level build. In particular, it uses its own .muddle/ directory to record its build state, and writes things to its own obj/ and install/ directories.

Also note that, as a consequence of the way domains work (and this is a good thing), it is only possible for a build to refer to labels in subdomains, not to those in sibling or parent build trees.

To allow for more flexibility in how domains are used, there are tools in the muddled package for manipulating the merged label-space. For instance, if one is using subdomains that originated as two different set-top-box builds, both might provide a linux kernel, and the top-level build probably wants to amend the labels such that both subdomains use one of the kernel packages (broadly, saying “this label should be used instead of that label”).

A worked example of using domains¶

So, we shall start in a new empty directory:

$ cd ..
$ mkdir example2
$ cd example2

and start with the same build that we used before:

$ muddle init svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio builds/01.py
> Make directory /home/tibs/sw/m3/example2/.muddle
Initialised build tree in /home/tibs/sw/m3/example2
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example2/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/builds builds
A    builds/01.py
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example2/.muddle/tags/checkout/builds
Done.

So, just as before, we now have the following directory structure:

.
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           `-- builds/
|               `-- checked_out
`-- src/
    `-- builds/
        |-- 01.py
        `-- 01.pyc

We can now edit the src/builds/01.py file to include a subdomain as part of the build. As described above, we need to specify how to retrieve the domain, giving the same information as for the “muddle init” command. We shall call our domain b (after the example we’re taking it from):

from muddled.mechanics import include_domain
include_domain(builder,
               domain_name = "b",
               domain_repo = "svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b",
               domain_desc = "builds/01.py")

We also have to say how to include the result into our deployment - a simple way to do this is:

import muddled.deployments.collect as collect
collect.deploy(builder, "everything")
collect.copy_from_role_install(builder, "everything",
                               role = "x86",
                               rel = "", dest = "",
                               domain = None)
collect.copy_from_role_install(builder, "everything",
                               role = "x86",
                               rel = "", dest = "usr",
                               domain = "b")

and then deploy the new deployment “everything”, instead of the old “cpio_dep”. Note that we’re not deploying to a CPIO archive this time, but just as normal files.

We should probably also change the introductory comment to:

# An example of building with a subdomain

So the “top level” build description is now:

#!  /usr/bin/env python
#
# An example of building with a subdomain

import muddled
import muddled.pkgs.make
import muddled.deployments.cpio
import muddled.checkouts.simple
import muddled.deployments.collect as collect
from muddled.mechanics import include_domain

def describe_to(builder):
    # Checkout ..
    muddled.checkouts.simple.relative(builder, "cpio_co")
    muddled.pkgs.make.simple(builder, "pkg_cpio", "x86", "cpio_co")

    include_domain(builder,
                   domain_name = "b",
                   domain_repo = "svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b",
                   domain_desc = "builds/01.py")

    collect.deploy(builder, "everything")
    collect.copy_from_role_install(builder, "everything",
                                   role = "x86",
                                   rel = "", dest = "",
                                   domain = None)
    collect.copy_from_role_install(builder, "everything",
                                   role = "x86",
                                   rel = "", dest = "usr",
                                   domain = "b")

    builder.add_default_role("x86")
    builder.by_default_deploy("everything")

# End file.

If we now do a “checkout _all”:

$ muddle checkout _all
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle
Initialised build tree in /home/tibs/sw/m3/example3/domains/b
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example3/domains/b/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/b/builds builds
A    builds/01.py
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/checkout/builds
There is no rule to build label checkout:b_co/checked_out
> Building checkout:cpio_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/cpio_co cpio_co
A    cpio_co/hello_world.c
A    cpio_co/Makefile
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/checkout/cpio_co

giving us:

$ ls -AF
domains/  .muddle/  src/

or, in more detail (omitting .svn directories):

.
|-- domains/
|   `-- b/
|       |-- .muddle/
|       |   |-- am_subdomain
|       |   |-- Description
|       |   |-- RootRepository
|       |   `-- tags/
|       |       `-- checkout/
|       |           `-- builds/
|       |               `-- checked_out
|       `-- src/
|           `-- builds/
|               |-- 01.py
|               `-- 01.pyc
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           |-- builds/
|           |   `-- checked_out
|           `-- cpio_co/
|               `-- checked_out
`-- src/
    |-- builds/
    |   |-- 01.py
    |   `-- 01.pyc
    `-- cpio_co/
        |-- hello_world.c
        `-- Makefile

Apart from the am_subdomain file in its .muddle/ directory, the domains/b/ directory looks like a perfectly normal build.

If we then do:

$ muddle build _all
Building package:(b)pkg_b{x86}/postinstalled package:pkg_cpio{x86}/postinstalled
> Building checkout:(b)b_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/b/b_co b_co
A    b_co/hello_world.c
A    b_co/Makefile
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/checkout/b_co
> Building package:(b)pkg_b{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86
> Make directory /home/tibs/sw/m3/example3/domains/b/install/x86
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/package/pkg_b
> Building package:(b)pkg_b{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:(b)pkg_b{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86/hello_world hello_world.c
> Building package:(b)pkg_b{x86}/installed
> make  -f Makefile install
install -m 0755 /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86/hello_world /home/tibs/sw/m3/example3/domains/b/install/x86/hello_world
> Building package:(b)pkg_b{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/obj/pkg_cpio/x86
> Make directory /home/tibs/sw/m3/example3/install/x86
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/package/pkg_cpio
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example3/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example3/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example3/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example3/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example3/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

we end up with the subdomain built in its directory structure:

domains/
`-- b/
    |-- install/
    |   `-- x86/
    |       `-- hello_world*
    |-- .muddle/
    |   |-- am_subdomain
    |   |-- Description
    |   |-- RootRepository
    |   `-- tags/
    |       |-- checkout/
    |       |   |-- b_co/
    |       |   |   `-- checked_out
    |       |   `-- builds/
    |       |       `-- checked_out
    |       `-- package/
    |           `-- pkg_b/
    |               |-- x86-built
    |               |-- x86-configured
    |               |-- x86-installed
    |               |-- x86-postinstalled
    |               `-- x86-preconfig
    |-- obj/
    |   `-- pkg_b/
    |       `-- x86/
    |           `-- hello_world*
    `-- src/
        |-- b_co/
        |   |-- hello_world.c
        |   `-- Makefile
        `-- builds/
            |-- 01.py
            `-- 01.pyc

and the top-level build in its:

`-- install/
|   `-- x86/
|       |-- bin/
|       |   `-- hello_world*
|       `-- hello_world.c
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       |-- checkout/
|       |   |-- builds/
|       |   |   `-- checked_out
|       |   `-- cpio_co/
|       |       `-- checked_out
|       `-- package/
|           `-- pkg_cpio/
|               |-- x86-built
|               |-- x86-configured
|               |-- x86-installed
|               |-- x86-postinstalled
|               `-- x86-preconfig
|-- obj/
|   `-- pkg_cpio/
|       `-- x86/
|           `-- hello_world*
`-- src/
    |-- builds/
    |   |-- 01.py
    |   `-- 01.pyc
    `-- cpio_co/
        |-- hello_world.c
        `-- Makefile

If we ask after packages, we have both sets:

$ muddle query packages
pkg_b
pkg_cpio

(this should arguably report the domain of each package name as well - there should probably be an option to select this).

If we ask what domains we have, we get:

$ muddle query domains

b

(the current printout of domains includes the “empty” or top-level domain in its listing, which is slightly unobvious, and will probably be fixed at some time).

Our dependency rules are now extended to include those from the subdomain as well:

$ muddle depend user-short
-----
checkout:builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:builds/up_to_date[T] ]
checkout:builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:builds/changes_committed ]
checkout:builds/pulled <-VcsCheckoutBuilder-- [ checkout:builds/checked_out, checkout:builds/up_to_date[T] ]
checkout:builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:builds/checked_out ]
checkout:cpio_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:cpio_co/changes_committed ]
checkout:cpio_co/pulled <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out, checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out ]
checkout:(b)b_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:(b)b_co/up_to_date[T] ]
checkout:(b)b_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:(b)b_co/changes_committed ]
checkout:(b)b_co/pulled <-VcsCheckoutBuilder-- [ checkout:(b)b_co/checked_out, checkout:(b)b_co/up_to_date[T] ]
checkout:(b)b_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:(b)b_co/checked_out ]
checkout:(b)builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:(b)builds/up_to_date[T] ]
checkout:(b)builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:(b)builds/changes_committed ]
checkout:(b)builds/pulled <-VcsCheckoutBuilder-- [ checkout:(b)builds/checked_out, checkout:(b)builds/up_to_date[T] ]
checkout:(b)builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:(b)builds/checked_out ]
deployment:everything/deployed <-CollectDeploymentBuilder-- [ package:*{x86}/postinstalled, package:(b)*{x86}/postinstalled ]
package:pkg_cpio{x86}/built <-MakeBuilder-- [ package:pkg_cpio{x86}/configured ]
package:pkg_cpio{x86}/configured <-MakeBuilder-- [ package:pkg_cpio{x86}/preconfig ]
package:pkg_cpio{x86}/installed <-MakeBuilder-- [ package:pkg_cpio{x86}/built ]
package:pkg_cpio{x86}/postinstalled <-MakeBuilder-- [ package:pkg_cpio{x86}/installed ]
package:pkg_cpio{x86}/preconfig <-MakeBuilder-- [ checkout:cpio_co/checked_out ]
package:(b)pkg_b{x86}/built <-MakeBuilder-- [ package:(b)pkg_b{x86}/configured ]
package:(b)pkg_b{x86}/configured <-MakeBuilder-- [ package:(b)pkg_b{x86}/preconfig ]
package:(b)pkg_b{x86}/installed <-MakeBuilder-- [ package:(b)pkg_b{x86}/built ]
package:(b)pkg_b{x86}/postinstalled <-MakeBuilder-- [ package:(b)pkg_b{x86}/installed ]
package:(b)pkg_b{x86}/preconfig <-MakeBuilder-- [ checkout:(b)b_co/checked_out ]
-----

and our deployment has the expected build order:

$ muddle query deps deployment:everything/deployed
Build order for deployment:everything/deployed ..
checkout:cpio_co/checked_out
checkout:(b)b_co/checked_out
package:(b)pkg_b{x86}/preconfig
package:pkg_cpio{x86}/preconfig
package:(b)pkg_b{x86}/configured
package:pkg_cpio{x86}/configured
package:(b)pkg_b{x86}/built
package:pkg_cpio{x86}/built
package:(b)pkg_b{x86}/installed
package:pkg_cpio{x86}/installed
package:(b)pkg_b{x86}/postinstalled
package:pkg_cpio{x86}/postinstalled
deployment:everything/deployed

Which means that when we deploy:

$ muddle deploy
Building deployment:everything/deployed
> Building deployment:everything/deployed
> Make directory /home/tibs/sw/m3/example3/deploy/everything
Copying /home/tibs/sw/m3/example3/install/x86/ to /home/tibs/sw/m3/example3/deploy/everything/
Copying /home/tibs/sw/m3/example3/domains/b/install/x86/ to /home/tibs/sw/m3/example3/deploy/everything/usr
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/deployment/everything

we end up with everything deployed in the deploy/ directory at the top-level:

deploy/
`-- everything/
    |-- bin/
    |   `-- hello_world*
    |-- hello_world.c
    `-- usr/
        `-- hello_world*

bin/hello_world comes from the top-level build, and usr/hello_world from the subdomain. The hello_world.c is also from the top-level build (the example Makefile for the cpio package copies the source file into the install/ directory).

Version stamps and domains¶

Version stamps work with domains as well, although one may only generate them for the top-level in a particular build tree.

Thus we can try:

$ muddle stamp version
Finding all checkouts... found 4
Processing Svn checkout 'builds'
builds: 'svnversion' reports checkout has revision '459M'
Processing Svn checkout 'cpio_co'
Processing Svn checkout '(b)b_co'
Processing Svn checkout '(b)builds'
Found domains: set([DomainTuple(name='b', repository='svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b', description='builds/01.py')])

Unable to work out revision ids for all the checkouts
- although we did work out 3 of 4
Problems were:
* builds: 'svnversion' reports checkout has revision '459M'
Problems prevent writing version stamp file

OK, that makes sense, since we had indeed edited that checkout, and had not commited the changes. Luckily, it is sufficient for our purposes to save the state without that edit:

$ muddle stamp save -force
Forcing original revision ids when necessary
Finding all checkouts... found 4
Processing Svn checkout 'builds'
builds: 'svnversion' reports checkout has revision '459M'
Processing Svn checkout 'cpio_co'
Processing Svn checkout '(b)b_co'
Processing Svn checkout '(b)builds'
Found domains: set([DomainTuple(name='b', repository='svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b', description='builds/01.py')])

Unable to work out revision ids for all the checkouts
- although we did work out 3 of 4
Problems were:
* builds: 'svnversion' reports checkout has revision '459M'
Writing to working.stamp
Wrote revision data to working.stamp
File has SHA1 hash 6425284f002c81c9849f2f23add025a710207509
Renaming working.stamp to 6425284f002c81c9849f2f23add025a710207509.partial

and that at least shows us that the stamp file does record details of any subdomains:

[ROOT]
description = builds/01.py
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio

[DOMAIN b]
description = builds/01.py
name = b
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b

[CHECKOUT (b)b_co]
domain = b
name = b_co
relative = b_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
revision = 459

[CHECKOUT (b)builds]
domain = b
name = builds
relative = builds
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
revision = 459

[CHECKOUT cpio_co]
name = cpio_co
relative = cpio_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
revision = 459

[PROBLEMS]
problem1 = builds: 'svnversion' reports checkout has revision '459M'

muddle and its directories¶

About this chapter¶

Notation in this chapter¶

An overview of the directory structure¶

A simple example build¶

The .muddle directory¶

The Description file¶

The RootRepository file¶

The tags/ directory¶

Other contents¶

The src/builds directory¶

The mechanics of the build - the build description¶

Some naming conventions¶

Introspection of the dependency tree¶

Checking out the source code¶

The mechanics of the build - the makefile¶

Building a package¶

Deployment¶

Rebuilding things¶

Do what I say, not what I do¶

Adding a new checkout/package¶

Version stamps¶

Domains¶

What domains are for¶

How domains work¶

A worked example of using domains¶

Version stamps and domains¶

The `.muddle` directory¶

The `Description` file¶

The `RootRepository` file¶

The `tags/` directory¶

The `src/builds` directory¶