Muddle workflows - or how to use muddle in real life

This section is intended to give some examples of how one actually uses muddle in normal programming life.

If you are using weld, see the chapter on working with weld.

Working with version control systems

or, “Should I use ``muddle push/pull`` or ``git push/pull``?”

Each checkout uses a particular VCS (above we’re assuming it’s git), but muddle also provides some VCS commands, notably muddle push, muddle pull and muddle merge.

So, when one could use the VCS-specific command, or the muddle command, which should one use?

My normal practise is to use:

  • muddle pull - to update a working directory from the repository
  • muddle push - to push changes once they’re done

and otherwise use the appropriate VCS commands to manipulate local state.

The reason for using the muddle commands for pull/push is that the build description keeps track of where the far repository is, and this can change - we’ve had occasion to do that whilst working with various build trees. Secondly, different VCS’s have different degrees of guessing correctly (in our usage) that the push and pull are to be for the same place, and muddle already knows that.

For any other operations, specific VCS commands are finer grained and generally more useful.

So, if I was editing the build description (01.py in this case), then I would typically:

  1. Edit src/builds/01.py.
  2. Test that muddle help still works - this is a good general test that the build description is at least still valid Python code, and doesn’t contain circular dependencies or anything else pathological.
  3. Test that everything (or some representative sample) still builds OK
  4. Use git to commit 01.py, with an appropriate commit message
  5. Use muddle to push it
  6. Use gitk to check I’ve done what I think I have

Similarly, if I were editing source code in a checkout, I would:

  1. cd or pushd into the checkout directory
  2. Perform the edit.
  3. Use muddle or muddle distrebuild to rebuild it, and if that fails, do (2) again (and so on).
  4. Use git (or whatever the appropriate VCS is) to commit the changes.
  5. Use muddle push to push them.

(And as to muddle merge - I’d use the VCS specific command in all cases, and it is possible that muddle merge may go away at some point in the future.)

Creating a new checkout, and adding it to (for example) git

When developing a new system, it is periodically necessary to add a new checkout to the system as a whole.

For the purpose of this illustration, I shall assume that the build is using git to store its repositories, and the muddle build was intialised using the command:

$ muddle init git+ssh://git@git.kynesim.co.uk/opt/projects/Fred builds/01.py

(note that this is not a real command line, there is no such server).

That is, we are expecting to connect to the server with SSH, all interaction is via the user git, and project repositories are to be found as subdirectories of /opt/projects/Fred on the server (typically named to reflect the layout of checkouts in the build trees src/ directory).

Each user will normally have an SSH key allowing them access to the server machine, as user git (that is, their public key will be in the git users .ssh/authorized_keys file).

So, to add a new checkout, we start by creating the repository on the server.

Logon to the server machine as git, something like:

$ ssh git@git.kynesim.co.uk

Then cd to the project root:

$ cd /opt/projects/Fred

We will assume we are adding the libpng library, which will live in src/libs, and we thus want to put it into a libs subdirectory on the server. We’re actually adding version 1.4.3 of the library, so we put that information into the checkout name (in fact, this will typically be how the directory would be named after retrieving it from its original home). This allows our build tree to change the version of a checkout in a clean fashion, later on.

So, create the new directory:

git@git.kynesim.co.uk$ mkdir -p libs/libpng-1.4.3

and tell git to set it up as a bare repository:

git@git.kynesim.co.uk$ cd libs/libpng-1.4.3
git@git.kynesim.co.uk$ git init --bare

You can then log out of the server (I surely don’t need to show that).

Back in the development tree, we can now add the new directory - in this instance we’re unpacking an archive we’d already downloaded:

$ cd src/libs
$ tar -zxvf ~/Downloads/libpng-1.4.3.tar.gz
$ cd libpng-1.4.3

It’s best to add the files to git exactly as received before doing anything else:

$ git init
$ git add *
$ git commit

and then in the editor, add a commit message something like:

New checkout libpng-1.4.3

Retrieved from http://www.libpng.org/pub/png/libpng.html, specifically
http://prdownloads.sourceforge.net/libpng/libpng-1.4.3.tar.gz?download

Of course, if this is a new and original checkout, there won’t be any content, and this stage will be omitted.

Then we will need a muddle Makefile - it’s best to add this as a separate transaction, to make it clear it is not part of the original code:

$ touch Makefile.muddle
$ git add Makefile.muddle
$ git commit Makefile.muddle -m 'A library needs a muddle Makefile'

The content of the muddle Makefile can be sorted out later.

The new library also needs to be added to the build description, or muddle won’t know about it. So, in this case, we edit src/builds/01.py and add:

# This is the version of the PNG library that works with our other code
make.twolevel(builder, name='libpng', roles=[graphics_role],
              co_dir='libs', co_name='libpng-1.4.3')

in an appropriate place.

Note that the package name does not include the version number - again, this makes it easier if we change the version of libpng we are using.

We can now use:

$ muddle query checkouts

to make sure the checkout has been succesfullly added to the build description. If it has, then libpng-1.4.3 should appear in the list that prints out.

Finally, we tell muddle that we have already checked this library out:

$ muddle import libpng-1.4.3

and we can then push the contents of our new checkout to the server:

$ muddle push libpng-1.4.3

The remaining tasks are:

  1. Add some dependencies on the library to the build description (there must be something that depends on it, or we’d not have included it).
  2. Write the actual muddle Makefile for it.
  3. Try building it with muddle build libpng.

How I use terminal windows with muddle

In a GUI environment (e.g., Gnome or whatever), I personally tend to have several terminal windows open:

  • One at the top level of the build tree.

    This is where I issue commands like muddle pull _all, or muddle query, or even visdep.

  • One for each checkout I’m working on, with the terminal cd-ed into the appropriate source directory.

    In these terminals, I can do muddle or muddle distrebuild or whatever without needing to specify the checkout or package I’m working on. This also facilitates using the appropriate VCS commands directly, as discussed above, or doing muddle push when I believe I’ve reached an appropriate stage.

Making a clean binary deployment/release

There are two approaches to this.

Using a clean build tree

$ cd <where the build trees live>
$ mkdir <new build tree directory>
$ cd <new build tree directory>
$ muddle init <repo> <desc>
$ muddle

or possible muddle deploy _all or whatever is appropriate.

<where the build trees live> is the directory that <new build tree directory> is going to be created in - in my work environment, I segregate these by project, but it may be as simple as a directory called ~/work. The important thing is that each build tree is a separate directory, with an appropriate name (perhaps by date).

<repo> and <desc> are normally known for a particular project (they may be documented on a wiki page), but are also to be found in:

  • .muddle/RootRepository
  • .muddle/Description

in a previous build tree.

This approach is always safe, but can take much more time (since the entire build tree needs to be checked out again).

Using an updated and “very clean” build tree

$ muddle pull _all
$ muddle veryclean
$ muddle

or possible muddle deploy _all or whatever is appropriate.

This approach is quicker, as it does not involve checking everything out again.

The muddle veryclean is to ensure that the obj, install and deploy directories are cleanly removed. Over time, during development, it is common for unused software and files to build up in theses directories - checkouts change the files they produce, or produce differently named files - and the muddle distclean operation does what the distclean: target in a muddle Makefile says it should do, which may not clean things as thoroughly as one might wish.

The disadvantage of this approach is that, for some projects, there can be problems with not building entirely from scratch. In particular:

  • If any of the packages actually do build or alter stuff in their checkout directory, then the build tree may not be as clean as one thought (this is why we strongly recommend out-of-tree building for all packages).

  • In older versions of muddle: If there are any shallow checkouts (typically specified in the build description as follows:

    pkg.set_checkout_vcs_option(builder, Label('checkout', 'kernel-source'),
                                shallow_checkout=True)
    

    where the Label part may, of course, differ), then these will not be updated by muddle pull, so it is possible to miss important changes.