The muddled package

Muddle - A VCS-agnostic package build and configuration management system

Note

This is a first pass at semi-automated documentation.

It is mostly derived from docstrings within the package, and these in could certainly do with expansion and clarification. Also, the organisation/layout of this section itself leaves something to be desired. Please be patient.

Modules available via import muddle

After doing:

>>> import muddled

you have available:

  • muddled
  • muddled.depend
  • muddled.pkg
  • muddled.repository
  • muddled.utils
  • muddled.vcs
  • muddled.version_control

Top-level modules

These are the “top-level” modules, i.e., those obtainable by (for instance):

>>> import muddled.depends

muddled.cmdline

Note

Internal use, handles the muddle command line

Main command line support for the muddle program.

See:

muddle help

for help on how to use it.

muddled.cmdline.cmdline(args, muddle_binary=None)

Work out what to do from a muddle command line.

‘args’ should be all of the “words” after the actual command name itself.

‘muddle_binary’ should be the __file__ value for the Python script that is calling us, or whatever other value we wish $(MUDDLE) to be set to by muddle itself. It is important to get this right, as it is used in Makefiles to run muddle itself. If it is given as None then we shall make up what should be a sensible value.

muddled.cmdline.find_and_load(specified_root, muddle_binary)

Find our .muddle root, and then load our builder, and return it.

muddled.cmdline.guess_cmd_in_build(builder, current_dir)

Returns a tuple (command_name, args)

muddled.cmdline.lookup_command(command_name, args, cmd_dict, subcmd_dict)

Look the command up, and return an instance of it and any remaining args

muddled.cmdline.our_cmd(cmd_list, error_ok=True)

Command processing for calculating muddle version

muddled.cmdline.show_version()

Show something akin to a version of this muddle.

Simply run git to do it for us. Of course, this will fail if we don’t have git...

muddled.commands

Note

Internal use, handles the muddle command line

Muddle commands - these get run more or less directly by the main muddle command and are abstracted out here in case your programs want to run them themselves

class muddled.commands.AnyLabelCommand

Bases: muddled.commands.Command

A Command that takes any sort of label. Always requires a build tree.

We don’t try to turn one sort of label into another, and we don’t alter the order of the labels given. At least one label must be provided.

build_these_labels(builder, checkouts)

Do whatever is necessary to each label

decode_args(builder, args, current_dir)

Turn the arguments into full labels.

with_build_tree(builder, current_dir, args)
class muddled.commands.Assert

Bases: muddled.commands.AnyLabelCommand

Syntax:muddle assert <label> [ <label> ... ]

Assert the given labels.

This sets the tags indicated by the specified label(s), and only those tags.

This is not the same as if muddle had performed the equivalent “muddle buildlabel” command, because setting the “/installed” tag in this way will not also set the “/built” (or any other) tag.

Thus this is mostly for use by experts and scripts.

Each <label> is a label fragment, in the normal manner. The <type> defaults to “package:”, and the <tag> defaults to the normal default <tag> for that type. Wildcards are expanded.

<label> may also be “_all”, “_default_deployments”, “_default_roles” or “_just_pulled”.

See “muddle help labels” for more help on label fragments and the “_xxx” values.

build_these_labels(builder, labels)
cmd_name = 'assert'
class muddled.commands.Bootstrap

Bases: muddled.commands.Command

Syntax:muddle bootstrap [-subdomain] <repo> <build_name>

Create a new build tree, from scratch, in the current directory. The current directory should ideally be empty.

  • <repo> should be the root URL for the repository which you will be using as the default remote location. It is assumed that it will contain a ‘builds’ and a ‘versions’ repository/subdirectory/whatever (this depends a bit on version control system being used).

    <repo> should be the same value that you would give to the ‘init’ command, if that was being used instead.

  • <build_name> is the name for the build. This should be a simple string, usable as a filename. It is strongly recommended that it contain only alphanumerics, underline, hyphen and dot (or period). Ideally it should be a meaningful (but not too long) description of the build.

For instance:

$ cd project33
$ muddle bootstrap git+http://example.com/fred/ build-27

You will end up with a build tree of the form:

.muddle/
    RootRepository      -- containing "git+http://example/com/fred/"
    Description         -- containing "builds/01.py"
    VersionsRepository  -- containing "git+http://example/com/fred/versions/"
src/
    builds/
        .git/           -- assuming "http://example/com/fred/builds/"
        01.py           -- wth a bare minimum of content
versions/
        .git/           -- assuming "http://example/com/fred/versions/"

Note that ‘src/builds/01.py’ will have been added to the VCS (locally), but will not have been committed (this may change in a future release).

Also, muddle cannot currently set up VCS support for Subversion in the subdirectories.

If you try to do this in a directory that is itself within an existing build tree (i.e., there’s a parent directory somewhere with a .muddle directory), then it will normally fail because you are trying to create a build within an existing build. If you are actually doing this because you are bootstrapping a subdomain, then specify the -subdomain switch.

Note that this command will never bootstrap a new build tree in the same directory as an existing .muddle directory.

allowed_in_release_build()
bootstrap(root_path, args)
cmd_name = 'bootstrap'
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)

Bootstrap a build tree.

class muddled.commands.BranchTree

Bases: muddled.commands.Command

Syntax:muddle branch-tree [-c[check] | -f[orce]] [-v] <branch>

Move all checkouts in the build tree (if they support it) to branch <branch>.

This works as follows:

  1. First inspect each checkout, and check if:

    1. the checkout is using a VCS which does not support this operation (which probably means it is not using git), or
    2. the build description explicitly specifies a particular revision, or
    3. the build description explicitly specifies a particular branch, or
    4. it is a shallow checkout, in which case there is little point branching it as it cannot be pushed, or
    5. the checkout already has a branch of the requested name.

    If any checkouts report problems, then the command will be aborted, and muddle will exit with status 1.

  2. Then, branch each checkout as requested, and change to that branch.

  3. Finally, remind the user to add “builder.follow_build_desc_branch = True” to the build description.

If the user specifies “-c” or “-check”, omit steps 2 and 3 (i.e., just do the checks).

If the user specifies “-f” or “-force”, omit step 1 (i.e., do not do the checks), and ignore any checkouts which do not support this operation, have an explicit revision specified, or are shallow. If a checkout already has a branch of the requested name, check it out.

If the ‘-v’ flag is used, report on each checkout (actually, each checkout directory) as it is entered.

It is recommended that <branch> include the build name (as specified using builder.build_name = <name> in the build description).

Muddle does not itself provide a means of branching only some checkouts. Use the appropriate VCS commands to do that (possibly in combination with ‘muddle runin’).

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.BranchTree, line 45)

Unexpected section title.

Normal usage
------------
  1. Choose a branch name that incorporates the build name and reason for branching. For instance, “acme_stb_alpha_v1.0_maintenance”.

  2. Perform the branch:

    muddle branch-tree acme_stb_alpha_v1.0_maintenance
    
  3. Edit the (newly branched) build description, and add:

    builder.follow_build_desc_branch = True
    

    to it, so that muddle knows the build tree has been branched as an entity.

  4. Commit all the new branches with an appropriate message. At the moment, there is no convenient single line way to do this with muddle, but the somewhat inconvenient muddle runin command can be used, for instance:

    muddle runin _all_checkouts git commit -a -m 'Branch for v1.0 maintenance'
    

    (this, of course, relies upon the fact that muddle only supports tree branching with git at the moment).

  5. Possibly do a “muddle push _all” at this point, or perhaps do some editing, some more committing, and then push.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.BranchTree, line 74)

Unexpected section title.

Dealing with problems
---------------------

If a checkout does not support lightweight branching, then the solution is to ‘-force’ the tree branch, and then “branch” the offending checkout by hand. The branched version of the build description will then need to be edited to indicate (in whatever manner is appropriate) the new “branch” to be used.

If the build description explicitly specifies a particular branch or revision for a checkout, then check the build out as normal, then branch the build description and remove the explicit branch or revision, and then use branch-tree to branch the checkout.

(If you can promise absolutely that it will never be necessary to edit the offending checkout, then it would also be OK to leave the explicit branch or revision, but experience proves this is often a mistake.)

If a checkout is marked as shallow, then the solution is to edit the branched build description and specify the required revision id explicitly, to guarantee that you keep on getting the particular shallow checkout that is needed (since the main build will continue to track HEAD).

If a checkout already has a branch of the name you wanted to use, then either use it, or change the branch name you ask for - this can only be decided by yourself, because you know what the name means.

Note that the check for a clashing branch name is done last. That means that if you change the build description so any of the previous checks no longer fail, you still need to re-run the “check” phase to re-check for clashing branch names.

allowed_switches = {'-force': 'force', '-v': 'verbose', '-f': 'force', '-check': 'check', '-c': 'check'}
branch_checkouts(builder, all_checkouts, branch, verbose)

Branch our checkouts.

If we can’t, say so but continue anyway.

If ‘verbose’, show each pushd into a checkout directory.

Returns the number of branched checkouts.

check_checkouts(builder, checkouts, branch, verbose)

Check if we can branch our checkouts.

Returns a list of problem reports, one per problem checkout.

cmd_name = 'branch-tree'
with_build_tree(builder, current_dir, args)
class muddled.commands.Build

Bases: muddled.commands.PackageCommand

Syntax:muddle build [ <package> ... ]

Build packages.

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/postinstalled”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

What is done depends upon the tags set for the package:

  1. If the package has not yet been preconfigured, any preconfigure actions will be done (for most packages, this is an empty step).
  2. If the package has not yet been configured, then it will be configured. This normally involves performing the actions for the “config” target in the muddle Makefile.
  3. If the package has not yet been built, then it will be built. This normally involves performing the actions for the “all” target in the muddle Makefile.
  4. If the package has not yet been installed, then it will be installed. This normally involves performing the actions for the “install” target in the muddle Makefile.
  5. If the package has not yet been post-installed, then it will be post-installed (for most packages, this is an empty step).

Steps 1. and 2. are identical to those in “muddle configure”.

This sequence is why a dependency on a package should normally be made on package:<name>{<role>}/postinstalled - that is the final stage of building any package.

build_these_labels(builder, labels)
cmd_name = 'build'
class muddled.commands.BuildLabel

Bases: muddled.commands.AnyLabelCommand

Syntax:muddle buildlabel <label> [ <label> ... ]

Performs the appropriate actions to ‘build’ each <label>.

Each <label> is a label fragment, in the normal manner. The <type> defaults to “package:”, and the <tag> defaults to the normal default <tag> for that type. Wildcards are expanded.

<label> may also be “_all”, “_default_deployments”, “_default_roles” or “_just_pulled”.

See “muddle help labels” for more help on label fragments and the “_xxx” values.

Unlikes the checkout, package or deployment specific commands, buildlabel does not try to guess what to do based on which directory the command is given in. At least one <label> must be specified.

This command is mainly used internally to build defaults (specifically, when you type a bare “muddle” command in the root directory) and the privileged half of instruction executions.

build_these_labels(builder, labels)
cmd_name = 'buildlabel'
class muddled.commands.CPDCommand

Bases: muddled.commands.Command

A command that takes checkout, package or deployment arguments.

This is purely an intermediate class for common code for the classes using it (I coult have done a mixin class instead)

build_these_labels(builder, checkouts)

Do whatever is necessary to each label

current_dir = None
decode_args(builder, args, current_dir)

Interpret ‘args’ as partial labels, and return a list of proper labels.

default_args(builder, current_dir)

Decide on default labels, based on where we are in the build tree.

expand_labels(builder, args)
interpret_labels(builder, args, initial_list)

Turn ‘initial_list’ into a list of labels of the required type.

This method should attempt to GiveUp with a useful message if it would otherwise return an empty list.

original_labels = []
required_tag = None
required_type = 'checkout'
with_build_tree(builder, current_dir, args)
class muddled.commands.Changed

Bases: muddled.commands.PackageCommand

Syntax:muddle changed <package> [ <package> ... ]

Mark packages as having been changed so that they will later be rebuilt by anything that needs to.

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/built”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For each label, unset the ‘/built’ tag for the label, and the tags of any labels that depend on it. Note that this will include the same label with its ‘/installed’ and ‘/postinstalled’ tags.

Note that we don’t reconfigure (or indeed clean) packages - we just clear the tags asserting that they’ve been built.

build_these_labels(builder, labels)
cmd_name = 'changed'
required_tag = 'built'
class muddled.commands.Checkout

Bases: muddled.commands.CheckoutCommand

Syntax:muddle checkout [ <checkout> ... ]

Checks out the specified checkouts.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/checked_out”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

Copies (clones/branches) the content of each checkout from its remote repository.

The value “_just_pulled” will be set to the labels of the checkouts whose working directories are altered by “muddle pull” or “muddle checkout” - i.e., those for which the “pull” or “checkout” operation did something tangible. One can then do “muddle rebuild _just_pulled” or “muddle distrebuild _just_pulled”.

(The value of _just_pulled is cleared at the start of “muddle pull” or “muddle checkout”, and set at the end - the list of checkout labels is actually stored in the file .muddle/_just_pulled.)
build_these_labels(builder, labels)
cmd_name = 'checkout'
class muddled.commands.CheckoutCommand

Bases: muddled.commands.CPDCommand

A Command that takes checkout arguments. Always requires a build tree.

If no explicit labels are given, then the default is to find all the checkouts below the current directory.

allowed_in_release_build()
default_args(builder, current_dir)

Decide on default labels, based on where we are in the build tree.

interpret_labels(builder, args, initial_list)

Turn ‘initial_list’ into a list of labels of the required type.

required_tag = 'checked_out'
required_type = 'checkout'
class muddled.commands.Clean

Bases: muddled.commands.PackageCommand

Syntax:muddle clean [ <package> ... ]

Clean packages. Subsequently, packages are regarded as having been configured but not built.

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/built”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For each label:

  1. Build the label with its tag set to ‘/clean’. This normally involves performing the actions for the “clean” target in the muddle Makefile.
  2. Unset the ‘/built’ tag for the label, and the tags of any labels that depend on it. Note that this will include the same label with its ‘/installed’ and ‘/postinstalled’ tags.
build_these_labels(builder, labels)
cmd_name = 'clean'
required_tag = 'built'
class muddled.commands.Cleandeploy

Bases: muddled.commands.DeploymentCommand

Syntax:muddle cleandeploy [<deployment> ... ]

Clean the named deployments, and remove their ‘/deployed’ tags.

Note that this also deletes the ‘deploy/<deployment>’ directory for each deployment named (but it does not delete the overall ‘deploy/’ directory).

It also sets the ‘clean’ tag for each deployment.

<deployment> should be a label fragment specifying a deployment, or one of _all and friends, as for any deployment command. The <type> defaults to “deployment”, and the deployment <tag> will be “/clean”. See “muddle help labels” for more information.

If no deployments are named, what we do depends on where we are in the build tree. See “muddle help labels”.

build_these_labels(builder, labels)
cmd_name = 'cleandeploy'
class muddled.commands.Command

Bases: object

Abstract base class for muddle commands. Stuffed with helpful functionality.

Each subclass is a muddle command, and its docstring is the “help” text for that command.

allowed_in_release_build()

Returns True iff this command is allowed in a release build (a build tree that has been created using “muddle release”).

allowed_switches = {}
check_for_broken_build(current_dir)

Check to see if there is a “partial” build in the current directory.

Intended for use in ‘without_build_tree()’, in classes such as UnStamp, which want to operate in an empty directory (or, at least, one without a muddle build tree in it).

The top-level muddle code does a simple check for whether there is a build tree in the current directory before calling a ‘without_build_tree()’ method, but sometimes we want to be a bit more careful and check for a “partial” build tree, presumably left by a previous, failed, command.

If it finds a problem, it prints out a description of the problem, and raises a GiveUp error with retcode 4, so that muddle will exit with exit code 4.

cmd_name = '<Undefined>'
help()
no_op()

Is this is a no-op (just print) operation?

remove_switches(args, allowed_more=True)

Find any switches, remember them, return the remaining arguments.

Switches are assumed to all come before any labels. We stop with an exception if we encounter something starting with ‘-‘ that is not a recognised switch.

If ‘allowed_more’ is False, then the command line must end after any switches.

requires_build_tree()

Returns True iff this command requires an initialised build tree, False otherwise.

set_old_env(old_env)

Take a copy of the environment before muddle sets its own variables - used by commands like subst to substitute the variables in place when muddle was called rather than those that would apply when the susbt command was executed.

set_options(opt_dict)

Set command options - usually from the options passed to mudddle.

switches = []
with_build_tree(builder, current_dir, args)

Run this command with a build tree.

Arguments are:

  • ‘builder’ is the Builder instance, as constructed from the build tree we are in.
  • ‘current_dir’ is the current directory.
  • ‘args’ - this is any other arguments given to muddle, that occurred after the command name.
without_build_tree(muddle_binary, current_dir, args)

Run this command without a build tree.

Arguments are:

  • ‘muddle_binary’ is the location of the muddle binary - this is only needed if “muddle” is going to be run explicitly, or if a Makefile.muddle is going to use $(MUDDLE_BINARY)
  • ‘current_dir’ is the current directory.
  • ‘args’ - this is any other arguments given to muddle, that occurred after the command name.
class muddled.commands.Commit

Bases: muddled.commands.CheckoutCommand

Syntax:muddle commit [ <checkout> ... ]

Commit the specified checkouts to their local repositories.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/changes_committed”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For a centralised VCS (e.g., Subversion) where the repository is remote, this will not do anything. See the update command.

build_these_labels(builder, labels)
cmd_name = 'commit'
required_tag = 'changes_committed'
class muddled.commands.Configure

Bases: muddled.commands.PackageCommand

Syntax:muddle configure [ <package> ... ]

Configure packages.

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/configured”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

What is done depends upon the tags set for the package:

  1. If the package has not yet been preconfigured, any preconfigure actions will be done.
  2. If the package has not yet been configured, then it will be configured. This normally involves performing the actions for the “config” target in the muddle Makefile.
build_these_labels(builder, labels)
cmd_name = 'configure'
required_tag = 'configured'
class muddled.commands.CopyWithout

Bases: muddled.commands.Command

Syntax:muddle copywithout [-f[orce]] <src-dir> <dst-dir> [ <without> ... ]

Many VCSs use ‘.XXX’ directories to hold metadata. When installing files in a Makefile, it’s often useful to have an operation which copies a hierarchy from one place to another without these dotfiles.

This is that operation. We copy everything from the source directory, <src-dir>, into the target directory, <dst-dir>, without copying anything which is in [ <without> ... ]. If you omit without, we just copy - this is a useful, working, version of ‘cp -a’

If you specify -f (or -force), then if a destination file cannot be overwritten because of its permissions, and attempt will be made to remove it, and then copy again. This is what ‘cp’ does for its ‘-f’ flag.

cmd_name = 'copywithout'
do_copy(args)
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.Deploy

Bases: muddled.commands.DeploymentCommand

Syntax:muddle deploy <deployment> [<deployment> ... ]

Build (deploy) the named deployments.

<deployment> should be a label fragment specifying a deployment, or one of _all and friends, as for any deployment command. The <type> defaults to “deployment”, and the deployment <tag> will be “/deployed”. See “muddle help labels” for more information.

If no deployments are named, what we do depends on where we are in the build tree. See “muddle help labels”.

build_these_labels(builder, labels)
cmd_name = 'deploy'
class muddled.commands.DeploymentCommand

Bases: muddled.commands.CPDCommand

A Command that takes deployment arguments. Always requires a build tree.

default_args(builder, current_dir)

Decide on default labels, based on where we are in the build tree.

interpret_labels(builder, args, initial_list)

Turn ‘initial_list’ into a list of labels of the required type.

required_tag = 'deployed'
required_type = 'deployment'
class muddled.commands.DistClean

Bases: muddled.commands.PackageCommand

Syntax:muddle distclean [ <package> ... ]

Distclean packages. Subsequently, packages are regarded as not having been configured or built.

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/built”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For each label:

  1. Build the label with its tag set to ‘/distclean’. This normally involves performing the actions for the “distclean” target in the muddle Makefile.
  2. Unset the ‘/preconfig’ tag for the label, and the tags of any labels that depend on it. Note that this will include the same label with its ‘/configured’,’/built’, ‘/installed’ and ‘/postinstalled’ tags.

Notes:

  • The “distclean” target in the Makefile is independent of the “clean” target - “muddle distclean” does not trigger the “clean” target.

  • “muddle distclean” itself does not delete the ‘obj/’ directory, although this is normally sensible. Muddle makefiles are thus recommended to do this themselves - for instance:

    .PHONY: distclean
    distclean:
        @rm -rf $(MUDDLE_OBJ)
    

    It is possible, though, that future versions of muddle might perform this deletion.

build_these_labels(builder, labels)
cmd_name = 'distclean'
required_tag = 'built'
class muddled.commands.Distrebuild

Bases: muddled.commands.PackageCommand

Syntax:muddle distrebuild [ <package> ... ]

A rebuild that does a distclean before attempting the rebuild.

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/postinstalled”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

  1. Do a “muddle distclean” for all the labels
  2. Do a “muddle build” for all the labels
build_these_labels(builder, labels)
cmd_name = 'distrebuild'
class muddled.commands.Distribute

Bases: muddled.commands.CPDCommand

Syntax:muddle distribute [<switches>|-no-muddle-makefile] <name> <target_directory> [<label> ...]
  • <switches> may be any of:

    • -with-versions
    • -with-vcs
    • -no-muddle-makefile

    See below for more information on each.

  • <name> is a distribution name.

    Several special distribution names exist:

    • “_source_release” is a distribution of all checkouts, without their VCS directories.

      “muddle distribute _source_release” is typically useful for generating a directory to archive with tar and send out as a source code release.

      “muddle distribute -with-vcs _source_release” is a way to get a “clean copy” of the current build tree, although perhaps not as clean as starting over again from “muddle init”,

    • “_binary_release” - this is a distribution of all the install directories, as well as the build description checkout(s) implied by the packages distributed and (unless -no-muddle-makefiles is given) the muddle Makefiles needed by each package (as the only file in each appropriate checkout directory)

    • “_deployment” - this is a distribution of the deploy directory, and all its subdirectories, as well as a version stamp. It can be useful for customers who do not yet have the ability to use muddle (as is necessary with a _binary_release).

      It is generally necessary to specify the deployment labels as <label> arguments, and the labels used will be written to a MANIFEST.txt file.

      Note that it is done by a different mechanism than the other commands, specificaly more or less as if the user had done:

      muddle deploy <label> ...
      mkdir -p <target_directory>
      cp -a deploy <target_directory>
      muddle stamp save <target_directory>/`muddle query name`.stamp
      cat "muddle distribute _deployment <target_directory> <labels>"                       > <target_directory>/MANIFEST.txt
      

      Consequently, the switches are not allowed with this variant.

    • “_for_gpl” is a distribution that satisfies the GPL licensing requirements. It is all checkouts that have an explicit GPL license (including LGPL), plus any licenses which depend on them, and do not explicitly state that they do not need distributing under the GPL terms, plus appropriate build descriptions.

      It will fail if “propagated” GPL-ness clashes with declared “binary” or “private” licenses for any checkouts.

    • “_all_open” is a distribution of all open-source licensed checkouts. It contains everything from “_for_gpl”, plus any other open source licensed checkouts.

      It will fail for the same reasons that _for_gpl” fails.

    • “_by_license” is a distribution of everything that is not licensed with a “private” license. It is equivalent to “_all_open” plus any proprietary source checkouts plus those parts of a “_binary_release” that are not licensed “private”.

      It will fail if “_all_open” would fail, or if any of the install directories to be distributed could contain results from building “private” packages (as determined by which packages are in the appropriate role).

  • <target_directory> is where to distribute to. If it already exists, it should preferably be an empty directory.

  • If given, each <label> is a label fragment specifying a deployment, package or checkout, or one of _all and friends. The <type> defaults to “deployment”. See “muddle help labels” for more information.

If specific labels are given, then the distribution will only concern those labels and those they depend on. Deployment labels will be expanded to all of the packages that the deployment depends upon. Package labels (including those implied by deployment labels) will be remembered, and also expanded to the checkouts that they depend directly upon. Checkout labels (including those implied by packages) will be remembered. When the distribution is calculated, only packages and checkouts that have been remembered will be candidates for distribution.

If no labels are given, then the whole of the build tree is considered.

If the -with-versions switch is specified, then if there is a stamp “versions/” directory it will also be copied. By default it is not.

If the -with-vcs switch is specified, then VCS “special” files (that is, ”.git”, ”.gitignore”, ”.gitmodules” for git, and so on) are requested:

  • for the build description directories
  • for the “versions/” directory, if it is being copied
  • to all checkouts in a “_source_release” distribution

It does not apply to checkouts specified with “distribute_checkout” in the build description, as they use the “copy_vcs_dirs” argument to that function instead.

If the -no-muddle-makefile switch is specified, then the _binary_release distribution will not include Muddle makefiles for each package distributed. It does not override the setting of the “with_muddle_makefile” argument explicitly set in any calls of “distribute_package” in the build description, nor does it stop distribution of any extra files explicitly chosen with “distribute_checkout_files” in the build description. It also does not affect the “_by_license” distribution.

Note that “muddle -n distribute” can be used in the normal manner to see what the command would do. It shows the labels that would be distributed, and the actions that would be used to do so. This is especially useful for the “_source_release” and “_binary_release” commands. Output will typically be something like:

$ m3 -n distribute -with-vcs _binary_release ../fred
Writing distribution _binary_release to ../fred
checkout:builds/distributed        DistributeBuildDescription: _binary_release[vcs]
checkout:main_co/distributed       DistributeCheckout: _binary_release[1], role-x86[*]
package:main_pkg{arm}/distributed  DistributePackage: _binary_release[install]
package:main_pkg{x86}/distributed  DistributePackage: _binary_release[install], role-x86[obj,install]
  • For each action, all the available distribution names are listed.
  • Each distribution name may be followed by values in [..], depending on what action it is associated with.
  • For a DistributeBuildDescription, the value may be [vcs], or [-<n>], or [vcs, -<n>]. ‘vcs’ means that VCS files will be distributed. A negative number indicates the number of “private” files that will not be distributed.
  • For a DistributeCheckout, the values are [*], [<n>], [,vcs] or [<n>,vcs]. ‘‘ means that all files will be distributed, a single integer (<n>) that just that many specific files have been selected for distribution. [1] typically means the muddle Makefile, or perhaps a license file. A ‘vcs’ means that the VCS files will be distributed.
  • For a DistributePackage, the values are [obj], [install] or [obj,install], indicating if “obj” or “install” directories are being distributed. It’s also possible (but not much use) to have a DistributePackage distribution name that doesn’t do either.

See also “muddle query checkout-licenses” for general information on the licenses in the current build, and “muddle query role-licenses” for how licenses are distributed between the roles in the build. Both of these will report on license clashes that appear to exist.

BEWARE: THIS COMMAND IS STILL NEW, AND DETAILS MAY CHANGE

In particular, the “-no-muddle-makefile” switch may go away, the details of use of the “-copy-vcs” switch may change. and the standard distribution names may change.
allowed_switches = {'-no-muddle-makefile': 'no-muddle-makefile', '-with-vcs': 'with-vcs', '-with-versions': 'with-versions'}
cmd_name = 'distribute'
deployment(builder, current_dir, target_dir, fragments)

Do “muddle distribute _deployment”.

interpret_labels(builder, args, initial_list)

Return selected packages and checkouts.

requires_build_tree()
with_build_tree(builder, current_dir, args)

We’re sufficiently unlike other commands to do this ourselves.

class muddled.commands.Doc

Bases: muddled.commands.Command

Syntax:muddle doc [<switch> ...] [<what>]

To get documentation on modules, classes, methods or functions in muddle, use:

muddle doc <name>               for help on <name>
muddle doc -contains <what>     to list all names that contain <what>

There are also (mainly for use in debugging muddle itself - beware, they all produce long output):

muddle doc -duplicates     to list all duplicate (partial) names
muddle doc -list           to list all the "full" names we know
muddle doc -dump           to dump the internal map of names/values

<switch> may also be:

-p[ager] <pager>    to specify a pager through which the text will be
                    piped. The default is $PAGER (if set) or else
                    'more'.
-nop[ager]          don't use a pager, just print the text out.
-pydoc              Use pydoc's rendering to output the text about the
                    item. This tends to produce more information. It
                    is also (more or less) the format that the older
                    "muddle doc" command used.

The plain “muddle doc <name>” can be used to find out about any muddle module, class, method or function. Leading parts of the name can be omitted (“Builder” and “mechanics.Builder” and “muddled.mechanics.Builder” are all the same), provided that doesn’t make <name> ambiguous, and if it does, you will be given a list of the possible alternatives. So, for instance:

$ muddle doc Builder

will report on muddled.mechanics.Builder, but:

$ muddle doc simple

will give a list of all the names that contain ‘simple’.

If you’re not sure of a name, then “-contains” can be used to look for all the (full names - i.e., starting with “muddled.”) that contain that string. For instance:

$ muddle doc -contains absolute
The following names contain "absolute":
  muddled.checkouts.multilevel.absolute
  muddled.checkouts.simple.absolute
  muddled.checkouts.twolevel.absolute

and one can then safely do:

$ muddle doc simple.absolute

For a module, the module docstring is reported, and then a list of the names of all the classes and functions in that module.

For a class, the classes it inherits from, its docstring and its __init__ method are all reported, followed by a list of the methods in that class.

For a method or function, its argument list (signature) and docstring are reported.

If “-pydoc” is specified, then the layout of various things will be different, and also the full documentation of internal items (methods inside classes, etc.) will be reported - this can lead to substantially longer output.

cmd_name = 'doc'
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.Env

Bases: muddled.commands.PackageCommand

Syntax:muddle env <language> <mode> <name> <label> [ <label> ... ]

Produce a setenv script in the requested language listing all the runtime environment variables bound to <label> (or the cumulation of the variables for several labels).

  • <language> may be ‘sh’, ‘c’, or ‘py’/’python’
  • <mode> may be ‘build’ (build time variables) or ‘run’ (run-time variables)
  • <name> is used in various ways depending upon the target language. It should be a legal name/symbol in the aforesaid target language (for instance, in C it will be uppercased and used as part of a macro name).
  • <label> should be a label fragment specifying a package, or one of _all and friends, as for any package command. See “muddle help labels” for more information.

So, for instance:

$ muddle env sh run 'encoder_settings' encoder > encoder_vars.sh

might produce a file encoder_vars.sh with the following content:

# setenv script for encoder_settings
# 2010-10-19 16:24:05

export BUILD_SDK=y
export MUDDLE_TARGET_LOCATION=/opt/encoder/sdk
export PKG_CONFIG_PATH=$MUDDLE_TARGET_LOCATION/lib/pkgconfig:$PKG_CONFIG_PATH
export PATH=$MUDDLE_TARGET_LOCATION/bin:$PATH
export LD_LIBRARY_PATH=$MUDDLE_TARGET_LOCATION/lib:$LD_LIBRARY_PATH

# End file.
build_these_labels(builder, args)
cmd_name = 'env'
with_build_tree(builder, current_dir, args)
class muddled.commands.Help

Bases: muddled.commands.Command

To get help on commands, use:

muddle help [<switch>] [<command>]

specifically:

muddle help <cmd>          for help on a command
muddle help <cmd> <subcmd> for help on a subcommand
muddle help _all           for help on all commands
muddle help <cmd> _all     for help on all <cmd> subcommands
muddle help commands       just list the (top level) commands
muddle help categories     shows command names sorted by category
muddle help labels         for help on using labels
muddle help subdomains     for help on subdomains
muddle help aliases        says which commands have more than one name
muddle help vcs [name]     name the supported version control systems,
                           or give details about one in particular
muddle help environment    list the environment variables muddle defines
                           for use in muddle Makefiles

<switch> may be:

-p[ager] <pager>    to specify a pager through which the help will be piped.
                    The default is $PAGER (if set) or else 'more'.
-nop[ager]          don't use a pager, just print the help out.
cmd_name = 'help'
command_line_help = 'Usage:\n\n muddle [<options>] <command> [<arg> ...]\n\nAvailable <options> are:\n\n --help, -h, -? This help text\n --tree <dir> Use the muddle build tree at <dir>.\n --just-print, -n Just print what muddle would have done. For commands that\n \'do something\', just print out the labels for which that\n action would be performed. For commands that "enquire"\n (or "find out") something, this switch is ignored.\n --version Show the version of muddle and the directory it is\n being run from. Note that this uses git to interrogate\n the .git/ directory in the muddle source directory.\n\nMuddle always starts by looking for a build tree, signified by the presence of\na .muddle directory. If you give --tree, then it will look in the directory\ngiven, otherwise, it will traverse directories from the current directory up\nto the root.\n'
get_help(args)

Return help for args, or a summary of all commands.

help_aliases()

Return a list of all commands with aliases

help_all()

Return help for all commands

help_categories()
help_command_list()

Return a list of the top-level commands.

help_environment(args)

Return help on MUDDLE_xxx environment variables

help_label_absent = '"muddle" with no label arguments\n--------------------------------\nMuddle tries quite hard to do the sensible thing if you type it without any\narguments, depending on the current directory. Specifically, for commands\nthat "build" a label (whether checkout, package or deployment):\n\n* at the very top of the build tree:\n\n muddle buildlabel _default_deployments _default_roles\n\n* within a \'src/\' directory, or within a non-checkout subdirectory inside\n \'src\'/, "muddle build" for each checkout that is below the current\n directory (i.e., build all packages using the checkouts below the\n current directory).\n* within a checkout directory, "muddle build" for the package(s) that use\n that checkout.\n* within an \'obj/\' directory, no defined action\n* within an \'obj/<package>\' directory, "muddle rebuild" for the named\n <package> in each of the default roles.\n* within an \'obj/<package>/<role>\' directory (or one of its subdirectories),\n "muddle rebuild package:<package>{<role>}".\n* within an \'install/\' directory, no defined action.\n* within an \'install/<role>\' directory (or one of its subdirectories),\n "muddle rebuild package:*{<role>}".\n* within a \'deploy/\' directory, no defined action.\n* within a \'deploy/<deployment>\' directory, "muddle redeploy <deployment>"\n (note, "redeploy" rather than "deploy", as this seems more likely to be\n useful).\n\nIf you have subdomains (see "muddle help subdomains"), then:\n\n* within a \'domains/\' directory, "muddle buildlabel" with arguments\n "deployment:(<domain>)*/deployed" and "package:(<domain>)*{*}/postinstalled"\n for each <domain> that has a directory (directly within) that \'domains/\'\n directory.\n* within a \'domains/<domain>\' directory, "muddle buildlabel" with arguments\n "deployment:(<domain>)*/deployed" and "package:(<domain>)*{*}/postinstalled".\n\nwhere <domain> is replaced by the subdomain\'s name (as given by "muddle where"\nin that directory).\n\nAnywhere else, "muddle" will say "Not sure what you want to build".\n\n"muddle <command>" with no label arguments\n------------------------------------------\nAgain, muddle tries to decide what to do based on the current directory.\n\nFor any command that "builds" a label (whether checkout, package or\ndeployment):\n\n* if "muddle where" gives a label, then that label will be used as the\n argument.\n* if the current directory is within \'src/\', then all the checkouts below\n the current directory will be found, and a checkout: label constructed\n for each, and those will be used as the arguments.\n\nOtherwise, muddle will say "Not sure what you want to build".\n'
help_label_all_and_friends = '_all and friends\n----------------\nThere are some special command line arguments that represent a set of labels.\n\n* _all represents all target labels of the appropriate type for the command.\n For example, in "muddle checkout _all" it expands to all "checkout:" labels.\n* _all_checkouts, _all_packages and _all_deployments repesent all labels of\n the appropriate type.\n* _default_roles represents package labels for all of the default roles, as\n given in the build description. Specifically, package:*{<role>}/postinstalled\n for each such <role>. You can find out what the default roles are with\n "muddle query default-roles".\n* _default_deployments represents deployment labels for each of the default\n deployments, as given in the build description. You can find out what the\n default deployments are with "muddle query default-deployments".\n* _just_pulled represents the checkouts that were (actually) pulled by the\n last "muddle pull" or "muddle merge" command.\n* _release represents the labels to be built for a release build, as set\n in the build description with "builder.add_to_release_build".\n\nThe help for particular commands will indicate if these values can be used,\nbut they are generally valid for all commands that "build" checkout, package\nor deployment labels. As normal, you can use "muddle -n <command> _xxx" to\nsee exactly what labels the "_xxx" value would expand to.\n'
help_label_fragments = 'Label fragments\n---------------\nTyping all of a label on the command line can be onerous. Muddle thus allows\nappropriate fragments of a label to be used, according to the particular\ncommand. In general, the aim is to require the label name, have a sensible\ndefault for the label type and tag, and (for packages) try all the default\nroles if none is specified.\n\nEach command says, in its help text, if it defaults to "checkout:", "package:"\nor "deployment".\n\nCheckout and deployment labels may not have roles.\n\nIf a package role is not given, then the label will be expanded into\npackage:<name>{<role>} for each <role> in the default roles. Default roles\nare defined in the build description. You can use "muddle query default-roles"\nto find out what they are.\n\nIf a tag is not given, then the "end tag" for the particular type of label will\nbe used. This is:\n\n* for checkout, \'/checked_out\'\n* for package, \'/postinstalled\'\n* for deployment, \'/deployed\'\n\nCheckout, package and deployment commands typically ignore the label tag\nof any checkout, package or deployment label they are given (whether because\nyou gave it exactly, or because it was deduced). Instead, they have a\nrequried tag, which is documented in their help text.\n\nOther commands default to the "end tag" for that particular label type.\n\nYou can always used "muddle -n <command> <fragment>" to see what labels the\n<command> will actually end up using.\n'
help_label_star = 'Unexpected results\n------------------\nNote: at a Unix shell, typing:\n\n $ muddle build *\n\nis unlikely to give the required result. The shell will expand the "*" to the\ncontents of the current directory, and if at top level of the built tree,\nmuddle will then typically complain that there is no package called \'deploy\'.\nInstead, escape the "*", for instance:\n\n $ muddle build \'*\'\n'
help_label_summary = 'More complete documentation on labels is available in the muddle documentation\nat http://muddle.readthedocs.org/. Information on the label class itself can\nbe obtained with "muddle doc depend.Label". This is a summary.\n\n(Nearly) everything in muddle is described by a label. A label looks like:\n\n <type>:<name>{<role>}/<tag>\n\nAll label components are made up of the characters [A-Z0-9a-z-_].\n<name>, <role> and <tag> may also be \'*\' (a wildcard), meaning all values.\nA <name> may not start with an underscore.\n\n (If your build tree contains *subdomains* then there is another label\n component - see "help subdomains" for more information if you need it.)\n\n<type> is one of checkout, package or deployment.\n\n* A checkout is checked out of version control. It lives (somewhere) under\n \'src/\'.\n* A package is built (under \'obj/<name>/<role>\') and installed (under\n \'install/<role>).\n* A deployment is deployed (ready for putting onto the target), and is found\n under \'deploy/<name>\'.\n\nLabels of type checkout and deployment do not use roles. Package labels\nalways need a role (although muddle will sometimes try to guess one for you).\n\n* For checkouts, <tag> is typically \'checked_out\', meaning the checkout has\n been checked out (cloned, branched, etc.). A checkout will be in a directory\n under the src/ directory, with the directory name given by the <name> from\n the checkout label.\n\n* For packages, <tag> is typically one of \'preconfig\', \'configured\', \'built\',\n \'installed\' or \'postinstalled\'.\n\n - preconfig - preconfiguration checks have been made on the package\n - configured - the package has been configured. The \'config\' target in its\n muddle Makefile has been run. This may have involved running GNU\n autotools \'./configure\', and perhaps copying source code if the checkout\n does not support building out-of-tree.\n - built - the package has been built (e.g., compiled and linked). The \'all\'\n target in its muddle Makefile has been run. The results of building end up\n in directory obj/<package-name>/<role>\n - installed - the package has been installed. The \'install\' target in its\n muddle Makefile has been run. All packages in a particular <role> install\n their results to somewhere in install/<role>\n - postinstalled - the package has been postinstalled. This is often an empty\n step.\n\n* For deployments, <tag> is typically \'deployed\', meaning a deployment has been\n created. This normally involves collecting files from particular\n install/<role> directories, and placing the result in\n deploy/<deployment-name>.\n\nSome muddle commands only operate on particular types of label. For instance,\ncommands in category "checkout" (see "muddle help categories") only operate\non checkout: labels.\n'
help_label_wrong = 'How "muddle" commands intepret labels of the "wrong" type\n---------------------------------------------------------\nMost muddle commands that want label arguments actually want labels of a\nparticular type. For instance, "muddle checkout" wants to operate on one\nor more checkout: labels.\n\nSometimes, however, it is more convenient to specify a label of a different\ntype. For instance: "muddle checkout package:fred", to checkout the checkouts\nneeded by package "fred".\n\nLabels of particular types are interpreted as follows:\n\n* in a checkout command:\n\n - checkout: -> itself\n - package: -> all the checkouts used *directly* by this package\n - deployment: -> all the checkouts needed by this deployment\n (this can be a bit slow to calculate)\n\n* in a package command:\n\n - checkout: -> the packages that depend directly upon this checkout\n (i.e., those that are "built" from it), but only if they are in one\n of the default roles.\n - package: -> itself\n - deployment: -> all the packages used *directly* by this deployment\n\n* in a deployment command\n\n - checkout: -> any deployments that depend upon this checkout (at any depth)\n - package: -> any deployments that depend upon this package (at any depth)\n - deployment: -> itself\n'
help_labels(args)

Help on labels within muddle.

“muddle help label” and “muddle help labels” are equivalent.

help label - show this text help label summary - a summary of how labels work help label fragments - using partial labels in commands help label fragment - the same help label absent - what “muddle” or “muddle <cmd>” does with no label

System Message: ERROR/3 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Help.help_labels, line 10)

Unexpected indentation.
arguments

System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Help.help_labels, line 11)

Block quote ends without a blank line; unexpected unindent.
help label wrong - what “muddle <cmd>” does with a label of the “wrong”
type (e.g., “muddle build checkout:something”)

System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Help.help_labels, line 13)

Definition list ends without a blank line; unexpected unindent.

help label _all help label <any name starting with underscore>

System Message: ERROR/3 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Help.help_labels, line 15)

Unexpected indentation.
  • these explain the “special” command line arguments

System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Help.help_labels, line 16)

Block quote ends without a blank line; unexpected unindent.

help label star - the unexpected result of “muddle build *” and its like help label everything - all the “muddle help label” subtopics

System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Help.help_labels, line 16); backlink

Inline emphasis start-string without end-string.
help_subcmd_all(cmd_name, sub_dict)

Return help for all commands in this dictionary

help_subdomains()

Return help on how to use subdomains

help_summary()

Return a summary of usage and a list of all commands

help_vcs(args)

Return help on supported VCS

print_help(args)
requires_build_tree()
subdomains_help = 'Your build contains subdomains if "muddle query domains" prints out subdomain\nnames. In this case, you will also have a top-level \'domains/\' directory, and\nthe top-level build description will contain calls to \'include_domain()\'.\n\nIn builds with subdomains, labels in the top-level build still look like:\n\n <type>:<name>{<role>}/<tag>\n\nbut labels from the subdomains will contain their domain name:\n\n <type>:(<domain>)<name>{<role>}/<tag>\n\nFor instance:\n\n * package:busybox{x86}/installed is in the toplevel build\n * package:(webkit)webkit{x86}/installed is in the \'webkit\' subdomain\n * package:(webkit(x11))xfonts{x11}/installed is in the \'x11\' subdomain,\n which in turn is a subdomain of \'webkit\'.\n\nNote that when typing labels containing domain names within Bash, it wil\nbe necessary to quote the whole label, otherwise Bash will try to interpret\nthe parentheses. So, for instance, use:\n\n $ muddle build \'(x11)xfonts{x11}\'\n '
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.Import

Bases: muddled.commands.CheckoutCommand

Syntax:muddle import [ <checkout> ... ]

Assert that the given checkouts (which may include the builds checkout) have been checked out.

This is mainly used when you’ve just written a package you plan to commit to the central repository - muddle obviously can’t check it out because the repository doesn’t exist yet, but you probably want to add it to the build description for testing (and in fact you may want to commit it with muddle push). For convenience in the expected use case, it goes on to prime the relevant VCS module (by way of “muddle reparent”) so it can be pushed once ready; this should be at worst harmless in all cases.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/checked_out”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

This command is really just a wrapper to “muddle assert” and “muddle reparent”, with the right magic label names.

build_these_labels(builder, labels)
cmd_name = 'import'
with_build_tree(builder, current_dir, args)
class muddled.commands.Init

Bases: muddled.commands.Command

Syntax:muddle init <repository> <build_description>
Or:muddle init -b[ranch] <branch_name> <repository> <build_description>

Initialise a new build tree with a given repository and build description. We check out the build description but don’t actually build anything. It is traditional to create a new muddle build tree in an empty directory.

For instance:

$ mkdir project32
$ cd project32
$ muddle init  git+file:///somewhere/else/examples/d  builds/01.py

This initialises a muddle build tree, creating two new directories:

  • ‘.muddle/’, which contains the build tree state, and
  • ‘src/’, which contains the build description in ‘src/builds/01.py’ (complex build desscriptions may use multiple Python files, and so other files may have been checked out into ‘src/builds/’)

You haven’t told muddle which actual repository the build description is in - you’ve only told it where the repository root is and where the build description file is. Muddle assumes that <repository> “git+file:///somewhere/else/examples/d” and <build_description> “builds/01.py” means repository “git+file:///somewhere/else/examples/d/builds” and file “01.py” therein.

If the -branch switch is given, then the named branch of the build description will be checked out. It thus an error if either the muddle support for the build description VCS does not support this (at the moment, that probably means “not git”), or if there is no such branch.

Note: if you find yourself trying to “muddle init” a subdomain, don’t. Instead, add the subdomain to the current build description (using a call of ‘include_domain()’), and it will automatically get checked out during the “muddle init” of the top-level build. Or see “muddle bootstrap -subdomain”.

cmd_name = 'init'
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)

Initialise a build tree.

class muddled.commands.Instruct

Bases: muddled.commands.Command

Syntax:muddle instruct <package>{<role>} <instruction-file>
Or:muddle instruct (<domain>)<package>{<role>} <instruction-file>

Sets the instruction file for the given package name and role to the file specified in instruction-file. The role must be explicitly given as it’s considered more likely that bugs will be introduced by the assumption of default roles than they are likely to prove useful.

This command is typically issued by ‘make install’ for a package, as:

$(MUDDLE_INSTRUCT) <instruction-file>

If you don’t specify an instruction file, we will unregister instructions for this package and role.

If you want to clear all instructions, you’ll have to edit the muddle database directly - this leaves the database in an inconsistent state - there’s no guarantee that the instruction files will ever be rebuilt correctly - so it is not a command.

You can list instruction files and their ordering with “muddle query inst-files”.

cmd_name = 'instruct'
decode_package_label(builder, arg, tag)

Convert a ‘package’ or ‘package{role}’ or ‘(domain)package{role} argument to a label.

If role or domain is not specified, use the default (which may be None).

requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.Merge

Bases: muddled.commands.CheckoutCommand

Syntax:muddle merge [-s[top]] [ <checkout> ... ]

Merge the specified checkouts from their remote repositories.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/merged”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For each checkout named, retrieve changes from the corresponding remote repository (as described by the build description) and merge them (into the checkout). The merge process is handled in a VCS specific manner, as each checkout is dealt with.

The value “_just_pulled” will be set to the labels of the checkouts whose working directories are altered by “muddle merge” - i.e., those for which the “merge” operation did something tangible. One can then do “muddle rebuild _just_pulled” or “muddle distrebuild _just_pulled”.

(The value of _just_pulled is cleared at the start of “muddle merge”, and set at the end - the list of checkout labels is actually stored in the file .muddle/_just_pulled.)

If ‘-s’ or ‘-stop’ is given, then we’ll stop at the first problem, otherwise an attempt will be made to process all the checkouts, and any problems will be re-reported at the end.

allowed_switches = {'-stop': 'stop', '-s': 'stop'}
build_these_labels(builder, labels)
cmd_name = 'merge'
required_tag = 'merged'
class muddled.commands.PackageCommand

Bases: muddled.commands.CPDCommand

A Command that takes package arguments. Always requires a build tree.

default_args(builder, current_dir)

Decide on default labels, based on where we are in the build tree.

interpret_labels(builder, args, initial_list)

Turn ‘initial_list’ into a list of labels of the required type.

required_tag = 'postinstalled'
required_type = 'package'
class muddled.commands.Pull

Bases: muddled.commands.CheckoutCommand

Syntax:muddle pull [-s[top]] [-noreload] [ <checkout> ... ]

Pull the specified checkouts from their remote repositories. Any problems will be (re)reported at the end.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/pulled”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For each checkout named, retrieve changes from the corresponding remote repository (as described by the build description) and apply them (to the checkout), but not if a merge would be required.

(For a VCS such as git, this actually means “not if a user-assisted merge would be required” - i.e., fast-forwards will be done.)

The value “_just_pulled” will be set to the labels of the checkouts whose working directories are altered by “muddle pull” or “muddle checkout” - i.e., those for which the “pull” or “checkout” operation did something tangible. One can then do “muddle rebuild _just_pulled” or “muddle distrebuild _just_pulled”.

(The value of _just_pulled is cleared at the start of “muddle pull” or “muddle checkout”, and set at the end - the list of checkout labels is actually stored in the file .muddle/_just_pulled.)

Normally, “muddle pull” will attempt to pull all the chosen checkouts, re-reporting any problems at the end. If ‘-s’ or ‘-stop’ is given, then it will instead stop at the first problem.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Pull, line 36)

Unexpected section title.

How build descriptions are treated specially
--------------------------------------------

If the build description is in the list of checkouts that should be pulled, either explicitly or after expanding one of _all and friends, then “muddle pull” will:

  1. Remember exactly what the user asked for on the command line.
  2. Pull the build description.
  3. If the build description changed (i.e., was pulled), then reload it, and re-expand the labels from the command line - so, for instance, _all might change if the new build description has added or removed checkouts.
  4. Remove the build description from this (new) list of checkouts, and pull any that are left.

Earlier versions of muddle (before v2.5.1) did not do this, which meant that one might do “muddle pull _all” and then have to do it again if the build description had changed. It was easy to forget to check for this, which could leave a build tree not as up-to-date as one might think.

If you do want the older, simpler mechanism, then:

muddle pull -noreload <arguments>

can be used, which will just pull the labels on the command line, without treating the build description specially.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Pull, line 63)

Unexpected section title.

What about subdomains?
----------------------

If your build contains subdomains, then all of the subdomain build descriptions will be treated specially. Specifically, each requested build description is pulled in domain order, reloading the top-level build description and re-evaluating the command line each time.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Pull, line 70)

Unexpected section title.

Other commands
--------------

Note that “muddle merge” and “muddle pull-upstream” do not behave in this manner, as it is believed that they are used in a more direct manner (with explicit labels).

allowed_switches = {'-stop': 'stop', '-noreload': 'noreload', '-s': 'stop'}
build_these_labels(builder, labels)
calc_build_descriptions(builder, done=None)

Calculate all the build descriptions in this build tree.

Remove any that have already been ‘done’

Return a list of build description checkout labels, ordered by domain.

cmd_name = 'pull'
delete_pyc_files(builder, co_label)

Delete .pyc files in this checkout

handle_build_descriptions_first(builder, labels)

Pull our build descriptions before anything else.

Returns:

  • the new top-level builder
  • an amended list of the checkout labels still to pull.
label_names(labels)
pull(builder, co_label)

Do the work of pulling checkout ‘co_label’

required_tag = 'pulled'
class muddled.commands.PullUpstream

Bases: muddled.commands.UpstreamCommand

Syntax:muddle pull-upstream [ <checkout> ... ] -u[pstream] <name> ...

For each checkout, pull from the named upstream repositories.

Specifically, retrieve changes from the corresponding remote repository, and apply them (to the checkout), but not if a merge would be required.

(For a VCS such as git, this actually means “not if a user-assisted merge would be required” - i.e., fast-forwards will be done.)

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/checked_out”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

The -u or -upstream switch is required, and must be followed by at least one upstream repository name. If a checkout does not have an upstream of that name, it will be ignored.

So, for instance:

pushd src/checkout1
muddle pull-upstream -u upstream1 upstream2

or:

muddle pull-upstream package:android{x86} -u upstream-android

Note that, unlike the normal “muddle pull” command, there is no -stop switch. Instead, we always stop at the first problem. Not finding an upstream with the right name does not count as a “problem” for this purpose.

Also, pull-upstream does not alter the meaning of “_just_pulled”.

Use “muddle query upstream-repos [<checkout>]” to find out about the available upstream repositories.

allowed_switches = {}
cmd_name = 'pull-upstream'
direction = 'from'
do_our_verb(builder, co_label, vcs_handler, upstream, repo)
required_tag = 'checked_out'
verb = 'pull'
verbing = 'Pulling'
class muddled.commands.Push

Bases: muddled.commands.CheckoutCommand

Syntax:muddle push [-s[top]] [ <checkout> ... ]

Push the specified checkouts to their remote repositories.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/changes_pushed”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

This updates the content of the remote repositories to match the local checkout.

If ‘-s’ or ‘-stop’ is given, then we’ll stop at the first problem, otherwise an attempt will be made to process all the checkouts, and any problems will be re-reported at the end.

“muddle push” will refuse to push if the checkout is not on the expected branch, either an explicit branch from the build description, or the build description branch if we are “following” it, or “master”.

allowed_switches = {'-stop': 'stop', '-s': 'stop'}
build_these_labels(builder, labels)
cmd_name = 'push'
required_tag = 'changes_pushed'
class muddled.commands.PushUpstream

Bases: muddled.commands.UpstreamCommand

Syntax:muddle push-upstream [ <checkout> ... ] -u[pstream] <name> ...

For each checkout, push to the named upstream repositories.

This updates the content of the remote repositories to match the local checkout.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/checked_out”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

The -u or -upstream switch is required, and must be followed by at least one upstream repository name. If a checkout does not have an upstream of that name, it will be ignored.

So, for instance:

pushd src/checkout1
muddle push-upstream -u upstream1 upstream2

or:

muddle push-upstream package:android{x86} -u upstream-android

Note that, unlike the normal “muddle push” command, there is no -stop switch. Instead, we always stop at the first problem. Not finding an upstream with the right name does not count as a “problem” for this purpose.

Use “muddle query upstream-repos [<checkout>]” to find out about the available upstream repositories.

allowed_switches = {}
cmd_name = 'push-upstream'
direction = 'to'
do_our_verb(builder, co_label, vcs_handler, upstream, repo)
required_tag = 'checked_out'
verb = 'push'
verbing = 'Pushing'
class muddled.commands.QueryBuildDescBranch

Bases: muddled.commands.QueryCommand

Syntax:muddle query build-desc-branch

Report the branch of the build description, and whether it is being used as the (default) branch for other checkouts.

If there are sub-domains in the build tree, then this reports the branch of the top-level build description, which is the only build description that can request checkouts to “follow” its branch.

cmd_name = 'query build-desc-branch'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCheckoutBranches

Bases: muddled.commands.QueryCommand

Syntax:muddle query checkout-branches

Print the known checkouts and their branches.

For each checkout, reports its current branch, and the branch implied (or explicitly requested) by the build description, and the branch it is “following” (if the build description has set this).

For instance:

--------  --------------  ---------------  ----------------
Checkout  Current branch  Original branch  Branch to follow
--------  --------------  ---------------  ----------------
builds    master          <none>           <not following>
co1       master          <none>           <not following>
co2       <can't tell>    <none>           <not following>

In this example, both checkouts are in git (which is the only VCS for which muddle really supports branches), and on master.

The “original branches” are both <none>. The “builds” checkout contains the build description, and its original checkout is <none> because “muddle init” did not specify a branch. The original checkouts for “co1” and “co2” are <none> because the build description did not specify explicit branches for them. We can’t tell what the current branch is for co2, which normally means that it has not yet been checked out.

Since the build description does not set “builder.follow_desc_build_branch = True”, all the checkouts show as <not following>.

Here is a slightly more complicated case:

--------  --------------   ---------------  ----------------
Checkout  Current branch   Original branch  Branch to follow
--------  --------------   ---------------  ----------------
builds    test-v0.1        branch0          <it's own>
co1       test-v0.1        <none>           test-v0.1
co2       branch1          branch1          <none>
co3       <none>           <none>           <none>
co4       <none>           branch1          <none>
co5       <not supported>  ...              <not following>

This build tree was created using “muddle init -branch branch0”, so the builds checkout shows “branch0” as its original branch. We can tell that the build description does have “builder.follow_desc_build_branch = True” because there are values in the “Branch to follow” column. The build description always follows itself.

“co1” doesn’t specify a particular branch in the build description, so its “original branch” is <none>. However, it does follow the build description, so has “test-v0.1” in the “Branch to follow” column.

“co2” explicitly specifies “branch1” in the build description. It got checked out on “branch1”, and is still on it. Having an explicit branch means it does not follow the build description.

“co3” explicitly specified a *revision” in the build description. This means that it got checked out on a detached HEAD, and thus its current branch is <none> - it really isn’t on a branch.

System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.QueryCheckoutBranches, line 57); backlink

Inline emphasis start-string without end-string.

“co4” explicitly specified a branch (“branch1”) and a revision in the build description. The revision id specified didn’t correspond to HEAD of the branch, so it too is on a detached HEAD. However, “branch1” still shows up as its original branch.

Finally, “co5” is not using git (it was actually using bzr), and thus muddle does not support branching it. However, it has set the “no_follow” VCS option in the build description, and thus the “Branch to follow” column shows as “<not following>” instead of ”...”.

(You can use “muddle query checkout-vcs” to see which VCS is being used for which checkout.)

cmd_name = 'query checkout-branches'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCheckoutDirs

Bases: muddled.commands.QueryCommand

Syntax:muddle query checkout-dirs

Print the known checkouts and their checkout paths (relative to ‘src/’)

cmd_name = 'query checkout-dirs'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCheckoutId

Bases: muddled.commands.QueryCommand

Syntax:muddle query checkout-id [<label>]

Report the VCS revision id (or equivalent) for the named checkout.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘checkout:’. If the label is a ‘package:’ label, and that package depends upon a single checkout, then report the id for that checkout.

If <label> is not given, and the current directory is within a checkout directory, then use that checkout.

The id returned is that which would be written to a stamp file as the checkout’s revision id.

cmd_name = 'query checkout-id'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCheckoutLicenses

Bases: muddled.commands.QueryCommand

Syntax:muddle query checkout-licenses

Print information including:

  • the known checkouts and their licenses
  • which checkouts (if any) have GPL licenses of some sort
  • which checkouts are “implicitly” GPL licensed because of depending on a GPL-licensed checkout
  • which packages have declared that they don’t actually need to be “implicitly” GPL
  • which checkouts have irreconcilable clashes between “implicit” GPL licenses and their actual license.

Note that “irreconcilable clashes” are only important if you intend to distribute the clashing items to third parties.

See also “muddle query role-licenses” for licenses applying to (packages in) each role.

cmd_name = 'query checkout-licenses'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCheckoutRepos

Bases: muddled.commands.QueryCommand

Syntax:muddle query checkout-repos [-u[rl]]

Print the known checkouts and their checkout repositories

With ‘-u’ or ‘-url’, print the repository URL. Otherwise, print the full spec of the Repository instance that represents the repository.

So, for instance, the standard printout produces lines of the form:

checkout:kernel/* -> Repository('git', 'ssh://git@server/project99/src', 'kernel', prefix='linuxbase', branch='linux-3.2.0')

but with ‘-u’ one would instead see:

checkout:kernel/* -> ssh://git@server/project99/src/linuxbase/kernel
allowed_switches = {'-u': 'url', '-url': 'url'}
cmd_name = 'query checkout-repos'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCheckoutVcs

Bases: muddled.commands.QueryCommand

Syntax:muddle query checkout-vcs

Print the known checkouts and their version control systems. Also prints the VCS options for the checkout, if there are any.

cmd_name = 'query checkout-vcs'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCheckouts

Bases: muddled.commands.QueryCommand

Syntax:muddle query checkouts [-j]

Print the names of all the checkouts described in the build description.

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query checkouts'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryCommand

Bases: muddled.commands.Command

The base class for ‘query’ commands

get_label(builder, args)
get_label_from_fragment(builder, args, default_type='package')
requires_build_tree()
class muddled.commands.QueryDefaultDeployments

Bases: muddled.commands.QueryCommand

Syntax:muddle query default-deployments [-j]

Print the names of the default deployments described in the build description (as defined using ‘builder.by_default_deploy()’).

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query default-deployments'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryDefaultRoles

Bases: muddled.commands.QueryCommand

Syntax:muddle query default-roles [-j]

Print the names of the default roles described in the build description (as defined using ‘builder.add_default_role()’).

These are the roles that will be assumed for ‘package:’ label fragments.

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query default-roles'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryDepend

Bases: muddled.commands.QueryCommand

Syntax:muddle query dependencies <what>
Or:muddle query dependencies <what> <label>

Print the current dependency sets.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

If no label is given, then all dependencies in the current build tree will be shown.

In order to show all dependency sets, even those where a given label does not actually depend on anything, <what> can be:

  • system - Print synthetic dependencies produced by the system
  • user - Print dependencies entered by the build description
  • all - Print all dependencies

To show only those dependencies where there is a dependency, add ‘-short’ (or ‘_short’) to <what>, i.e.:

  • system-short - Print synthetic dependencies produced by the system
  • user-short - Print dependencies entered by the build description
  • all-short - Print all dependencies
cmd_name = 'query dependencies'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryDeployments

Bases: muddled.commands.QueryCommand

Syntax:muddle query deployments [-j]

Print the names of all the deployments described in the build description.

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query deployments'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryDir

Bases: muddled.commands.QueryCommand

Syntax:muddle query dir <label>

Print a directory:

  • for checkout labels, the checkout directory
  • for package labels, the install directory
  • for deployment labels, the deployment directory

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

Typically used in a muddle Makefile, as for instance:

KBUS_INSTALLDIR:=$(shell $(MUDDLE) query dir package:kbus{*})
cmd_name = 'query dir'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryDistributions

Bases: muddled.commands.QueryCommand

Syntax:muddle query distributions

List the names of the distributions defined by the build description, and the license categories that each distributes.

cmd_name = 'query distributions'
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.QueryDomains

Bases: muddled.commands.QueryCommand

Syntax:muddle query domains [-j]

Print the names of all the subdomains described in the build description (and recursively in the subdomain build descriptions).

Note that it does not report the ‘’ (top level) domain, as that is assumed.

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query domains'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryEnv

Bases: muddled.commands.QueryCommand

Syntax:muddle query env <label>

Print the environment in which this label will be run.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query env'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryEnvs

Bases: muddled.commands.QueryCommand

Syntax:muddle query all-env <label>

Print a list of the environments that will be merged to create the resulting environment for this label.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query all-env'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryInstDetails

Bases: muddled.commands.QueryCommand

Syntax:muddle query inst-details <label>

Print the list of actual instructions for this label, in the order in which they will be applied.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query inst-details'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryInstFiles

Bases: muddled.commands.QueryCommand

Syntax:muddle query inst-files <label>

Print the list of currently registered instruction files, in the order in which they will be applied.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query inst-files'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryKernelver

Bases: muddled.commands.QueryCommand

Syntax:muddle query kernelver <label>

Determine the Linux kernel version.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

<label> should be the package label for the kernel version. This command looks in <obj>/obj/include/linux/version.h (where <obj> is the directory returned by “muddle query objdir <label>”) for the LINUX_VERSION_CODE definition, and attempts to decode that.

It prints out the Linux version, e.g.:

muddle query kernelver package:linux_kernel{boot}/built
2.6.29
cmd_name = 'query kernelver'
kernel_version(builder, kernel_pkg)

Given the label for the kernel, determine its version.

with_build_tree(builder, current_dir, args)
class muddled.commands.QueryLicenses

Bases: muddled.commands.QueryCommand

Syntax:muddle query licenses

Print the standard licenses we know about.

See “muddle query checkout-licenses” to find out about any licenses defined, and used, in the build description.

cmd_name = 'query licenses'
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.QueryLocalRoot

Bases: muddled.commands.QueryCommand

Syntax:muddle query localroot <label>

Print the “local root” directory for a label.

For a label representing a checkout, package or deployment in the top-level, prints out the normal root directory (as “muddle query root”).

For a label in a subdomain, printes out the root directory for said subdomain (i.e., the directory containing its .muddle/ directory).

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query localroot'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryMakeEnv

Bases: muddled.commands.QueryCommand

Syntax:muddle query make-env <label>

Print the environment in which “make” will be called for this label.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

Specifically, print what muddle adds to the environment (so it leaves out anything that was already in the environment when muddle was called). Note that various things (lists of directories) only get set up when the directories actually exists - so, for instance, MUDDLE_INCLUDE_DIRS will only include directories for the packages depended on that have already been built. This means that this command shows the environment actually as would be used if one did muddle buildlabel, but not necessarily as it would be for muddle build, when the dependencies themselves would be built first. (It would be difficult to do otherwise, as the environment built is always as small as possible, and it is not until a package has been built that muddle can tell which directories will be present.

cmd_name = 'query make-env'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryMatch

Bases: muddled.commands.QueryCommand

Syntax:muddle query match <label>

Print out any labels that match the label given. If the label is not wildcarded, this just reports if the label is known.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query match'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryName

Bases: muddled.commands.QueryCommand

Syntax:muddle query name

Print the build name, as specified in the build description with:

builder.build_name = "Project32"

This prints just the name, so that one can use it in the shell - for instance in bash:

export PROJECT_NAME=$(muddle query name)

or in a muddle Makefile:

build_name:=$(shell $(MUDDLE) query name)
cmd_name = 'query name'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryNeededBy

Bases: muddled.commands.QueryCommand

Syntax:muddle query needed-by <label>

Print what we need to build to build this label.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query needed-by'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryNeeds

Bases: muddled.commands.QueryCommand

Syntax:muddle query needs <label>

Print what this label is required to build.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query needs'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryObjdir

Bases: muddled.commands.QueryCommand

Syntax:muddle query objdir <label>

Print the object directory for a label.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

Typically used in a muddle Makefile, as for instance:

KBUS_OBJDIR:=$(shell $(MUDDLE) query objdir package:kbus{*})
cmd_name = 'query objdir'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryPackageRoles

Bases: muddled.commands.QueryCommand

Syntax:muddle query package-roles [-j]

Print the names of all the packages, and their roles, as described in the build description.

Note that if there is a rule for a package with a wildcarded name, like “package:{x86}/”, then ‘*’ will be included in the names printed.

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query package-roles'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryPackages

Bases: muddled.commands.QueryCommand

Syntax:muddle query packages [-j]

Print the names of all the packages described in the build description.

Note that if there is a rule for a package with a wildcarded name, like “package:{x86}/”, then ‘*’ will be included in the names printed.

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query packages'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryPreciseEnv

Bases: muddled.commands.QueryCommand

Syntax:muddle query precise-env <label>

Print the environment pertaining to exactly this label (no fuzzy matches)

cmd_name = 'query precise-env'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryRelease

Bases: muddled.commands.QueryCommand

Syntax:muddle query release [-labels]

Print information about this build as a release, including the release specification, and the “translation” of the special “_release” argument.

(That content, what is to be released, is defined in the build description, using ‘builder.add_to_release_build()’.)

For instance:

$ muddle query release
This is a release build
Release spec:
  name        = simple
  version     = v1.0
  archive     = tar
  compression = gzip
  hash        = c7c10cf4d6da4519714ac334a983ab518c68c5d1
What to release (the meaning of "_release", before expansion):
  _default_deployments
  package:(subdomain2)second_pkg{x86}/*

or:

$ muddle query release
This is NOT a release build
Release spec:
  name        = None
  version     = None
  archive     = tar
  compression = gzip
  hash        = None
What to release (the meaning of "_release", before expansion):
  _default_deployments
  package:(subdomain2)second_pkg{x86}/*

If nothing has been designated for release, then that final clause will be replaced with:

What to release (the meaning of "_release"):
  <nothing defined>

With the ‘-labels’ switch, just prints out that last list of “what to release”:

$ muddle query release -labels
_default_deployments
package:(subdomain2)second_pkg{x86}/*

The ‘-labels’ variant prints nothing out if nothing has been designated for release.

allowed_switches = {'-labels': 'labels'}
cmd_name = 'query release'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryRoleLicenses

Bases: muddled.commands.QueryCommand

Syntax:muddle query role-licenses [-no-clashes]

Print the known roles and the licenses used within them (i.e., by checkouts used by packages with those roles).

If -no-clashes is given, then don’t report binary/private license clashes (which might cause problems when doing a “_by_license” distribution).

See also “muddle query checkout-licenses” for information on licenses with respect to checkouts.

allowed_switches = {'-no-clashes': 'no-clashes'}
cmd_name = 'query role-licenses'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryRoles

Bases: muddled.commands.QueryCommand

Syntax:muddle query roles [-j]

Print the names of all the roles described in the build description.

With ‘-j’, print them all on one line, separated by spaces.

allowed_switches = {'-j': 'join'}
cmd_name = 'query roles'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryRoot

Bases: muddled.commands.QueryCommand

Syntax:muddle query root

Print the root path, the path of the directory containing the ‘.muddle/’ directory.

For a build containing subdomains, this means the root directory of the top-level build.

The root is where “muddle where” will print “Root of the build tree”.

cmd_name = 'query root'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryRules

Bases: muddled.commands.QueryCommand

Syntax:muddle query rules <label>

Print the rules covering building this label.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query rules'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryTargets

Bases: muddled.commands.QueryCommand

Syntax:muddle query targets <label>

Print the targets that would be built by an attempt to build this label.

<label> is a label or label fragment (see “muddle help labels”). The default type is ‘package:’.

cmd_name = 'query targets'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryUnused

Bases: muddled.commands.QueryCommand

Syntax:muddle query unused [<label> [...]]

Report on labels that are defined in the build description, but are not “used” by the targets. With no arguments, the targets are the default deployables. The argument “_all” means all available deployables (not just the defaults). Otherwise, arguments are labels.

cmd_name = 'query unused'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryUpstreamRepos

Bases: muddled.commands.QueryCommand

Syntax:muddle query upstream-repos [-u[rl]] [<co_label>]

Print information about upstream repositories.

If <co_label> is given then it should be a checkout label or label fragment (see “muddle help labels”).

If a label or labels are given, then the repositories, and any upstream repositories, for those labels are reported. Otherwise, those repositories that have upstream repositories are reported.

With ‘-u’ or ‘-url’, print repository URLs. Otherwise, print the full spec of each Repository instance.

XXX Examples to be provided

allowed_switches = {'-u': 'url', '-url': 'url'}
cmd_name = 'query upstream-repos'
with_build_tree(builder, current_dir, args)
class muddled.commands.QueryVCS

Bases: muddled.commands.QueryCommand

Syntax:muddle query vcs

List the version control systems supported by this version of muddle, together with their VCS specifiers.

cmd_name = 'query vcs'
do_command()
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.Rebuild

Bases: muddled.commands.PackageCommand

Syntax:muddle rebuild [ <package> ... ]

Rebuild packages. Just like build except that we clear any ‘/built’ tags first (and their dependencies).

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/postinstalled”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

  1. For each label, clear its ‘/built’ tag, and then clear the tags for all the labels that depend on it. Note that this will include the same label with its ‘/installed’ and ‘/postinstalled’ tags.
  2. For each label, build its ‘/postinstalled’ tag (so essentially, do the equivalent of “muddle build”).
build_these_labels(builder, labels)
cmd_name = 'rebuild'
class muddled.commands.Reconfigure

Bases: muddled.commands.PackageCommand

Syntax:muddle reconfigure [ <package> ... ]

Reconfigure packages. Just like configure except that we clear any ‘/configured’ tags first (and their dependencies).

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/configured”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

  1. For each label, clear its ‘/configured’ tag, and then clear the tags for all the labels that depend on it. Note that this will include the same label with its ‘/built’, ‘/installed’ and ‘/postinstalled’ tags.
  2. Do “muddle configure” for each label.
build_these_labels(builder, labels)
cmd_name = 'reconfigure'
required_tag = 'configured'
class muddled.commands.Redeploy

Bases: muddled.commands.DeploymentCommand

Syntax:muddle redeploy [<deployment> ... ]

Clean the named deployments (deleting their ‘deploy/’ directory), remove their ‘/deployed’ tags, and then rebuild (deploy) them.

This is exactly equivalent to doing “muddle cleandeploy” for all the labels, followed by “muddle deploy” for them all.

<deployment> should be a label fragment specifying a deployment, or one of _all and friends, as for any deployment command. The <type> defaults to “deployment”, and the deployment <tag> will be “/deployed”. See “muddle help labels” for more information.

If no deployments are named, what we do depends on where we are in the build tree. See “muddle help labels”.

build_these_labels(builder, labels)
cmd_name = 'redeploy'
class muddled.commands.Reinstall

Bases: muddled.commands.PackageCommand

Syntax:muddle reinstall [ <package> ... ]

Reinstall packages (but don’t rebuild them).

<package> should be a label fragment specifying a package, or one of _all and friends, as for any package command. The <type> defaults to “package”, and the package <tag> will be “/postinstalled”. See “muddle help labels” for more information.

If no packages are named, what we do depends on where we are in the build tree. See “muddle help labels”.

  1. For each label, clear its ‘/installed’ tag, and then clear the tags for all the labels that depend on it. Note that this will include the same label with its ‘/postinstalled’ tag.
  2. For each label, build its ‘/postinstalled’ tag (so essentially, do the equivalent of “muddle build”).
build_these_labels(builder, labels)
cmd_name = 'reinstall'
class muddled.commands.Release

Bases: muddled.commands.Command

Produce a customer release from a release stamp file.

Syntax:muddle release <release-file>
Or:muddle release -test <release-file>

For example:

$ muddle release project99-1.2.3.release

This:

  1. Checks the current directory is empty, and refuses to proceed if it is not.

    We always recommend doing muddle init or muddle bootstrap in an empty directory, but muddle insists that muddle release must be done in an empty directory.

  2. Does muddle unstamp <release-file>,

  3. Copies the release file to .muddle/Release.

    The existence of this file indicates that this is a release build tree, and “normal” muddle will refuse to build in it.

  4. Copies the release specification to .muddle/ReleaseSpec.

  5. Sets some extra environment variables, which can be used in the normal manner in muddle Makefiles:

    • MUDDLE_RELEASE_NAME is the release name, from the release file.
    • MUDDLE_RELEASE_VERSION is the release version, from the release file.
    • MUDDLE_RELEASE_HASH is the SHA1 hash of the release file

    “Normal” muddle will also create those environment variables, but they will be set to (unset).

  6. Does mudddle build _release.

    The meaning of “_release” is defined in the build description, using builder.add_to_release_build(). See:

    $ muddle doc mechanics.Builder.add_to_release_build
    

    for more information on that method, and “muddle query release” for the current setting.

    Note that, if youi have subdomains, only calls of add_to_release_build() in the top-level build description will be effective.

  7. Creates the release directory, which will be called <release-name>_<release-version>_<release-sha1>. It copies the release file therein.

  8. Calls the release_from(builder, release_dir) function in the build description, which is responsible for copying across whatever else needs to be put into the release directory.

    (Obviously it is an error if the build description does not have such a function.)

    Note that, if you have subdomains, only the release_from() function in the top-level build will be called.

  9. Creates a compressed tarball of the release directory, using the compression mechanism specified in the release file. It will have the same basename as the release directory.

If the -test switch is given, then items 1..2 are not done. This allows testing a release build in the current build directory. The produce of such a test must not be treated as a proper release, as it has not involved a clean build of the build tree. Note that if you want to make your build tree back into a normal muddle build tree, then you will need to delete the .muddle/Release file yourself, by hand.

allowed_switches = {'-test': 'test'}
calc_tf_name(release, release_dir)

Work out the name and mode of the archive file we want to generate.

cmd_name = 'release'
do_release(muddle_binary, current_dir, release_file, testing)
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.Reparent

Bases: muddled.commands.CheckoutCommand

Syntax:muddle reparent [-f[orce]] [ <checkout> ... ]

Re-associate the specified checkouts with their remote repositories.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/pulled”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

Some distributed VCSs (notably, Bazaar) can “forget” the remote repository for a checkout. In Bazaar, this typically means not remembering the “parent” repository, and thus not being able to pull. It appears to be possible to end up in this situation if network disconnection happens in an inopportune manner.

This command attempts to reassociate each checkout to the remote repository as named in the muddle build description. If ‘-force’ is given, then this will be done even if the remote repository is already known, otherwise it will only be done if it is necessary.

For Bazaar: Reads and (maybe) edits .bzr/branch/branch.conf.

  • If “parent_branch” is unset, sets it.
  • With ‘-force’, sets “parent_branch” regardless, and also unsets “push_branch”.
allowed_in_release_build()
allowed_switches = {'-force': 'force', '-f': 'force'}
build_these_labels(builder, labels)
cmd_name = 'reparent'
required_tag = 'pulled'
class muddled.commands.Retract

Bases: muddled.commands.AnyLabelCommand

Syntax:muddle retract <label> [ <label> ... ]

Retract the given labels and their consequents.

This unsets the tags specified in the given labels, and also the tags for all labels which each label depended on. For instance, if the label package:fred{x86}/built was given, then package:fred{x86}/configured would also be retracted, as /built (normally) depends on /configured for the same package.

This command is mostly for use by experts and scripts.

Each <label> is a label fragment, in the normal manner. The <type> defaults to “package:”, and the <tag> defaults to the normal default <tag> for that type. Wildcards are expanded.

<label> may also be “_all”, “_default_deployments”, “_default_roles” or “_just_pulled”.

See “muddle help labels” for more help on label fragments and the “_xxx” values.

build_these_labels(builder, labels)
cmd_name = 'retract'
class muddled.commands.Retry

Bases: muddled.commands.AnyLabelCommand

Syntax:muddle retry <label> [ <label> ... ]

First this unsets the tags implied by the specified label(s), and only those tags. Then it rebuilds the labels.

Note that unsetting the tags only unsets exactly the tags named, and not any others.

This is sometimes useful when you’re messing about with package rebuild rules.

Each <label> is a label fragment, in the normal manner. The <type> defaults to “package:”, and the <tag> defaults to the normal default <tag> for that type. Wildcards are expanded.

<label> may also be “_all”, “_default_deployments”, “_default_roles” or “_just_pulled”.

See “muddle help labels” for more help on label fragments and the “_xxx” values.

build_these_labels(builder, labels)
cmd_name = 'retry'
class muddled.commands.RunIn

Bases: muddled.commands.Command

Syntax:muddle runin <label> <command> [ ... ]

Run the command “<command> [ ...]” in the directory corresponding to every label matching <label>.

  • Checkout labels are run in the directory corresponding to their checkout.
  • Package labels are run in the directory corresponding to their object files.
  • Deployment labels are run in the directory corresponding to their deployments.

We only ever run the command in any directory once.

<label> may be a label fragment, or one of the _xxx arguments If it is a label fragment, and the label type is not given, then “checkout:” is assumed. If it is an _xxx argument, then it may not be _all, since it would not be clear what type of label to expand it to). See “muddle help label _all” for more information on these values.

In practice, it is often simplest to use a shell script for <command>, rather than trying to work out the appropriate quoting rules for whatever command is actually wanted.

cmd_name = 'runin'
requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.StampDiff

Bases: muddled.commands.Command

Syntax:muddle stamp diff [<style>] <path1> <path2> [<output_file>]

Compare two builds, as version stamps.

Each of <path1> and <path2> may be an existing stamp file, or the top-level directory of a muddle build tree (i.e., the directory that contains the ‘.muddle’ and ‘src’ directories).

If <output_file> is given, then the results of the comparison will be written to it, otherwise they will be written to standard output.

<style> specifies the way the comparison is done:

  • -u, -unified - output a unified difference between stamp files.
  • -c, -context - output a context difference between stamp files. This uses a “before/after” style of presentation.
  • -n, -ndiff - use Python’s “ndiff” to output the difference between stamp files. This is normally a more human-friendly set of differences, but outputs the parts of the files that match as well as those that do not.
  • -h, -html - output the difference between stamp files as an HTML page, displaying the files in two columns, with differences highlighted by colour.
  • -d, -direct - output the difference between two VersionStamp datastructures (this is the datastructure used to hold a stamp file internally within muddle). This is the default.

NOTE that at the moment ‘-d’ only compares checkout information, not repository and domain information. It also ignores any “problems” in the stamp file.

For textual comparisons between stamp files, “muddle stamp diff” will first generate a temporary stamp file, if necessary (i.e., if <path1> or <path2> is a build tree), using the equivalent of “muddle stamp save”.

For direct (‘-d’) comparison, a VersionStamp will be created from the build tree or read from the stamp file, as appropriate.

cmd_name = 'stamp diff'
compare_stamps(muddle_binary, args)
diff(path1, path2, file1, file2, diff_style='unified', fd=<open file '<stdout>', mode 'w'>)

Output a comparison of two stamp files

diff_direct(path1_is_build, path2_is_build, path1, path2, stamp1, stamp2, fd)

Output comparison using VersionStamp instances.

Currently, only compares the checkouts.

XXX TODO It should compare everything (including any problems!)

print_syntax()
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.StampPull

Bases: muddled.commands.Command

Syntax:muddle stamp pull [<repository_url>]

This performs a VCS “pull” operation for the “versions/” directory. This assumes that the versions repository is defined in .muddle/VersionsRepository.

If a <repository_url> is given, then that is used as the remote repository for the pull, and also saved as the “current” remote repository in .muddle/VersionsRepository.

(If the VCS being used is Subversion, then <repository> is ignored by the actual “pull”, but will still be used to update the VersionsRepository file. So be careful.)

If a <repository_url> is not given, then the repository URL named in .muddle/VersionsRepository is used. If there is no repository specified there, then the operation will fail.

See ‘unstamp’ for restoring from stamp files.

allowed_in_release_build()
cmd_name = 'stamp pull'
requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.StampPush

Bases: muddled.commands.Command

Syntax:muddle stamp push [<repository_url>]

This performs a VCS “push” operation for the “versions/” directory. This assumes that the versions repository is defined in .muddle/VersionsRepository.

If a <repository_url> is given, then that is used as the remote repository for the push, and also saved as the “current” remote repository in .muddle/VersionsRepository.

(If the VCS being used is Subversion, then <repository> is ignored by the actual “push”, but will still be used to update the VersionsRepository file. So be careful.)

If a <repository_url> is not given, then the repository URL named in .muddle/VersionsRepository is used. If there is no repository specified there, then the operation will fail.

‘stamp push’ does not (re)create a stamp file in the “versions/” directory - use ‘stamp version’ to do that separately.

See ‘unstamp’ for restoring from stamp files.

allowed_in_release_build()
cmd_name = 'stamp push'
requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.StampRelease

Bases: muddled.commands.Command

Syntax:muddle stamp release [<switches>] <release-name> <release-version>
Syntax:muddle stamp release [<switches>] <release-name> -next
Or:muddle stamp release [<switches>] -template

This is similar to “stamp version”, but saves a release stamp file - a stamp file that describes a release of the build tree.

The release stamp file written will be called:

versions/<release_name>_<release_version>.release

The “versions/” directory is at the build root (i.e., it is a sibling of the ”.muddle/” and “src/” directories). If it does not exist, it will be created.

If the VersionsRepository is set (in the .muddle/ directory), and it is a distributed VCS (e.g., git or bzr) then git init (or bzr init, or the equivalent) will be done in the directory if necessary, and then the file will be added to the local working set in that directory. For subversion, the file adding will be done, but no attempt will be made to initialise the directory.

If the -next option is used, then the version number will be guessed. Muddle will look in the “versions/” directory for all the ”.release” files whose names start with <release_name>_v, and will work out the last version number (as <major>.minor>) present (not that 1.01 is the same as 1.1). It will then use 0.0 if it didn’t find anything, or will use the next <minor> value. So if the user asked for muddle stamp release Fred -next and the files in “versions/” were:

Fred_v1.1.release
Fred_v3.02.release
Graham_v9.9.release

then the next version number would be guessed as v3.3.

If the -template option is used, then the file created will be called:

versions/this-is-not-a-file-name.release

and both the release name and release version values in the file will be set to <REPLACE THIS>. The user will have to rename the file, and edit both of those to sensible values, before using it (well, we don’t enforce renaming the file, but...).

<switches> may be:

  • -archive <name>

    This specifies how the release will be archived. At the moment the only permitted value is “tar”.

  • -compression <name>

    This specifies how the archive will be compressed. The default is “gzip”, and at the moment the only other alternative is “bzip2”.

See “muddle release” for using release files to build a release.

Note that release files are also valid stamp files, so “muddle unstamp” can be used to retore a build tree from them.

allowed_in_release_build()
cmd_name = 'stamp release'
guess_next_version_number(version_dir, name)

Return our best guess as to the next (minor) version number.

requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.StampSave

Bases: muddled.commands.Command

Syntax:muddle stamp save [<switches>] [<filename>]

Go through each checkout, and save its remote repository and current brach/revision id/number to a file.

This is intended to be enough information to allow reconstruction of the entire build tree, as-is.

<switches> may be:

  • -before <when> - use the (last) revision id at or before <when>
  • -f, -force - “force” a revision id
  • -h, -head - use HEAD for all checkouts
  • -v <version>, -version <version> - specify the version of stamp file

These are explained more below. Switches may occur before or after <filename>.

If a <filename> is specified, then output will be written to a file called either <filename>.stamp or <filename>.partial. If <filename> already ended in ‘.stamp’ or ‘.partial’, then the old extension will be removed before deciding on whether to use ‘.stamp’ or ‘.partial’.

If a <filename> is not specified, then a file called <sha1-hash>.stamp or <sha1-hash>.partial will be used, where <sha1-hash> is a hexstring representation of the hash of the content of the file.

The ‘.partial’ extension will be used if it was not possible to write a full stamp file (revisions could not be determined for all checkouts, and neither ‘-force’ nor ‘-head’ was specified). An attempt will be made to give useful information about what the problems are.

If a file already exists with the name ultimately chosen, that file will be overwritten.

If ‘-before’ is specified, then use the (last) revision id at or before that date and time. <when> is left a bit unspecified at the moment, and thus this feature is experimental.

XXX At the moment ‘-before’ is only supported for git and bzr, and
XXX thus any form of date/time/revision id that git and/or will accept
XXX may be used for <when>. The simple for “yyyy-mm-dd hh:mm:ss” seems
XXX acceptabl to both.

For instance:

muddle stamp save -before "2012-06-26 23:00:00"

If ‘-f’ or ‘-force’ is specified, then attempt to “force” a revision id, even if it is not necessarily correct. For instance, if a local working directory contains uncommitted changes, then ignore this and use the revision id of the committed data. If it is actually impossible to determine a sensible revision id, then use the revision specified by the build description (which defaults to HEAD). For really serious problems, this may refuse to guess a revision id, in which case the ‘stamp save’ process should stop with the relevant checkout.

(Typical use of ‘-f’ is expected to be when a ‘stamp save’ reports problems in particular checkouts, but inspection shows that these are artefacts that may be ignored, such as an executable built in the source directory.)

Note that if ‘-before’ is specified, ‘-force’ will be ignored.

If ‘-h’ or ‘-head’ is specified, then HEAD will be used for all checkouts. In this case, the repository specified in the build description is used, and the revision id and status of each checkout is not checked.

By default, a version 2 stamp file will be created. This is equivalent to specifying ‘-version 2’. If ‘-version 1’ is specified, then a version 1 stamp file will be created instead. This is the version of stamp file understood by muddle before it was able to create version 2 stamp files (see ‘muddle help stamp save’ to see if this is the case for a particular version of muddle or not). Note that the version 1 stamp file created by muddle 2.3 and above is not absolutely guaranteed to be correct.

See “muddle unstamp” for restoring from stamp files.

cmd_name = 'stamp save'
decide_stamp_filename(hash, basename=None, partial=False)

Return filename, given a SHA1 hash hexstring, and maybe a basename.

If ‘partial’, then the returned filename will have extension ‘.partial’, otherwise ‘.stamp’.

If the basename is not given, then the main part of the filename will be <hash>.

If the basename is given, then if it ends with ‘.stamp’ or ‘.partial’ then that will be removed before it is used.

requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.StampVersion

Bases: muddled.commands.Command

Syntax:muddle stamp version [-f[orce]|-v[ersion] <version>]

This is similar to “stamp save”, but using a pre-determined stamp filename.

Specifically, the stamp file written will be called:

versions/<build_name>.stamp

or, if the build description has asked for checkouts to follow its branch with builder.follow_build_desc_branch = True:

versions/<build_name>.<branch_name>.stamp

The “versions/” directory is at the build root (i.e., it is a sibling of the ”.muddle/” and “src/” directories). If it does not exist, it will be created.

If the VersionsRepository is set (in the .muddle/ directory), and it is a distributed VCS (e.g., git or bzr) then git init (or bzr init, or the equivalent) will be done in the directory if necessary, and then the file will be added to the local working set in that directory. For subversion, the file adding will be done, but no attempt will be made to initialise the directory.

<build_name> is the name of this build, as specified by the build description (by setting builder.build_name). If the build description does not set the build name, then the name will be taken from the build description file name. You can use “muddle query name” to find the build name for a particular build.

If a full stamp file cannot be written (i.e., if the result would have extension ”.partial”), then the version stamp file will not be written.

Note that ‘-f’ is supported (although perhaps not recommended), but ‘-h’ is not.

By default, a version 2 stamp file will be created. This is equivalent to specifying ‘-version 2’. If ‘-version 1’ is specified, then a version 1 stamp file will be created instead. This is the version of stamp file understood by muddle before it was able to create version 2 stamp files (see ‘muddle help stamp version’ to see if this is the case for a particular version of muddle or not). Note that the version 1 stamp file created by muddle 2.3 and above is not absolutely guaranteed to be correct.

See “muddle unstamp” for restoring from stamp files.

allowed_in_release_build()
cmd_name = 'stamp version'
requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.Status

Bases: muddled.commands.CheckoutCommand

Syntax:muddle status [-v] [-j] [-quick] [ <checkout> ... ]

Report on the status of checkouts that need attention.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/pulled”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

If ‘-v’ is given, report each checkout label as it is checked (allowing a sense of progress if there are many bazaar checkouts, for instance).

Runs the equivalent of git status or bzr status on each repository, and tries to only report those which have significant status.

Note: For subversion and bazaar, the (remote) repository is queried, which may be slow. For git, the HEAD of the remote may be queried.

Be aware that “muddle status” will report on the currently checked out checkouts. “muddle status _all” will (attempt to) report on all the checkouts described by the build, even if they have not yet been checked out. This will fail on the first checkout directory it can’t “cd” into (i.e., the first checkout that isn’t there yet).

This muddle command exits with status 0 if all checkouts appear alright, and with status 1 if no checkouts were specified, if an exception occurred, or if some checkouts need attention.

At the end, if any checkouts need attention, their names are reported. With ‘-j’, print them all on one line, separated by spaces.

The ‘-quick’ switch tells muddle not to make any (potentially slow) queries across the network, and is only supported for “git”. It will only look at the local information it already has, which means that the information it can gives depends upon whatever was last fetched into the local repository. It can typically inform you if there are local updates to be pushed, but will not (cannot) warn you if there are commits to be pulled.

allowed_in_release_build()
allowed_switches = {'-quick': 'quick', '-v': 'verbose', '-j': 'join'}
build_these_labels(builder, labels)
cmd_name = 'status'
required_tag = 'checked_out'
class muddled.commands.Subst

Bases: muddled.commands.Command

Syntax:muddle subst <src_file> [<xml_file>] <output_file>

Reads in <src_file>, and replaces any strings of the form “${..}” with values from the XML file (if any) or from the environment.

For the examples, I’m assuming we’re building a release build, using “muddle release”, and thus the MUDDLE_RELEASE_xxx environment variables are set.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Subst, line 11)

Unexpected section title.

Without an XML file
-------------------

So, in the two argument form, we might run:

$ muddle subst version.h.in version.h

on version.h.in:

#ifndef PROJECT99_VERSION_FILE
#define PROJECT99_VERSION_FILE
#define BUILD_VERSION "${MUDDLE_RELEASE_NAME}: $(MUDDLE_RELEASE_VERSION}"
#endif

to produce:

#ifndef PROJECT99_VERSION_FILE
#define PROJECT99_VERSION_FILE
#define BUILD_VERSION "simple: v1.0"
#endif

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.Subst, line 31)

Unexpected section title.

With an XML file
----------------

In the three argument form, values will first be looked up in the XML file, and then, if they’re not found, in the environment. So given values.xml:

<?xml version="1.0" ?>
<values>
    <version>Kynesim version 99</version>
    <more>
        <value1>This is value 1</value1>
        <value2>This is value 2</value2>
    </more>
</values>

and values.h.in:

#ifndef KYNESIM_VALUES
#define KYNESIM_VALUES
#define KYNESIM_VERSION "${/values/version}"
#define RELEASE_VERSION "Release version ${MUDDLE_RELEASE_VERSION}"
#endif

then running:

$ muddle subst values.h values.xml values.h.in

would give us values.h:

#ifndef KYNESIM_VALUES
#define KYNESIM_VALUES
#define KYNESIM_VERSION "Kynesim version 99"
#define RELEASE_VERSION "Release version v1.0"
#endif

XML queries are used in the “${..}” to extract particular values from the XML. These look a bit like XPath queries - “/elem/elem/elem...”, so for instance:

${/values/more/value2}

would be replaced by:

This is value 2

You can escape a “${ .. }” by passing “$${ .. }”, so:

$${/values/more/value1}

becomes:

${/values/more/value1}

Both ${/version} and ${“/version”} give the same result.

You can also nest evaluations. With the environment variable THING set to “/values/version”, then:

${ ${THING} }

will evaluate to:

Kynesim version 99

You can call functions with “${fn: .. }”. Parameters can be surrounded by matching double quotes - these will be stripped before the parameter is evaluated. The available functions are:

  • “${fn:val(something)}”

    This expands to the value of ‘something’ as a query (either as an environment variable or XPath)

  • “${fn:ifeq(something,b)c}”

    If ${something} evaluates to b, then this expands to c. Both b and c may contain “${..}” sequences.

    Note that ‘something’ is expanded without you needing to specify such, but b and c are not.

    It is allowed to do things like:

    ${fn:ifeq(/values/version,"Kynesim version 99")
        def missing_function(a):
            # 'Version ${/values/version} of the software does not provide
            # this function, so we do so here
            <implementation code>
     }
    
  • “${fn:ifneq(something,b)c}”

    The same, but you get c if evaluating ‘something’ does not give b.

  • “${fn:echo(a,b,c,...)}”

    Evaluates each parameter (a, b, c, ...) in turn. Spaces between parameters are ignored. So:

    ${fn:echo(a, " space ", ${/values/more/value1}}
    

    would give:

    a space This is value 1
    
cmd_name = 'subst'
do_subst(args)
requires_build_tree()
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.Sync

Bases: muddled.commands.CheckoutCommand

Syntax:muddle sync [ <checkout> ... ]
Or:muddle sync [-v[erbose]] [ <checkout> ... ]
Or:muddle sync [-show] [ <checkout> ...]

“Synchronise” each checkout onto the branch it should be on...

Less succinctly, for each checkout, do the first applicable of the following:

  • If this is the top-level build description, then:
    • if it has “builder.follow_build_desc_branch = True”, then nothing needs to be done, as we’re already there.
    • if it does not have “builder.follow_build_desc_branch = True”, but a branch was specified for it (i.e., via “muddle init -branch”), then go to that branch.
    • if it does not have “builder.follow_build_desc_branch = True”, and no branch was specified (at “muddle init”), then go to “master”.
  • If the build description specifies a revision for this checkout, go to that revision.
  • If the build description specifies a branch for this checkout, and the checkout VCS supports going to a specific branch, go to that branch
  • If the build description specifies that this checkout should not follow the build description (both Subversion and Bazaar support the “no_follow” option), then go to “master”.
  • If the build description specifies that this checkout is shallow, then give up.
  • If the checkout’s VCS does not support lightweight branching, then give up (the following choices require this).
  • If the build description has “builder.follow_build_desc_branch = True”, then go to the same branch as the build description.
  • Otherwise, go to “master”.

With ‘-v’ or ‘-verbose’, report in detail on what the “sync” operation is doing, and why.

With ‘-show’, report on its decision making process, but don’t actually do anything.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/changes_committed”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

allowed_switches = {'-v': 'verbose', '-verbose': 'verbose', '-show': 'show'}
build_these_labels(builder, labels)
cmd_name = 'sync'
required_tag = 'changes_committed'
class muddled.commands.UnCheckout

Bases: muddled.commands.CheckoutCommand

Syntax:muddle uncheckout [ <checkout> ... ]

Tell muddle that the given checkouts no longer exist in the ‘src/’ directory hierarchy, and will need to be checked out again before they can be used.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/checked_out”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For each label, unset the ‘/checked_out’ tag for that label, and the tags of any labels that depend on it. Note that this will include all the tags for any packages that directly depend on this checkout. However, it will not perform any “clean” or “distclean” actions for those packages.

Note that muddle itself does not check whether the checkout directory has been deleted or not. Attempting to do “muddle checkout” for a checkout directory that (still) exists will generally fail.

build_these_labels(builder, labels)
cmd_name = 'uncheckout'
class muddled.commands.UnStamp

Bases: muddled.commands.Command

To create a build tree from a stamp file:

Syntax:muddle unstamp <file>
Or:muddle unstamp <url>
Or:muddle unstamp <vcs>+<url>
Or:muddle unstamp <vcs>+<repo_url> <version_desc>

To update a build tree from a stamp file:

Syntax:muddle unstamp -u[pdate] <file>

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.UnStamp, line 13)

Unexpected section title.

Creating a build tree from a stamp file
---------------------------------------

The normal “unstamp” command reads the contents of a “stamp” file, as produced by the “muddle stamp” command, and:

  1. Retrieves each checkout mentioned
  2. Reconstructs the corresponding muddle directory structure
  3. Confirms that the muddle build description is compatible with the checkouts.

This form of the command cannot be used within an existing muddle build tree, as its intent is to create a new build tree.

The file may be specified as:

  • The local path to a stamp file.

    For instance:

    muddle stamp  thing.stamp
    mkdir /tmp/thing
    cp thing.stamp /tmp/thing
    cd /tmp/thing
    muddle unstamp  thing.stamp
    
  • The URL for a stamp file. In this case, the file will first be copied to the current directory.

    For instance:

    muddle unstamp  http://some.url/some/path/thing.stamp
    

    which would first copy “thing.stamp” to the current directory, and then use it. If the file already exists, it will be overwritten.

  • The “revision control specific” URL for a stamp file. This names the VCS to use as part of the URL - for instance:

    muddle unstamp  bzr+ssh://kynesim.co.uk/repo/thing.stamp
    

    This also copies the stamp file to the current directory before using it. Note that not all VCS mechanisms support this (at time of writing, muddle’s git support does not). If the file already exists, it will be overwritten.

  • The “revision control specific” URL for a repository, and the path to the version stamp file therein.

    For instance:

    muddle unstamp  bzr+ssh://kynesim.co.uk/repo  versions/ProjectThing.stamp
    

    This is intended to act somewhat similarly to “muddle init”, in that it will checkout:

    bzr+ssh://kynesim.co.uk/repo/versions
    

    and then unstamp the ProjectThing.stamp file therein.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/commands.py:docstring of muddled.commands.UnStamp, line 71)

Unexpected section title.

Updating a build tree from a stamp file
---------------------------------------

The “-update” form (“unstamp -update” or “unstamp -u”) also reads the contents of a “stamp” file, but it then tries to amend the current build tree to match the stamp file.

This form of the command must be used within an existing muddle build tree, as its intent is to alter it.

The stamp file must be specified as a local path - the URL forms are not supported.

The command looks up each checkout described in the stamp file. If it already exists, then it sets it to the correct revision, using “muddle pull”. This last means that the value “_just_pulled” will be set to those checkouts which have been pulled, so one can do, for instance, “muddle distrebuild _just_pulled”.

XXX Future versions of this command will also be able to change the branch of a checkout. This is not yet supported.

If the checkout does not exist, then it will be cloned, using “muddle checkout”. Newly cloned checkouts will not be represented in “_just_pulled”.

In the simplest case, the “unstamp -update” operation may just involve choosing different revisions on some checkouts.

Before using this form of the command, it is probably worth using:

muddle stamp diff . <file>

to determine what changes will be made.

After using this form of the command, it is highly recommended to use:

muddle veryclean

to delete the directories built from the checkout sources.

allowed_in_release_build()
allowed_switches = {'-u': 'update', '-update': 'update'}
check_build(current_dir, checkouts, muddle_binary)

Check that the build tree we now have on disk looks a bit like what we want...

cmd_name = 'unstamp'
print_syntax()
requires_build_tree()
restore_stamp(builder, current_dir, domains, checkouts)

Given the information from our stamp file, restore things.

unstamp_from_file(muddle_binary, current_dir, thing)

Unstamp from a file (local, over the network, or from a repository)

unstamp_from_repo(muddle_binary, current_dir, repo, version_path)

Unstamp from a repository and version path.

unstamp_from_stamp(muddle_binary, current_dir, stamp, versions_repo=None)

Given a stamp file, do our work.

update_from_file(builder, filename)

Update our build from the given stamp file.

update_from_stamp(builder, domains, checkouts)

Given the information from our stamp file, update the current build.

with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
class muddled.commands.Unimport

Bases: muddled.commands.CheckoutCommand

Syntax:muddle unimport [ <checkout> ... ]

Assert that the given checkouts haven’t been checked out and must therefore be checked out.

<checkout> should be a label fragment specifying a checkout, or one of _all and friends, as for any checkout command. The <type> defaults to “checkout”, and the checkout <tag> will be “/checked_out”. See “muddle help labels” for more information.

If no checkouts are named, what we do depends on where we are in the build tree. See “muddle help labels”.

For each label, unset the ‘/checked_out’ tag for that label.

This command does not do anything to labels that depend on the given labels - if you want that, see “muddle removed”.

Note that muddle itself does not check whether the checkout directory has been deleted or not. Attempting to do “muddle checkout” for a checkout directory that (still) exists will generally fail.

build_these_labels(builder, labels)
cmd_name = 'unimport'
class muddled.commands.UpstreamCommand

Bases: muddled.commands.CheckoutCommand

The parent class for the push/pull-upstream commands.

allowed_switches = {}
build_these_labels(builder, labels, upstream_names, no_op)
direction = 'to'
do_our_verb(builder, co_label, vcs_handler, upstream, repo)

Each subclass needs to implement this.

handle_label(builder, co_label, upstream_name, repo)
required_tag = 'checked_out'
verb = 'push'
verbing = 'Pushing'
with_build_tree(builder, current_dir, args)

Our command line is somewhat differently shaped.

So we have to handle it ourselves.

class muddled.commands.VeryClean

Bases: muddled.commands.Command

Syntax:muddle veryclean

Sets the muddle build tree back to just checked out sources.

  1. Delete the obj/, install/ and deploy/ directories.
  2. Removes all the package tags, so muddle thinks that packages have not had anything done to them.
  3. Removes all the deployment tags, so muddle thinks that all deployments have not been deployed.

For a build tree without subdomains, this is equivalent to:

$ rm -rf obj install deploy
$ rm -rf .muddle/tags/package
$ rm -rf .muddle/tags/deployment

It’s a bit more complicated if there are any subdomains. If there is a ‘domains/’ directory, then this command will recurse down into it and perform the same operation for each subdomain it finds. This does not depend on whether the subdomain is defined in the build description - it is done purely on the basis of what directories are actually present.

As usual, ‘muddle -n veryclean’ will report on what it would do, without actually doing it.

Using this helps prevent unwanted built/installed software “building up” in the obj and install (and, to a lesser extent, deploy) infrastructure. Note that it does not remove checkouts that are no longer in use, nor can it do anything about any build artefacts inside checkout directories.

cmd_name = 'veryclean'
requires_build_tree()
with_build_tree(builder, current_dir, args)
class muddled.commands.Whereami

Bases: muddled.commands.Command

Syntax:muddle where [-detail]
Or:muddle whereami [-detail]

Looks at the current directory and tries to identify where it is within the enclosing muddle build tree. If it can calculate a label corresponding to the location, it will also report that (as <name> and, if appropriate, <role>).

For instance:

$ muddle where
Root of the build tree
$ cd src; muddle where
Checkout directory
$ cd main_co; muddle where
Checkout directory for checkout:main_co/*
$ cd ../../obj/main_pkg; muddle where
Package object directory for package:main_pkg{*}/*

If you’re not in a muddle build tree, it will say so:

You are here. Here is not in a muddle build tree.

If the ‘-detail’ switch is given, output suitable for parsing is output, in the form:

<what> <label> <domain>

i.e., a space-separated triple of items that don’t themselves contain whitespace. For instance:

$ muddle where
Checkout directory for checkout:screen-4.0.3/*
$ muddle where -detail
Checkout checkout:screen-4.0.3/* None
cmd_name = 'where'
requires_build_tree()
want_detail(args)
with_build_tree(builder, current_dir, args)
without_build_tree(muddle_binary, current_dir, args)
muddled.commands.build_a_kill_b(builder, labels, build_this, kill_this)

For every label in labels, build the label formed by replacing tag in label with build_this and then kill the tag in label with kill_this.

We have to interleave these operations so an error doesn’t lead to too much or too little of a kill.

muddled.commands.build_labels(builder, to_build)
muddled.commands.command(command_name, category, aliases=None)

A simple decorator to remmember a class by its command name.

‘category’ indicates which type of command this is

muddled.commands.in_category(command_name, category)
muddled.commands.kill_labels(builder, to_kill)
muddled.commands.subcommand(main_command, sub_command, category, aliases=None)

Remember the class for <main_command> <subcommand>.

muddled.cpiofile

Note

Provides CPIO file support

Utilities to write cpio archives.

There is apparently no standard way to do this from python. Ugh.

class muddled.cpiofile.Archive

Bases: object

Represents a CPIO archive.

Files are represented by their header information and the name of a file in the filesystem where they live. When render() is called, the appropriate data is written out to disc.

add_file(a_file)

DANGER WILL ROBINSON! You need to add files in the right order here or cpio will get very confused.

Todo

Reorder files in render() so that we get them in the right order, and remember to create intermediate directories.

add_files(in_files)
render(to_file, logProgress=False)

Render a CPIO archive to the given file.

class muddled.cpiofile.CpioFileDataProvider(hierarchy)

Bases: muddled.filespec.FileSpecDataProvider

Given a file map like that returned from files_from_fs() and a root name, create a filespec data provider.

name is the root of the effective hierarchy.

abs_match(filespec, vroot=None)

Return a list of the file object for each file that matches filespec.

list_files_under(dir, recursively=False, vroot=None)

Return a list of the files under dir.

class muddled.cpiofile.File

Bases: object

Represents a file in a CPIO archive.

Create a new, empty file. Some more or less sensible defaults are set up so that if you do try to synthesise a cpio archive from a default-constructed File you don’t get utter rubbish. No guarantees you get a valid archive either though ..

  • key_name is the name of the key under which this file is stored in the parent hierarchy. It’s a complete hack, but essential for finding a file in the key map quickly without which deletion becomes an O(n^2) operation (and N = number of files in the root fs, so it’s quite high).
  • self.name - is the name of the file in the target archive.
  • self.fs_name - is the name of the file in the underlying filesystem.
  • self.orig_file - is the name of the file from which the data in this file object comes.
S_BLK = 24576
S_CHAR = 8192
S_DIR = 16384
S_REG = 32768
S_SGID = 1024
S_SOCKET = 49152
S_STICKY = 512
S_SUID = 2048
as_str(fs_relative=None)

Report ourself, but present the file system name relative to ‘fs_relative’

delete_child_with_name(in_name)
rename(name)
set_contents(data)
set_contents_from_file(file_name)
class muddled.cpiofile.Hierarchy(map, roots)

Bases: object

  • self.map - maps names in the target archive to file objects.
  • self.roots - is a subset of self.map that just maps the root objects.
as_str(fs_relative=None)

Report ourself, but present the file system names relative to ‘fs_relative’

erase_target(file_name)

Recursively remove file_name and all its descendants from the hierarchy.

merge(other)

Merge other with self.

We need to keep the hierarchy sensibly updated.

We merge the maps. We then kill all children and iterate over everything in the resulting map, finding a parent to add it to. This fails to preserve the order of files in directories, but the result is correct in every other way.

Anything in the result that does not have a parent is a root.

normalise()

Normalise the hierarchy into one with a single root.

We do this by taking each root in turn and removing a component, creating a directory in the process. If the resulting root is in roots, we add it to the children of that root and eliminate it from the map.

Iterate until there is only one root left.

parent_from_key(key_name)
put_target_file(name, obj)

Put a file into the archive. The directory for it must already exist.

  • name - The name of the file in the target archive.
  • obj - The file object to insert.
render(to_file, logProgress=False)
muddled.cpiofile.file_for_dir(name)

Create a vague attempt at a directory entry.

muddled.cpiofile.file_from_data(name, data)

Creates a File object from some explicit data you give it.

muddled.cpiofile.file_from_fs(orig_file, new_name=None)

Create a file object from a file on disc. You’ll want to rename() it, unless you meant the file in the CPIO archive to have the same name as the one you passed in.

muddled.cpiofile.hierarchy_from_fs(name, base_name)

Create a hierarchy of files from a named object in the filesystem.

The files will be named with ‘base_name’ substituted for ‘name’.

Returns a Hierarchy with everything filled in.

muddled.cpiofile.merge_maps(dest, src)

Merge src into dest. This needs special handling because we need to keep the hierarchy intact

We merge dest and src and then just rebuild the entire hierarchy - it’s the easiest way, frankly.

For everything in the merged list, zap its children.

Now iterate over everything, os.path.split() it to find its parent and add it to its parents’ child list.

If its parent doesn’t exist, it’s a root - mark and ignore it.

muddled.cpiofile.trace_files(file_list, root)

Given a File, add it and all its children, top-down, into file_list. Used as a utility routine by Hierarchy.

muddled.db

Note

Handles the content of the .muddle directory

Contains code which maintains the muddle database, held in root/.muddle

class muddled.db.CheckoutData(vcs_handler, repo, co_dir, co_leaf)

Bases: object

  • location - The directory the checkout is in, relative to the root of the build tree. For instance:

    src/builds
    domains/subdomain1/src/first_co
    
  • dir and leaf - The same information, as it was originally specified in the build description. This is primarily of use in version stamping. The dir may be None, and the leaf defaults to the checkout labels name.

    We expect to take the repository described in ‘repo’ and check it out into:

    • src/<co_leaf> or
    • src/<co_dir>/<co_leaf>

    depending on whether <co_dir> is None.

  • repo - A Repository instance, representing where the checkout is checked out from. For example (eliding the actual URL):

    Repository('git', 'http://.../main', 'builds')
    Repository('git', 'http://.../subdomain1', 'first_co')
    

    This the Repository as defined in the build description.

  • vcs_handler - A VCS handler, which knows how to do version control operations for this checkout.

  • options - Any specialised optons needed by the VCS handler. At the moment, the only option available is whether a git checkout is shallow or not. This is a dictionary.

move_to_subdomain(other_domain_name)
set_option(name, value)

Add/replace the named VCS option ‘name’.

For reasons mostly to do with how stamping/unstamping works, we require option values to be either boolean, integer or string.

Also, only those option names that are explicitly allowed for a particular VCS may be used.

class muddled.db.Database(root_path)

Bases: object

Represents the muddle database

Historically, this class represented the muddle database as stored in the .muddle directory (on disk). Since we expect the user (and code) to edit these files frequently, we deliberately do not cache their values (other than, well, as themselves in the .muddle directory).

Since then however, we have also gained some dictionaries linking checkout labels to particular quantities.

It’s useful to have a single place for most such dictionaries because when we do subdomain manipulation (i.e., taking a build description and including its build tree into another as a subdomain) we need to change all the labels in the new subdomain to reflect that fact. The fewer places we have to worry about that, the better.

So, we remember:

  • root_path - The path to the root of the build tree.

Various PathFile instances:

  • RootRepository_pathfile - for the ‘.muddle/RootRepository’ file
  • Description_pathfile - for the ‘.muddle/Description’ file
  • VersionsRepository_pathfile - for the ‘.muddle/VersionsRepository’ file
  • DescriptionBranch_pathfile - for the ‘.muddle/DescriptionBranch’ file

which describe what the user requested via the original “muddle init”.

and:

  • local_labels - Transient labels which are “asserted”, via ‘set_tag()’, and queried via ‘is_tag()’. This functionality is used inside the Builder’s “build_label()” mechanism, and is only intended for use within muddle itself.

Also, a variety of dictionaries that take (mostly) checkout labels as keys. Note that:

  1. All the keys are “normalised” to have an unset label tag.
  2. Thus it is assumed that the dictionaries will only be accessed via the methods supplied for this purpose.
  3. The existence of an entry does not necessarily imply that the particular checkout is actually used, as the need for it may have gone away during a builder.unify() operation.

and, perhaps most importantly, the user should treat all of these as READ ONLY, since muddle itself maintains their content.

The dictionaries we use are:

  • checkout_data - This maps checkout labels to the information we need to do checkout actions.

  • checkout_vcs - This is a cache remembering the VCS for a checkout. It is not intended for direct access.

  • checkout_licenses - This maps a checkout label to a License instance, representing the source code license under which this checkout’s source code is being used. For instance:

    checkout:builds/*               -> License('MPL 1.1', 'open-source')
    checkout:(subdomain1)first_co/* -> License('LGPL v3', 'gpl')
    

    In the case of a checkout that has multiple licenses, the license that is being “used” should be indicated.

    Note that not all checkouts will necessarily have licenses associated with them.

  • checkout_license_files - Some licenses require distribution of a license file from within the checkout, even in binary distributions. In such cases, this dictionary maps the checkout label to the name of the license file, relative to the checkout directory.

  • license_not_affected_by is a dictionary of the form:

    { package_label : set( gpl_checkout_labels ) }

    Each gpl_checkout_label is a label whose license (typically GPL) might be expected to “propagate” to any package built against that checkout. This dictionary is used to tell the system that, whilst ‘package_label’ may depend upon any of the ‘gpl_checkout_labels’, it doesn’t build against them in a way that actually causes that propagation.

    So, for instance, if we have LGPL checkout label which our package links to as a dynamic library, we’d want to tell muddle that the package depends on the checkout, but doesn’t get affected by the GPL nature of its license.

    This sort of thing is necessary because muddle itself has no way of telling.

    Note that ALL labels in this dictionary and its constituent sets should have their tags set to ‘*’, so it is expected that this dictionary will be accessed using set_license_not_affected_by() and get_license_not_affected_by().

  • nothing_builds_against is a set of checkout labels, each of which is a checkout that (presumably) has a GPL license, but against which no package links in a way that will cause GPL license “propagation”.

  • upstream_repositories is a dictionary of the form:

    { repo : { repo, set(names) } }

    That is, the key is a Repository instance (normally expected to be the same as one of the values in the checkout_repos dictionary), and the value is a dictionary whose keys are other repositories (“upstream” repositories) and some names associated with them.

    The same names may be associated with more than one upstream repository. It is also conceivable that an upstream repository might also act as a key, if it in turn has upstream repositories (whether this is strictly necessary is unclear - XXX still to decide whether to support this).

    XXX Note that if a checkout is branched because it is following the build description branch, the checkout Repository in checkot_repositories will get a new .branch value, at which point it will not necessarily compare equal to the Repository we’re using as a key. This uncertainty MAY be a problem, and it may be better if add_upstream_repo() takes a copy of the Repository object before using it as a key, to force the issue.

  • domain_build_desc_label is a dictionary of the form:

    { domain : build-desc-label }

    where domain is the domain from a label (so None or a suitable string), and build-desc-label is the (normalised) checkout label for the build description checkout for that domain.

    Clearly, there is always at least one entry, with key None, for the top-level build description.

Initialise a muddle database with the given root_path.

add_upstream_repo(orig_repo, upstream_repo, names)

Add an upstream repo to ‘orig_repo’.

  • ‘orig_repo’ is the original Repository that we are adding an upstream for.
  • ‘upstream_repo’ is the upstream Repository. It is an error if that repository is already an upstream of ‘orig_repo’.
  • ‘names’ is a either a single string, or a sequence of strings, that can be used to select this (and possibly other) upstream repositories.

Upstream repository names must be formed of A-Z, a-z, 0-9 and underscore or hyphen.

build_desc_file_name()

Return the filename of the build description.

checkout_has_license(checkout_label)

Return True if the named checkout has a license registered

clear_all_instructions(domain=None)

Clear all instructions - essentially only ever called from the command line.

clear_tag(label)
commit()

Commit changes to the db back to disc.

Remember to call this function when anything of note happens - don’t assume you aren’t about to hit an exception.

db_file_name(rel)

The full path name of the given relative filename in the current build tree.

dump_checkout_licenses(just_name=False)

Report on the licenses associated with our checkouts.

If ‘just_name’ is true, then report the licenses name, otherwise report the full License definition.

dump_checkout_paths()
dump_checkout_repos(just_url=False)

Report on the repositories associated with our checkouts.

If ‘just_url’ is true, then report the repository URL, otherwise report the full Repository definition (which shows branch and revision as well).

dump_checkout_vcs()

Report on the version control systems associated with our checkouts, and any VCS options.

dump_domain_build_desc_labels()
dump_upstream_repos(just_url=False)

Report on the upstream repositories associated “default” repositories

If ‘just_url’ is true, then report the repository URL, otherwise report the full Repository definition (which shows branch and revision as well).

get_checkout_data(checkout_label)
get_checkout_dir_and_leaf(checkout_label)
get_checkout_license(checkout_label, absent_is_None=False)

Returns the License instance for this checkout label

If ‘absent_is_None’ is true, then if ‘checkout_label’ does not have an entry in the licenses dictionary, None will be returned. Otherwise, an appropriate GiveUp exception will be raised.

get_checkout_license_file(checkout_label, absent_is_None=False)

Returns the License file for this checkout label

If ‘absent_is_None’ is true, then if ‘checkout_label’ does not have an entry in the license files dictionary, None will be returned. Otherwise, an appropriate GiveUp exception will be raised.

get_checkout_location(checkout_label)

‘checkout_label’ is a “checkout:” Label, or None

If it is None, then “src” is returned.

Otherwise, the path to the checkout directory for this label, relative to the root of the build tree, is calculated and returned.

If you want the full path to the checkout directory, then use get_checkout_path().

get_checkout_path(checkout_label)

‘checkout_label’ is a “checkout:” Label, or None

If it is None, then “<root path>/src” is returned.

Otherwise, the path to the checkout directory for this label is calculated and returned.

If you want the path relative to the root of the build tree (i.e., a path starting “src/”), then use get_checkout_location().

get_checkout_repo(checkout_label)

Returns the Repository instance for this checkout label

get_checkout_vcs(checkout_label)

‘checkout_label’ is a “checkout:” Label.

Returns the VCS handler for the given checkout.

Raises GiveUp (containing an explanatory message) if we cannot find that checkout label.

get_checkout_vcs_options(checkout_label)

‘checkout_label’ is a “checkout:” Label.

Returns the options for the given checkout, as a (possibly empty) dictionary.

Since most checkouts will not have options, and will thus have no entry for such, cannot return an error if there is no such checkout in the build.

get_domain_build_desc_label(domain)

‘domain’ is a domain as taken from a label, so None or a string

If it is None, then the checkout label for the top-level build description is returned.

Otherwise, the checkout label for the build description for that domain is returned.

In either case, the checkout label will be normalised (so its tag will be ‘*’)

Raises GiveUp with an appropriate message if ‘domain’ is not recognised.

get_license_not_affected_by(this_label)

Find what is registered as not affecting this label’s license

That is, the things on which this package depends, that appear to be GPL and propagate, but against which we have been told we do not actually build, so the license is not, in fact, propagated.

Returns a (possibly empty) set of checkout labels, each with tag ‘*’.

get_nothing_builds_against(co_label)

Return True if this label is in the “not linked against” set.

get_subdomain_info(domain_name)

Return the root repository and build description for a subdomain.

Reads the RootRepository and Description files in the (sub)domain’s ”.muddle” directory.

get_upstream_repos(orig_repo, names=None)

Retrieve the upstream repositories for ‘orig_repo’

If ‘names’ is given, it must be a sequence of strings, in which case only those upstream repositories annotated with any of the names will be returned.

Returns a list of tuples of the form:

(upstream repositories, matching names)

This will be empty if there are no upstream repositories for ‘orig_repo’, or none with any of the names in ‘names’ (if given).

In the case of ‘names’ being empty, ‘matching names’ will contain the names registered for that upstream repository.

NB: ‘matching names’ is a tuple with the names sorted, and the list returned is also sorted.

include_domain(other_builder, other_domain_name)

Include data from other_builder, built in other_domain_name

This method is the main reason why this class gets to hold so much information - it gives us a single place to concentrate much of the knowledge about including subdomains.

Note we rely upon all the labels in the other domain already having been altered to reflect their subdomain-ness

This should only be called by muddle itself.

instruction_file_dir(domain=None)

Return the name of the directory in which we keep the instruction files

instruction_file_name(label)

If this label were to be associated with a database file containing the (absolute) filename of an instruction file to use for this package and role, what would it be?

is_tag(label)

Is this label asserted?

print_upstream_repo_info(orig_repo, co_labels, just_url)

Print upstream repository information.

‘orig_repo’ is the “main” repository, the one that is not upstream

‘co_labels’ is a sequence of 0 or more checkout labels, which are associated with that repository.

If ‘just_url’ is true, then report the repository URL, otherwise report the full Repository definition (which shows branch and revision as well).

scan_instructions(lbl)

Returns a list of pairs (label, filename) indicating the list of instruction files matching lbl. It’s up to you to load and sort them (but load_instructions() will help with that).

set_checkout_data(checkout_label, co_data)
set_checkout_license(checkout_label, license)
set_checkout_license_file(checkout_label, license_file)

Set the license file for this checkout.

set_checkout_vcs_option(checkout_label, option_name, option_value)

‘checkout_label’ is a “checkout:” Label.

set_domain_build_desc_label(checkout_label)

This should only be called by muddle itself.

set_domain_marker(domain_name)

Mark this as a (sub)domain

In a (sub)domain, we have a file called .muddle/am_subdomain, which acts as a useful flag that we are a (sub)domain.

set_instructions(label, instr_file)

Set the name of a file containing instructions for the deployment mechanism.

  • label -
  • instr_file - The InstructionFile object to set.

If instr_file is None, we unset the instructions.

set_license_not_affected_by(this_label, co_label)

Asserts that the license for ‘co_label’ does not affect ‘pkg_label’

We assume that:

  1. ‘this_label’ is a package that depends (perhaps indirectly) on ‘co_label’, or is a checkout directly required by such a package.
  2. ‘co_label’ is a checkout with a “propagating” license (i.e., some form of GPL license).
  3. Thus by default the “GPL”ness would propagate from ‘co_label’ to ‘this_label’ (and, if it is a package, to the checkouts it is (directly) built from).

However, this function asserts that, in fact, this is not true. Our checkout is (or our checkouts are) not built in such a way as to cause the license for ‘co_label’ to propagate.

Or, putting it another way, for a normal GPL license, we’re not linking with anything from ‘co_label’, or using its header files, or copying GPL’ed files from it, and so on.

If ‘co_label’ is under LGPL, then that would reduce to saying we’re not static linking against ‘co_label’ (or anything else not allowed by the LGPL).

Note that we may be called before ‘co_label’ has registered its license, so we cannot actually check that ‘co_label’ has a propagating license (or, indeed, that it exists or is depended upon by ‘pkg_label’).

set_nothing_builds_against(co_label)

Indicate that no-one links against this checkout.

...or, at least, not in a way to cause GPL license “propagation”.

set_tag(label)

Assert this label.

setup(repo_location, build_desc, versions_repo=None, branch=None)

Set values for the files in .muddle that describe our intial state.

  • ‘repo_location’ is written to .muddle/RootRepository

  • ‘build_desc’ is written to .muddle/Description

  • If ‘versions_repo’ is not None, it is written to .muddle/VersionsRepository. Note that “not None” means that a value of ‘’ will be written to the file.

    If ‘versions_repo’ is None, and ‘repo_location’ is not a centralised VCS (i.e., subversion), then it will be written to .muddle/VersionsRepository instead.

  • If ‘branch’ is not None, then it will be written to .muddle/DescriptionBranch

This method should only be called by muddle itself.

tag_file_name(label)

If this file exists, the given label is asserted.

To make life a bit easier, we group labels.

upstream_name_re = <_sre.SRE_Pattern object>
class muddled.db.Instruction

Bases: object

Something stored in an InstructionFile.

Subtypes of this type are mainly defined in the instr.py module.

clone_from_xml(xmlNode)

Given an XML node, create a clone of yourself, initialised from that XML or raise an error.

equal(other)

Return True iff self and other represent the same instruction.

Not __eq__() because we want the python identity to be object identity as always.

outer_elem_name()

What’s the outer element name for this instructiont type?

to_xml(doc)

Given an XML document, return a node which represents this instruction

class muddled.db.InstructionFactory

Bases: object

An instruction factory.

from_xml(xmlNode)

Given an xmlNode, manufacture an Instruction from it or return None if none could be built

class muddled.db.InstructionFile(file_name, factory)

Bases: object

An XML file containing a sequence of instructions for deployments. Each instruction is a subtype of Instruction.

file_name Where this file is stored values A list of instructions. Note that instructions are ordered.

add(instr)

Add an instruction.

clear()
commit(file_name)

Commit an instruction list file back to disc.

equal(other)

Return True iff self and other represent the same set of instructions. False if they don’t.

get()

Retrieve the value of this instruction file.

get_xml()

Return an XML representation of this set of instructions as a string.

read()

Read our instructions from disc. The XML file in question looks like:

<?xml version="1.0"?>
<instructions priority=100>
 <instr-name>
   <stuff .. />
 </instr-name>
</instructions>

The priority is used by deployments when deciding in what order to apply instructions. Higher priorities get applied last (which is the logical way around, if you think about it).

save_as(file_name)
class muddled.db.JustPulledFile(file_name)

Bases: object

Our memory of the checkouts that have just been pulled.

Set the path to the _just_pulled file.

add(label)

Add the label to our local memory.

The label is not added to the _just_pulled file until commit() is called.

clear()

Clear the contents of the _just_pulled file, and our local memory.

If the _just_pulled files does not exist, does nothing

commit()

Commit our local memory to the _just_pulled file.

The labels are sorted before being written to the file.

Leaves the local memory intact after writing (it does not clear it).

get_from_disk()

Retrieve the contents of the _just_pulled file as a list of labels.

First clears the local memory, then reads the labels in the _just_pulled file into local memory, then returns that set as a sorted list.

is_pulled(label)
class muddled.db.PathFile(file_name)

Bases: object

Manipulates a file containing a single path name.

Create a PathFile object with the given filename.

commit()

Write the value of the PathFile to disc.

from_disc()

Retrieve the current value of the PathFile, directly from disc.

Returns None if there is a problem reading the PathFile.

Caches the value if there was one.

get()

Retrieve the current value of the PathFile, or None if there isn’t one.

Uses the cached value if that is believed valid.

get_if_it_exists()

Retrieve the current value of the PathFile, if it exists (on disk).

This variant does not try to cache the value.

set(val)

Set the value of the PathFile (possibly to None).

class muddled.db.TagFile(file_name)

Bases: object

An XML file containing a set of tags (statements).

clear(tag_value)

Clear the relevant tag value.

commit()

Commit an XML tagfile back to a file.

erase()

Erase this tag file.

get()

Retrieve the value of this tagfile.

read()

Read data in from the disc.

The XML file in question looks a bit like:

<?xml version="1.0"?>
<tags>
  <X />
  <Y />
</tags>
set(tag_value)

Set the relevant tag value.

muddled.db.load_instruction_helper(x, y)

Given two triples (l,f,i), compare i.prio followed by f.

muddled.db.load_instructions(in_instructions, a_factory)

Given a list of pairs (label, filename) and a factory, load each instruction file, sort the result by priority and filename (the filename just to ensure that the sort is stable across fs operations), and return a list of triples (label, filename, instructionfile).

  • in_instructions -
  • a_factory - An instruction factory - typically instr.factory.

Returns a list of triples (label, filename, instructionfile object)

muddled.depend

Note

The core of the dependency system, and Labels

Dependency sets and dependency management

class muddled.depend.Action

Bases: object

Represents an object you can call to “build” a tag.

build_label(builder, label)

Build the given label. Your dependencies have been satisfied.

  • in_deps - Is the set whose dependencies have been satisified.

Returns True on success, False or throw otherwise.

class muddled.depend.Label(type, name, role=None, tag='*', transient=False, system=False, domain=None)

Bases: object

A label denotes an entity in muddle’s dependency hierarchy.

A label is structured as:

<type>:<name>{<role>}/<tag>[<flags>]

or:

<type>:(<domain>)<name>{<role>}/<tag>[<flags>]

The <type>, <name>, <role> and <tag> parts are composed of the characters [A-Za-z0-9-+_], or the wildcard character ‘*’.

The <domain> name is composed of the same plus ‘(‘ and ‘)’.

The domain, role and flags are all optional.

Note

The label strings “type:name/tag” and “type:name{}/tag[]” are identical, although the former is the more usual form.)

The ‘+’ is allowed in label parts to allow for names like “g++”.

Domains are used when relating sub-builds to each other, and are not necessary when relating labels within the same build. It is not allowed to specify the empty domain as “()” - just omit the parentheses.

If necessary “sub-domains” are specified using nested domains – for instance:

(outer)
(outer(inner))
(outer(inner(even.innerer)))

This is intended to be unambiguous rather than pretty.

Note that wildcarding of a domain name currently only supports one level (i.e., the top “(*)”), and not wildcarding of nested domains.

If you do find yourself using multi-level domains, we would strongly suggest reconsidering your overall build design.

The “core” part of a label is the <name>{<role>} or (<domain>)<name>{<role>}. The <type> and <tag> can (typically) be thought of as tracking the progress of the “core” entity through its lifecycle (build sequence, etc.).

Names beginning with an underscore are reserved by muddle, so do not use them for other purposes.

(Why is the ‘domain’ argument at the end of the argument list? Because it was added after the other arguments were already well-established, and some uses of Label use positional arguments.)

Label instances are treated as immutable by the muddle system, although the implementation does not currently enforce this. Please don’t try to abuse this, as Bad Things will happen.

If you’re implementing a new copy-constructor and changing the new instance’s label before returning it, don’t forget to call rehash(). If you don’t, Really Bad Things will happen.

Note

The flags on a label are not immutable, and are regarded as transient annotations.

Note

When a domain is included as a subdomain, all of its labels are “adjusted” to have the new, appropriate domain name. This is clearly a special meaning of the word “immutable”. However, it should only be the muddle system itself doing this.

Because of this (potential change in content of a label), the domain name does not contribute to a label’s hash value. Thus a label that whose domain name is changed will continue to work as the same key in a dictionary (for instance).

Type:What kind of label this is. The standard muddle values are “checkout”, “package” and “deployment”. These values are defined programmatically via muddled.utils.LabelType. Thus the ‘type’ is conventionally used to indicate what general “stage” of the build process a label belongs to.
Name:The name of this checkout/package/whatever. This should be a useful mnemonic for the labels purpose.
Role:The role for this checkout/package/whatever. A role might delimit the target architecture of the labels it is used in (roles such as “x86”, “arm”, “beagleboard”), or the sort of purpose (“role” in the more traditionale sense, such as “boot”, “firmware”, “packages”), or some other useful delineation of a partition in the general label namespace (thinking of labels as points in an N-dimensional space).
Tag:A tag indicating more precisely what stage the label belongs to within each ‘type’. There are different conventional values according to the ‘type’ of the label (for instance, “checked_out”, “built”, “installed”, etc.). These values are defined programmatically via muddled.utils.LabelTag.
Transient:If true, changes to this tag will not be persistent in the muddle database. ‘transient’ is used to denote something which will go away when muddle is terminated - e.g. environment variables.
System:If true, marks this label as a system label and not to be reported (by ‘muddle depend’) unless asked for. System labels are labels “invented” by muddle itself to satisfy implicit dependencies, or to allow the build system as a whole to work.
Domain:The domain is used to specify which build or sub-build this label corresponds to. Nested “tail recursive” parenthesised components may be used to specify sub-domains (but this is not recommended). The domain defaults to the current build.

The role may be None, indicating (for instance) that roles are not relevant to this particular label.

The domain may be None, indicating that the label belongs to the current build. If the domain is given as ‘’ (empty string) then this is equivalent to None, and is stored as such. Do not specify domains unless you need to.

The kind, name, role and tag may be wildcarded, by being set to ‘*’. When evaluating dependencies between labels, for instance, a wildcard indicates “for any value of this part of the label”.

Domains can be wildcarded, and that probably means the obvious (that the label applies across all domains), but this may not yet be implemented. Wildcarding of sub-domains may never be supported.

Note that label flags (including specifically ‘transient’ and ‘system’) are not equality-preserving properties of a label - two labels are not made unequal just because they have different flags.

(In fact, no two labels should ever have different values for transience, for obvious reasons, and the system flag is intended only to limit over-reporting of information.)

For instance:

>>> Label('package', 'busybox')
Label('package', 'busybox', role=None, tag='*')
>>> str(_)
'package:busybox/*'
>>> Label('package', 'busybox', tag='installed')
Label('package', 'busybox', role=None, tag='installed')
>>> str(_)
'package:busybox/installed'
>>> Label('package', 'busybox', role='rootfs', tag='installed')
Label('package', 'busybox', role='rootfs', tag='installed')
>>> str(_)
'package:busybox{rootfs}/installed'
>>> Label('package', 'busybox', 'rootfs', 'installed')
Label('package', 'busybox', role='rootfs', tag='installed')
>>> str(_)
'package:busybox{rootfs}/installed'
>>> Label('package', 'busybox', role='rootfs', tag='installed', domain="arm.helloworld")
Label('package', 'busybox', role='rootfs', tag='installed', domain='arm.helloworld')
>>> str(_)
'package:(arm.helloworld)busybox{rootfs}/installed'
>>> Label('package', 'busybox', role='rootfs', tag='installed', domain="arm(helloworld)")
Label('package', 'busybox', role='rootfs', tag='installed', domain='arm(helloworld)')
>>> str(_)
'package:(arm(helloworld))busybox{rootfs}/installed'
FLAG_DOMAIN_SWEEP = 'D'
FLAG_SYSTEM = 'S'
FLAG_TRANSIENT = 'T'
copy()

Return a copy of this label.

copy_and_unify_with(target)

Return a copy of ourserlves, unified with the target.

All the non-wildcard parts of ‘target’ are copied, to overwrite the equivalent parts of the new label.

copy_with_domain(new_domain)

Return a copy of self, with the domain changed to new_domain.

Note that if ‘new_domain’ is given as “” (empty string) then that is treated as if it were given as None.

copy_with_role(new_role)

Return a copy of self, with the role changed to new_role.

copy_with_tag(new_tag, system=None, transient=None)

Return a copy of self, with the tag changed to new_tag.

domain
domain_part = '[()A-Za-z0-9._+-]+|\\*'
domain_part_re = <_sre.SRE_Pattern object>
fragment_re = <_sre.SRE_Pattern object at 0x28e60e0>
static from_fragment(fragment, default_type, default_role=None, default_domain=None)

Given a string containing a label fragment, return a Label.

The caller indicates the default type, role and domain.

The fragment must contain a <name>, but otherwise may contain any of:

  • <type>: - if this is not given, the default is used
  • (<domain>) - if this is not given, the default is used.
  • {<role>} - if this is not given, the default is used.
  • /<tag> - if this is not given, a tag appropriate to the <type> is chosen (checked_out, postinstalled or deployed)

Any of the default_xx values may be None.

static from_string(label_string)

Construct a Label from its string representation.

The string should be of the correct form:

  • <type>:<name>/<tag>
  • <type>:<name>{<role>}/<tag>
  • <type>:<name>/<tag>[<flags>]
  • <type>:<name>{<role>}/<tag>[<flags>]
  • <type>:(<domain>)<name>/<tag>
  • <type>:(<domain>)<name>{<role>}/<tag>
  • <type>:(<domain>)<name>/<tag>[<flags>]
  • <type>:(<domain>)<name>{<role>}/<tag>[<flags>]

See the docstring for Label itself for the meaning of the various parts of a label.

<flags> is a set of individual characters indicated as flags. There are two flags that will be recognised and used, ‘T’ for Transience and ‘S’ for System. Any other flag characters will be ignored.

If the label string is valid, a corresponding Label will be returned, otherwise a utils.GiveUp exception will be raised.

>>> Label.from_string('package:busybox/installed')
Label('package', 'busybox', role=None, tag='installed')
>>> Label.from_string('package:busybox{firmware}/installed[ABT]')
Label('package', 'busybox', role='firmware', tag='installed', transient=True)
>>> Label.from_string('package:(arm.hello)busybox{firmware}/installed[ABT]')
Label('package', 'busybox', role='firmware', tag='installed', transient=True, domain='arm.hello')
>>> Label.from_string('*:(*)*{*}/*')
Label('*', '*', role='*', tag='*', domain='*')
>>> Label.from_string('*:*{*}/*')
Label('*', '*', role='*', tag='*')
>>> Label.from_string('foo:bar{baz}/wombat[T]')
Label('foo', 'bar', role='baz', tag='wombat', transient=True)
>>> Label.from_string('foo:(ick)bar{baz}/wombat[T]')
Label('foo', 'bar', role='baz', tag='wombat', transient=True, domain='ick')
>>> Label.from_string('foo:(ick(ack))bar{baz}/wombat[T]')
Label('foo', 'bar', role='baz', tag='wombat', transient=True, domain='ick(ack)')

A tag must be supplied:

>>> Label.from_string('package:busybox')
Traceback (most recent call last):
...
GiveUp: Label string 'package:busybox' is not a valid Label

If you specify a domain, it may not be “empty”:

>>> Label.from_string('package:()busybox/*')
Traceback (most recent call last):
...
GiveUp: Label string 'package:()busybox/*' is not a valid Label
is_definite()

Return True iff this label contains no wildcards

is_wildcard()

Return True iff this label contains at least one wildcard.

This is the dual of is_definite(), but is provided so whichever seems more appropriate to the task at hand can be chosen.

just_match(other)

Return True if the labels match, False if they do not

label_part = '[A-Za-z0-9._+-]+|\\*'
label_part_re = <_sre.SRE_Pattern object>
label_string_re = <_sre.SRE_Pattern object at 0x266a770>
match(other)

Return an integer indicating the match specicifity - which we do by counting ‘*’ s and subtracting from 0.

Returns the match specicifity, None if there wasn’t one.

match_without_tag(other)

Returns True if other matches self without the tag, False otherwise

Specifically, tests whether the two Labels have identical type, domain, name and role.

middle()

Return the “middle” portion of our name, between type and tag.

That is, the domain, name and role (as appropriate).

For instance:

>>> checkout('fred').middle()
'fred'
>>> checkout('jim', domain='a(b)', tag='*').middle()
'(a(b))jim'
>>> package('fred', role='bob', domain='a').middle()
'(a)fred{bob}'
name
rehash()

Calculate the hash for a label.

Ignores the domain name (since that may be changed) and the transient and system flags (since they are defined to be, well, transient).

role
static split_domain(value)

Split a domain into its parts and check that it is valid.

Note that the ‘value’ should not include the outermost parentheses (see the examples below).

Raises a utils.GiveUp exception if it’s Bad.

For instance:

>>> Label.split_domain('fred')
['fred']
>>> Label.split_domain('fred(jim)')
['fred', 'jim']
>>> Label.split_domain('fred(jim(bob))')
['fred', 'jim', 'bob']
>>> Label.split_domain('')
Traceback (most recent call last):
...
GiveUp: Label domain '()' is not allowed
>>> Label.split_domain('()')
Traceback (most recent call last):
...
GiveUp: Label domain '(())' starts with zero length domain, '(()', i.e. '(('
>>> Label.split_domain('(')
Traceback (most recent call last):
...
GiveUp: Label domain part '(()' has unbalanced parentheses, '('
>>> Label.split_domain(')')
Traceback (most recent call last):
...
GiveUp: Label domain '())' has unbalanced parentheses, ')'
>>> Label.split_domain('fred(jim')
Traceback (most recent call last):
...
GiveUp: Label domain part '(fred(jim)' has unbalanced parentheses, 'fred(jim'
>>> Label.split_domain('fred((jim(bob)))')
Traceback (most recent call last):
...
GiveUp: Label domain '(fred((jim(bob))))' starts with zero length domain, '((jim(bob))', i.e. '(('
split_domains()

Returns a list of the domains for this Label, in order.

If there are no subdomains, then a zero length list is returned.

Raises a utils.GiveUp exception if the parentheses do not match up (the check is only fairly crude), or if there are two adjacent opening parentheses.

This is similar to utils.split_domain, but behaves somewhat differently.

tag
type
unifies(other)

Returns True if and only if every field in self is either equal to a field in other , or if other is a wildcard. Wildcards in self do not match anything but a wildcard in other.

class muddled.depend.Rule(target_dep, action)

Bases: object

A rule or “dependency set”.

Every Rule has:

  • a target Label (its desired result),
  • an optional Action object (to do the work to produce that result),
  • and a set of Labels on which the target depends (which must have been satisfied before this Rule can be triggered).

In other words, once all the dependency Labels are satisfied, the object can be called to ‘build’ the target Label.

(And if there is no object, the target is automatically satisfied.)

Note that the “set of Labels” is indeed a set, so adding the same Label more than once will not have any effect (caveat: adding a label with different flags from a previous label may have an effect, but it’s not something that should be relied on).

Note

The actual “satisfying” of labels is done in muddled.mechanics. For instance, Builder.build_label() “builds” a label in the context of the rest of its environment, and uses ‘action’ to “build” the label.

  • target_dep is the Label this Rule intends to “make”.
  • action is None or an Action, which will be used to “make” the target_dep.
add(label)

Add a dependency on the given Label.

catenate_and_merge(other_rule, complainOnDuplicate=False, replaceOnDuplicate=True)

Merge ourselves with the given rule.

If replaceOnDuplicate is true, other_rule get priority - this is the target for a unify() and makes the source build instructions go away.

depend_checkout(co_name, tag)

Add a dependency on label “checkout:<co_name>/<tag>”.

depend_deploy(dep_name, tag)

Add a dependency on label “deployment:<dep_name>/tag”.

depend_pkg(pkg, role, tag)

Add a dependency on label “package:<pkg>{<role>}/tag”.

merge(deps)

Merge another Rule with this one.

Adds all the dependency labels from deps to this Rule.

If deps.action is not None, replaces our action with the one from deps.

replace_target(new_t)
to_string(showSystem=True, showUser=True)

Return a string representing this dependency set.

If showSystem is true, include dependency labels with the System tag (i.e., dependencies inserted by the muddle system itself), otherwise ignore such.

If showUser is true, include dependency labels without the System tag (i.e., “normal” dependencies, explicitly added by the user), otherwise ignore such.

The default is to show all of the dependencies.

For instance (not a very realistic example):

>>> tgt = Label.from_string('package:fred{jim}/*')
>>> r = Rule(tgt,None)
>>> r.to_string()
'package:fred{jim}/* <- [ ]'
>>> r.add(Label.from_string('package:bob{bob}/built'))
>>> r.depend_checkout('fred','jim')
>>> r.depend_pkg('albert','jim','built')
>>> r.depend_deploy('hall','deployed')
>>> r.to_string()
'package:fred{jim}/* <- [ checkout:fred/jim, deployment:hall/deployed, package:albert{jim}/built, package:bob{bob}/built ]'

The “<-” is to be read “depends on”.

Note that the order of the dependencies in the output is sorted by label.

unify_dependencies(source, target)

Whenever source appears in our dependencies, replace it with source.unify(target)

class muddled.depend.RuleSet

Bases: object

A collection of rules that encapsulate how you can get from A to B.

Formally, this is just a mapping of labels to Rules. Duplicate targets are merged - it’s assumed that the objects will be the same.

Informally, we cache some of the label look-ups for a major improvement in build time; the half billion lookups were taking over a hundred seconds to do!

CAVEAT: Be aware that new rules (when added) can be merged into existing rules. Since we don’t copy rules when we add them, this could be a cause of unexpected side effects...

add(rule)

Add the Rule ‘rule’.

Specifically, if we already have a rule for this rule’s target label, merge the new rule into the old (see Rule.merge).

If this rule is for a new target, just remember it.

merge(other_deps)

Merge another RuleSet into this one.

Simply adds each rule from the other RuleSet to this one (see the ‘RuleSet.add’ method)

rule_for_target(target, createIfNotPresent=False)

Return the rule for this target - this contains all the labels that need to be asserted in order to build the target.

If createIfNotPresent is true, and there is no rule for this target, then we will create (and add to our internal map) an empty Rule for this target.

Otherwise, if there is no rule for this target, we return None

rules_for_target(label, useTags=True, useMatch=True)

Return the set of rules for any target(s) matching the given label.

  • If useTags is true, then we should take account of tags when matching, otherwise we should ignore them. If useMatch is true, then useTags is ignored.
  • If useMatch is true, then we allow wildcards in ‘label’, otherwise we do not.

Returns the set of Rules found, or an empty set if none were found.

rules_which_depend_on(label, useTags=True, useMatch=True)

Given a label, return a set of the rules which have it as one of their dependencies.

If there are no rules which have this label as one of their dependencies, we return the empty set.

  • If useTags is true, then we should take account of tags when matching, otherwise we should ignore them. If useMatch is true, then useTags is ignored.
  • If useMatch is true, then we allow wildcards in ‘label’, otherwise we do not.
targets_match(target, useMatch=True)

Return the set of targets matching the given ‘target’ label.

If useMatch is true, allow wildcards in ‘target’ (in which case more than one result may be obtained). If useMatch is false, then at most one match can be found (‘target’ itself).

Returns a set of suitable targets, or an empty set if there are none.

to_string(matchLabel=None, showUser=True, showSystem=True, ignore_empty=False)

Return a string representing this rule set.

If showSystem is true, include dependency labels with the System tag (i.e., dependencies inserted by the muddle system itself), otherwise ignore such.

If showUser is true, include dependency labels without the System tag (i.e., “normal” dependencies, explicitly added by the user), otherwise ignore such.

The default is to show all of the dependencies.

For instance (not a very realistic example):

>>> l = Label.from_string('package:fred{bob}/initial')
>>> r = RuleSet()
>>> depend_chain(None, l, ['built', 'bamboozled'], r)
>>> print str(r)
-----
package:fred{bob}/bamboozled <- [ package:fred{bob}/built ]
package:fred{bob}/built <- [ package:fred{bob}/initial ]
package:fred{bob}/initial <- [ ]
-----

The “<-” is to be read “depends on”.

Note that the order of the rules in the output is sorted by target label, and is thus reproducible.

unify(source, target)

Merge source into target.

This is a pain, and depends heavily on CatenatedObject

wrap_actions(generator, label)
class muddled.depend.SequentialAction(a, b)

Bases: object

Invoke two actions in turn

build_label(builder, label)
muddled.depend.checkout(name, tag='*', domain=None)

A simple convenience function to return a checkout label

  • ‘name’ is the checkout name
  • ‘tag’ is the label tag, defaulting to “*”
  • ‘domain’ is the label domain name, defaulting to None

For instance:

>>> l = checkout('fred')
>>> l == Label.from_string('checkout:fred/*')
True
muddled.depend.depend_chain(action, label, tags, ruleset)

Add a chain of dependencies to the given ruleset.

This is perhaps best explained with an example:

>>> l = Label.from_string('package:fred{bob}/initial')
>>> r = RuleSet()
>>> depend_chain(None, l, ['built', 'bamboozled'], r)
>>> print str(r)
-----
package:fred{bob}/bamboozled <- [ package:fred{bob}/built ]
package:fred{bob}/built <- [ package:fred{bob}/initial ]
package:fred{bob}/initial <- [ ]
-----
muddled.depend.depend_empty(action, label)

Create a dependency set with no prerequisites - simply signals that a tag is available to be built at any time.

muddled.depend.depend_none(action, label)

Quick rule that makes label depend on nothing.

muddled.depend.depend_one(action, label, dep_label)

Quick rule that makes label depend only on dep_label.

muddled.depend.depend_self(action, label, old_tag)

Make a quick dependency set that depends just on you. Used by some of the standard package and checkout classes to quickly build standard dependency sets.

muddled.depend.deployment(name, tag='*', domain=None)

A simple convenience function to return a deployment label

  • ‘name’ is the deployment name
  • ‘tag’ is the label tag, defaulting to “*”
  • ‘domain’ is the label domain name, defaulting to None

For instance:

>>> l = deployment('fred')
>>> l == Label.from_string('deployment:fred/*')
True
muddled.depend.label_from_string(str)

Do not use this!!! Can you say “deprecated”?

This function was originally removed, since it is replaced by Label.from_string. However, too many old builds still attempt to import it, which can cause problems at the “muddle init” stage, and also with “muddle unstamp”.

Please do not use this function in new builds.

muddled.depend.label_list_to_string(labels, join_with=' ')
muddled.depend.label_set_to_string(label_set, start_with='[', end_with=']', join_with=', ')

Utility function to convert a label set to a string.

muddled.depend.needed_to_build(ruleset, target, useTags=True, useMatch=False)

Given a rule set and a target, return a complete list of the rules needed to build the target.

  • If useTags is true, then we should take account of tags when looking for the rules for this ‘target’, otherwise we should ignore them.
  • If useMatch is true, then we allow wildcards in ‘target’, otherwise we do not.

Returns a list of rules.

muddled.depend.normalise_checkout_label(label, tag='*')

Given a checkout label with random “other” fields, normalise it.

Returns a normalised checkout label, with the role unset and the tag set to ‘tag’ (normally, “*”). This may be the same label (if it was already normalised), or it may be a new label. No guarantee is given of either.

Raise a MuddleBug exception if the label is not a checkout label.

A normalised checkout label:

  1. Has the given tag (normally ‘*’, sometimes LabelTag.CheckedOut)
  2. Does not have a role (checkout labels do not use the role)
  3. Does not have the system or transient flags set
  4. Has the same name and (if present) domain
muddled.depend.package(name, role, tag='*', domain=None)

A simple convenience function to return a package label

  • ‘name’ is the package name
  • ‘role’ is the package role
  • ‘tag’ is the label tag, defaulting to “*”
  • ‘domain’ is the label domain name, defaulting to None

For instance:

>>> l = package('fred', 'jim')
>>> l == Label.from_string('package:fred{jim}/*')
True
muddled.depend.required_by(ruleset, label, useTags=True, useMatch=True)

Given a ruleset and a label, form the list of labels that (directly or indirectly) depend on label. We deliberately do not give you the associated rules since you will want to call needed_to_build() individually to ensure that other prerequisites are satisfied.

The order in which we give you the labels gives you a hint as to a logical order to rebuild in (i.e. one the user will vaguely understand).

  • useMatch - If True, do wildcard matches, else do an exact comparison.
  • useTags - If False, we discount the value of a tag - this effectively
    results in a wildcard tag search.

Returns a set of labels to build.

muddled.depend.retag_label_list(labels, new_tag)

Does what it says on the tin, returning the new label list.

That is, returns a list formed by copying each Label in ‘labels’ and setting its tag to the given ‘new_tag’.

muddled.depend.rule_list_to_string(rule_list)

Utility function to convert a rule list to a string.

muddled.depend.rule_target_str(rule)

Take a rule and return its target as a string. Mainly used as an argument for map so we can print lists of rules sensibly.

muddled.depend.rule_with_least_dependencies(rules)

Given a (Python) set of rules, find the ‘best’ one to use.

This is actually impossible by any rational metric, so you usually only expect to call this function with a set of size 1, in which case our metric really doesn’t matter.

However, in a vague attempt to be somewhat intelligent, we return the element with the fewest direct dependencies.

muddled.deployment

Note

The base of deployment support

Common rules for deployments - basically just the clean rules.

class muddled.deployment.CleanDeploymentBuilder

Bases: muddled.depend.Action

build_label(builder, label)
muddled.deployment.deployment_depends_on_deployment(builder, what, depends_on, domain=None)

Inter-deployment dependencies. Aren’t you glad we have a general purpose dependency solver?

muddled.deployment.deployment_depends_on_roles(builder, deployment, roles, domain=None)

Make the deployment of the deployment with the given name depend on the installation of every package in the given role

muddled.deployment.deployment_rule_from_name(builder, name)

Return the rule for target label “deployment:<name>{}/deployed”.

Raises an exception if there is more than one such rule.

muddled.deployment.inform_deployment_path(builder, name, deployment, roles, domain=None)

Sets an environment variable to tell the given roles about the location of the given deployment.

Useful when e.g. some tools need to run other tools and therefore want to know where they are at build (rather than run)time.

muddled.deployment.pkg_depends_on_deployment(builder, pkg, roles, deployment, domain=None)

Make this package depend on the given deployment

Specifically, given each role ‘r’ in ‘roles’, make the label “package:<pkg>{<r>}/preconfig” depend on the label “deployment:<deployment>/deployed”.

If ‘domain’ is given, this is (currently) just used for the deploymet label.

muddled.deployment.register_cleanup(builder, deployment)

Register the rule you need to clean a deployment.

Cleaning a deployment basically means we remove the directory and its deployed tag.

muddled.deployment.role_depends_on_deployment(builder, role, deployment, domain=None)

Make every package in the given role depend on the given deployment

muddled.deployment.set_env(builder, deployment, name, value)

Set NAME=VALUE in the environment for this deployment.

muddled.distribute

Note

Distribution support - generating subsets of the build tree for distribution to others.

Actions and mechanisms relating to distributing build trees

class muddled.distribute.DistributeAction(name, data)

Bases: muddled.depend.Action

An action that distributes a something-or-other.

Intended as a base class for actions that know what they’re doing.

We contain one thing: a dictionary of {name : distribution information}.

‘name’ is the name of a distribution that this action supports.

‘data’ is the data that describes how we do the distribution - this will differ according to whether we are distributing a checkout or package.

add_distribution(name, copy_vcs=None, private_files=None)

Add a new named distribution.

It is an error if there is already a distribution of this name.

build_label(builder, label)

Override this to do the actual task of distribution.

distribution_names()

Return the distribution names we know about.

does_distribution(name)

Return True if we know this distribution name, False if not.

get_distribution(name)

Return the data for distribution ‘name’, or raise MuddleBug

merge_names(other)

Merge in the distribution names dictionary from another action.

Any names that the other action has that we don’t will be copied over.

Any names we already have will be ignored.

class muddled.distribute.DistributeBuildDescription(name, copy_vcs=None, private_files=None, replacement_build_desc=None)

Bases: muddled.distribute.DistributeAction

This is a bit like DistributeCheckoutAction, but without ‘just’.

‘name’ is the name of a DistributionContext. When created, we are told which DistributionContext we can be distributed by. Later on, other names may be added...

If ‘copy_vcs’ is False, then don’t copy any VCS “special” files ([‘.git’, ‘.gitignore’, ...] or [‘.bzr’], etc., depending on the VCS used for this checkout).

If ‘copy_vcs’ is True, then always copy such files.

If ‘copy_vcs’ is None, then do whatever “muddle distribute” indicates.

Note

Note that copy_vcs may only distinguish true and false for the moment, so None may be equivalvent to False.

If ‘private_files’ is given, it is a sequence of Python files, relative to the build description checkout directory, that must be replaced by empty files when the distribution is done. It is always copied (as a set). If it is None, then an empty set will be used.

If ‘replacement_build_desc’ is given, then it is a file (relative to the checkout directory) to be used instead of all the rest of the content of the build description checkout. It will be named using the appropriate name (as in .muddle/Description) when it is distributed. If a replacement build description is named, then ‘copy_vcs’ will be ignored, and no VCS will be copied. Similarly, ‘private_files’ will be ignored.

add_distribution(name, copy_vcs=None, private_files=None, replacement_build_desc=None)

Add a new named distribution.

It is an error if there is already a distribution of this name.

add_private_files(name, private_files)

Add some specific private files to distribution ‘name’.

Distribution ‘name’ must already be present on this action.

Does nothing if the files are already added

build_label(builder, label)
copying_vcs(name)

Are we distributing the VCS directory?

set_copy_vcs(name, copy_vcs)

Change the value of copy_vcs for distribution ‘name’.

set_replacement_build_desc(name, replacement_build_desc)

Change the value of replacement_build_desc for distribution ‘name’.

Note that ‘None’ is a perfectly sensible value.

class muddled.distribute.DistributeCheckout(name, copy_vcs=False, just=None)

Bases: muddled.distribute.DistributeAction

An action that distributes a checkout.

By default it copies the whole of the checkout source directory, not including any VCS files (.git/, etc.)

A mechanism for only copying some files is also included. This is typically used by “binary” packages that want to copy (for instance) the muddle Makefile for a checkout, but not all the source code.

Each checkout distribution is associated with a data tuple of the form:

(copy_vcs, specific_files)

where specific_files is None or a set of specific files for distibution.

‘name’ is the name of a DistributuionContext. When created, we are told which DistributionContext we can be distributed by. Later on, other names may be added...

If ‘copy_vcs’ is false, then don’t copy any VCS “special” files ([‘.git’, ‘.gitignore’, ...] or [‘.bzr’], etc., depending on the VCS used for this checkout).

If ‘just’ is None, then we wish to distribute all the source files in the checkout (apart from VCS files, for which see ‘copy_vcs’).

If ‘just’ is not None, then it must be a sequence of source paths, relative to the checkout directory, which are the specific files to be distributed for this checkout.

Note that we distinguish between ‘just=None’ and ‘just=[]’: the former instructs us to distribute all source files, the latter instructs us to distribute no source files.

add_distribution(name, copy_vcs=None, just=None)

Add a new named distribution.

The arguments are interpreted as when creating an instance.

It is an error if there is already a distribution of this name.

add_source_files(name, source_files)

Add some specific source files to distribution ‘name’.

If we’re already distributing the whole checkout, then this does nothing, as we’re already outputting all the source files.

Don’t try to use this to add the VCS directory to a distribution of all source files that was instantiated with copy_vcs, as the clause above will make that fail...

build_label(builder, label)
copying_all_source_files(name)

Are we distributing all the soruce files?

copying_vcs(name)

Are we distributing the VCS directory?

override(name, copy_vcs=None, just=None)

Override the definition of an existing named distribution.

request_all_source_files(name)

Request that distribution ‘name’ distribute all source files.

This is a tidy way of undoing any selection of specific files.

set_copy_vcs(name, copy_vcs)

Change the value of copy_vcs for distribution ‘name’.

class muddled.distribute.DistributePackage(name, obj=True, install=True)

Bases: muddled.distribute.DistributeAction

An action that distributes a package.

If the package is being distributed as binary, then this action copies the obj/ and install/ directories for the package, as well as any instructions (and anything else I haven’t yet thought of).

If the package is being distributed as source, then this action copies the source directory for each checkout that is directly used by the package.

In either case, associated and appropriate muddle tags are also copied.

Destination directories that do not exist are created as necessary.

‘name’ is the name of a DistributuionContext. When created, we are told which DistributionContext we can be distributed by. Later on, other names may be added...

If ‘obj’ is true, then the obj/ directory (and the associated muddle tags) should be copied.

If ‘install’ is true, then the install/ directory (and the associated muddle tags) should be copied

Notes:

  1. We don’t forbid having any particular combinations of ‘obj’ and ‘install’, although both False is not terribly useful.
add_distribution(name, obj=True, install=True)

Add a new named distribution.

It is an error if there is already a distribution of this name.

add_or_set_distribution(name, obj=True, install=True)

Add a distribution if it’s not there, or replace it if it is.

build_label(builder, label)
muddled.distribute.distribute(builder, name, target_dir, with_versions_dir=False, with_vcs=False, no_muddle_makefile=False, no_op=False, package_labels=None, checkout_labels=None)

Distribute using distribution context ‘name’, to ‘target_dir’.

The DistributeContext called ‘name’ must exist.

All distributions described in that DistributeContext will be made.

‘name’ is the name of the distribution to, erm, distribute. The special names:

  • _source_release (all checkout source directories)
  • _binary_release (all install directories, maybe plus extras)
  • _for_gpl (just GPL and GPL-propagated source directories)
  • _all_open (all open licensed source directories)
  • _by_license (source or install directories by license, nothing private)

are always recognised. See the code or “muddle help distribute” for a more complete description of these.

‘target_dir’ is where to put the distribution. It will be created if necessary.

If ‘with_versions_dir’ is true, then any stamp “versions/” directory will also be distributed.

If “with_vcs” is true, then the VCS directory (.git/ for git, etc.) will be copied for:

  • the build description(s)
  • the “versions/” directory (if it is distributed)
  • all checkouts in a “_source_release” distribution

If ‘no_muddle_makefile’ is true, then the appropriate muddle Makefile (in the appropriate checkout) will not be distributed with a package.

If ‘no_op’ is true, then we just report on what we would do - this lists the labels that would be distributed, and the action that would be used to do so.

If ‘package_labels’ and/or ‘checkout_labels’ is not None, then the labels selected for distribution will be “filtered” through those sequences, and only labels that occur in one or the other will be added to the distribution. Note that this filtering is done before adding in build descriptions. Passing them both as empty sets is likely to give a very small distribution...

NB: We assume that each package label in ‘package_labels’ has had the checkouts it directly depends upon added to ‘checkout_labels’ by the caller. Also, all labels must have their tag as ‘*’.

muddled.distribute.distribute_build_desc(builder, name, label, copy_vcs=False)

Request the distribution of the given build description checkout.

  • ‘name’ is the name of the distribution we’re adding this build description to.

    Note that this function is normally used by muddle itself, and it does not support any wildcarding of ‘name’.

  • ‘label’ must be a checkout label, but the tag is not important.

  • ‘copy_vcs’ says whether we should copy VCS “special” files (so, for

    git this includes at least the ‘.git’ directory, and any ‘.gitignore’ or ‘.gitmodules’ files). The default is not to do so.

Notes:

  1. If there was already a DistributeBuildDescription action defined for this build description’s checkout, then we will amend it to look as if we created it (but leaving any “private” file requests untouched).
  2. If there was already a DistributeBuildCheckout action defined for this build description’s checkout, then we will replace it with a DistributeBuildDescription action. All we’ll copy over from the older action is the distribution names.

_distribution/<name>.py files

If the build description checkout contains a file called _distribution/<name>.py, where <name> is the ‘name’ of the distribution we’re building, then that file will be distributed as the build description (using the appropriate name found from the .muddle/Description file), and all other files in the build description checkout will be ignored. Note that this also means that in this case any calls of `set_private_build_files() will be ignored.

Since the name of the file is specifically tied to the distribution name, no license checking is done in this case - if you are doing a “_for_gpl” distribution, and provide a _distribution/_for_gpl.py file, then it is assumed that this was deliberate, whatever license the main build description may have.

Also, ‘copy_vcs’ will be ignored in this situation, and any VCS data will not be copied.

muddled.distribute.distribute_checkout(builder, name, label, copy_vcs=False)

Request the distribution of the specified checkout(s).

  • ‘name’ is the name of the distribution we’re adding this checkout to, or a “shell pattern” matching existing (already named) distributions. In that case:

    *       matches everything
    ?       matches any single character
    [seq]   matches any character in seq
    [!seq]  matches any char not in seq
    
  • ‘label’ must be

    1. a checkout label, in which case that checkout will be distributed
    2. a package label, in which case all the checkouts directly used by the package will be distributed (this is identical to calling ‘distribute_checkout’ on each of them in turn). Note that in this case the same value of ‘copy_vcs’ will be used for all the checkouts. Either the package name or package role may be wildcarded, in which case the checkouts directly used by each matching label will be distributed.

    In either case, the label tag is ignored.

  • ‘copy_vcs’ says whether we should copy VCS “special” files (so, for

    git this includes at least the ‘.git’ directory, and any ‘.gitignore’ or ‘.gitmodules’ files). The default is not to do so.

All files and directories within the specified checkout(s) will be distributed, except for the VCS “special” files, whose distribution depends on ‘copy_vcs’.

Notes:

  1. If we already described a distribution called ‘name’ for a given checkout label, then this will silently overwrite it.
muddled.distribute.distribute_checkout_files(builder, name, label, source_files)

Request the distribution of extra files from a particular checkout.

  • ‘name’ is the name of the distribution we’re adding this checkout to, or a “shell pattern” matching existing (already named) distributions. In that case:

    *       matches everything
    ?       matches any single character
    [seq]   matches any character in seq
    [!seq]  matches any char not in seq
    
  • ‘label’ must be a checkout label. The label tag is not important.

  • ‘specified_files’ is a sequence of file paths, relative to the checkout directory.

The intent of this function is to allow adding a small number of source files from a checkout to a binary package distribution, typically so that the necessary Makefiles and other build infrastructure is distributed. So, for instance:

label = Label.from_string
distribute_package(builder, 'marmalade', label('package:binapp{x86}/*'),
                   obj=True, install=True, with_muddle_makefile=True)
distribute_checkout(builder, 'marmalade', label('checkout:binapp-1.2/*'),
                    ['Makefile', 'src/Makefile', 'src/rules'])

Notes:

  1. If we already described a distribution called ‘name’ for a given checkout label, then this will, if necessary, add the given source files to that distribution.
  2. However, if that previous distribution was distributing “all files” (i.e., created with ‘distribute_checkout()’), then we will not alter the action. This means that this call may not be used to override the ‘copy_vcs’ choice by trying to specify the VCS directory as an extra source path...
muddled.distribute.distribute_package(builder, name, label, obj=False, install=True, with_muddle_makefile=True)

Request the distribution of the given package.

  • ‘name’ is the name of the distribution we’re adding this checkout to, or a “shell pattern” matching existing (already named) distributions. In that case:

    *       matches everything
    ?       matches any single character
    [seq]   matches any character in seq
    [!seq]  matches any char not in seq
    
  • ‘label’ must be a package label. Either the name or the role may be wildcarded, in which case this function will be called on each matching label. The label tag is ignored.

  • If ‘obj’ is true, then the obj/ directory (and the associated muddle tags) should be copied.

  • If ‘install’ is true, then the install/ directory (and the associated muddle tags) should be copied

  • If ‘with_muddle_makefile’ is true, then the muddle Makefile associated with building this package will also be distributed.

    This is implemented by looking up the MakeBuilder action used to build the package, finding the checkout and Makefile name from that, and then calling ‘distribute_checkout_files()’ to add that file in that checkout to the distribution.

    For most distributions with obj=False, install=True, this is probably a useful option.

The ‘with_muddle_makefile=True’ mechanism is a fair attempt at allowing the distributed obj/ and install/ directory contents to be built, but doesn’t support things like calling a different makefile or including other files directly in the muddle Makefile.

If you need to specify extra files, that can be done with additional calls to ‘distribute_checkout_files()’.

Notes:

  1. We don’t forbid having any particular combinations of ‘obj’ and ‘install’, although both False is not terribly useful.
  2. If we already described a distribution called ‘name’ for ‘label’, then this will silently overwrite it.
muddled.distribute.get_distribution_names(builder=None)

Return the known distribution names.

Note that ‘builder’ is optional.

muddled.distribute.get_distributions_by_category(builder)

Return a dictionary of distribution names according to license category.

The dictionary returned has license category names as keys, and sets of distribution names as the values.

muddled.distribute.get_distributions_for(builder, categories)

Return distributions that distribute all the given ‘categories’

That is, for each distribution, look and see if the license categories it distributes for include all the values in ‘categories’, and if it does, add its name to the result.

For instance, we know we should always have at least two distributions that work for category ‘binary’:

>>> dists = get_distributions_for(None, ['binary'])
>>> '_by_license' in dists
True
>>> '_binary_release' in dists
True
>>> '_for_gpl' in dists
False

‘builder’ is ignored at the moment, but should be the build tree “builder” if available.

muddled.distribute.get_distributions_not_for(builder, categories)

Return distributions that distribute none of the given ‘categories’

That is, for each distribution, look and see if the license categories it distributes for include any of the values in ‘categories’, and if it does not, add its name to the result.

For instance, we know that we have at least one distribution that is not for ‘binary’ and ‘secure’:

>>> dists = get_distributions_not_for(None, ['binary', 'secure'])
>>> '_for_gpl' in dists
True
>>> '_source_release' in dists
False

Asking for distributions that don’t do anything should hopefully return an empty list:

>>> get_distributions_not_for(None, ALL_LICENSE_CATEGORIES)
[]

‘builder’ is ignored at the moment, but should be the build tree “builder” if available.

muddled.distribute.get_used_distribution_names(builder)

Return a set of all the distribution names that are actually in use

“in use” is taken to mean that some rule in the dependency tree has an action that is defined for that particular distribution name.

muddled.distribute.name_distribution(builder, name, categories=None)

Declare that a distribution called ‘name’ exists.

Also specify which license categories are distributed.

If ‘categories’ is None, then all license categories are distributed.

Otherwise ‘categories’ must be a sequence of category names, taken from ‘gpl’, ‘open-source’, binary’ and ‘private’.

The user may assume that the standard distributions (see “muddle help distribute”) already exist, but otherwise must name a distribution before it is used.

It is not an error to name a distribution more than once (although it won’t have any effect), but the categories named must be identical.

>>> name_distribution(None, '_all_open', ['gpl', 'open-source'])  # same categories
>>> name_distribution(None, '_all_open', ['open-source']) # different categories
Traceback (most recent call last):
...
GiveUp: Attempt to name distribution "_all_open" with categories "open-source" but it already has "gpl", "open-source"

It is an error to try to use a distribution before it has been named. This includes adding checkouts and packages to distributions. Wildcard operations will only take account of the distributions that have already been named.

Distribution names that start with an underscore are reserved by muddle to define as it wishes, although we don’t stop you naming a distribution that starts with an underscore (just remember muddle may take the name later without warning).

muddled.distribute.select_all_binary_nonprivate_packages(builder, name, with_muddle_makefile, just_from=None)

Select all packages with a “binary” license for distribution.

‘name’ is the name of our distribution, for error reporting.

If ‘with_muddle_makefile’ is true, then we’ll make an attempt to add distribution information for each package’s muddle Makefile (in the appropriate checkout)

We do not want “private” packages, and as such this function checks to see if any “private” packages may be present in the install/ directories that we are proposing to distribute.

If ‘just_from’ is given, then we’ll only consider the (package) labels therein.

muddled.distribute.select_all_gpl_checkouts(builder, name, with_vcs, just_from=None)

Select all checkouts with some sort of “gpl” license for distribution

(or any checkout that has had “gpl”-ness propagated to it)

(or any checkout that one of those depends on)

‘name’ is the name of our distribution.

‘with_vcs’ is true if we want VCS “special” files in our distributed checkouts.

If ‘just_from’ is given, then we’ll only consider the labels therein.

muddled.distribute.select_all_open_checkouts(builder, name, with_vcs, just_from=None)

Select all checkouts with an “open” license for distribution.

This includes all “gpl” checkouts, and all checkouts made implicitly “gpl”.

‘name’ is the name of our distribution.

‘with_vcs’ is true if we want VCS “special” files in our distributed checkouts.

If ‘just_from’ is given, then we’ll only consider the labels therein.

muddled.distribute.select_all_prop_source_checkouts(builder, name, with_vcs, just_from=None)

Select all checkouts with a “prop-source” license for distribution.

‘name’ is the name of our distribution.

‘with_vcs’ is true if we want VCS “special” files in our distributed checkouts.

If ‘just_from’ is given, then we’ll only consider the labels therein.

muddled.distribute.set_private_build_files(builder, name, private_files)

Set some private build files for the (current) build description.

These are files within the build description directory that will replaced by dummy files when doing the distribution.

  • ‘name’ is the name of the distribution we’re adding this checkout to, or a “shell pattern” matching existing (already named) distributions. In that case:

    *       matches everything
    ?       matches any single character
    [seq]   matches any character in seq
    [!seq]  matches any char not in seq
    
  • ‘private_files’ is the list of the files that must be distributed as dummy files. They are relative to the build description checkout directory.

The “original” private files must exist and must work by providing a function with signature:

def describe_private(builder, *args, **kwargs):
    ...

The dummy files will also containg such a function, but its body will be pass.

muddled.env_store

Note

Internal environment handling

An environment store holds a set of instructions for manipulating a set of environment variables

Sometimes we need to generate these for C. This is particularly evil because C neither has good environment variable lookup nor good string handling support.

See the boilerplate in resources/c_env.c for how we handle this. It’s not pretty ..

class muddled.env_store.EnvBuilder(external=False)

Bases: object

Represents a way of building an environment variable value from a series of instructions.

  • prepend_list - List of paths to prepend to the value
  • retain_old_value - Retain the old value?
  • append_list - List of things to append to the old value.
  • env_type - Type of this environment variable
  • erased - Have we been erased?
  • external - This variable is defined externally.

All paths are now of EnvExprs.

append(val)
append_expr(val)

Append val to this environment value.

copy()
dependencies()

Return a set of the environment variables that this value depends on.

empty()

Is this environment builder empty? i.e. does it have an empty value?

ensure_appended(val)
ensure_appended_expr(val)

Make sure val is appended to the value or append it.

Returns True if we added the value, False if it was already there

ensure_prepended(val)
ensure_prepended_expr(val)

Make sure val is part of the value or prepend it. What you usually want for paths.

Returns True if we added the value, False if it was already there.

erase()
get(inOldValue, language)
get_c(var, prefix, variable_name)

Return a string containing C code which leaves the value of this builder in ‘var’.

The string does not declare var, - that’s the caller’s job.

variable_name is the variable name whose value we’re processing - it’s needed so we can refer to its previous value.

get_py(inOldValue, env_name='os.environ')

Like get, but in python syntax.

  • inOldValue - A python expression which gives the old value.
get_sh(inOldValue, doQuote)

The old value of this variable was inOldValue; what is its new value?

  • doQuote - if doQuote is true, we’ll use shell quoting. We never quote inOldValue since it’s probably $PATH or something else that shouldn’t be quoted.
get_value(inOldValue, env={})
merge(other)

Merge another environment builder with this one.

prepend(val)
prepend_expr(val)

Prepend val to this environment value.

set(val)
set_expr(val)

Set val to this environment value.

set_external(external=True)
set_type(type)
class muddled.env_store.EnvExpr(type, val=None)

Bases: object

An environment variable expression. This allows us to symbolically represent things like catenating one variable value with another.

CatType = 'Cat'
RefType = 'Ref'
StringType = 'String'
append(other)
append_ref(ref)
append_str(str)
augment_dependency_set(a_set)

Add the environment variables this expression depends on to a_set.

same_as(other)

Decide if two EnvExprs will produce the same value on output.

to_c(var, prefix)

Returns a list of strings you can join together to make C code for constructing the value of this expression

  • var - The name of the C variable we’re creating.
  • prefix - The name of the prefix to prepend to function calls.
to_py(env_var)

Return a list of expressions you can put into a literal python list after "".join() to write the correct value for this variable.

to_sh(doQuote)
to_value(env)
class muddled.env_store.EnvLanguage

Bases: object

Languages in which we can generate setenv files.

C = 3
Python = 1
Sh = 0
Value = 2
class muddled.env_store.EnvMode

Bases: object

Ways of manipulating environment variables.

Append = 0
Prepend = 2
Replace = 1
class muddled.env_store.EnvType

Bases: object

Types of environment variable.

SimpleValue:is just a value (the default)
Path:colon-separated path
Path = 1
SimpleValue = 0
class muddled.env_store.Store

Bases: object

Maintains a store of environment variables and allows us to apply them to any given environment dictionary.

append(name, value)

Append a value to a variable.

append_expr(name, expr)

Append an EnvExpr to a variable.

apply(in_env)

Apply the modifications here to the environment in dict.

builder_for_name(name)

Return a builder for the given variable, inventing one if there isn’t already one

copy()
dependency_sort()

Sort self.vars.items() in as close to dependency order as you can.

empty(name)

Return True iff name has a builder with an empty value, False otherwise.

(i.e. if it’s likely to actually generate an environment variable setting)

ensure_appended(name, value)
ensure_prepended(name, value)
erase(name)

Explicitly erase a variable.

external(name)
get_c_subst_var(prefix)

Returns the block of C to use as a substitute for body_impl in resources/c_env.c

get_setvars_c(builder, prefix)
get_setvars_py(name)

Write some statements that will set the relevant environment variables in python.

Returns a string containing the relevant python code.

get_setvars_script(builder, name, lang)

Write a setvars script in the chosen language.

get_setvars_sh(name)

Write a setvars script.

Returns a string containing the script.

merge(other)

Merge another environment store into this one. Instructions from the new store will override or augment those in self.

op(name, mode, value)

Perform mode (an EnvMode) on name with value.

prepend(name, value)

Prepend a value to a variable.

prepend_expr(name, expr)

Prepend an EnvExpr to a variable.

set(name, value)

Set a value for a name.

set_expr(name, expr)

Set a variable to an EnvExpr.

set_external(name)
set_type(name, type)
muddled.env_store.add_install_dir_env(env, var_name)

Add an install directory, whose base is held in var_name, to:

  • PATH
  • LD_LIBRARY_PATH
  • PKG_CONFIG_PATH
muddled.env_store.append_expr(var, str)

Create an environment expression consisting of the given string appended to the given variable name.

muddled.env_store.prepend_expr(astr, var)
Create an environment expression consisting of a string prepended to a
variable
muddled.env_store.print_deps(deps)

Given a dictionary mapping environment variable names to sets of dependencies, return a string representing the map.

muddled.env_store.set_expr(var)

Create an environment expression consisting of a reference to the given variable name

muddled.env_store.string_expr(var)

Create an environment expression consisting of a literal string.

muddled.filespec

Note

Internal handling of file specifications, for deployment

FileSpecs: a cheap and cheerful way to specify files for the purposes of deployment instructions

class muddled.filespec.FSFileSpecDataProvider(base_dir)

Bases: object

A FileSpecDataProvider rooted at a particular point in the filesystem

abs_match(filespec)

Match the filespec to this data provider and return a list of actual absolute filenames on which to operate

list_files_under(dir, recursively=False, vroot=None)
class muddled.filespec.FileSpec(root, spec, allUnder=False, allRegex=False)

Bases: object

Represents a (possibly recursive) file specification. Filespecs are essentially python regular expressions with a recursion flag.

Matching for these objects is slightly special, since they need to apply to objects in the filesystem. Filespecs contain a root, which bounds the spec, a specifier, which is a regular expression indicating which files in that spec should match, and two recursion flags:

  • all_under - This filespec applies to all files under any directories that match the base filespec.
  • all_regex - The specifier applies as a regex to all files under the root. This can be very slow if there are many files under the filespec root.
clone_from_xml(xmlNode)

Clone a filespec from some XML like:

<filespec>
 <root>..</root>
 <spec> ..</spec>
 <all-under />
 <all-regex />
</filespec>
equal(other)
is_filespec_node(inXmlNode)

Given an XML node, decide if this is likely to be a filespec (really just checks if it’s an element of the right name). Useful for parsing documents that may contain filespecs.

match(data_provider, vroot=None)

Match this filespec with a data provider, returning a set of file- and directory- names upon which to operate.

Since we have no idea what the root path of the data provider might be, you probably need to stitch together the filenames yourself.

FSFileSpecDataProvider.abs_match() is probably your friend - if you’re using the filesystem to provide data for a filespec, call it, not us.

vroot is a ‘virtual root’ - the data provider transparently returns the subset of files that would be present if ‘vroot’ in the data provider were the root. It is used by remappings in the cpio deployment, among others.

outer_elem_name()
to_xml(doc)

Create some XML from this filespec.

class muddled.filespec.FileSpecDataProvider

Bases: object

Provides data to a filespec so it can decide what it matches.

list_files_under(dir, recursively=False, vroot=None)

Return a list of the files under dir. If dir is not a directory, returns an empty list.

The files are returned without ‘dir’, so:

list_files_under("/fred/wombat", False)

gives:

[ "a", "b", "c" ]

not:

[ "/fred/wombat/a" .. ]

whilst:

list_files_under("/fred", True)

gives:

[ "wombat", "wombat/a", "wombat/b", "wombat/c" ]
class muddled.filespec.ListFileSpecDataProvider(file_list)

Bases: object

A FileSpecDataProvider that uses a file list. Used to test the FileSpec matching code.

list_files_under(dir, recursively=False, vroot=None)

muddled.instr

Note

Internal handling of instrunction files, for deployment

Routines and classes which cope with instructions

class muddled.instr.BuiltinInstructionFactory

Bases: muddled.db.InstructionFactory

An instruction factory that can build all the built-in instructions. You can extend or augment this class to generate a factory which builds your favourite add-on instructions.

(though note that your favourite deployment will need to understand them in order to to obey them)

instr_map Maps instruction names to prototype classes, which can then be cloned

from_xml(xmlNode)
register(name, instruction)
class muddled.instr.ChangeModeInstruction(filespec, new_mode, name)

Bases: muddled.db.Instruction

Change the mode of a filespec (chown).

clone_from_xml(node)
equal(other)
outer_elem_name()
to_xml(doc)
class muddled.instr.ChangeUserInstruction(filespec, new_user, new_group, name)

Bases: muddled.db.Instruction

An instruction that takes a username, groupname and filespec.

This is the base class for chown and chgrp.

clone_from_xml(node)
equal(other)
outer_elem_name()
to_xml(doc)
class muddled.instr.MakeDeviceInstruction

Bases: muddled.db.Instruction

Create a device file - this is essentially mknod.

clone_from_xml(node)
equal(other)
outer_elem_name()
to_xml(doc)
validate()
muddled.instr.sanitise_filename(name)

Sanitise a <name>filename</name>.

We want to make sure that the name is relative to the muddle directories. Specifically, we want to make sure that if the filename is <name>/etc/passwd</name> then we do not try to access the host system’s /etc/passwd file, but rather a local .../etc/passwd.

It turns out the simplest thing to do is to remove any initial “/”, rendering the name relative...

muddled.licenses

Note

License annotation - indicating the type of license that applies to checkouts, for use in making distributions.

Matters relating to attributing licenses to checkouts

class muddled.licenses.License(name, category, version=None)

Bases: object

The representation of a source license.

License instances should be:

  1. Singular
  2. Immutable

but I don’t particularly propose to work hard to enforce those...

Use the subclasses to create your actual License instance, so that you can use any appropriate extra methods...

Initialise a new License.

The ‘name’ is the name of this license, as it is normally recognised.

‘category’ is meant to be a broad categorisation of the type of the license. Currently that is one of:

  • ‘gpl’ - some sort of GPL license, which propagate the need to distribute source code to other “adjacent” entities
  • ‘open-source’ - an open source license, anything that is not ‘gpl’. Source code may, but need not be, distributed.
  • ‘prop-source’ - a proprietary source license, not an open source license. This might, for instance, be used for /etc files, which are distributed as “source code” (i.e., text), but are not in fact licensed under an open source license.
  • ‘binary’ - a binary license, indicating that the source code is not to be distributed, but binary (the contents of the “install” directory) may be.
  • ‘private’ - a marker that the checkout should not be distributed at all.

‘version’ may be a version string. If it is None, then it will not be shown in the str() or repr() for a license.

copy_with_version(version)

Make a copy of this license with a different version string.

distribute_as_source()

Returns True if we this license is for source distribution.

Currently, equivalent to having a category of open-source, gpl or prop-source.

is_binary()

Is this a binary-distribution-only license?

is_gpl()

Returns True if this is some sort of GPL license.

is_lgpl()

Returns True if this is some sort of LGPL license.

This only works for the LicenseLGpl class (and any subclasses of it, of course).

is_open()

Returns True if this is some sort of open-source license.

Note: this includes GPL and LGPL licenses.

is_open_not_gpl()

Returns True if this license is ‘open-source’ but not ‘gpl’.

is_private()

Is this a private-do-not-distribute license?

is_proprietary_source()

Returns True if this is some sort of propetary source license.

(i.e., has category ‘prop-source’)

Note: this does not include ‘open-source’ or ‘gpl’.

propagates()

Does this license “propagate” to other checkouts?

In other words, if checkout A has this license, and checkout B depends on checkout A, does the license have an effect on what you can do with checkout B?

For non-GPL licenses, the answer is assumed “no”, and we thus return False.

For GPL licenses with a linking exception (e.g., the GCC runtime library, or some Java libraries with CLASSPATH exceptions), the answer is also “no”, and we return False.

However, for most GPL licenses (and this includes LGPL), the answer if “yes”, there is some form of propagation (remember, LGPL allows dynamic linking, and use of header files, but not static linking), and we return True.

If we return True, it is then up to the user to decide if this means anything in this particular case - muddle doesn’t know why one checkout depends on another.

class muddled.licenses.LicenseBinary(name, version=None)

Bases: muddled.licenses.License

A binary license - we distribute binary only, not source code

class muddled.licenses.LicenseGPL(name, version=None, with_exception=False)

Bases: muddled.licenses.License

Some sort of GPL license.

(Why LicenseGPL rather than GPLLicense? Because I find the later more confusing with the adjacent ‘L’s, and I want to keep GPL uppercase...)

Initialise a new GPL License.

The ‘name’ is the name of this license, as it is normally recognised.

Some GNU libraries provide a linking exception, which allows software to “link” (depending on the exception) to the library, without needing to be GPL-compatible themselves. One example of this (more or less) is the LGPL, for which we have a separate class. Another example is the GCC Runtime Library.

is_gpl()

Returns True if this is some sort of GPL license.

propagates()

Does this license “propagate” to other checkouts?

In other words, if checkout A has this license, and checkout B depends on checkout A, does the license have an effect on what you can do with checkout B?

For non-GPL licenses, the answer is assumed “no”, and we thus return False.

For GPL licenses with a linking exception (e.g., the GCC runtime library, or some Java libraries with CLASSPATH exceptions), the answer is also “no”, and we return False.

However, for most GPL licenses (and this includes LGPL), the answer if “yes”, there is some form of propagation (remember, LGPL allows dynamic linking, and use of header files, but not static linking), and we return True.

If we return True, it is then up to the user to decide if this means anything in this particular case - muddle doesn’t know why one checkout depends on another.

class muddled.licenses.LicenseLGPL(name, version=None, with_exception=False)

Bases: muddled.licenses.LicenseGPL

Some sort of Lesser GPL (LGPL) license.

The lesser GPL implies that it is OK to link to this checkout as a shared library, or to include its header files, but not link statically. We don’t treat that as a “with_exception” case specifically, since it is up to the user to decide if an individual checkout that depends on a checkout with this license is affected by our GPL-ness.

Initialise a new LGPL License.

The ‘name’ is the name of this license, as it is normally recognised.

is_lgpl()

Returns True if this is some sort of LGPL license.

class muddled.licenses.LicenseOpen(name, version=None)

Bases: muddled.licenses.License

Some non-GPL open source license.

This should probably be named “LicenseOpenSource”, but that is rather long.

class muddled.licenses.LicensePrivate(name, version=None)

Bases: muddled.licenses.License

A “private” license - we do not want to distribute anything

class muddled.licenses.LicenseProprietarySource(name, version=None)

Bases: muddled.licenses.License

A source license, but not open source.

This is separate from the “open” licenses mainly because it is not, in fact, representing an open license, so even if it were to be treated identically in all matters, it would still be wrong.

The class name is rather long, but it is hard to think of a shorter name that explains what it is for.

muddled.licenses.checkout_license_allowed(builder, co_label, categories)

Does this checkout have a license in the given categories?

Returns True if the checkout has a license that is in any of the given categories, or if it does not have a license.

Returns False if it is licensed, but its license is not in any of the given categories.

muddled.licenses.get_binary_checkouts(builder)

Return a set of all the “binary” licensed checkouts.

muddled.licenses.get_gpl_checkouts(builder)

Return a set of all the GPL licensed checkouts.

That’s checkouts with any sort of GPL license.

muddled.licenses.get_implicit_gpl_checkouts(builder)

Find all the checkouts to which GPL-ness propagates.

Returns a tuple, (result, because), where:

  • ‘result’ is a set of the checkout labels that are implicitly made “GPL” by propagation, and
  • ‘because’ is a dictionary linking each such label to a set of strings
    explaining the reason for the labels inclusion
muddled.licenses.get_license(builder, co_label, absent_is_None=True)

Get the license for a checkout.

If ‘absent_is_None’ is true, then if ‘co_label’ does not have an entry in the licenses dictionary, None will be returned. Otherwise, an appropriate GiveUp exception will be raised.

This is a simple wrapper around builder.db.get_checkout_license.

muddled.licenses.get_license_clashes(builder, implicit_gpl_checkouts)

Return clashes between actual license and “implicit GPL” licensing.

get_implicit_gpl_checkouts() returns those checkouts that are implicitly “made” GPL by propagation. However, if the checkouts concerned were already licensed with either “binary” or “private” licenses, then it is likely that the caller would like to know about it, as it is probably a mistake (or at best an infelicity).

This function returns two sets, (bad_binary, bad_private), of checkouts named in implicit_gpl_checkouts that have an explicit “binary” or “private” license.

muddled.licenses.get_license_clashes_in_role(builder, role)

Find license clashes in the install/ directory of ‘role’.

Returns two dictionaries (binary_items, private_items)

‘binary_items’ is a dictionary of {checkout_label : binary_license}

‘private_items’ is a dictionary of {checkout_label : private_license}

If private_items has content, then there is a licensing clash in the given role, as one cannot do a binary distribution of both “binary” and “private” licensed content in the same “install” directory.

muddled.licenses.get_not_licensed_checkouts(builder)

Return the set of all checkouts which do not have a license.

(Actually, a set of checkout labels, with the label tag “/checked_out”).

muddled.licenses.get_open_checkouts(builder)

Return a set of all the open licensed checkouts.

That’s checkouts with any sort of GPL or open license.

muddled.licenses.get_open_not_gpl_checkouts(builder)

Return a set of all the open licensed checkouts that are not GPL.

Note sure why anyone would want this, but it’s easy to provide.

muddled.licenses.get_private_checkouts(builder)

Return a set of all the “private” licensed checkouts.

muddled.licenses.get_prop_source_checkouts(builder)

Return a set of all the “proprietary source” licensed checkouts.

muddled.licenses.licenses_in_role(builder, role)

Given a role, what licenses are used by the packages (checkouts) therein?

Returns a set of License instances. May also include None in the values in the set, if some of the checkouts are not licensed.

muddled.licenses.print_standard_licenses()
muddled.licenses.report_license_clashes(builder, report_binary=True, report_private=True, just_for=None)

Report any license clashes.

This wraps get_implicit_gpl_checkouts() and check_for_license_clashes(), plus some appropriate text reporting any problems.

It returns True if there were any clashes, False if there were not.

It reports clashes with “binary” licenses if ‘report_binary’ is True.

It reports clashes with “private” licenses if ‘report_private’ is True.

If both are False, it is silent.

If ‘just_for’ is None, it looks at all the implicit GPL checkouts. Otherwise, it only considers those labels in ‘just_for’ that are also implicitly GPL.

muddled.licenses.report_license_clashes_in_role(builder, role, just_report_private=True)

Report license clashes in the install/ directory of ‘role’.

Basically, this function allows us to be unhappy if there are a mixture of “binary” and “private” things being put into the same “install/” directory.

If ‘just_report_private’ is true, then we will only talk about the private entities, otherwise we’ll report the “binary” licensed packages that end up there as well.

If there was a clash reported, we return True, and otherwise we return False.

muddled.licenses.set_license(builder, co_label, license, license_file=None, not_built_against=False)

Set the license for a checkout.

‘co_label’ is either a checkout label, or the name of a checkout.

(Specifying a checkout label allows a domain name to be specified as well. The tag of the checkout label is ignored.)

‘license’ must either be a License instance, or the mnemonic for one of the standard licenses.

Some licenses (for instance, ‘BSD-3-clause’) require inclusion of their license file in binary distributions. ‘license_file’ allows the relevant file to be named (relative to the root of the checkout directory), and implies that said file should be included in all distributions.

If ‘not_built_against’ is True, then it will be noted that nothing “builds against” this checkout. See ‘set_nothing_builds_against()’ for more information - this parameter is a useful convenience to save an extra call.

muddled.licenses.set_license_for_names(builder, co_names, license)

A convenience function to set one license for several checkout names.

Since this uses checkout names rather than labels, it is not domain aware.

It calls ‘set_license()’ for each checkout name, passing it a checkout label constructed from the checkout name, with no domain.

muddled.licenses.set_license_not_affected_by(builder, this_label, co_label)

Asserts that the license for co_label does not affect this_label

We assume that:

  1. ‘this_label’ is a package that depends (perhaps indirectly) on ‘co_label’, or is a checkout directly required by such a package.
  2. ‘co_label’ is a checkout with a “propagating” license (i.e., some form of GPL license).
  3. Thus by default the “GPL”ness would propagate from ‘co_label’ to ‘this_label’ (and, if it is a package, to the checkouts it is (directly) built from).

However, this function asserts that, in fact, this is not true. Our checkout is (or our checkouts are) not built in such a way as to cause the license for ‘co_label’ to propagate.

Or, putting it another way, for a normal GPL license, we’re not linking with anything from ‘co_label’, or using its header files, or copying GPL’ed files from it, and so on.

If ‘co_label’ is under LGPL, then that would reduce to saying we’re not static linking against ‘co_label’ (or anything else not allowed by the LGPL).

Note that we may be called before ‘co_label’ has registered its license, so we cannot actually check that ‘co_label’ has a propagating license (or, indeed, that it exists or is depended upon by ‘pkg_label’).

This is a simple wrapper around builder.db.set_license_not_affected_by.

muddled.licenses.set_nothing_builds_against(builder, co_label)

Asserts that no packages “build against” this checkout.

We assume that co_label is a checkout with a “propagating” license (i.e., some form of GPL license).

This function tells the distribution/licensing system that there are no packages that build against (link against) this checkout, in a way which would cause GPL license “propagation”.

Typically used to mark checkouts that (just) provide or build:

  • an application (a program)
  • a kernel module
  • a text file (e.g., something to be placed in /etc)

An example might be busybox, which is GPL-2 licensed, but which builds a set of independent programs.

muddled.mechanics

Note

The main machinery that makes muddle go - the Builder class.

Contains the mechanics of muddle.

class muddled.mechanics.BuildDescriptionAction(file_name, build_co)

Bases: muddled.depend.Action

Load the build description.

This action is used to read a build description. As such, it does not need to be domain aware - it is only ever done in the (current) top-level domain.

build_label(builder, label)

Actually load the build description.

Note that the Python path (sys.path) will have the build description checkout directory added to its start, so that the release_from() function itself can import things therefrom.

class muddled.mechanics.Builder(root_path, muddle_binary, domain_params=None, default_domain=None)

Bases: object

A builder does stuff following rules derived from a build description.

Don’t construct a Builder directly, always use the ‘load_builder()’ function, or ‘minimal_build_tree()’ if that is more appropriate.

  • self.db - The metadata database for this project.
  • self.ruleset - The rules describing this build
  • self.env - A dictionary of label to environment
  • self.default_roles - The roles to build when you don’t specify any. These will also be used for “guessing” a role for a package when one is not specified. ‘_default_roles’ is calculated from this.
  • self.default_deployment_labels - The deployments to deploy when you don’t specify any specific roles. This is the meaning of ‘_default_deployments’.

There are then a series of values used in managing subdomains:

  • self.banned_roles - An array of pairs of the form (role, domain) which aren’t allowed to share libraries.
  • self.domain_params - Maps domain names to dictionaries storing parameters that other domains can retrieve. This is used to communicate values from a build to its subdomains.
  • self.unifications - This is a list of tuples of the form (source-label, target-label), where one “replaces” the other in the build tree.

And:

  • self.what_to_release - a set of entities to build for a release build. It may contain labels and also “special” names, such as ‘_default_deployments’ or even ‘_all’. You may not include ‘_release’ (did you need to ask?). “_just_pulled” is not allowed either.

Construct a fresh Builder with a .muddle directory at the given ‘root_path’.

‘muddle_binary’ is the full path to our muddle “binary” - the program that is muddle. This is needed when running muddle Makefiles that invoke $(MUDDLE).

‘domain_params’ is the set of domain parameters in effect when this builder is loaded. It’s used to communicate values down to sub-domains.

Note that you MUST NOT set ‘domain_params’ Null unless you are the top-level domain - it MUST come from the enclosing domain’s builder or modifications made by the subdomain’s buidler will be lost, and as this is the only way to communicate values to a parent domain, this would be bad. Ugh.

‘default_domain’ is the default domain value to add to anything in local_pkgs , etc - it’s used to make sure that if you’re cd’d into a domain subdirectory, we build the right labels.

add_default_deployment_label(label)

Set the label that’s built when you call muddle from the root directory

add_default_role(role)

Add role to the list of roles built when you don’t ask for something different.

Returns False if we didn’t actually add the role (it was already there), True if we did.

add_default_roles(roles)

Add the given roles to the list of default roles for this build.

add_to_release_build(thing)

Add a thing to the set of entities to build for a release build.

‘thing’ must be:

  • a Label
  • one of the “special” names, “_all”, “_default_deployments”, “_default_roles”.
  • a sequence of either/both

It may not be “_release” (!) or “_just_pulled”.

Special names are expanded after all build descriptions have been read.

The special name “_release” corresponds to this set.

all_checkout_labels(tag=None)

Return a set of the labels of all the checkouts in our rule set.

Note that if ‘tag’ is None then all the labels will be of the form:

checkout:<co_name>/*

otherwise ‘tag’ will be used as the checkout label tag:

checkout:<co_name>/<tag>
all_checkout_rules()

Returns a set of the labels of all the checkouts in our rule set.

Specifically, all the rules for labels of the form:

checkout:*{*}/checked_out
checkout:(*)*{*}/checked_out

Returns a set of labels, thus allowing one to know the domain of each checkout as well as its name.

all_checkouts()

Return a set of the names of all the checkouts in our rule set.

Returns a set of strings.

This is not domain aware. Consider using all_checkout_labels(), which is.

all_deployment_labels(required_tag)

Return a set of all the deployment labels in our ruleset.

If ‘required_tag’ is given, then the labels returned will all have that tag (this, of course, may result in a smaller set of labels).

all_deployments()

Return a set of the names of all the deployments in our rule set.

Returns a set of strings.

all_domains()

Return a set of the names of all the domains in our rule set.

Returns a set of strings. The ‘None’ domain (the unnamed, top-level domain) is returned as the empty string, “”.

all_package_labels()

Return a set of the labels of all the packages in our rule set.

all_packages()

Return a set of the names of all the packages in our rule set.

Returns a set of strings.

Note that if ‘*’ is one of the package “names” in the ruleset, then it will be included in the names returned.

all_packages_with_roles()

Return a set of the names of all the packages/roles in our rule set.

Returns a set of strings.

Note that if ‘*’ is one of the package “names” in the ruleset, then it will be included in the names returned.

However, any labels with role ‘*’ will be ignored.

all_roles()

Return a set of the names of all the roles in our rule set.

Returns a set of strings.

apply_unifications(source)
build_co_and_path()

Return a pair (build_co, build_path).

build_desc_repo

The Repository for our build description.

This used to be a simple value, but is now a shim around looking it up in builder.db.checkout_data - so that we only have the information stored in one place.

Returns None if there is no Repository registered yet

build_label(label, silent=False)

The fundamental operation of a builder - build this label.

build_label_with_options(label, useDepends=True, useTags=True, silent=False)

The fundamental operation of a builder - build this label.

  • useDepends - Use dependencies?
build_name

The build name is meant to be a short description of the purpose of a build. It might thus be something like “ProjectBlue_intel_STB” or “XWing-minimal”.

The name may only contain alphanumerics, underlines and hyphens - this is to facilitate its use in version stamp filenames. Also, it is a superset of the allowed characters in a Python module name, which means that the build description filename (excluding its ”.py”) will be a legal build name (so we can use that as a default).

by_default_deploy(deployment)

Add a deployment label to the deployments to build by default.

by_default_deploy_list(deployments)

Now we’ve got a list of default labels, we can just add them ..

checkout_label_exists(label)

Return True if this checkout label is in any rules (i.e., is used).

Note that this method does not understand wildcards, so the match must be exact.

checkout_path(label)

Return the path in which the given checkout resides.

This is a simple wrapper around builder.db.get_checkout_path(), provided for use in scripts and build descriptions, since it “matches” builder.package_obj_path, builder.deploy_path, and so on.

checkouts_for_package(pkg_label)

Return a set of the checkouts that the given package depends upon

This only looks at direct dependencies (so if a package depends on something that in turn depends on a checkout that it does not directly depend on, then that indirect checkout will not be returned).

It does, however, expand wildcards.

deploy_path(label)

Where should deployment ‘label’ deploy to?

diagnose_unused_labels(labels, arg, required_type=None, required_tag='*')

Concoct a useful report on why none of ‘labels’ is used.

We rely on ‘labels’ having been generated by our label_from_fragment() method, which means that all the labels will have the same type

We assume quite a lot of knowledge about how that method works.

effective_environment_for(label)

Return an environment which embodies the settings that should be used for the given label. It’s the in-order merge of the output of list_environments_for().

expand_release()

Expand the command line argument “_release”

expand_underscore_arg(word, type_for_all=None)

Given a command line argument (‘word’) that starts with an underscore, try to expand it to a list of labels.

If the argument is _all, then if ‘type_for_all’ is given, expand it to all labels of that type, and otherwise reject it.

Raises a GiveUp exception if the argument is not recognised.

expand_wildcards(label, default_to_obvious_tag=True)

Given a label which may contain wildcards, return a set of labels that match.

As per the normal definition of labels, the <type>, <name>, <role> and <tag> parts of the label may be wildcarded.

If default_to_obvious_tag is true, then if label has a tag of ‘*’, it will be replaced by the “obvious” (final) tag for this label type, before any searching (so for a checkout: label, /checked_out would be used).

find_local_package_labels(dir, tag)

This is slightly horrible because if you’re in a source checkout (as you normally will be), there could be several packages.

Returns a list of the package labels involved. Uses the given tag for the labels.

find_location_in_tree(dir)

Find the directory type and name of subdirectory in a repository. This is used by the find_local_package_labels method to work out which packages to rebuild

  • dir - The directory to analyse

If nothing sensible can be determined, we return None. Otherwise we return a tuple of the form:

(DirType, label, domain_name)

where:

  • ‘DirType’ is a utils.DirType value,
  • ‘label’ is None or a label describing our location,
  • ‘domain_name’ None or the subdomain we are in and

If ‘label’ and ‘domain_name’ are both given, they will name the same domain.

follow_build_desc_branch
follows_build_desc_branch
get_all_checkout_labels_below(dir)

Get the labels of all the checkouts in or below directory ‘dir’

NOTE that this will not work if you are in a subdirectory of a checkout. It’s not meant to. Consider using find_location_in_tree() to determine that, before calling this method.

get_build_desc_branch(verbose=False)

Return the current branch of the top-level build description.

(Returns None if the build description is not on a branch, or if its VCS does not support this operation.)

get_default_domain()
get_dependent_package_dirs(label)

Find all the dependent packages for label and return a set of the object directories for each. Mainly used as a helper function by set_default_variables().

get_distribution()

Retrieve the current distribution name and target directory.

Raises GiveUp if there is no current distribution set.

get_domain_parameter(domain, name)
get_domain_parameters(domain)
get_environment_for(label)

Return the environment store for the given label, inventing one if necessary.

get_labels_in_default_roles()

Return a list of the package labels in the default roles.

get_parameter(name)

Returns the given domain parameter, or None if it wasn’t defined.

include_domain(domain_builder, domain_name)

Import the builder domain_builder into the current builder, giving it domain_name.

We first import the db, then we rename None to domain_name in banned_roles

instruct(pkg, role, instruction_file, domain=None)

Register the existence or non-existence of an instruction file. If instruction_file is None, we unregister the instruction file.

  • instruction_file - A db.InstructionFile object to save.
is_release_build()

Are we a release build (i.e., a build tree created by “muddle release”)?

We look to see if there is a file called .muddle/Release

kill_label(label, useTags=True, useMatch=True)

Kill everything that matches the given label and all its consequents.

label_from_fragment(fragment, default_type)

A variant of Label.from_fragment that understands types and wildcards

In particular, it knows that:

  1. packages have roles, but checkouts and deployments do not.
  2. wildcards expand to their appropriate values

Returns a list of labels. This method does not check that all of the labels returned actually exist as targets in the dependency tree.

labels_for_role(kind, role, tag, domain=None)

Find all the target labels with the specified kind, role and tag and return them in a set.

If ‘domain’ is specified, also require the domain to match.

list_environments_for(label)

Return a list of environments that contribute to the environment for the given label.

Returns a list of triples (match level, label, environment), in order.

load_instructions(label)

Load the instructions which apply to the given label (usually a wildcard on a role, from a deployment) and return a list of triples (label, filename, instructionfile).

map_unifications(source_list)
mark_domain(domain_name)

Write a file that marks this directory as a domain so we don’t mistake it for the root.

note_unification(source, target)
package_install_path(label)

Where should pkg install itself, by default?

package_obj_path(label)

Where should the package with this label build its object files?

packages_for_deployment(dep_label)

Return a set of the packages that the given deployment depends upon

This only looks at direct dependencies (so if a deployment depends on something that in turn depends on a package that it does not directly depend on, then that indirect package will not be returned).

It does, however, expand wildcards.

packages_using_checkout(co_label)

Return a set of the packages which directly use a checkout (this does not include dependencies)

print_banned_roles()
resource_body(file_name)

Return the body of a resource as a string.

resource_file_name(file_name)
role_combination_acceptable_for_lib(r1, r2, domain1=None, domain2=None)

True only if (r1,r2) does not appear in the list of banned roles.

role_install_path(role, domain=None)

Where should this role find its install to deploy?

roles_do_not_share_libraries(r1, r2, domain1=None, domain2=None)

Assert that roles a and b do not share libraries

Either a or b may be * to mean wildcard

Add (r1,r2) to the list of role pairs that do not share their libraries.

set_default_variables(label, store)

Muddle defines a variety of environment variables which are available whilst a label is being built. The particular variables provided depend on the type of label being built, or the type of build.

Package labels are associated with muddle Makefiles, so any environment variable specific to a package label will be available within a muddle Makefile (i.e., commands such as “muddle build” work on package labels).

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/mechanics.py:docstring of muddled.mechanics.Builder.set_default_variables, line 10)

Unexpected section title.

All labels
----------
MUDDLE

The muddle executable itself. This can be used in muddle Makefiles, for instance:

fred_objdir = $(shell $(MUDDLE) query objdir package:fred{base})
MUDDLE_ROOT
The absolute path to the root of the build tree (where the ‘.muddle’ and ‘src’ directories are).
MUDDLE_LABEL
The label currently being built.
MUDDLE_KIND, MUDDLE_NAME, MUDDLE_ROLE, MUDDLE_TAG, MUDDLE_DOMAIN
Broken-down bits of the label being built. Values will not exist if the label does not contain them (so if ‘label’ is a checkout label, MUDDLE_ROLE will not be set).
MUDDLE_OBJ
Where we should build object files for this label - the obj directory for packages, the src directory for checkouts, and the deploy directory for deployments. See “muddle query objdir”.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/mechanics.py:docstring of muddled.mechanics.Builder.set_default_variables, line 35)

Unexpected section title.

Package labels
--------------

For package labels, we also set:

MUDDLE_OBJ_OBJ, MUDDLE_OBJ_INCLUDE, MUDDLE_OBJ_LIB, MUDDLE_OBJ_BIN
$(MUDDLE_OBJ)/obj, $(MUDDLE_OBJ)/include, $(MUDDLE_OBJ)/lib, $(MUDDLE_OBJ)/bin, respectively. Note that we do not create these directories - it is up to the muddle Makefile to do so.
MUDDLE_INSTALL
Where we should install package files to.
MUDDLE_INSTRUCT
A shortcut to the ‘muddle instruct’ command for this package. Essentially “$(MUDDLE) instruct $(MUDDLE_LABEL)”
MUDDLE_UNINSTRUCT
A shortcut to the ‘muddle uninstruct’ command for this package. Essentially “$(MUDDLE) uninstruct $(MUDDLE_LABEL)”
MUDDLE_PKGCONFIG_DIRS
A path suitable for passing to pkg-config to tell it to look only at packages this label is declared to be dependent on. It will be empty if the label doesn’t have any dependencies.
MUDDLE_PKGCONFIG_DIRS_AS_PATH
The same as MUDDLE_PKGCONFIG_DIRS, for historical reasons.
MUDDLE_INCLUDE_DIRS

A space separated list of include directories, constructed from the packages that this label depends on. Names will have been intelligently escaped for the shell. Only directories that actually exist will be included.

Typically used in a muddle Makefile as:

CFLAGS += $(MUDDLE_INCLUDE_DIRS:%=-I%)
MUDDLE_LIB_DIRS

A space separated list of library directories, constructed from the packages that this label depends on. Names will have been intelligently escaped for the shell. Only directories that actually exist will be included.

Typically used in a muddle Makefile as:

LDFLAGS += $(MUDDLE_LIB_DIRS:%=-L%)
MUDDLE_LD_LIBRARY_PATH
The same values as in MUDDLE_LIB_DIRS, but with items separated by colons. This is useful for passing (as LD_LIBRARY_PATH) to configure scripts that try to look for libraries when linking test programs.
MUDDLE_KERNEL_DIR

If any of the packages that this label depends on has a directory called kerneldir in its obj dir (so, in its own terms, $(MUDDLE_OBJ)/kerneldir), then we set this value to that directory. Otherwise it is not set. If there is more than one candidate, then the last found is used (but the order of search is not defined, so this would be confusing).

If the build tree is building a Linux kernel, it can be useful to build the kernel into a directory of this name.

MUDDLE_KERNEL_SOURCE_DIR
Like MUDDLE_KERNEL_DIR, but it looks for a directory called kernelsource. The same comments apply.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/mechanics.py:docstring of muddled.mechanics.Builder.set_default_variables, line 105)

Unexpected section title.

Deployment labels
-----------------

For deployment labels we also set:

MUDDLE_DEPLOY_FROM
Where we should deploy from (probably just MUDDLE_INSTALL with the last component removed)
MUDDLE_DEPLOY_TO
Where we should deploy to, if we’re a deployment.

System Message: SEVERE/4 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/mechanics.py:docstring of muddled.mechanics.Builder.set_default_variables, line 116)

Unexpected section title.

Release build values
--------------------

When building in a release tree (typically by use of “muddle release”) extra environment variables are set to allow the build to know useful information about the release. In a non-release build, these will all be set to “(unset)”.

MUDDLE_RELEASE_NAME
The release name.
MUDDLE_RELEASE_VERSION
The release version.
MUDDLE_RELEASE_HASH

The hash of the release stamp file. This acts as a useful unique identifier for a particular release, as it is calculated from the stamp file information describing all the release checkouts.

Two releases with the same name and version, but with different checkout information, will have different release hashes.

set_distribution(name, target_dir)

Set the current distribution name and target directory.

set_domain_parameter(domain, name, value)
set_parameter(name, value)
Set a domain parameter: Danger Will Robinson! This is a
very odd thing to do - domain parameters are typically set by their enclosing domains. Setting your own is an odd idea and liable to get you into trouble. It is, however, the only way of communicating values back from a domain to its parent (and you shouldn’t really be doing that either!)
setup_environment(label, src_env)

Modify src_env to reflect the environments which apply to label, in match order.

target_label_exists(label)

Return True if this label is a target.

If it is not, then we are not going to be able to build it.

Note that this method does not understand wildcards, so the match must be exact.

unify_environments(source, target)

Given a source label and a target label, find all the environments which might apply to source and make them also apply to target.

This is (slightly) easier than one might imagine ..

unify_labels(source, target)

Unify the ‘source’ label with/into the ‘target’ label.

Given a dependency tree containing rules to build both ‘source’ and ‘target’, this edits the tree such that the any occurrences of ‘source’ are replaced by ‘target’, and dependencies are merged as appropriate.

Free variables (i.e. wildcards in the labels) are untouched - if you need to understand that, see depend.py for quite how this works.

Why is it called “unify” rather than “replace”? Mainly because it does more than replacement, as it has to merge the rules/dependencies together. In retrospect, though, some variation on “merge” might have been easier to remember (if also still inaccurate).

uninstruct_all()
exception muddled.mechanics.ErrorInBuildDescription(message=None, retcode=1)

Bases: muddled.utils.GiveUp

We want to be able to distinguish this exception in this module

We don’t expect anyone outside this module to care.

muddled.mechanics.build_co_and_path_from_str(str)

Turn a BuildDescription text into checkout name and inner path.

That is, we assume the string we’re given (which was presumably read from a BuildDescription) is of the form:

<checkout-name>/<path-to-build-desc>

For instance:

>>> build_co_and_path_from_str('builds/01.py')
('builds', '01.py')
>>> build_co_and_path_from_str('strawberry/jam/toast.py')
('strawberry', 'jam/toast.py')
muddled.mechanics.check_build_name(name)

Check a build name for legality.

Raises a GiveUp exception if the name is not allowed.

muddled.mechanics.dynamic_load_build_desc(builder)

Dynamically load the build description for this builder.

Specifically, load the top-level build description. At the moment we do not provide support for (re)loading subdomain build descriptions.

(When each build description is first read, it is always the top-level build description of its own build.)

Note that the Python path (sys.path) will have the build description checkout directory added to its start, so that the build description can import things therefrom.

Returns the apropriate module

muddled.mechanics.include_domain(builder, domain_name, domain_repo, domain_desc)

Include the named domain as a sub-build of this builder.

  • ‘domain_name’ is the name of the domain
  • ‘domain_repo’ is the string defining the repository for the domain’s build.
  • ‘domain_build_desc’ is then the path to the domain’s build description, within that.

If the domain has not yet been retrieved from ‘domain_repo’ (more specifically, if domains/<domain_name>/.muddle/ doesn’t yet exist), then it will be retrieved. This will normally happen when muddle init is done.

Note that, as a short-hand convenience, sub-domains are marked as such by having a .muddle/am_subdomain file. This will be created by include_domain() if necessary.

muddled.mechanics.load_builder(root_path, muddle_binary, params=None, default_domain=None)

Load a builder from the given root path.

This should only ever be called internally within muddle itself.

  • ‘root_path’ is the path to the root of our build tree
  • ‘muddle_binary’ is the path to our muddle binary. This is used when doing $(MUDDLE) in muddle Makefiles, and is not needed otherwise.
  • If given, ‘params’ specifies the domain parameters to pass down to the new Builder instance we’re creating - it maps domain names to dictionaries of parameters so that other domains can retrieve them. This is used within the calltree of include_domain().
  • If given, ‘default_domain’ is the default domain name for this (sub) build tree. This is used by the “muddle unstamp” command, and the muddle_patch.py script.
muddled.mechanics.minimal_build_tree(muddle_binary, root_path, repo_location, build_desc, desc_branch=None, versions_repo=None)

Setup the very minimum of a build tree.

This should give a .muddle directory with its main files, but will not actually try to retrieve any checkouts.

muddled.mechanics.run_release_from(builder, release_dir)

Run the build descriptions “release_from()” function.

‘release_dir’ is the path to the directory where the release is being assembled, which will become the final release archive/tarball.

The function is called as:

release_from(builder, release_dir)

We only run the “release_from()” in the top-level build description.

Note that the Python path (sys.path) will have the build description checkout directory added to its start, so that the release_from() function itself can import things therefrom.

muddled.pkg

Note

Package actions and other low level setup

Routines for manipulating packages and checkouts.

class muddled.pkg.ArchSpecificAction(underlying, arch)

Bases: object

Allow an action to be invoked if and only if you’re on the right architecture

build_label(builder, label)
class muddled.pkg.ArchSpecificActionGenerator(arch)

Bases: object

generate(underlying)
class muddled.pkg.Deployment

Bases: muddled.depend.Action

Represents a deployment. Deployments (typically) package code into release packages

build_label(builder, tag)

Whatever’s needed to build the relevant tag for this deployment.

class muddled.pkg.NoAction

Bases: muddled.depend.Action

An action which does nothing - used largely for testing.

build_label(builder, label)
class muddled.pkg.NullPackageBuilder(name, role)

Bases: muddled.pkg.PackageBuilder

A package that does nothing.

This can be useful when a build wants to force some checkouts to be present (and checked out), but there is nothing to build in them. Examples include documentation and meta-information that is just being kept in the build tree so that it doesn’t get lost.

Use the ‘null_package’ function to construct a useful instance.

Construct a package.

self.name
The name of this package
self.deps
The dependency set for this package. The dependency set contains mappings from role to ( package, role ). A role of ‘*’ indicates a wildcard.
build_label(builder, label)
class muddled.pkg.PackageBuilder(name, role)

Bases: muddled.depend.Action

Describes a package.

Construct a package.

self.name
The name of this package
self.deps
The dependency set for this package. The dependency set contains mappings from role to ( package, role ). A role of ‘*’ indicates a wildcard.
build_label(builder, label)
class muddled.pkg.Profile(name, role)

Bases: object

A profile ties together a role, a deployment and an installation directory. Profiles aren’t actions - they modify the builder.

There are two things you can do to a profile: you can assume() it, in which case you build that profile, or you can use() it, in which case that profile’s build results (if any) become available to you.

assume(builder)
use(builder)
class muddled.pkg.VcsCheckoutBuilder(vcs)

Bases: muddled.depend.Action

This class represents the actions available on a checkout.

‘self.vcs’ is the VCS handler which knows how to do version control operations on a checkout.

build_label(builder, co_label)
must_pull_before_commit(builder, co_label)

Must we update in order to commit? Only the VCS handler knows ..

muddled.pkg.add_checkout_rules(builder, co_label, action)

Add the standard checkout rules to a ruleset for a checkout with name co_label. ‘action’ should be an instance of VcsCheckoutBuilder, which knows how to build a checkout: label, depending on its tag.

muddled.pkg.add_package_rules(ruleset, pkg_name, role_name, action)

Add the standard package rules to a ruleset.

muddled.pkg.append_env_for_package(builder, pkg_name, pkg_roles, name, value, domain=None, type=None)

Set the environment variable name to value in the given package built in the given roles. Useful for customising package behaviour in particular roles in the build description.

muddled.pkg.depend_across_roles(ruleset, pkg_name, role_names, depends_on_pkgs, depends_on_role)

Register that pkg_name{role_name}’s preconfig depends on depends_on_pkg{depends_on_role} having been postinstalled.

muddled.pkg.do_depend(builder, pkg_name, role_names, deps)

Make pkg_name in role_names depend on the contents of deps.

deps is a list of 2-tuples (pkg_name, role_name)

If the role name is None, we depend on the pkg name in the role we’re currently using, so do_depend(a, ['b', 'c'], [ ('d', None) ]) leads to a{b} depending on d{b} and a{c} depending on d{c}.

If role_names is a string, we will implicitly convert it into the singleton list [ role_names ].

muddled.pkg.do_depend_label(builder, pkg_name, role_names, label)

Make pkg_name in role_names depend on the given label

If role_names is a string, we will implicitly convert it into the singleton list [ role_names ].

muddled.pkg.null_package(builder, name, role)

Create a Null package, a package that does nothing.

Uses NullPackageBuilder to construct our package, and then calls add_package_rules() to add the standard rules for a package.

Returns the new package instance.

Use like this:

# We have documentation in this checkout
checkouts.simple.relative(builder, co_name='docs')

# And we'd like it always to be checked out
# For this, we use a Null package that doesn't build itself
null_pkg = null_package(builder, name='docs', role='meta')
pkg.package_depends_on_checkout(builder.ruleset,
                                pkg_name='docs', role_name='meta',
                                co_name='docs')

# And add that to our default roles
builder.add_default_role('meta')
muddled.pkg.package_depends_on_checkout(ruleset, pkg_name, role_name, co_name, action=None)

Make the given package depend on the given checkout

  • ruleset - The ruleset to use - builder.ruleset, for example.
  • pkg_name - The package which depends.
  • role_name - The role which depends. Can be ‘*’ for a wildcard.
  • co_name - The checkout which this package and role depends on.
  • action - If non-None, specifies an Action to be invoked to get from the checkout to the package preconfig. If you are a normal (outside muddle itself) caller, then you will normally leave this None unless you are doing something deeply weird.
muddled.pkg.package_depends_on_packages(ruleset, pkg_name, role, tag_name, deps)

Make pkg_name depend on the list in deps.

pkg_name’s tag_name tag ends up depending on the deps having been installed - this can be PreConfig or Built, depending on whether you need that dependency to configure yourself or not.

muddled.pkg.prepend_env_for_package(builder, pkg_name, pkg_roles, name, value, domain=None, type=None)

Set the environment variable name to value in the given package built in the given roles. Useful for customising package behaviour in particular roles in the build description.

muddled.pkg.set_checkout_vcs_option(builder, co_label, **kwargs)

Sets extra VCS options for a checkout (identified by its label).

For reasons mostly to do with how stamping/unstamping works, we require option values to be either boolean, integer or string.

For example:

pkg.set_checkout_vcs_option(builder, depend.checkout('kernel-source'),
                            shallow_checkout=True, something_else=99)

This is a wrapper around:

builder.db.set_checkout_vcs_option(depend.checkout('kernel-source'),
                                   'shallow', True)
builder.db.set_checkout_vcs_option(depend.checkout('kernel-source'),
                                   'something_else', 99)

“muddle help vcs <name>” should document the available options for the version control system <name> (see “muddle help vcs” for the supported version control systems).

muddled.pkg.set_env_for_package(builder, pkg_name, pkg_roles, name, value, domain=None)

Set the environment variable name to value in the given package built in the given roles. Useful for customising package behaviour in particular roles in the build description.

muddled.repository

Note

Repository definition and handling

A new way of handling repositories

class muddled.repository.Repository(vcs, base_url, repo_name, prefix=None, prefix_as_is=False, suffix=None, inner_path=None, revision=None, branch=None, handler='guess', push=True, pull=True)

Bases: object

The representation of a single repository.

At minimum, a repository is specified by a VCS, a base URL, and a checkout name. For instance:

>>> r = Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/',
...                'builds')

As you might expect, it remembers what you told it:

>>> r.vcs
'git'
>>> r.base_url
'ssh://git@project-server/opt/kynesim/projects/042/git/'
>>> r.repo_name
'builds'

but it also calculates the full URL for accessing the repository:

>>> r.url
'ssh://git@project-server/opt/kynesim/projects/042/git/builds'

(this is calculated when the object is created because it is expected to be queried many times, and a Repository object is intended to be treated as immutable, even though we don’t enforce that).

Why do we split the repository into a base URL and a checkout name? Mainly because we assume that we are going to have more than one repository with the same base URL, so we can use the copy_with_changes method to construct new instances without having to constantly propagate the base URL explicitly.

Note that it is possible for some project hosts to be treated differently - for instance, we have a built-in rule for google code projects:

>>> g1 = Repository('git', 'https://code.google.com/p/grump', 'default')
>>> g1.url
'https://code.google.com/p/grump'
>>> g2 = Repository('git', 'https://code.google.com/p/grump', 'wiki')
>>> g2.url
'https://code.google.com/p/grump.wiki'

which is detected automatically, and the appropriate handler used. If we don’t want that, we can explicitly say so:

>>> g3 = Repository('git', 'https://code.google.com/p/grump', 'default',
...                 handler=None)
>>> g3.url
'https://code.google.com/p/grump/default'

or we can ask for it explicitly by name:

>>> g4 = Repository('git', 'https://code.google.com/p/grump', 'default',
...                 handler='code.google.com')
>>> g4.url
'https://code.google.com/p/grump'
>>> g4.handler
'code.google.com'

The default handler name is actually ‘guess’, which tries to decide by looking at the repository URL and the VCS - basically, if the repository starts with “https://code.google.com/p/” and the VCS is ‘git’, then it will use the ‘code.google.com’ handler, and otherwise it won’t. The ‘handler’ value reflects which handler was actually used:

>>> g2.handler
'code.google.com'
>>> print g3.handler
None
>>> g4.handler
'code.google.com'

Sometimes, we need some extra “path” between the repository base path and the checkout name. For instance:

>>> r = Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/',
...                'busybox-1.18.5', prefix='core')
>>> r.base_url
'ssh://git@project-server/opt/kynesim/projects/042/git/'
>>> r.url
'ssh://git@project-server/opt/kynesim/projects/042/git/core/busybox-1.18.5'

By default, that prefix is added in as a pseudo-file-path - i.e., with ‘/’ before and after it. If that’s the wrong thing to do, then specifying ‘use_prefix_as_is=True’ will cause the ‘prefix’ to be used as-is (does anyone actually support this sort of thing?):

>>> r = Repository('git', 'http://example.com',
...                'busybox-1.18.5', prefix='?repo=', prefix_as_is=True)
>>> r.base_url
'http://example.com'
>>> r.url
'http://example.com?repo=busybox-1.18.5'

Bazaar, in particular, sometimes wants to add trailing text to the checkout name, commonly to indicate a branch (bzr doesn’t really support branches as such, but instead sometimes uses conventions on how different repositories are named). So, for instance:

>>> r = Repository('bzr', 'ssh://bzr@project-server/opt/kynesim/projects/042/bzr/',
...                'repo42', suffix='/fixit_branch')
>>> r.url
'ssh://bzr@project-server/opt/kynesim/projects/042/bzr/repo42/fixit_branch'

Note that we had to specify the ‘/’ in ‘suffix’, it wasn’t assumed.

Git servers sometimes want us to put ‘.git’ on the end of a checkout name. This can be done using the same mechanism:

>>> r = Repository('git', 'git@github.com:tibs', 'withdir', suffix='.git')
>>> r.url
'git@github.com:tibs/withdir.git'

(although note that github will cope with or without the ‘.git’ at the end). Note that we had to specify the ‘.’ in ‘.git’, it wasn’t assumed.

Subversion allows retrieving part of a repository, by specifying the internal path leading to the entity to be retrieved. So, for instance:

>>> r = Repository('svn', 'ssh://svn@project-server/opt/kynesim/projects/042/svn',
...                'all_our_code', inner_path='core/busybox-1.18.4')
>>> r.url
'ssh://svn@project-server/opt/kynesim/projects/042/svn/all_our_code/core/busybox-1.18.4'

It is not intended that <inner_path> and <suffix> be used together, and the result if they are is not guaranteed.

Finally, it is possible to specify a revision and branch. These are both handled as strings, with no defined interpretation (and are not always relevant to a particular VCS - see the discussion of Bazaar above).

Creating a new Repository instance

  • ‘vcs’ is the (short name) for the Version Control System being used to access this repository. For instance, “git” or “svn”.
  • ‘base_url’ is the first part of the URL for the repository. This is separated out because it is common for different repositories to share the first part of their URL, not because it necessarily has any greater meaning.
  • ‘repo_name’ is the part that names this particular repository. It is often the same as the name of the checkout using this repository (but that depends on how the reposiory is accessed).
  • ‘prefix’ is a string to put between the ‘base_url’ and ‘repo_name’, when constructing the full repository URL.
  • if ‘prefix_as_is’ is true, then ‘prefix’ is inserted between ‘base_url’ and ‘repo_name’ exactly as it is given, otherwise it is delimited by “/” characters. It is ignored if ‘prefix’ is None.
  • ‘suffix’ is a string to put after the ‘repo_name’ when constructing the full repository URL.
  • ‘inner_path’ is used to specify a path inside the repository, for version control systems that allow this (typically Subversion).
  • ‘revision’ is the revision to use. This should always be a string, regardless of what the VCS expects. It may look like an integer (e.g., “123”) or an expression (“-r123” or “date:20120101”) depending on the VCS.
  • ‘branch’ is the branch to use.
  • ‘handler’ is either None, or “guess” (the default) or the name of a registered handler for constructing the full repository URL if the normal mechanisms are not adequate.
  • ‘push’ is true if we can push to this repository. The default is True.
  • ‘pull’ is true if we can pull from this repository. The default is True.

If there is no handler in action, the full repository URL is thus either:

  • <base_url>/<prefix>/<repo_name>/<inner_path><suffix> or
  • <base_url><prefix><repo_name>/<inner_path><suffix> or

depending on the value of ‘prefix_as_is’, and with values that are None being turned into empty strings. Note that it is not really intended that both <inner_path> and <suffix> be used on the same repository.

copy_with_changed_branch(branch, revision=None)

Return a Repository that differs only in its branch (and revision).

A simple copy is taken, and then the branch and revision are changed. Typically, the revision is just unset.

Note that we don’t check that you don’t set the revision to the same value again, although it seems unlikely to be sensible in most version control systems to do this.

For instance:

>>> r = Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/',
...                'builds', branch='fred', revision='23')
>>> r
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'builds', revision='23', branch='fred')
>>> s = r.copy_with_changed_branch('jim')
>>> s
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'builds', branch='jim')
>>> s = r.copy_with_changed_branch('jim', revision='99')
>>> s
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'builds', revision='99', branch='jim')

Note that the copy will have the same value of ‘from_url_string’ as the original.

copy_with_changed_revision(revision)

Return a Repository that differs only in its revision.

A simple copy is taken, and then the revision is changed. This is used in version stamping.

For instance:

>>> r = Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/',
...                'builds', revision='23')
>>> r
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'builds', revision='23')
>>> s = r.copy_with_changed_revision('27')
>>> s
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'builds', revision='27')

Note that the copy will have the same value of ‘from_url_string’ as the original.

copy_with_changes(repo_name, prefix=None, suffix=None, inner_path=None, revision=None, branch=None, push=None, pull=None)

Return a new instance based on this one.

A simple copy is taken, and then any amendments are made to it.

‘repo_name’ must be given.

This is expected to be (typically) useful for working out a repository relative to another (for instance, relative to the default, builds, repository). For instance:

>>> r = Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/',
...                'builds')
>>> r
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'builds')
>>> s = r.copy_with_changes('fred')
>>> s
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'fred')
>>> s = r.copy_with_changes('jim', suffix='bob')
>>> s
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'jim', suffix='bob')

Thus it does not default to using the branch or revision from the original object:

>>> x = Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/',
...                'builds', revision='123', branch='fred')
>>> x
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'builds', revision='123', branch='fred')
>>> x.copy_with_changes('jim')
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'jim')
>>> x.copy_with_changes('jim', branch='thrim')
Repository('git', 'ssh://git@project-server/opt/kynesim/projects/042/git/', 'jim', branch='thrim')

Looking at a google code example:

>>> g = Repository('git', 'https://code.google.com/p/raw-cctv-replay', 'builds')
>>> g.url
'https://code.google.com/p/raw-cctv-replay.builds'
>>> g2 = g.copy_with_changes('stuff')
>>> g2.url
'https://code.google.com/p/raw-cctv-replay.stuff'

Note that, by its nature, calling this will result in a Repository instance that has ‘from_url_string’ unset.

default_path()

Return the default repository path, calculated from all the parts.

It is returned in a manner suitable for passing to the appropriate command line tool for cloning the repository.

static from_url(vcs, repo_url, revision=None, branch=None, push=True, pull=True)

Construct a Repository instance from a URL.

  • ‘vcs’ is the version control system (‘git’, ‘svn’, etc.)
  • ‘repo_url’ is the complete URL used to access the repository
  • ‘revision’ and ‘branch’ are the revision and branch to use, just as for the normal constructor.
  • ‘push’ and ‘pull’ indicate whether one can push to and/or pull from the repository, again as for the normal constructor.

We make a good guess as to the ‘checkout name’, assuming it is the last component of the path in the URL. But, of course, no path handler gets called, so the result may not be quite what is expected.

Thus:

>>> r = Repository.from_url('git', 'http://example.com/fred/jim.git?branch=a')
>>> r
Repository.from_url('git', 'http://example.com/fred/jim.git?branch=a')
>>> r.url
'http://example.com/fred/jim.git?branch=a'
>>> r.repo_name
'jim.git'
>>> r.suffix
'?branch=a'

and even:

>>> f = Repository.from_url('file', 'file:///home/tibs/versions')
>>> f
Repository.from_url('file', 'file:///home/tibs/versions')
>>> f.url
'file:///home/tibs/versions'
>>> f.repo_name
'versions'

We do, however, set the ‘from_url_string’ value to the original URL as given:

>>> f.from_url_string
'file:///home/tibs/versions'
>>> r.from_url_string
'http://example.com/fred/jim.git?branch=a'

which allows one to tell definitively what URL was requested, and also distinguishes this from a Repository instance created in the normal manner (in a normal Repository instance. ‘from_url_string’ will be None).

Also, the handler will always be None for a Repository created with this method:

>>> print f.handler
None
>>> print r.handler
None
static get_path_handler(vcs, starts_with)

Retrieve the handler for ‘vcs’ and ‘starts_with’.

Returns None if there isn’t one (which is actually the primary reason for providing this function).

path_handlers = {('git', 'code.google.com'): <function google_code_handler>}
static register_path_handler(vcs, starts_with, handler)

Register a special handler for a particular repository location.

  • ‘vcs’ is the short name of the VCS we’re interested in, as returned by split_vcs_url().
  • ‘starts_with’ is what a repository ‘base_url’ must start with in order for us to use the handler
  • ‘handler’ is a function that takes a Repository instance and returns a path suitable for putting into the Repository instance’s ‘url’ value method.

For instance, the google code handler for git might be registered using:

Repository.register_path_handle('git', 'https://code.google.com/p/',
                                google_code_git_handler)

The handler is associated with both ‘vcs’ and ‘starts_with’. Calling this function again with the same ‘vcs’ and ‘starts_with’, but a different ‘handler’, will silently override the previous entry.

same_ignoring_revision(other)

Requires equality except for the revision.

muddled.repository.add_upstream_repo(builder, orig_repo, upstream_repo, names)

Add an upstream repo to ‘orig_repo’.

  • ‘orig_repo’ is the original Repository that we are adding an upstream for.
  • ‘upstream_repo’ is the upstream Repository. It is an error if that repository is already an upstream of ‘orig_repo’.
  • ‘names’ is a either a single string, or a sequence of strings, that can be used to select this (and possibly other) upstream repositories.

Upstream repository names must be formed of A-Z, a-z, 0-9 and underscore or hyphen.

A convenience wrapper around ‘builder.db.add_upstream_repo’.

muddled.repository.get_checkout_repo(builder, co_label)

Returns the Repository instance for this checkout label

A convenience wrapper around ‘builder.db.get_checkout_repo’.

muddled.repository.get_upstream_repos(builder, orig_repo, names=None)

Retrieve the upstream repositories for ‘orig_repo’

If ‘names’ is given, it must be a sequence of strings, in which case only those upstream repositories annotated with any of the names will be returned.

Returns a set of upstream repositories. This will be empty if there are no upstream repositories for ‘orig_repo’, or none with any of the names in ‘names’ (if given).

A convenience wrapper around ‘builder.db.get_upstream_repos’.

muddled.repository.google_code_handler(repo)

A repository path handler for google code projects.

Google code project repository URLs are of the form:

This is registered as the “code.google.com” handler for “git”.

muddled.rewrite

This module provides for rewrites of .al and pkgconfig files to reflect the realities of existing inside a muddle build tree.

Specifically, it allows you to rewrite the .la and .pkgconfig files from an autoconf’d package (typically created with make install DESTDIR=$(MUDDLE_OBJ_DIR) ) so that future packages will pick up libraries in the right places.

It is, of course, up to those packages to not use -rpath-link to force your build tree locations into the target filesystem.

muddled.rewrite.fix_up_pkgconfig_and_la(builder, dir, subdir=None, libPath=None, includePath=None, execPrefix=None)

Given a directory, dir, in which there may be .pc and .la files lurking, identify the .pc and .la files and rewrite them.

  • subdir, if present, is the subdirectory to search.

  • libPath, if present, is the directory in which the target libraries are installed (typically dir/lib)

  • includePath, if present, is where the target include files are installed (typically dir/include).

  • execPrefix is where the package will be installed.

    This is a bit tricky for us, since we’re cross-compiling: we will actually define it, by default, to be dir, since in practice most packages want this (they use it to locate tools, not -rpath-link). But you can set it to whatever you like :-)

For the moment, we work by rewriting.

  • In a .la file:
    • libdir - gets prefixed with ‘dir’
  • In a .pc file:
    • prefix -> dir
    • exec_prefix -> execDir
    • libdir -> libPath (if present)
    • includedir -> includePath (if present)
muddled.rewrite.parse_line(l)

Parse the given line, returning (key, value) or (None, None) if it wasn’t valid

muddled.rewrite.subst_la(builder, current, dir, libPath, includePath, execPrefix)

Substitute a .la file.

muddled.rewrite.subst_pc(builder, current, dir, libPath, includePath, execPrefix)

Substitute a pkgconfig (.pc) file.

muddled.rrw

Note

A general collection of toolchain utilities, named after Richard for historical reasons

rrw’s development library of _experimental_ muddle entry points. This file will go away in the next major release of muddle - in the meantime, it provides a useful library of code for reuse.

muddled.rrw.append_env(builder, roles, bindings, domain=None, setType=None)

Set environment variable <var> = <value> for every package in the given roles

Bindings is a series of (<var>, <value>) pairs.

If setType is specified, we set the type of the environment variable to one
of the types in env_store.py: the most popular of these is EnvType.SimpleValue which marks the variable as not a path-type variable.
muddled.rrw.append_to_path(builder, roles, val)

Append the given value to the PATH for the given roles

muddled.rrw.apt_get_install(builder, pkg_list, required_by, pkg_name='dev_pkgs', role='dev_pkgs')

Make sure the host has installed the given packages.

Uses apt-get install (or equivalent).

  • ‘pkg_list’ is the list of the names of the packages to check for. The names should be as they are expected by apt-get.
  • ‘required_by’ is the list of roles that require the development packages to be installed.

This is essentially a convenience wrapper for muddled.pkgs.aptget.medium(), with sensible default values for ‘pkg_name’ and ‘role’.

For instance:

apt_get_install(builder, ["bison", "flex", "libtool"], ["text", "graphics"])
muddled.rrw.build_role_on_architecture(builder, role, arch)

Wraps all the actions in a given role inside an ArchSpecificAction generator.

This requires all the actions in that role (“package:{<role>}/”) to be built on architecture <arch>.

muddled.rrw.build_with_helper(builder, helpers, pkg_name, checkout, roles, makefileName=None, co_dir=None, repoRelative=None, rev=None)

Builds a package called ‘pkg_name’ from a makefile in a helpers checkout called ‘helpers’, involving the use of the checkout ‘checkout’, which is a relative checkout with optional second level name co_dir, repo relative name repoRelative, and revision rev.

In other words, declares that ‘pkg_name’ in the given ‘roles’ will be built with the Makefile called:

<helpers>/<makefileName>

If ‘co_dir’ is None, this will be checked out using checkouts.simple.relative(), otherwise it will be checked out using checkouts.twolevel.relative(). The ‘co_dir’, ‘repoRelative’ and ‘rev’ arguments will be used in the obvious ways.

muddled.rrw.get_domain_param(builder, domain, name)

A convenience wrapper around builder.get_domain_parameter().

It’s slightly shorter to type...

muddled.rrw.package_requires(builder, in_pkg, pkg_roles, reqs)

Register the information that ‘in_pkg’, built in all of ‘pkg_roles’, require (depends on) ‘reqs’, which is a list of pairs (<package_name>, <role>)

muddled.rrw.packages_use_role(builder, pkgs, in_role, use_role, domain=None)

Specifies that one role uses the results of another role; this is most often used to allow roles built for a target to use roles built for the host.

We would normally want a role_uses_role(), but this tends to lead to undesirable excessive rebuilding of entire roles when the host tools change.

muddled.rrw.set_domain_param(builder, domain, name, value)

A convenience wrapper around builder.set_domain_parameter().

It’s slightly shorter to type...

muddled.rrw.set_env(builder, roles, bindings, domain=None)

Set environment variable <var> = <value> for every package in the given roles

Bindings is a series of (<var>, <value>) pairs.

muddled.rrw.set_global_package_env(builder, name, value, roles=['*'])

Set an environment variable ‘name = value’ for all of the named roles.

(The default sets the environment variable globally, i.e., for all roles.)

muddled.rrw.set_gnu_tools(builder, roles, env_prefix, prefix, cflags=None, ldflags=None, asflags=None, archspec=None, archname=None, archroles=['*'], domain=None, dirname=None, cppflags=None, cxxflags=None)

This is a utility function which sets up the given roles to use the given compiler prefix (typically the empty string “” for host tools, or something like “arm-linux-none-gnueabi-” for ARM)

Environment variables like:

<env_prefix>GCC

end up with values like:

<prefix>gcc
  1. If ‘env_prefix’ is not None, then we set up the following environment variables:

    • <env_prefix>CC is <prefix>gcc
    • <env_prefix>CXX is <prefix>g++
    • <env_prefix>CPP is <prefix>gpp
    • <env_prefix>LD is <prefix>ld
    • <env_prefix>AR is <prefix>ar
    • <env_prefix>AS is <prefix>as
    • <env_prefix>NM is <prefix>nm
    • <env_prefix>OBJDUMP is <prefix>objdump
    • <env_prefix>OBJCOPY is <prefix>objcopy
    • <env_prefix>RANLIB is <prefix>ranlib
    • <env_prefix>PFX is the <prefix> itself
    • if ‘archspec’ is not None, <env_prefix>ARCHSPEC is set to it
    • if ‘cflags’ is not None, <env_prefix>CFLAGS is set to it
    • if ‘cppflags’ is not None, <env_prefix>CPPFLAGS is set to it
    • if ‘cxxflags’ is not None, <env_prefix>CXXFLAGS is set to it
    • if ‘ldflags’ is not None, <env_prefix>LDFLAGS is set to it
    • if ‘asflags’ is not None, <env_prefix>ASFLAGS is set to it
    • if ‘dirname’ is not None, <env_prefix>COMPILER_TOOLS_DIR is set to it

    in all of the ‘roles’ named.

    Note that it is perfectly possible to have ‘env_prefix’ as the empty string (“”) if one wishes to set ${CC}, etc.

  2. If ‘archname’ is not None, we also set <archname>_<XX> to the same set of values, in all of the roles named in ‘archroles’. Thus roles which are, for instance, building for the host can access toolchains for other processors in the system.

For instance:

set_gnu_tools(builder, ['tools'], '', HOST_TOOLS_PREFIX,
              archname='HOST', archroles=['firmware'])

set_gnu_tools(builder, ['firmware'], '', ARM_TOOLS_PREFIX)

After this:

  • in role ‘tools’ ${CC} will refer to the version of gcc in HOST_TOOLS_PREFIX.
  • in role ‘firmware’, ${CC} will refer to the version of gcc in ARM_TOOLS_PREFIX, and ${HOST_CC} will refer to the “host” gcc in HOST_TOOLS_PREFIX.
muddled.rrw.setup_helpers(builder, helper_name)

Set up a helper checkout to be used in subsequent calls to build_with_helper

Basically a wrapper around:

checkouts.simple.relative(builder, helper_name, helper_name)
muddled.rrw.setup_tools(builder, roles_that_use_tools=['*'], tools_roles=['tools'], tools_dep='tools', tools_path_env='TOOLS_PATH', tools_install=None)

Setup the “post-build” environment for particular roles.

This sets up the deployment paths for roles, and also the runtime environment variables. This can typicaly be used to distinguish roles which run in the host environment (using programs and shared libraries from the host) and roles which run in the environment being built (using programs and shared libraries from the muddle deployment directories).

  • ‘roles_that_use_tools’ is a list of the roles that will be using the named tools. So, if the tools are GCC and its friends, this would typically be all of the roles that contain things to be built with (that) GCC. These roles will depend on the tools being deployed.
  • ‘tools_roles’ is a list of the roles that provide the tools. These do not share libraries with any other roles (so, GCC on the host does not use the same libraries as the roles that will be installed on the target).
  • ‘tools_dep’ is the deployment name for this set-of-tools. It corresponds to a label “deployment:<name>{}/deployed” in the ruleset.
  • ‘tools_path_env’ is the name of an environment variable that will be set to tell each of the roles in ‘roles_that_use_tools’ about the location of the ‘tools_dep’ deployment.
  • ‘tools_install’ is currently ignored.

Specifically:

  1. Register a tools deployment called ‘tools_dep’, used by the ‘roles_that_use_this’, and provided by packages in the ‘tools_roles’.

  2. In each of the ‘roles_that_use_tools’, set the environment variable ‘tools_path_env’ to the deployment path for ‘tools_dep’.

  3. In each of the ‘roles_that_use_tools’, amend the following environment variables as follows, where “$role_deploy” is the deployment path for ‘tools_dep’:

    • LD_LIBRARY_PATH - Prepend $role_deployl/lib
    • PATH - Append $role_deploy/bin
    • PKG_CONFIG_PATH - Prepend $role_deploy/lib/pkgconfig
    • <role>_TOOLS_PATH (where <role> is upper-cased) - Prepend $role_deploy/bin
  4. Tell each of the ‘tools_roles’ that it does not share libraries with any other roles.

muddled.subst

Substitutes a file with query parameters. These can come from environment variables or from an (optional) XML file.

Queries are a bit like XPath:

/elem/elem ...

An implicit ::text() is appended so you get all the text in the specified element.

class muddled.subst.PushbackInputStream(str)

Bases: object

A pushback input stream based on a string. Used in our recursive descent parser

get_line(line_no)

Return line ‘line_no’. Line numbers start at 1.

next()
peek()
print_what_we_just_read()
push_back(c)
report()
class muddled.subst.TreeNode(in_type, input_stream)

Bases: object

A TreeNode contains itself, followed by all its children, so this is essentially a left tree.

ContainerType = 'container'
InstructionType = 'instruction'
StringType = 'string'
append_child(n)
append_children(xml_doc, env, output_list)
echo(xml_doc, env, output_list)
eval(xml_doc, env, output_list)

Evaluate this node with respect to xml_doc, env and place your output in output_list - a list of strings.

eval_str(xml_doc, env)

Evaluate this node and return the result as a string

fnval(xml_doc, env, output_list)
ifeq(xml_doc, env, output_list, polarity)
set_fn(fn_name, params, rest)

fn_name is the name of the function params and rest are lists of nodes

set_string(inStr)
set_val(v)

v is the value which should be evaluated to get the value to evaluate.

val(xml_doc, env, output_list)
muddled.subst.flatten_literal_node(in_node)

Flatten a literal node into a string. Raise GiveUp if we, um, fail.

muddled.subst.get_text_in_xml_node(node)

Given an XML node, collect all the text in it.

muddled.subst.parse_document(input_stream, node, end_chars, has_escapes)

Parse a document into a tree node. Ends with end_char (which may be -1)

Leaves the input stream positioned at end_char.

muddled.subst.parse_instruction(input_stream, node)

An instruction ends at }, and contains:

fn:<name>(<args>, .. ) rest}

or

<stuff>}

muddled.subst.parse_literal(input_stream, echars)

Given a set of end chars, parse a literal.

muddled.subst.parse_param(input_stream, node, echars)

Parse a parameter: may be quoted (in which case ends at ”) else ends at echars

muddled.subst.query_result(keys, doc_node)

Given a list of keys and a document node, return the XML node which matches the query, or None if there isn’t one.

muddled.subst.query_string_value(xml_doc, env, k)

Given a string-valued query, work out what the result was

muddled.subst.skip_whitespace(in_stream)

Skip some whitespace

muddled.subst.split_query(query)

Split a query into a series of keys suitable to be passed to query_result().

muddled.subst.subst_file(in_file, out_file, xml_doc, env)
muddled.subst.subst_str(in_str, xml_doc, env)

Substitute ${...} in in_str with the appropriate objects - if XML doesn’t match, try an environment variable.

Unescape $${...} in case someone actually wanted ${...}` in the output.

Functions can be called with: ${fn:NAME(ARGS) REST}

name can be: ifeq(query,value) - in which case REST is substituted.
val(query) - just looks up query.

muddled.utils

Note

Core utilities used throughout muddle

Muddle utilities.

class muddled.utils.Choice(choices)

Bases: object

A choice “sequence”.

A choice sequence is:

  • a string or dictionary, the only choice. For instance:

    choice = Choice("libxml-dev2")
    assert choice.choose('any string at all') == 'libxml-dev2)
    
  • a sequence of the form [ (pattern, value), ... ]; that is a sequence of one or more ‘(pattern, value)’ pairs, where each ‘pattern’ is an fnmatch pattern (see below) and each ‘value’ is a string or dict.

    The patterns are compared to ‘what_to_match’ in turn, and if one matches, the corresponding ‘value’ is returned. If none match, a ValueError is raised.

    For instance:

    choice = Choice([ ('ubuntu-12.*', 'package-v12'),
                      ('ubuntu-1?.*', 'package-v10') ])
    try:
        match = choice.choose_to_match_os()
    except ValueError:
        print 'No package matched OS %s'%get_os_version_name()
    
  • a sequence of the form [ (pattern, value), ..., default ]; that is a sequence of one or more pairs (as above), with a final “default” value, which must be a string or dict or None.

    The patterns are compared to ‘what_to_match’ in turn, and if one matches, the corresponding ‘value’ is returned. If none match, the final default value is returned. None is allowed so the caller can easily tell that no choice was actually made.

    choice = Choice([ (‘ubuntu-12.*’, ‘package-v12’),

    (‘ubuntu-1?.*’, ‘package-v10’), ‘package-v09’ ])

    System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/utils.py:docstring of muddled.utils.Choice, line 39)

    Definition list ends without a blank line; unexpected unindent.

    # ‘match’ will always have a “sensible” value match = choice.choose_to_match_os()

    choice = Choice([ (‘ubuntu-12.*’, ‘package-v12’),

    (‘ubuntu-1?.*’, ‘package-v10’), None ])

    System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/utils.py:docstring of muddled.utils.Choice, line 45)

    Definition list ends without a blank line; unexpected unindent.

    match = choice.choose_to_match_os() if match is None:

    System Message: ERROR/3 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/utils.py:docstring of muddled.utils.Choice, line 47)

    Unexpected indentation.

    return # We know there was no given value

  • as a result of the previous, we also allow [default], although [None] is of questionable utility.

    choice = Choice([“libxml-dev2”]) assert choice.choose(‘any string at all’) == ‘libxml-dev2)

    choice = Choice([“None”]) assert choice.choose(‘any string at all’) is None

    (although that latter is the only way of “forcing” a Choice that will always return None, if you did need such a thing...)

Why not just use a list of pairs (possibly with a default string at the end, essentially just what we pass to Choice)? Well, it turns out that if you want to do something like:

pkgs.apt_get(["fromble1",
              Choice([ ('ubuntu-12.*', 'fromble'),
                       ('ubuntu-11.*', 'alex'),
                       None ]),
              "ribbit",
              Choice([ ('ubuntu-12.*', 'package-v12'),
                       ('ubuntu-1?.*', 'package-v10'),
                       'package-v7' ]),
              "libxml-dev2",
            ])

it is (a) really hard to type it right if it is just nested sequences, and (b) terribly hard to give useful error messages when the user doesn’t get it right. There are already enough brackets of various sorts, and if we don’t have the “Choice” delimiters, it just gets harder to keep track.

choose(what_to_match)

Try to match ‘what_to_match’, and return the appropriate value.

Raises ValueError if there is no match.

Returns None if (and only if) that was given as a fall-back default value.

choose_to_match_os(version_name=None)

A special case of ‘decide’ to match OS id/version

If ‘version_name’ is None, then it looks up the system ‘id’ (the “name” of the OS, e.g., “ubuntu”), and ‘version’ of the OS (e.g., “12.10”) from /etc/os-release, and concatenates them separated by a space (so “ubuntu 12.10”).

It returns the result of calling:

choose(version_name)

So, on an Ubuntu system (which also includes a Linux Mint system, since its /etc/os-release identifies it as the underlying Ubuntu system), one might do:

choice = Choice([ (‘ubuntu-12.*’, ‘package-v12’),
(‘ubuntu-1?.*’, ‘package-v10’), ‘package-v7’ ])

System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/muddled/utils.py:docstring of muddled.utils.Choice.choose_to_match_os, line 19)

Definition list ends without a blank line; unexpected unindent.

choice.choose_to_match_os()

to choose the appropriate version of “package” depending on the OS.

muddled.utils.Error

alias of MuddleBug

muddled.utils.Failure

alias of GiveUp

exception muddled.utils.GiveUp(message=None, retcode=1)

Bases: exceptions.Exception

Use this to indicate that something has gone wrong and we are giving up.

This is not an error in muddle itself, however, so there is no need for a traceback.

By default, a return code of 1 is indicated by the ‘retcode’ value - this can be set by the caller to another value, which __main__.py should then use as its return code if the exception reaches it.

retcode = 1
class muddled.utils.HashFile(name, mode='r', ignore_comments=False, ignore_blank_lines=False)

Bases: object

A very simple class for handling files and calculating their SHA1 hash.

We support a subset of the normal file class, but as lines are read or written, we calculate a SHA1 hash for the file.

Optionally, comment lines and/or blank lines can be ignored in calculating the hash, where comment lines are those starting with a ‘#’, or whitespace and a ‘#’, and blank lines are those which contain only whitespace (which includes empty lines).

Open the file, for read or write.

  • ‘name’ is the name (path) of the file to open
  • ‘mode’ is either ‘r’ (for read) or ‘w’ (for write). If ‘w’ is specified, then if the file doesn’t exist, it will be created, otherwise it will be truncated.
  • if ‘ignore_comments’ is true, then lines starting with a ‘#’ (or whitespace and a ‘#’) will not be used in calculating the hash.
  • if ‘ignore_blank_lines’ is true, then lines that are empty (zero length), or contain only whitespace, will not be used in calculating the hash.

Note that this “ignore” doesn’t mean “don’t write to the file”, it just means “ignore when calculating the hash”.

close()

Close the file.

hash()

Return the SHA1 hash, calculated from the lines so far, as a hex string.

next()
readline()

Read the next line from the file, and add it to the SHA1 hash as well.

Returns ‘’ if there is no next line (i.e., EOF is reached).

write(text)

Write the give text to the file, and add it to the SHA1 hash as well.

(Unless we are ignoring comment lines and it is a comment line, or we are ignoring blank lines and it is a blank line, in which case it will be written to the file but not added to the hash.)

As is normal for file writes, the ‘n’ at the end of a line must be specified.

exception muddled.utils.MuddleBug(message=None, retcode=1)

Bases: muddled.utils.GiveUp

Use this to indicate that something has gone wrong with muddle itself.

We thus expect that a traceback will be produced.

class muddled.utils.MuddleOrderedDict

Bases: _abcoll.MutableMapping

A simple dictionary-like class that returns keys in order of (first) insertion.

class muddled.utils.MuddleSortedDict

Bases: _abcoll.MutableMapping

A simple dictionary-like class that returns keys in sorted order.

exception muddled.utils.ShellError(cmd, retcode, output=None)

Bases: muddled.utils.GiveUp

exception muddled.utils.Unsupported(message=None, retcode=1)

Bases: muddled.utils.GiveUp

Use this to indicate that an action is unsupported.

This is used, for instance, when git reports that it will not pull to a shallow clone, which is not an error, but the user will want to know.

This is deliberately a subclass of GiveUp, because it is telling muddle to give up an operation.

class muddled.utils.VersionNumber(major=0, minor=0)

Bases: object

Simple support for two part “semantic version” numbers.

Such version numbers are of the form <major>.<minor>

static from_string(s)
next()

Return the next (minor) version number.

static unset()

Return an unset version number.

Unset version numbers compare less than proper ones.

muddled.utils.arch_name()

Retrieve the name of the architecture on which we’re running. Some builds require packages to be built on a particular (odd) architecture.

muddled.utils.c_escape(v)

Escape sensitive characters in v.

muddled.utils.calc_file_hash(filename)

Calculate and return the SHA1 hash for the named file.

muddled.utils.copy_file(from_path, to_path, object_exactly=False, preserve=False, force=False)

Copy a file (either a “proper” file, not a directory, or a symbolic link).

Just like recursively_copy, only not recursive :-)

If the target file already exists, it is overwritten.

Caveat: if the target file is a directory, it will not be overwritten. If the source file is a link, being copied as a link, and the target file is not a link, it will not be overwritten.

If ‘object_exactly’ is true, then if ‘from_path’ is a symbolic link, it will be copied as a link, otherwise the referenced file will be copied.

If ‘preserve’ is true, then the file’s mode, ownership and timestamp will be copied, if possible. Note that on Un*x file ownership can only be copied if the process is running as ‘root’ (or within ‘sudo’).

If ‘force’ is true, then if a target file is not writeable, try removing it and then copying it.

muddled.utils.copy_file_metadata(from_path, to_path)

Copy file metadata.

If ‘to_path’ is a link, then it tries to copy whatever it can from ‘from_path’, treated as a link.

If ‘to_path’ is not a link, then it copies from ‘from_path’, or, if ‘from_path’ is a link, whatever ‘from_path’ references.

Metadata is: mode bits, atime, mtime, flags and (if the process has an effective UID of 0) the ownership (uid and gid).

muddled.utils.copy_name_list_with_dirs(file_list, old_root, new_root, object_exactly=True, preserve=False)

Given file_list, create file_list[new_root/old_root], creating any directories you need on the way.

file_list is a list of full path names. old_root is the old root directory new_root is where we want them copied

muddled.utils.copy_without(src, dst, without=None, object_exactly=True, preserve=False, force=False, verbose=True)

Copy files from the ‘src’ directory to the ‘dst’ directory, without those in ‘without’

If given, ‘without’ should be a sequence of filenames - for instance, [‘.bzr’, ‘.svn’].

If ‘object_exactly’ is true, then symbolic links will be copied as links, otherwise the referenced file will be copied.

If ‘preserve’ is true, then the file’s mode, ownership and timestamp will be copied, if possible. Note that on Un*x file ownership can only be copied if the process is running as ‘root’ (or within ‘sudo’).

If ‘force’ is true, then if a target file is not writeable, try removing it and then copying it.

If ‘verbose’ is true (the default), print out what we’re copying.

Creates directories in the destination, if necessary.

Uses copy_file() to copy each file.

muddled.utils.current_machine_name()

Return the identity of the current machine - possibly including the domain name, possibly not

muddled.utils.current_user()

Return the identity of the current user, as an email address if possible, but otherwise as a UNIX uid

muddled.utils.debian_version_is(test, ref)

Return 1 if test > ref, -1 if ref > test, 0 if they are equal

muddled.utils.do_shell_quote(str)
muddled.utils.domain_subpath(domain_name)

Calculate the sub-path for a given domain name.

For instance:

>>> domain_subpath('a')
'domains/a'
>>> domain_subpath('a(b)')
'domains/a/domains/b'
>>> domain_subpath('a(b(c))')
'domains/a/domains/b/domains/c'
>>> domain_subpath('a(b(c)')
Traceback (most recent call last):
...
GiveUp: Domain name "a(b(c)" has mis-matched parentheses
muddled.utils.dynamic_load(filename)
muddled.utils.ensure_dir(dir, verbose=True)

Ensure that dir exists and is a directory, or throw an error.

muddled.utils.find_by_predicate(source_dir, accept_fn, links_are_symbolic=True)
Given a source directory and an acceptance function
fn(source_base, file_name) -> result

Obtain a list of [result] if result is not None.

muddled.utils.find_domain(root_dir, dir)

Find the domain of ‘dir’.

‘root_dir’ is the root of the (entire) muddle build tree.

This function basically works backwards through the path of ‘dir’, until it reaches ‘root_dir’. As it goes, it assembles the full domain name for the domain enclosing ‘dir’.

Returns the domain name, or None if ‘dir’ is not within a subdomain, and the directory of the root of the domain. That is:

(domain_name, domain_dir) or (None, None)
muddled.utils.find_label_dir(builder, label)

Given a label, find the corresponding directory.

  • for checkout labels, the checkout directory
  • for package labels, the install directory
  • for deployment labels, the deployment directory

This is the heart of “muddle query dir”.

muddled.utils.find_local_relative_root(builder, label)

Given a label, find its “local” root directory, relative to toplevel.

Calls find_local_root() and then calculates the location of that relative to the root of the entire muddle build tree.

muddled.utils.find_local_root(builder, label)

Given a label, find its “local” root directory.

For a normal label, this will be the normal muddle root directory (where the top-level .muddle/ directory is).

For a label in a subdomain, it will be the root directory of that subdomain - again, where its .muddle/ directory is.

muddled.utils.find_root_and_domain(dir)

Find the build tree root containing ‘dir’, and find the domain of ‘dir’.

This function basically works backwards through the path of ‘dir’, until it finds a directory containing a ‘.muddle/’ directory, that is not within a subdomain. As it goes, it assembles the full domain name for the domain enclosing ‘dir’.

Returns a pair (root_dir, current_domain).

If ‘dir’ is not within a subdomain, then ‘current_domain’ will be None.

If ‘dir’ is not within a muddle build tree, then ‘root_dir’ will also be None.

muddled.utils.get_cmd_data(thing, env=None, show_command=False)

Run the command ‘thing’, and return its output.

‘thing’ may be a string (e.g., “ls -l”) or a sequence (e.g., [“ls”, “-l”]). Internally, a string will be converted into a sequence before it is used. Any non-string items in a ‘thing’ sequence will be converted to strings using ‘str()’ (e.g., if a Label instance is given).

If ‘env’ is given, then it is the environment to use when running ‘thing’, otherwise ‘os.environ’ is used.

Note that the output of the command is not shown whilst the command is running.

If the command returns a non-zero exit code, then we raise a ShellError.

(This is basically a muddle-flavoured wrapper around subprocess.check_output)

muddled.utils.get_domain_name_from(dir)

Given a directory ‘dir’, extract the domain name.

‘dir’ should not end with a trailing slash.

It is assumed that ‘dir’ is of the form “<something>/domains/<domain_name>”, and we want to return <domain_name>.

muddled.utils.get_os_version_name()

Retrieve a string identifying this version of the operating system

Looks in /etc/os-release, which gives a different result than platform.py, which looks in /etc/lsb-release.

muddled.utils.get_prefix_pair(prefix_one, value_one, prefix_two, value_two)

Returns a pair (prefix_onevalue_one, prefix_twovalue_two) - used by rrw.py as a utility function

muddled.utils.indent(text, indent)

Return the text indented with the ‘indent’ string.

(i.e., place ‘indent’ in front of each line of text).

muddled.utils.is_release_build(dir)

Check if the given ‘dir’ is the top level of a release build.

‘dir’ should be the path to the directory contining the build’s .muddle directory (the “top” of the build).

The build is assumed to be a release build if there is a file called .muddle/Release.

muddled.utils.is_subdomain(dir)

Check if the given ‘dir’ is a (sub)domain.

‘dir’ should be the path to the directory contining the build’s .muddle directory (the “top” of the build).

The build is assumed to be a (sub)domain if there is a file called .muddle/am_subdomain.

muddled.utils.iso_time()

Retrieve the current time and date in ISO style YYYY-MM-DD HH:MM:SS.

muddled.utils.join_domain(domain_parts)

Re-join a domain name we split with split_domain.

muddled.utils.mark_as_domain(dir, domain_name)

Mark the build in ‘dir’ as a (sub)domain

This is done by creating a file .muddle/am_subdomain

‘dir’ should be the path to the directory contining the sub-build’s .muddle directory (the “top” of the sub-build).

‘dir’ should thus be of the form “<somewhere>/domains/<domain_name>”, but we do not check this.

The given ‘domain_name’ is written to the file, but this should not be particularly trusted - refer to the containing directory structure for the canonical domain name.

muddled.utils.maybe_shell_quote(str, doQuote)

If doQuote is False, do nothing, else shell-quote str.

Annoyingly, shell quoting things correctly must use backslashes, since quotes can (and will) be misinterpreted. Bah.

NB: Despite the name, this is actually “escaping”, rather then “quoting”. Specifically, any single quote, double quote or backslash characters in the original string will be converted to a backslash followed by the original character, in the final string.

muddled.utils.normalise_dir(dir)
muddled.utils.normalise_path(dir)
muddled.utils.num_cols()

How many columns on our terminal?

If it can’t tell (e.g., because it curses is not available), returns 70.

muddled.utils.pad_to(str, val, pad_with=' ')

Pad the given string to the given number of characters with the given string.

muddled.utils.page_text(progname, text)

Try paging ‘text’ by piping it through ‘progname’.

Looks for ‘progname’ on the PATH, and if os.environ[‘PATH’] doesn’t exist, tries looking for it on os.defpath.

If an executable version of ‘progname’ can’t be found, just prints the text out.

If ‘progname’ is None (or an empty string, or otherwise false), then just print ‘text’.

muddled.utils.parse_etc_os_release()

Parse /etc/os-release and return a dictionary

This is not a good parser by any means - it is the quickest and simplest thing I could do.

Note that a line like:

FRED='Fred's name'

will give us:

key 'Fred' -> value r"Fred's name"

i.e., we do not treat backslashes in a string in any way at all. In fact, we don’t do anything with strings other than throw away paired outer “’” or ‘”’.

Oh, also we don’t check whether the names before the ‘=’ signs are those that are expected, although we do provide a default value for ‘ID’ if it is not given (as the documentation for /etc/os-release’ specified).

(Note that the standard library platform.py (in 2.7) looks at /etc/lsb-release, instead of /etc/os-release, which gives different results.)

muddled.utils.parse_gid(builder, text_gid)

Todo

One day, we should do something more intelligent than just assuming your gid is numeric

muddled.utils.parse_mode(in_mode)

Parse a UNIX mode specification into a pair (clear_bits, set_bits).

muddled.utils.parse_uid(builder, text_uid)

Todo

One day, we should do something more intelligent than just assuming your uid is numeric

muddled.utils.print_string_set(ss)

Given a string set, return a string representing it.

muddled.utils.quote_list(lst)

Given a list, quote each element of it and return them, space separated

muddled.utils.recursively_copy(from_dir, to_dir, object_exactly=False, preserve=True, force=False)

Take everything in from_dir and copy it to to_dir, overwriting anything that might already be there.

Dot files are included in the copying.

If object_exactly is true, then symbolic links will be copied as links, otherwise the referenced file will be copied.

If preserve is true, then the file’s mode, ownership and timestamp will be copied, if possible. This is only really useful when copying as a privileged user.

If ‘force’ is true, then if a target file is not writeable, try removing it and then copying it.

muddled.utils.recursively_remove(a_dir)

Recursively demove a directory.

muddled.utils.rel_join(vroot, path)

Find what path would be called if it existed inside vroot. Differs from os.path.join() in that if path contains a leading ‘/’, it is not assumed to override vroot.

If vroot is none, we just return path.

muddled.utils.replace_root_name(base, replacement, filename)

Given a filename, a base and a replacement, replace base with replacement at the start of filename.

muddled.utils.run0(thing, env=None, show_command=True, show_output=True)

Run the command ‘thing’, returning nothing.

(Run and return 0 values)

‘thing’ may be a string (e.g., “ls -l”) or a sequence (e.g., [“ls”, “-l”]). Internally, a string will be converted into a sequence before it is used.

If ‘env’ is given, then it is the environment to use when running ‘thing’, otherwise ‘os.environ’ is used.

If ‘show_command’ is true, then “> <thing>” will be printed out before running the command.

If ‘show_output’ is true, then the output of the command (both stdout and stderr) will be printed out as the command runs. Note that this is the default.

If the command returns a non-zero return code, then a ShellError will be raised, containing the returncode, the command string and any output that occurred.

muddled.utils.run1(thing, env=None, show_command=True, show_output=False)

Run the command ‘thing’, returning its output.

(Run and return 1 value)

‘thing’ may be a string (e.g., “ls -l”) or a sequence (e.g., [“ls”, “-l”]). Internally, a string will be converted into a sequence before it is used. Any non-string items in a ‘thing’ sequence will be converted to strings using ‘str()’ (e.g., if a Label instance is given).

If ‘env’ is given, then it is the environment to use when running ‘thing’, otherwise ‘os.environ’ is used.

If ‘show_command’ is true, then “> <thing>” will be printed out before running the command.

If ‘show_output’ is true, then the output of the command (both stdout and stderr) will be printed out as the command runs.

If the command returns a non-zero return code, then a ShellError will be raised, containing the returncode, the command string and any output that occurred.

Otherwise, the command output (stdout and stderr combined) is returned.

muddled.utils.run2(thing, env=None, show_command=True, show_output=False)

Run the command ‘thing’, returning the return code and output.

(Run and return 2 values)

‘thing’ may be a string (e.g., “ls -l”) or a sequence (e.g., [“ls”, “-l”]). Internally, a string will be converted into a sequence before it is used. Any non-string items in a ‘thing’ sequence will be converted to strings using ‘str()’ (e.g., if a Label instance is given).

If ‘show_command’ is true, then “> <thing>” will be printed out before running the command.

If ‘show_output’ is true, then the output of the command (both stdout and stderr) will be printed out as the command runs.

The output of the command (stdout and stderr) goes to the normal stdout whilst the command is running.

The command return code and output are returned as a tuple:

(retcode, output)
muddled.utils.run3(thing, env=None, show_command=True, show_output=False)

Run the command ‘thing’, returning the return code, stdout and stderr.

(Run and return 3 values)

‘thing’ may be a string (e.g., “ls -l”) or a sequence (e.g., [“ls”, “-l”]). Internally, a string will be converted into a sequence before it is used. Any non-string items in a ‘thing’ sequence will be converted to strings using ‘str()’ (e.g., if a Label instance is given).

If ‘env’ is given, then it is the environment to use when running ‘thing’, otherwise ‘os.environ’ is used.

If ‘show_command’ is true, then “> <thing>” will be printed out before running the command.

If ‘show_output’ is true, then the output of the command (both stdout and stderr) will be printed out as the command runs.

The output of the command is shown whilst the command is running; its stdout goes to the normal stdout, and its stderr to stderr.

The command return code, stdout and stderr are returned as a tuple:

(retcode, stdout, stderr)
muddled.utils.shell(thing, env=None, show_command=True)

Run the command ‘thing’ in the shell.

If ‘thing’ is a string (e.g., “ls -l”), then it will be used as it is given.

If ‘thing’ is a sequence (e.g., [“ls”, “-l”]), then each component will be escaped with pipes.quote(), and the result concatenated (with spaces between) to give the command line to run.

If ‘env’ is given, then it is the environment to use when running ‘thing’, otherwise ‘os.environ’ is used.

If ‘show_command’ is true, then “> <thing>” will be printed out before running the command.

The output of the command will always be printed out as it runs.

If the command returns a non-zero return code, then a ShellError will be raised, containing the returncode, the command string and any output that occurred.

Unlike the various ‘runX’ functions, this calls subprocess.Popen with ‘shell=True’. This makes things like “cd” available in ‘thing’, and use of shell specific things like value expansion. It also, morei mportantly for muddle, allows commands like “git clone” to do their progress report “rolling” output. However, the warnings in the Python subprocess documentation should be heeded about not using unsafe command lines.

NB: If you do want to do “cd xxx; yyy”, you’re probably better doing:

with Directory("xxx"):
    shell("yyy")
muddled.utils.sort_domains(domains)

Given a sequence of domain names, return them sorted by depth.

So, given some random domain names (and we forgot to forbid strange names starting with ‘+’ or ‘-‘):

>>> a = ['a', '+1', '-2', 'a(b(c2))', 'a(b(c1))', '+1(+2(+4(+4)))',
...    'b(b)', 'b', 'b(a)', 'a(a)', '+1(+2)', '+1(+2(+4))', '+1(+3)']

sorting “alphabetically” gives the wrong result:

>>> sorted(a)
['+1', '+1(+2(+4(+4)))', '+1(+2(+4))', '+1(+2)', '+1(+3)', '-2', 'a', 'a(a)', 'a(b(c1))', 'a(b(c2))', 'b', 'b(a)', 'b(b)']

so we needed this function:

>>> sort_domains(a)
['+1', '+1(+2)', '+1(+2(+4))', '+1(+2(+4(+4)))', '+1(+3)', '-2', 'a', 'a(a)', 'a(b(c1))', 'a(b(c2))', 'b', 'b(a)', 'b(b)']

If we’re given a domain name that is None, we’ll replace it with ‘’.

muddled.utils.split_debian_version(v)

Takes a debian-style version string - <major>.<minor>.<subminor>-<issue><additional> - and turns it into a dictionary with those keys.

muddled.utils.split_domain(domain_name)

Given a domain name, return a tuple of the hierarchy of sub-domains.

For instance:

>>> split_domain('a')
['a']
>>> split_domain('a(b)')
['a', 'b']
>>> split_domain('a(b(c))')
['a', 'b', 'c']
>>> split_domain('a(b(c)')
Traceback (most recent call last):
...
GiveUp: Domain name "a(b(c)" has mis-matched parentheses

We don’t actually allow “sibling” sub-domains, so we try to complain helpfully:

>>> split_domain('a(b(c)(d))')
Traceback (most recent call last):
...
GiveUp: Domain name "a(b(c)(d))" has 'sibling' sub-domains

If we’re given ‘’ or None, we return [‘’], “normalising” the domain name.

>>> split_domain('')
['']
>>> split_domain(None)
['']
muddled.utils.split_path_left(in_path)

Given a path a/b/c ..., return a pair (a, b/c..) - ie. like os.path.split(), but leftward.

What we actually do here is to split the path until we have nothing left, then take the head and rest of the resulting list.

For instance:

>>> split_path_left('a/b/c')
('a', 'b/c')
>>> split_path_left('a/b')
('a', 'b')

For a single element, behave in sympathy (but, of course, reversed) to os.path.split:

>>> import os
>>> os.path.split('a')
('', 'a')
>>> split_path_left('a')
('a', '')

The empty string isn’t really a sensible input, but we cope:

>>> split_path_left('')
('', '')

And we take some care with delimiters (hopefully the right sort of care):

>>> split_path_left('/a///b/c')
('', 'a/b/c')
>>> split_path_left('//a/b/c')
('', 'a/b/c')
>>> split_path_left('///a/b/c')
('', 'a/b/c')
muddled.utils.split_vcs_url(url)

Split a URL into a vcs and a repository URL. If there’s no VCS specifier, return (None, None).

muddled.utils.string_cmp(a, b)

Return -1 if a < b, 0 if a == b, +1 if a > b.

muddled.utils.text_in_node(in_xml_node)

Return all the text in this node.

muddled.utils.total_ordering(cls)

Class decorator that fills-in missing ordering methods

muddled.utils.truncate(text, columns=None, less=0)

Truncate the given text to fit the terminal.

More specifically:

  1. Split on newlines
  2. If the first line is too long, cut it and add ‘...’ to the end.
  3. Return the first line

If ‘columns’ is 0, then don’t do the truncation of the first line.

If ‘columns’ is None, then try to work out the current terminal width (using “curses”), and otherwise use 80.

If ‘less’ is specified, then the actual width used will be the calculated or given width, minus ‘less’ (so if columns=80 and less=2, then the maximum line length would be 78). Clearly this is ignored if ‘columns’ is 0.

muddled.utils.unescape_backslashes(str)

Replace every string ‘X’ with X, as if you were a shell

muddled.utils.unix_time()

Return the current UNIX time since the epoch.

muddled.utils.unquote_list(lst)

Given a list of objects, potentially enclosed in quotation marks or other shell weirdness, return a list of the actual objects.

muddled.utils.well_formed_dot_muddle_dir(dir)

Return True if this seems to be a well-formed .muddle directory

We’re not trying to be absolutely rigorous, but do want to detect (for instance) an erroneous file with that name, or an empty directory

muddled.utils.wrap(text, width=None, **kwargs)

A convenience wrapper around textwrap.wrap()

(basically because muddled users will have imported utils already).

muddled.utils.xml_elem_with_child(doc, elem_name, child_text)

Return an element ‘elem_name’ containing the text child_text in doc.

muddled.version_control

Note

The top-level VCS infrastructure, using VCS specific plugins to do the actual work.

Routines which deal with version control.

class muddled.version_control.VersionControlHandler(vcs)

Bases: object

Handle all version control operations for a checkout.

The VersionControlSystem class knows how to do individual VCS operations, but is not required to know anything muddle-specific beyond how a Repository object works (or, at least, that is the aim).

This class acts as a translator between muddle actions on a checkout label and the underlying VCS actions.

Each underlying VCS (git, bzr, etc.) is used via an instance of this class.

  • ‘vcs’ is the class corresponding to the particular version control system - e.g., Git.
branch_exists(builder, co_label, branch, verbose=False, show_pushd=False)

Returns True if a branch of that name exists.

This allowed to be conservative - e.g., in git the existence of a remote branch with the given name can be counted as True.

Will be called in the actual checkout’s directory.

If ‘show_pushd’ is false, then we won’t report as we “pushd” into the checkout directory.

branch_to_follow(builder, co_label)

Determine what branch is actually wanted.

Returns a branch name or None. None means that the description of this checkout in the build description is adequate - i.e., we do not need to override it with the branch of the build description.

BEWARE: this duplicates some of the code in sync().

checkout(builder, co_label, verbose=True)

Check this checkout out of version control.

The actual operation we perform is commonly called “clone” in actual version control systems. We retain the name “checkout” because it instantiates a muddle checkout.

commit(builder, co_label, verbose=True)

Commit any changes in the local working copy to the local repository.

In a centralised VCS, like subverson, this does not do anything, as there is no local repository.

create_branch(builder, co_label, branch, verbose=False, show_pushd=False)

Create a (new) branch of the given name.

Will be called in the actual checkout’s directory.

If ‘show_pushd’ is false, then we won’t report as we “pushd” into the checkout directory.

get_current_branch(builder, co_label, verbose=False, show_pushd=False)

Return the name of the current branch.

Will be called in the actual checkout’s directory.

Return the name of the current branch (e.g., “master” or “Fred”), or None if there is no current branch.

If ‘show_pushd’ is false, then we won’t report as we “pushd” into the checkout directory.

Raises a GiveUp exception if the VCS does not support this operation, or if something goes wrong.

get_file_content(url, verbose=True)

Retrieve a file’s content via a VCS.

get_vcs_special_files()

Return the names of the ‘special’ files/directories used by this VCS.

For instance, if ‘url’ starts with “git+” then we might return [”.git”, ”.gitignore”, ”.gitmodules”]

Returns an empty list if there is no such concept.

goto_branch(builder, co_label, branch, verbose=False, show_pushd=False)

Make the named branch the current branch.

Will be called in the actual checkout’s directory.

If ‘show_pushd’ is false, then we won’t report as we “pushd” into the checkout directory.

goto_revision(builder, co_label, revision, branch=None, verbose=False, show_pushd=False)

Go to the specified revision.

If branch is given, this may alter the behaviour.

Will be called in the actual checkout’s directory.

If ‘show_pushd’ is false, then we won’t report as we “pushd” into the checkout directory.

long_name
merge(builder, co_label, verbose=True)

Retrieve changes from the remote repository, and apply them to the local working copy, performing a merge operation if necessary.

Returns True if it changes its checkout (changes the files visible to the user), False otherwise.

must_pull_before_commit(builder, co_label)

Do we need to pull before we can commit?

This may depend on the options chosen for this checkout.

pull(builder, co_label, upstream=None, repo=None, verbose=True)

Retrieve changes from the remote repository, and apply them to the local working copy, but not if a merge operation would be required, in which case an exception shall be raised.

If ‘upstream’ and ‘repo’ are given, then they specify the upstream repository we should pull from, instead of using the ‘normal’ repository from the build description.

Returns True if it changes its checkout (changes the files visible to the user), False otherwise.

push(builder, co_label, upstream=None, repo=None, verbose=True)

Push changes in the local repository to the remote repository.

If ‘upstream’ and ‘repo’ are given, then they specify the upstream repository we should push to, instead of using the ‘normal’ repository from the build description.

Note that in a centralised VCS, like subversion, this is typically called “commit”, since there is no local repository.

This operaton does not do a ‘commit’.

reparent(builder, co_label, force=False, verbose=True)

Re-associate the local repository with its original remote repository,

(This is not relevant for all VCS systems, and will only be overridden for those where it does make sense - notably Bazaar)

This re-associates the local repository with the remote repository named in the muddle build description.

If ‘force’ is true, it does this regardless. If ‘force’ is false, then it only does it if the checkout is actually not so associated.

revision_to_checkout(builder, co_label, force=False, before=None, verbose=False, show_pushd=True)

Determine a revision id for this checkout, usable to check it out again.

The revision id we want is that we could use to check out an identical checkout.

If the local working set/repository/whatever appears to have been altered from the remove repository, or otherwise does not yield a satisfactory revision id (this is something only the subclass can tell), then the method should raise GiveUp, with as clear an explanation of the problem as possible.

If ‘force’ is true, then if the revision cannot be determined, return the orginal revision that was specified when the checkout was checked out.

(Individual version control classes may opt to ignore the ‘force’ argument, either because it is not useful in their context, or because they can tell that the checkout is seriously astray/broken.)

If ‘before’ is given, it should be a string describing a date/time, and the revision id chosen will be the last revision at or before that date/time.

Note

This depends upon what the VCS concerned actually supports. This feature is experimental.

NB: if ‘before’ is specified, ‘force’ is ignored.

If ‘show_pushd’ is false, then we won’t report as we “pushd” into the checkout directory.

NB: If the VCS class does not override this method, then the default implementation will raise a GiveUp unless ‘force’ is true, in which case it will return the string ‘0’.

short_name
status(builder, co_label, verbose=False, quick=False)

Report on the status of the checkout, in a VCS-appropriate manner

If there is nothing to be done for this repository, returns None.

Otherwise, returns a string comprising a report on the status of the repository, in a VCS appropriate manner.

If ‘verbose’, then report each checkout label as it is checked.

The reliability and accuracy of this varies by VCS, but the idea is that a checkout is ‘safe’ if:

  • there are no files in the local checkout that are not also in the (local) repository, unless explicitly marked to be ignored
  • there are no files that need committing to the local repository

In general, if a checkout is ‘safe’ then it should be OK to ‘merge’ the remote repository into it.

sync(builder, co_label, verbose=False, sync=True)

Attempt to go to the branch indicated by the build description.

  • ‘co_label’ is the label for which to sync
  • if ‘verbose’ is True, then report extra information about what is being done and why
  • if ‘sync’ is False, don’t actually do the sync - this is expected to be used with “verbose=True” to report on what we would do and why.

Do the first applicable of the following

  • If this is the top-level build description, then:
    • if it has “builder.follow_build_desc_branch = True”, then nothing needs to be done, as we’re already there.
    • if it does not have “builder.follow_build_desc_branch = True”, but a branch was specified for it (i.e., via “muddle init -branch”), then go to that branch.
    • if it does not have “builder.follow_build_desc_branch = True”, and no branch was specified (at “muddle init”), then go to “master”.
  • If the build description specifies a revision for this checkout, go to that revision.
  • If the build description specifies a branch for this checkout, and the checkout VCS supports going to a specific branch, go to that branch
  • If the build description specifies that this checkout should not follow the build description (both Subversion and Bazaar support the “no_follow” option), then go to “master”.
  • If the build description specifies that this checkout is shallow, then give up.
  • If the checkout’s VCS does not support lightweight branching, then give up (the following choices require this).
  • If the build description has “builder.follow_build_desc_branch = True”, then go to the same branch as the build description.
  • Otherwise, go to “master”.

BEWARE: this duplicates some of the code in branch_to_follow().

class muddled.version_control.VersionControlSystem

Bases: object

Provide version control operations for a particular VCS.

This is a super-class, acting as a template for the actual classes.

The intent is that implementors of this interface do not need to know much about muddle, although they will have to have some understanding of the contents of a Repository object.

add_files(files=None, verbose=True)

If files are given, add them, but do not commit.

Will be called in the actual checkout’s directory.

allowed_options = set(['no_follow', 'shallow_checkout'])
allows_relative_in_repo()

Does this VCS allow relative locations within the repository to be checked out?

Subversion does. Distributed revision control systems tend not to.

branch_exists(branch)

Returns True if a branch of that name exists.

This allowed to be conservative - e.g., in git the existence of a remote branch with the given name can be counted as True.

Will be called in the actual checkout’s directory.

Raises Unsupported if the VCS does not support this operation.

checkout(repo, co_leaf, options, verbose=True)

Checkout (clone) a given checkout.

Will be called in the parent directory of the checkout.

Expected to create a directory called <co_leaf> therein.

Any exception raised will be “wrapped” by the calling handler.

commit(repo, options, verbose=True)

Will be called in the actual checkout’s directory.

Any exception raised will be “wrapped” by the calling handler.

create_branch(branch, verbose=False)

Create a (new) branch of the given name.

Will be called in the actual checkout’s directory.

Raises Unsupported if the VCS does not support this operation.

Raises GiveUp if a branch of that name already exists.

get_current_branch()

Return the name of the current branch.

Will be called in the actual checkout’s directory.

Returns the name of the branch, or None if there is no current branch. Note that the master branch is thus returned as “master”, not as None.

Raises Unsupported if the VCS does not support this operation.

get_file_content(url, verbose=True)

Retrieve a file’s content via a VCS.

get_vcs_special_files()

Return the names of the ‘special’ files/directories used by this VCS.

For instance, if ‘url’ starts with “git+” then we might return [”.git”, ”.gitignore”, ”.gitmodules”]

Returns an empty list if there is no such concept.

goto_branch(branch, verbose=False)

Make the named branch the current branch.

Will be called in the actual checkout’s directory.

Raises Unsupported if the VCS does not support this operation.

Raises GiveUp if there is no existing branch of that name.

goto_revision(revision, branch=None, repo=None, verbose=False)

Make the specified revision current.

Note that this may leave the working data (the actual checkout directory) in an odd state, in which it is not sensible to commit, depending on the VCS and the revision.

Will be called in the actual checkout’s directory.

If the VCS supports it, may also take a branch name.

Raises Unsupported if the VCS does not support this operation.

Raises GiveUp if there is no such revision.

init_directory(files=None, verbose=True)

If the directory does not appear to have had ‘<vcs> init’ run in it, then do so first.

Will be called in the actual checkout’s directory.

merge(other_repo, options, verbose=True)

Will be called in the actual checkout’s directory.

Any exception raised will be “wrapped” by the calling handler.

Returns True if it changes its checkout (changes the files visible to the user), False otherwise.

must_pull_before_commit(options)

Do we need to ‘pull’ before we ‘commit’?

In a centralised VCS like subverson, this is highly recommended.

In a distributed VCS like bazaar or git, it is unnecessary.

We shall default to the distributed answer, and individual VCS support can override if necessary.

pull(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

Any exception raised will be “wrapped” by the calling handler.

Returns True if it changes its checkout (changes the files visible to the user), False otherwise.

push(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

Any exception raised will be “wrapped” by the calling handler.

reparent(co_leaf, remote_repo, options, force=False, verbose=True)

Will be called in the actual checkout’s directory.

revision_to_checkout(repo, co_leaf, options, force=False, before=None, verbose=True)

Will be called in the actual checkout’s directory.

status(repo, options, quick=False)

Will be called in the actual checkout’s directory.

Return status text or None if there is no interesting status.

supports_branching()

Does this VCS support “lightweight” branching like git?

Git clearly does. Subversion does not (branches are a different path inside the repository - if we ever wanted/needed to, we might accomodate that, but at the moment we don’t), and bazaar does not (branches are clones/checkouts), although I believe there are proposals to handle lightweight branching as well (but that is “as well”, so particular user might or might not want to use that facility).

Thus the default is False.

muddled.version_control.checkout_from_repo(builder, co_label, repo, co_dir=None, co_leaf=None)

Declare that the checkout for ‘co_label’ comes from Repository ‘repo’

We will take the repository described in ‘repo’ and check it out into:

  • src/<co_label>.name or
  • src/<co_dir>/<co_label>.name or
  • src/<co_dir>/<co_leaf> or
  • src/<co_leaf>

depending on whether <co_dir> and/or <co_leaf> are given. We will assign the label <co_label> to this directory/repository combination.

muddled.version_control.get_vcs_docs(vcs)

Given a VCS short name, return the docs for how muddle handles it

muddled.version_control.get_vcs_instance(vcs)

Given a VCS short name, return a VCS instance.

muddled.version_control.get_vcs_instance_from_string(repo_str)

Given a <vcs>+<url> string, return a VCS instance and <url>.

muddled.version_control.list_registered(indent='')

Return a list of registered version control systems.

muddled.version_control.register_vcs(scheme, vcs_instance, docs=None, options=None)

Register a VCS instance with a VCS scheme prefix.

Also, preferably, register the VCS documentation on how muddle handles it, and maybe also any allowed options.

muddled.version_control.vcs_get_directory(url, directory=None)

Retrieve (clone) the directory identified by the URL, via its VCS.

If ‘directory’ is given, then clones to the named directory.

Looks at the first few characters of the URL to determine the VCS to use - so, e.g., “bzr” for “bzr+ssh://whatever”.

Raises KeyError if the scheme is not one for which we have a registered handler.

muddled.version_control.vcs_get_file_data(url)

Return the content of the file identified by the URL, via its VCS.

Looks at the first few characters of the URL to determine the VCS to use - so, e.g., “bzr” for “bzr+ssh://whatever”.

Returns a string (the content of the file).

Raises KeyError if the scheme is not one we have a registered file getter for.

muddled.version_control.vcs_handler_for(builder, co_label)

Create a VCS handler for the given checkout label.

We look up the repository for this label, and which VCS to use is determined by interpreting the initial part of the repository URI’s protocol.

We then create a handler that will call the appropriate VCS-specific mechanisms for any VCS operations on this checkout.

  • co_label - The label for this checkout. This includes the name and domain (if any) for the checkout
muddled.version_control.vcs_init_directory(scheme, files=None)

Initialised the current directory for this VCS, and add the given list of files.

‘scheme’ is “git”, “bzr”, etc. - as taken from the first few characters of the muddle repository URL - so, e.g., “bzr” for “bzr+ssh://whatever”.

Raises KeyError if the scheme is not one for which we have a registered handler.

muddled.version_control.vcs_pull_directory(url)

Pull the current directory from the repository indicated by the URL

Looks at the first few characters of the URL to determine the VCS to use - so, e.g., “bzr” for “bzr+ssh://whatever”.

Raises KeyError if the scheme is not one for which we have a registered handler.

muddled.version_control.vcs_push_directory(url)

Push the current directory to the repository indicated by the URL

Looks at the first few characters of the URL to determine the VCS to use - so, e.g., “bzr” for “bzr+ssh://whatever”.

Raises KeyError if the scheme is not one for which we have a registered handler.

muddled.version_control.vcs_special_files(url)

Return the names of the ‘special’ files/directories used by this VCS.

For instance, if ‘url’ starts with “git+” then we might return [”.git”, ”.gitignore”, ”.gitmodules”]

muddled.version_stamp

Note

Stamp files - a record of the content of a build tree.

VersionStamp and stamp file support.

Stamp files

Stamp files are INI files, implemented using the Python ConfigParser module (specifically, RawConfigParser).

INI files are composed of one or more sections, each of the form:

[header]
key = value
key = value

Indentation is allowed, but will be ignored.

Comment lines may be introduced with either ‘#’ or ‘;’, and in-line comments may be added by following ‘[header]’ or ‘key = value’ with whitespace and a ‘;’ (but not a ‘#’).

Muddle treats ‘#’ comment lines specially in stamp files, but is not aware of ‘;’ comments, and we recommend not using them in muddle stamp files.

When a stamp file is written or read, a SHA1 hash is calculated from the lines of text being written/read. Since mid-July 2012, comment lines starting with ‘#’, and blank lines (whitespace only) are not included in that SHA1 hash. Since a stamp file uniquely describes the current state of a muddle build tree, the hash calculated from its content can be regarded as a version id for the build tree.

It can sometimes be useful to edit a stamp file. If you do so, it is advisable to add a comment or comments indicating what has been done and why.

All muddle commands for handling stamp files are subcommands of either “muddle stamp” or “muddle unstamp”.

Version 1 Stamp Files

We retain limited support for version 1 stamp files, mainly to allow reading existing (legacy) files. There is provision for writing version 1 stamp files, but it is not guaranteed to be accurate.

Version 1 stamp files were replaced because they could not unambiguously store all of the information needed by the Repository class we now use to remember where a checkout “came from”.

Support for version 1 stamp files will be removed at some future time.

Version 2 Stamp Files

Version 2 stamp files are the current form of stamp file.

This section describes the current content of a stamp file, as currently written by muddle. For the rest of this section, “stamp files” should be taken to mean “version 2 stamp files”.

All stamp files start with some standard comment lines:

# Muddle stamp file
# Written at 2012-07-12 13:13:13
#            2012-07-12 12:13:13 UTC

The date and time stamps are in local time and UTC respectively.

This is followed by a general section:

[STAMP]
version = 2

which at the moment just identifies the version of the stamp file.

Next comes a section identifying the build description - this repeats information taken from the RootRepository, Description and VersionsRepository files in the .muddle/ directory of the build. For instance:

[ROOT]
repository = git+ssh://git@palmera.c//opt/kynesim/projects/001
description = builds/01.py
versions_repo = git+file:///home/tibs/temp/r/versions

and, if “muddle init -branch” was originally used, also the DescriptionBranch file:

[ROOT]
repository = git+ssh://git@palmera.c//opt/kynesim/projects/001
description = builds/01.py
description_branch = test-v0.1
versions_repo = git+file:///home/tibs/temp/r/versions

The keys (‘repository’, etc.) are always presented in the same order - this is a general principle within stamp file sections, so that stamp files themselves can be comparable with simple tools such as diff.

Next, if the build tree has subdomains, will come sections describing those domains. For instance:

[DOMAIN subdomain1]
description = builds/01.py
name = subdomain1
repository = git+file:///home/tibs/sw/muddle/tests/transient/repo/subdomain1

[DOMAIN subdomain1(subdomain3)]
description = builds/01.py
name = subdomain1(subdomain3)
repository = git+file:///home/tibs/sw/muddle/tests/transient/repo/subdomain3

Domain sections occur in “C” sort order of the domain names. Note that there will not be any domain sections if there are no subdomains (i.e., if there is no domains/ directory in the build tree).

Note

The [DOMAIN] sections do not record a ‘description_branch’ for each subdomain. We do not currently support specifying a particular branch for the subdomain build description in the same manner as is done at the top level with “muddle init -branch”.

Next come sections for each checkout. For instance:

[CHECKOUT (subdomain1)builds]
co_label = checkout:(subdomain1)builds/checked_out
co_leaf = builds
repo_vcs = git
repo_from_url_string = None
repo_base_url = file:///home/tibs/sw/muddle/tests/transient/repo/subdomain1
repo_name = builds
repo_prefix_as_is = False
repo_revision = 7d8377a18efcd8b3d7b788de8cee26aa6d770005

and:

[CHECKOUT builds]
co_label = checkout:builds/checked_out
co_leaf = builds
repo_vcs = git
repo_from_url_string = None
repo_base_url = ssh://git@palmera.c//opt/kynesim/projects/001
repo_name = builds
repo_prefix_as_is = False
repo_revision = dfb06f29c81828bbdfc48ce53d7bc4203b68a459

[CHECKOUT busybox-1.17.1]
co_label = checkout:busybox-1.17.1/checked_out
co_dir = linux
co_leaf = busybox-1.17.1
repo_vcs = git
repo_from_url_string = None
repo_base_url = ssh://git@palmera.c//opt/kynesim/projects/001
repo_name = busybox-1.17.1
repo_prefix = linux
repo_prefix_as_is = False
repo_revision = 965b4809b5ca93df0a4973e043a5a9af0ecf50e3
repo_branch = issue99-fix

The values presented give the checkout label, its location in the build tree (‘co_dir’ and ‘co_leaf’), and its Repository instance (the ‘repo_xxx’ values. See “muddle doc version_control.checkout_from_repo” for some information on how the ‘co_xxx’ values are used, and “muddle doc repository.Repository” for more information on the Repository class).

Note

The ‘repo_revision’ is the current revision of the checkout, and ‘repo_branch’ is its current branch if that is not “master”, and if the VCS for the checkout supports setting branches.

Again, these are presented in “C” sort order of the checkout name, including the domain component - this means that subdomain checkouts will occur before toplevel checkouts. Also, while not all checkout sections will contain the same values, the order in which they are presented will always be the same.

Finally, if “muddle stamp” reported any problems in creating the stamp file, these will be saved in a problems section. For instance:

[PROBLEMS]
problem1 = builds: 'svnversion' reports checkout has revision '459M'

(a subversion revision ending in ‘M’ means that the checkout was modified and not yet committed). Problems are presented as ‘problem1’, ‘problem2’, etc. In general, the checkout name should be the first part of the message occurring as the problem value.

Release stamp files

Release stamp files are an extension of normal stamp files that also specify a release. As such, they have an extra section (after the [STAMP] section) of the form:

[RELEASE]
name = project99
version = 1.2.3
archive = tar
compression = gzip

This indicates that the rest of the stamp file describes a release called (or for) “project99”, and that it is version 1.2.3. The release will be archived using tar, and the tar file will be compressed using gzip.

Both the name and version specified must start with an ASCII alphanumeric, and may only contain ASCII alphanumerics, and the characters ‘.’, ‘-‘ or ‘_’. For instance:

version = v1-2.11
version = xvii
version = 0.9.3alpha1

The archive value must currently be tar - other values may perhaps be allowed in the future.

The compression value must be either gzip or bzip2.

Note that any release stamp file can be read as a “normal” stamp file - the extra release-specific information will just be ignored.

class muddled.version_stamp.CheckoutTupleV1(name, repo, rev, rel, dir, domain, co_leaf, branch)

Bases: tuple

Create new instance of CheckoutTupleV1(name, repo, rev, rel, dir, domain, co_leaf, branch)

branch

Alias for field number 7

co_leaf

Alias for field number 6

dir

Alias for field number 4

domain

Alias for field number 5

name

Alias for field number 0

rel

Alias for field number 3

repo

Alias for field number 1

rev

Alias for field number 2

class muddled.version_stamp.ReleaseSpec(name=None, version=None, archive=None, compression=None, hash=None)

Bases: object

The basic specification of a Release.

The following correspond to the [RELEASE] section in a release stamp file:

  • ‘name’
  • ‘version’
  • ‘archive’
  • ‘compression’

‘name’ and ‘version’ may be set to None, or to a valid name/version.

When a ‘name’ or ‘version’ of None is written out to a release file, it is written out as ‘<REPLACE THIS>’, a carefully invalid value, indicating that the user needs to edit the file.

Otherwise, ‘name’ and ‘version’ values must start with ASCII alphanumeric and continue with ASCII alphanumeric, hyphen, underscore or dot.

‘archive’ and ‘compression’ may be set to None or a value from the ‘allowed_‘ lists. If they are set to None, they in fact get set to the first ‘allowed_‘ value, so this is a simple way of selecting the default.

We also allow the SHA1 hash of a stamp file to be remembered:

  • ‘hash’

There is no special handling for this, and it defaults to None.

allowed_archive_values = ['tar']
allowed_compression_values = ['gzip', 'bzip2']
archive
compression
static from_file(filename)

Read a simple representation of ourself from a file.

name
name_re = <_sre.SRE_Pattern object>
version
version_re = <_sre.SRE_Pattern object>
write_to_file(filename)

Write a simple representation of ourself out to a file.

class muddled.version_stamp.ReleaseStamp

Bases: muddled.version_stamp.VersionStamp

A VersionStamp with extra stuff to describe a release.

static from_builder(builder, quiet=False)

Construct a ReleaseStamp from a muddle build description.

‘builder’ is the muddle Builder for our build description.

If ‘quiet’ is True, then we will not print information about what we are doing, and we will not print out problems as they are found.

Returns a tuple of:

  • the new ReleaseStamp instance
  • a (possibly empty) list of problem summaries. If this is empty, then the stamp was calculated fully. Note that this is the same list as held within the ReleaseStamp instance itself.
static from_file(filename)

Construct a ReleaseStamp by reading in a stamp file.

Returns a new ReleaseStamp instance.

Note that the SHA1 computed and reported for that ReleaseStamp does not include blank lines or comment lines that start with a ‘#’ (or whitespace and a ‘#’).

write_to_file_object(fd, version=2)

Write our data out to a file-like object (one with a ‘write’ method).

class muddled.version_stamp.VersionStamp

Bases: object

A representation of the revision state of a build tree’s checkouts.

Our internal data is:

  • ‘repository’ is a string giving the default repository (as stored in .muddle/RootRepository)

  • description is a string naming the build description (as stored in .muddle/Description)

  • description_branch is None or a string naming the branch of the build description (as stored in .muddle/DescriptionBranch, and originally specified by “muddle init -branch”)

  • ‘versions_repo’ is a string giving the versions repository (as stored in .muddle/VersionsRepository)

  • ‘domains’ is a dictionary mapping domain names to tuples of the form (domain_repo, domain_desc), where:

    • domain_repo is the default repository for the domain
    • domain_desc is the build description for the domain
  • ‘checkouts’ is a dictionary mapping checkout labels to tuples of the form (co_dir, co_leaf, repo), where:

    • co_dir is the sub-path between src/ and the co_leaf
    • co_leaf is the name of the directory within src/ that actually contains the checkout
    • repo is a Repository instance, where to find the checkout remotely

    These are the appropriate arguments for the checkout_from_repo() function in version_control.py.

    The checkout will be in src/<co_dir>/<co_leaf> (if <co_dir> is set), or src/<co_leaf> (if it is not).

  • ‘options’ is a dictionary mapping checkout labels to dictionaries of the form {option_name : option_value}. There will only be entries for those checkouts which do have options.

  • ‘problems’ is a list of problems in determining the stamp information. This will be of zero length if the stamp if accurate, but will otherwise contain a string for each checkout whose revision could not be accurately determined.

    Note that when problems descriptions are written to a stamp file, they are truncated.

MAX_PROBLEM_LEN = 100
compare_checkouts(other, fd=<open file '<stdout>', mode 'w'>)

Compare the checkouts in this VersionStamp with those in another.

‘other’ is another VersionStamp.

‘fd’ is where any messages should be written, defaulting to stdout. If this is None, then no messages will be writen.

Note that this only compares the checkouts (including their options) - it does not compare any of the other fields in a VersionStamp.

Returns a tuple of (deleted, new, changed, problems) sequences, where these are:

  • a sequence of tuples of the form:

    (co_label, co_dir, co_leaf, repo)

    for checkouts that are in this VersionStamp but not in the ‘other’ - i.e., “deleted” checkouts

  • a sequence of tuples of the form:

    (co_label, co_dir, co_leaf, repo)

    for checkouts that are in the ‘other’ VersionStamp but not in this - i.e., “new” checkouts

  • a sequence of tuples for any checkouts with differing revisions, of the form:

    (co_label, revision1, revision2)

    where ‘this_repo’ and ‘other_repo’ are relevant Repository instances.

  • a sequence of tuples of the form:

    (co_label, problem_string)

    for checkouts that are present in both VersionStamps, but differ in something other than revision.

static from_builder(builder, force=False, just_use_head=False, before=None, quiet=False)

Construct a VersionStamp from a muddle build description.

‘builder’ is the muddle Builder for our build description.

If ‘force’ is true, then attempt to “force” a revision id, even if it is not necessarily correct. For instance, if a local working directory contains uncommitted changes, then ignore this and use the revision id of the committed data. If it is actually impossible to determine a sensible revision id, then use the revision specified by the build description (which defaults to HEAD). For really serious problems, this may refuse to guess a revision id.

(Typical use of this is expected to be when a trying to calculate a stamp reports problems in particular checkouts, but inspection shows that these are artefacts that may be ignored, such as an executable built in the source directory.)

If ‘just_use_head’ is true, then HEAD will be used for all checkouts. In this case, the repository specified in the build description is used, and the revision id and status of each checkout is not checked.

If ‘before’ is given, it should be a string describing a date/time, and the revision id chosen for each checkout will be the last revision at or before that date/time.

Note

This depends upon what the VCS concerned actually supports. This feature is experimental.

If ‘quiet’ is True, then we will not print information about what we are doing, and we will not print out problems as they are found.

Returns a tuple of:

  • the new VersionStamp instance
  • a (possibly empty) list of problem summaries. If this is empty, then the stamp was calculated fully. Note that this is the same list as held within the VersionStamp instance itself.
static from_file(filename)

Construct a VersionStamp by reading in a stamp file.

Returns a new VersionStamp instance.

Note that the SHA1 computed and reported for that VersionStamp does not include blank lines or comment lines that start with a ‘#’ (or whitespace and a ‘#’).

print_problems(output=None, truncate=None, indent='')

Print out any problems.

If ‘output’ is not specified, then it will be STDOUT, otherwise it should be a file-like object (supporting ‘write’).

If ‘truncate’ is None (or zero, non-true, etc.) then the problems will be truncated to the same length as when writing them to a stamp file.

‘indent’ should be a string to print in front of every line.

If there are no problems, this method does not print anything out.

write_to_file(filename, version=2)

Write our data out to a file.

By default, writes out a version 2 stamp file, as opposed to the older version 1 format.

Returns the SHA1 hash for the file.

write_to_file_object(fd, version=2)

Write our data out to a file-like object (one with a ‘write’ method).

By default, writes out a version 2 stamp file, as opposed to the older version 1 format.

Returns the SHA1 hash for the file.

Note that the SHA1 does not include blank lines or comment lines that start with a ‘#’ (or whitespace and a ‘#’).

muddled.version_stamp.get_and_remove_option(config, section, name)

Get an option from a section, as a string, and also remove it.

muddled.version_stamp.make_RawConfigParser(ordered=False, sorted=False)

Make a RawConfigParrser.

Always tell it we want to preserve the case of keys.

If ‘ordered’, then use a MuddleOrderedDict inside it, so that things remember their insertion order, and that is used on output.

If ‘sorted’, use a MuddleSortedDict, so that things are sorted, and thus output in sorted order.

If neither, don’t specify a dict, and random stuff might happen

muddled.version_stamp.maybe_get_option(config, section, name, remove=False)

Get an option from a section, as a string, or as None.

If the option is present, return it, otherwise return None.

If remove is true, and the option was present, also remove it.

muddled.version_stamp.maybe_set_option(config, section, name, value)

Set an option in a section, if its value is not None.

muddled.xmlconfig

A utility module which allows you to read XML files and then query them as XPath-like paths.

Call readXml() to read an XML file and return an xmlConfig object which you can then pass queries like:

/elem1/elem2/elem3

This is essentially a (restricted) XPath query with an implicit ::text() appended.

This file is hereby placed in the public domain -
Richard Watts, <rrw@kynesim.co.uk> 2009-10-23.
(as this is about the third time I have had to write
it and I am getting quite bored .. )
class muddled.xmlconfig.Config(in_file)

Bases: object

Represents a configuration file

Parse an XML config file into a local representation

exists(key)

Given a key, decide if its value exists in the configuration file.

query(keys)

Perform a query on this list of keys and return the node which matches (and which you can then call text() on)

query_bool(key)

Given an XPath-like expression, return a boolean value based on a text value of ‘true’ (True) or anything else (False)

If the node doesn’t exist, throw.

query_hashlist(key, subkeys)

Given an XPath-like expression and a list of subkeys, take the list denoted by key and return a list of hashes pointing the subkeys at their values.

query_int(key)

Given an XPath-like expression a/b/c.., return the text in the final node interpreted as an integer.

If the node doesn’t exist, throw.

query_list(key)

Given an XPath-like expression, return a list containing the text from all values key0..keyN that actually exist

query_string(key)

Given an XPath-like expression /a/b/c , return the text in the final node. If the node doesn’t exist, throw.

split_key(instring)

Take a series of components and split them by ‘/’.

text(node)

Collect all the text in an XML node

exception muddled.xmlconfig.ConfigError

Bases: exceptions.Exception

Lower-level modules

muddled.checkouts

muddled.checkouts.simple

Simple entry points so that descriptions can assert the existence of checkouts easily

muddled.checkouts.simple.absolute(builder, co_name, repo_url, rev=None, branch=None)

Check out a repository from an absolute URL.

<repo_url> must be of the form <vcs>+<url>, where <vcs> is one of the support version control systems (e.g., ‘git’, ‘svn’).

<rev> may be a revision (specified as a string). “HEAD” (or its equivalent) is assumed by default.

<branch> may be a branch. “master” (or its equivalent) is assumed by default.

The repository <repo_url>/<co_name> will be checked out into src/<co_name>.

muddled.checkouts.simple.relative(builder, co_name, repo_relative=None, rev=None, branch=None)

A simple, VCS-controlled, checkout.

<rev> may be a revision (specified as a string). “HEAD” (or its equivalent) is assumed by default.

<branch> may be a branch. “master” (or its equivalent) is assumed by default.

If <repo_relative> is None then the repository <base_url>/<co_name> will be checked out into src/<co_name>, where <base_url> is the base URL as specified in .muddle/RootRepository (i.e., the base URL of the build description, as used in “muddle init”).

For example:

<base_url>/<co_name>  -->  src/<co_name>

If <repo_relative> is not None, then the repository <base_url>/<repo_relative> will be checked out instead:

<base_url>/<repo_relative>  -->  src/<co_name>

muddled.checkouts.twolevel

Two-level checkouts. Makes it slightly easier to separate checkouts out into roles. I’ve deliberately not implemented arbitrary-level checkouts for fear of complicating the checkout tree.

muddled.checkouts.twolevel.absolute(builder, co_dir, co_name, repo_url, rev=None, branch=None)

Check out a twolevel repository from an absolute URL.

<repo_url> must be of the form <vcs>+<url>, where <vcs> is one of the support version control systems (e.g., ‘git’, ‘svn’).

<rev> may be a revision (specified as a string). “HEAD” (or its equivalent) is assumed by default.

<branch> may be a branch. “master” (or its equivalent) is assumed by default.

The repository <repo_url>/<co_name> will be checked out into src/<co_dir>/<co_name>.

muddled.checkouts.twolevel.relative(builder, co_dir, co_name, repo_relative=None, rev=None, branch=None)

A two-level version of checkout.simple.relative().

It attempts to check out <co_dir>/<co_name> (but see below).

<rev> may be a revision (specified as a string). “HEAD” (or its equivalent) is assumed by default.

<branch> may be a branch. “master” (or its equivalent) is assumed by default.

If <repo_relative> is None then the repository <base_url>/<co_name> will be checked out, where <base_url> is the base URL as specified in .muddle/RootRepository (i.e., the base URL of the build description, as used in “muddle init”).

If <repo_relative> is not None, then the repository <base_url>/<repo_relative> ...

In the normal case, the location in the repository and in the checkout is assumed the same (i.e., <co_dir>/<co_name>). So, for instance, with co_dir=”A” and co_name=”B”, the repository would have:

<base_url>/A/B

which we would check out into:

src/A/B

Occasionally, though, the repository is organised differently, so for instance, one might want to checkout:

<base_url>/B

into:

src/A/B

In this latter case, one can use the ‘repo_relative’ argument, to say where the checkout is relative to the repository’s “base”. So, in the example above, we still have co_dir=”A” and co_name=”B”, but we also want to say repo_relative=B.

muddled.checkouts.twolevel.twolevel(builder, co_dir, co_name, repo_relative=None, rev=None, branch=None)

A two-level version of checkout.simple.relative().

It attempts to check out <co_dir>/<co_name> (but see below).

<rev> may be a revision (specified as a string). “HEAD” (or its equivalent) is assumed by default.

<branch> may be a branch. “master” (or its equivalent) is assumed by default.

If <repo_relative> is None then the repository <base_url>/<co_name> will be checked out, where <base_url> is the base URL as specified in .muddle/RootRepository (i.e., the base URL of the build description, as used in “muddle init”).

If <repo_relative> is not None, then the repository <base_url>/<repo_relative> ...

In the normal case, the location in the repository and in the checkout is assumed the same (i.e., <co_dir>/<co_name>). So, for instance, with co_dir=”A” and co_name=”B”, the repository would have:

<base_url>/A/B

which we would check out into:

src/A/B

Occasionally, though, the repository is organised differently, so for instance, one might want to checkout:

<base_url>/B

into:

src/A/B

In this latter case, one can use the ‘repo_relative’ argument, to say where the checkout is relative to the repository’s “base”. So, in the example above, we still have co_dir=”A” and co_name=”B”, but we also want to say repo_relative=B.

muddled.checkouts.multilevel

Multi-level checkouts. Required for embedding things like android, which have a lot of deep internal checkouts.

muddled.checkouts.multilevel.absolute(builder, co_dir, co_name, repo_url, rev=None, branch=None)

Check out a multilevel repository from an absolute URL.

<repo_url> must be of the form <vcs>+<url>, where <vcs> is one of the support version control systems (e.g., ‘git’, ‘svn’).

<rev> may be a revision (specified as a string). “HEAD” (or its equivalent) is assumed by default.

<branch> may be a branch. “master” (or its equivalent) is assumed by default.

The repository <repo_url> will be checked out into src/<co_dir>. The checkout will be identified by the label checkout:<co_name>/checked_out.

muddled.checkouts.multilevel.relative(builder, co_dir, co_name, repo_relative=None, rev=None, branch=None)

A multilevel checkout, with checkout name unrelated to checkout directory.

Sometimes it is necessary to cope with checkouts that either:

  1. are more than two directories below src/, or
  2. have a checkout name that is not the same as the “leaf” directory in their path

Both of these can happen when trying to represent an Android build, for instance.

Thus:

multilevel.relative(builder, co_dir='this/is/here', co_name='checkout1')

will look for the repository <base_url>/this/is/here and check it out into src/this/is/here, but give it label checkout:checkout1/checked_out.

(<base_url> is the base URL as specified in .muddle/RootRepository (i.e., the base URL of the build description, as used in “muddle init”).

For the moment, <repo_relative> is ignored.

muddled.deployments

muddled.deployments.collect

Collect deployment.

This deployment is used to collect elements from:

  • checkout directories
  • package ‘obj’ directories
  • package role ‘install’ directories
  • other deployments

into deployment directories, usually to be processed by some external tool.

class muddled.deployments.collect.AssemblyDescriptor(from_label, from_rel, to_name, recursive=True, failOnAbsentSource=False, copyExactly=True, usingRSync=False, obeyInstructions=True)

Bases: object

Construct an assembly descriptor.

We copy from the directory from_rel in from_label (package, deployment, checkout) to the name to_name under the deployment.

Give a package of ‘*’ to copy from the install directory for a given role.

If recursive is True, we’ll copy recursively.

  • failOnAbsentSource - If True, we’ll fail if the source doesn’t exist.
  • copyExactly - If True, keeps links. If false, copies the file they point to.
get_source_dir(builder)
class muddled.deployments.collect.CollectApplyChmod

Bases: muddled.deployments.collect.InstructionImplementor

apply(builder, instr, role, path)
needs_privilege(builder, instr, role, path)
prepare(builder, instr, role, path)
class muddled.deployments.collect.CollectApplyChown

Bases: muddled.deployments.collect.InstructionImplementor

apply(builder, instr, role, path)
needs_privilege(builder, instr, role, path)
prepare(builder, instr, role, path)
class muddled.deployments.collect.CollectDeploymentBuilder

Bases: muddled.depend.Action

Builds the specified collect deployment.

add_assembly(assembly_descriptor)
apply_instructions(builder, label, prepare, deploy_path)
build_label(builder, label)

Actually do the copies ..

deploy(builder, label, target_base)
sort_out_and_run_instructions(builder, label)
class muddled.deployments.collect.InstructionImplementor

Bases: object

apply(builder, instruction, role, path)
needs_privilege(builder, instr, role, path)
prepare(builder, instruction, role, path)

Prepares for rsync. This means fixing up the destination file (e.g. removing it if it may have changed uid by a previous deploy) so we will be able to rsync it.

muddled.deployments.collect.copy_from_checkout(builder, name, checkout, rel, dest, recursive=True, failOnAbsentSource=False, copyExactly=True, domain=None, usingRSync=False)
muddled.deployments.collect.copy_from_deployment(builder, name, dep_name, rel, dest, recursive=True, failOnAbsentSource=False, copyExactly=True, domain=None, usingRSync=False)
usingRSync - set to True to copy with rsync - substantially faster than
cp
muddled.deployments.collect.copy_from_package_obj(builder, name, pkg_name, pkg_role, rel, dest, recursive=True, failOnAbsentSource=False, copyExactly=True, domain=None, usingRSync=False)
  • If ‘usingRSync’ is true, copy with rsync - substantially faster than
    cp, if you have rsync. Not very functional if you don’t :-)
muddled.deployments.collect.copy_from_role_install(builder, name, role, rel, dest, recursive=True, failOnAbsentSource=False, copyExactly=True, domain=None, usingRSync=False, obeyInstructions=True)

Add a requirement to copy from the given role’s install to the named deployment.

‘name’ is the name of the collecting deployment, as created by:

deploy(builder, name)

which is remembered as a rule whose target is deployment:<name>/deployed, where <name> is the ‘name’ given.

‘role’ is the role to copy from. Copying will be based from ‘rel’ within the role’s install, to ‘dest’ within the deployment.

The label package:(<domain>)*{<role>}/postinstalled will be added as a dependency of the collecting deployment rule.

An AssemblyDescriptor will be created to copy from ‘rel’ in the install directory of the label package:*{<role>}/postinstalled, to ‘dest’ within the deployment directory of ‘name’, and added to the rule’s actions.

So, for instance:

copy_from_role_install(builder,'fred','data','public','data/public',
                       True, False, True)

might copy (recursively) from:

install/data/public

to:

deploy/fred/data/public

‘rel’ may be the empty string (‘’) to copy all files in the install directory.

  • If ‘recursive’ is true, then copying is recursive, otherwise it is not.
  • If ‘failOnAbsentSource’ is true, then copying will fail if the source does not exist.
  • If ‘copyExactly’ is true, then symbolic links will be copied as such, otherwise the linked file will be copied.
  • If ‘usingRSync’ is true, copy with rsync - substantially faster than
    cp, if you have rsync. Not very functional if you don’t :-)
  • If ‘obeyInstructions’ is False, don’t obey any applicable instructions.
muddled.deployments.collect.deploy(builder, name)

Create a collection deployment builder.

This adds a new rule linking the label deployment:<name>/deployed to the collection deployment builder.

You can then add assembly descriptors using the other utility functions in this module.

Dependencies get registered when you add an assembly descriptor.

muddled.deployments.cpio

cpio deployment.

Most commonly used to create Linux ramdisks, this deployment creates a CPIO archive from the relevant install directory and applies the relevant instructions.

Because python has no native CPIO support, we need to do this by creating a tar archive and then invoking cpio in copy-through mode to convert the archive to cpio. Ugh.

class muddled.deployments.cpio.CIApplyChmod

Bases: muddled.deployments.cpio.CpioInstructionImplementor

apply(builder, instr, role, target_base, hierarchy)
class muddled.deployments.cpio.CIApplyChown

Bases: muddled.deployments.cpio.CpioInstructionImplementor

apply(builder, instr, role, target_base, hierarchy)
class muddled.deployments.cpio.CIApplyMknod

Bases: muddled.deployments.cpio.CpioInstructionImplementor

apply(builder, instr, role, target_base, hierarchy)
class muddled.deployments.cpio.CpioDeploymentBuilder(target_file, target_base, compressionMethod=None, pruneFunc=None)

Bases: muddled.depend.Action

Builds the specified CPIO deployment.

  • ‘target_file’ is the CPIO file to construct.
  • ‘target_base’ is an array of pairs mapping labels to target locations, or (label, src) -> location
  • ‘compressionMethod’ is the compression method to use, if any - gzip -> gzip, bzip2 -> bzip2.
  • if ‘pruneFunc’ is not None, it is a function to be called like pruneFunc(Hierarchy) to prune the hierarchy prior to packing. Usually something like deb.deb_prune, it’s intended to remove spurious stuff like manpages from initrds and the like.
attach_env(builder)

Attaches an environment containing:

MUDDLE_TARGET_LOCATION - the location in the target filesystem where this deployment will end up.

to every package label in this role.

build_label(builder, label)

Actually cpio everything up, following instructions appropriately.

class muddled.deployments.cpio.CpioInstructionImplementor

Bases: object

apply(builder, instruction, role, path)
class muddled.deployments.cpio.CpioWrapper(builder, action, label)

Bases: object

copy_from_role(from_role, from_fragment, to_fragment, with_base=None)

Copy the relative path from_fragment in from_role to to_fragment in the CPIO package or deployment given by ‘action’

Use ‘with_base’ to change the base offset we apply when executing instructions; this is useful when using repeated copy_from_role() invocations to copy a subset of one role to a package/deployment.

done()

Call this once you’ve added all the roles you want; it attaches the deployment environment to them and generally finishes up

muddled.deployments.cpio.create(builder, target_file, name, compressionMethod=None, pruneFunc=None)

Create a CPIO deployment and return it.

  • ‘builder’ is the muddle builder that is driving us

  • ‘target_file’ is the name of the CPIO file we want to create. Note that this may include a sub-path (for instance, “fred/file.cpio” or even “/fred/file.cpio”).

  • ‘name’ is either:

    1. The name of the deployment that will contain this CPIO file (in the builder’s default domain), or
    2. A deployment or package label, ditto
  • ‘comporessionMethod’ is the compression method to use:

    • None means no compression
    • ‘gzip’ means gzip
    • ‘bzip2’ means bzip2
  • if ‘pruneFunc’ is not None, it is a function to be called like pruneFunc(Hierarchy) to prune the hierarchy prior to packing. Usually something like deb.deb_prune, it’s intended to remove spurious stuff like manpages from initrds and the like.

Normal usage is thus something like:

fw = cpio.create(builder, 'firmware.cpio', deployment)
fw.copy_from_role(role1, '', '/')
fw.copy_from_role(role2, 'bin', '/bin')
fw.done()

or:

fw = cpio.create(builder, 'firmware.cpio', package('firmware', role))
fw.copy_from_role(role, '', '/')
fw.done()

muddled.deployments.filedep

File deployment. This deployment just copies files into a role subdirectory in the /deployed directory, applying appropriate instructions.

class muddled.deployments.filedep.FIApplyChmod

Bases: muddled.deployments.collect.CollectApplyChmod

needs_privilege(builder, instr, role, path)
class muddled.deployments.filedep.FIApplyMknod

Bases: muddled.deployments.collect.InstructionImplementor

apply(builder, instr, role, path)
needs_privilege(builder, instr, role, path)
prepare(builder, instr, role, path)
class muddled.deployments.filedep.FileDeploymentBuilder(roles, target_dir)

Bases: muddled.depend.Action

Builds the specified file deployment

role is actually a list of (role, domain) pairs.

apply_instructions(builder, label)
attach_env(builder)

Attaches an environment containing:

MUDDLE_TARGET_LOCATION - the location in the target filesystem where this deployment will end up.

to every package label in this role.

build_label(builder, label)

Performs the actual build.

We actually do need to copy all files from install/ (where unprivileged processes can modify them) to deploy/ (where they can’t).

Then we apply instructions to deploy.

deploy(builder, label)
muddled.deployments.filedep.deploy(builder, target_dir, name, roles)

Register a file deployment.

This is a convenience wrapper around deploy_with_domains().

‘roles’ is a sequence of role names. The deployment will take the roles specified, and build them into a deployment at deploy/[name].

More specifically, a rule will be created for label:

“deployment:<name>/deployed”

which depends on “package:*{<role}/postinstalled” (in the builder’s default domain) for each <role> in ‘roles’.

In other words, the deployment called ‘name’ will depend on the given roles (in the default domain) having been “finished” (postinstalled).

An “instructions applied” label “deployment:<name>/instructionsapplied” will also be created.

The deployment should eventually be located at ‘target_dir’.

muddled.deployments.filedep.deploy_with_domains(builder, target_dir, name, role_domains)

Register a file deployment.

‘role_domains’ is a sequence of (role, domain) pairs. The deployment will take the roles and domains specified, and build them into a deployment at deploy/[name].

More specifically, a rule will be created for label:

“deployment:<name>/deployed”

which depends on “package:(<domain>)*{<role}/postinstalled” for each (<role>, <domain>) pair in ‘role_domains’.

In other words, the deployment called ‘name’ will depend on the given roles in the appropriate domains having been “finished” (postinstalled).

An “instructions applied” label “deployment:<name>/instructionsapplied” will also be created.

The deployment should eventually be located at ‘target_dir’.

muddled.deployments.tools

Tools deployment. This deployment merely adds the appropriate environment variables to use the tools in the given role install directories to everything in another list of deployments.

Instructions are ignored - there’s no reason to follow them (yet) and it’s simpler not to.

XXX Do we support this anymore?

class muddled.deployments.tools.ToolsDeploymentBuilder(dependent_roles)

Bases: muddled.depend.Action

Copy the dependent roles into the tools deployment.

build_label(builder, label)
deploy(builder, label)
muddled.deployments.tools.attach_env(builder, role, env, name)

Attach suitable environment variables for the given input role to the given environment store.

We set:

  • LD_LIBRARY_PATH - Prepend $role_installl/lib
  • PATH - Append $role_install/bin
  • PKG_CONFIG_PATH - Prepend $role_install/lib/pkgconfig
  • $role_TOOLS_PATH - Prepend $role_install/bin

The PATH/TOOLS_PATH stuff is so you can still locate tools which were in the path even if they’ve been overridden with your built tools.

muddled.deployments.tools.deploy(builder, name, rolesThatUseThis=[], rolesNeededForThis=[])

Register a tools deployment.

This is used to:

  1. Set the environment for each role in ‘rolesThatUseThis’ so that PATH, LD_LIBRARY_PATH and PKG_CONFIG_PATH include the ‘name’ deployment
  2. Make deployment:<name>/deployed depend upon the ‘rolesNeededForThis’
  3. Register cleanup for this deployment

The intent is that we have a “tools” deployment, which provides useful host tools (for instance, something to mangle a file in a particular manner). Those roles which need to use such tools in their builds (normally in a Makefile.muddle) then need to have the environment set appropriately to allow them to find the tools (and ideally, not system provided tools which mighth have the same name).

muddled.pkgs

muddled.pkgs.aptget

An apt-get package. When you try to build it, this package pulls in a pre-canned set of packages via apt-get.

class muddled.pkgs.aptget.AptGetBuilder(name, role, pkgs_to_install, os_version=None)

Bases: muddled.pkg.PackageBuilder

Make sure that particular OS packages have been installed.

The “build” action for AptGetBuilder uses the Debian tool apt-get to ensure that each package is installed.

Our arguments are:

  • ‘name’ - the name of this builder
  • ‘role’ - the role to which it belongs
  • ‘pkgs_to_install’ - a sequence specifying which packages are to be installed.

Each item in the sequence ‘pkgs_to_install’ can be:

  • the name of an OS package to install - for instance, ‘libxml2-dev’

    (this is backwards compatible with how this class worked in the past)

  • a Choice allowing a particular package to be selected according to the operating system.

    See “muddle doc Choice” for details on the Choice class.

    Note that a choice resulting in None (i.e., where the default value is None, and the default is selected) will not do anything.

    If ‘os_version’ is given, then it will be used as the version name, otherwise the result of calling utils.get_os_version_name() will be used.

We also allow a single string, or a single Choice, treated as if they were wrapped in a list.

already_installed(pkg)

Decide if the quoted debian package is already installed.

We use dpkg-query:

$ dpkg-query -W -f=\${Status}\n libreadline-dev
install ok installed

That third word means what it says (installed). Contrast with a package that is either not recognised or has not been downloaded at all:

$ dpkg-query -W -f=\${Status}\n a0d
dpkg-query: no packages found matching a0d

So we do some fairly simple processing of the output...

build_label(builder, label)

This time, build is the only one we care about.

muddled.pkgs.aptget.depends_on_aptget(builder, name, role, pkg, pkg_role)

Make a package dependant on a particular apt-builder.

  • pkg - The package we want to add a dependency to. ‘*’ is a good thing to add here ..
muddled.pkgs.aptget.medium(builder, name, role, apt_pkgs, roles, os_version=None)

Construct an apt-get package and make every package in the named roles depend on it.

Note that apt_pkgs can be an OS package name or a choices sequence - see the documentation for AptGetBuilder.

muddled.pkgs.aptget.simple(builder, name, role, apt_pkgs, os_version=None)

Construct an apt-get package in the given role with the given apt_pkgs.

Note that apt_pkgs can be an OS package name or a Choice - see the documentation for AptGetBuilder for more details.

For instance (note: not a real example - the dependencies don’t make sense!):

from muddled.utils import Choice
from muddled.pkgs import aptget
aptget.simple(builder, "host_packages", "host_environment",
       [
       "gcc-multilib",
       "g++-multilib",
       "lib32ncurses5-dev",
       "lib32z1-dev",
       "bison",
       "flex",
       "gperf",
       "libx11-dev",
       # On Ubuntu 11 or 12, choose icedtea-7, otherwise icedtea-6
       Choice([ ("ubuntu 1[12].*", "icedtea-7-jre"),
                ("ubuntu *", "icedtea-6-jre") ]),
       # On Ubuntu 10 or later, use libgtiff5
       # On Ubuntu 3 through 9, use libgtiff4
       # Otherwise, just don't try to use libgtiff
       Choice([ ("ubuntu 1?", "libgtiff5"),
                ("ubuntu [3456789]", "libgtiff4"),
                None ])
       ])

muddled.pkgs.deb

Some code which sneakily steals binaries from Debian/Ubuntu.

Quite a lot of code for embedded systems can be grabbed pretty much directly from the relevant Ubuntu binary packages - this won’t work with complex packages like exim4 without some external frobulation, since they have relatively complex postinstall steps, but it works supported architecture it’s a quick route to externally maintained binaries which actually work and it avoids having to build absolutely everything in your linux yourself.

This package allows you to ‘build’ a package from a source file in a checkout which is a .deb. We run dpkg with enough force options to install it in the relevant install directory.

You still need to provide any relevant instruction files (we’ll register <filename>.instructions.xml for you automatically if it exists).

We basically ignore the package database (there is one, but it’s always empty and stored in the object directory).

class muddled.pkgs.deb.DebAction(name, role, co, pkg_name, pkg_file, instr_name=None, postInstallMakefile=None)

Bases: muddled.pkg.PackageBuilder

Use dpkg to extract debian archives from the given checkout into the install directory.

  • co - is the checkout name in which the package resides.

  • pkg_name - is the name of the package (dpkg needs it)

  • pkg_file - is the name of the file the package is in, relative to the checkout directory.

  • instr_name - is the name of the instruction file, if any.

  • postInstallMakefile - if not None:

    make -f postInstallMakefile <pkg-name>
    

    will be run at post-install time to make links, etc.

build_label(builder, label)

Build the relevant label.

ensure_dirs(builder, label)
class muddled.pkgs.deb.DebDevAction(name, role, co, pkg_name, pkg_file, instr_name=None, postInstallMakefile=None, nonDevCoName=None, nonDevPkgFile=None)

Bases: muddled.pkg.PackageBuilder

Use dpkg to extract debian archives into obj/include and obj/lib directories so we can use them to build other packages.

As for a DebAction, really.

build_label(builder, label)

Actually install the dev package.

ensure_dirs(builder, label)
muddled.pkgs.deb.deb_prune(h)

Given a cpiofile hierarchy, prune it so that only the useful stuff is left.

We do this by lopping off directories, which is easy enough in cpiofile heirarchies.

muddled.pkgs.deb.dev(builder, coName, name, roles, depends_on=[], pkgFile=None, debName=None, nonDevCoName=None, nonDevDebName=None, instrFile=None, postInstallMakefile=None)

A wrapper for ‘deb.simple’, with the “idDev” flag set True.

  • nonDevCoName is the checkout in which the non-dev version of the package resides.
  • nonDevDebName is the non-dev version of the package; this is sometimes needed because of the odd way in which debian packages the ‘.so’ link in the dev package and the sofiles themselves into the non-dev.
muddled.pkgs.deb.extract_into_obj(inv, co_name, label, pkg_file)
muddled.pkgs.deb.simple(builder, coName, name, roles, depends_on=[], pkgFile=None, debName=None, instrFile=None, postInstallMakefile=None, isDev=False, nonDevCoName=None, nonDevPkgFile=None)

Build a package called ‘name’ from co_name / pkg_file with an instruction file called instr_file.

‘name’ is the name of the muddle package and of the debian package. if you want them different, set deb_name to something other than None.

Set isDev to True for a dev package, False for an ordinary binary package. Dev packages are installed into the object directory where MUDDLE_INC_DIRS etc. expects to look for them. Actual packages are installed into the installation directory where they will be transported to the target system.

muddled.pkgs.depmode_merge

Merge depmod databases.

Linux’s depmod tool is absurdly stupid. It:

  • Will not merge module databases from more than one location
  • Will not deduce kernel versions from its naming scheme
  • Will not accept absolute directory names in its configuration file
  • Will not document its database format so I can use it directly

This module contains a depmod_merge() package. You create one with create() and then add every package which produces kernel modules to it with add_deps()/add_roles()/add() - whichever is most convenient.

By default, we expect your packages to leave their modules in <pkg_install_dir>/lib/modules/KERNEL_VERSION/... - you can change this by specifying ‘subdir’ to the add_XXX() routines and we will then expect <pkg_install_dir>/<subdir>/KERNEL_VERSION/...

(the KERNEL_VERSION is sadly a requirement of depmod. Go, um, depmod)

Your modules must have a ‘.ko’ extension.

This package then depends on all those packages and when built will create a temporary database in its object directory containing all the .kos from all of the packages it’s been told to look at.

It then scans <objdir>/lib/modules/d+.d+.* for all the kernel versions and depmod’s them all.

It then copies module.* (i.e. all the module dependency files produced) back into <install_dir>/<subdir>/KERNEL_VERSION/ .

The dependency mechanism means that so long as you have your roles set up correctly, even if your sub-packages attempt to depmod on their own (as the kernel does), this package will always run later and overwrite the bad module dependencies with new, good ones.

If you are deploying multiple roles which each compute their dependencies separately, you will need to use the cpio ordering feature to make sure the right module database gets copied into your final cpio archive - we can’t do this for you (yet) because there are no facilities yet for post-deployment operations and we can’t create the dependency database on the way because the files don’t all exist in the same place that we’d need them to to run depmod.

Grr. Aargh. beat head against wall. Etc.

class muddled.pkgs.depmod_merge.MergeDepModBuilder(name, role, custom_depmod=None)

Bases: muddled.pkg.PackageBuilder

Use depmod_merge to merge several depmod databases into a

single result. We do this by writing a depmod.conf in our object directory and then invoking depmod.

custom_depmod tells us we want to use a custom depmod.

Constructor for the depmod package

self.components is a list of (label, subdir) pairs which we combine in our
object directory to form a unified module database on which we can run depmod.
add_label(label, subdir)
build_label(builder, label)
muddled.pkgs.depmod_merge.add(builder, merger, pkg, role, subdir='/lib/modules')
muddled.pkgs.depmod_merge.add_deps(builder, merger, deps, subdir='/lib/modules')

Add a set of packages and roles to this merger as packages which create linux kernel modules. deps is a list of (pkg,role)

muddled.pkgs.depmod_merge.add_roles(builder, merger, pkg, roles, subdir='/lib/modules')
muddled.pkgs.depmod_merge.create(builder, name, role, pkgs_and_roles, custom_depmod=None, subdir='/lib/modules')

Create a depmod_merge . It will depend on each of the mentioned packages.

pkgs is a list of (pkg, role) pairs.

We return the depmod_merge we’ve created.

muddled.pkgs.depmod_merge.predicate_is_kernel_module(name)
muddled.pkgs.depmod_merge.predicate_is_module_db(name)

Decide if a full path name is list modules.*

muddled.pkgs.initscripts

Write an initialisation script into $(MUDDLE_TARGET_LOCATION)/bin/$(something)

This is really just using utils.subst_file() with the current environment, on a resource stored in resources/.

We also write a setvars script with a suitable set of variables for running code in the context of the deployment, and any variables you’ve set in the environment store retrieved with get_env_store().

class muddled.pkgs.initscripts.InitScriptBuilder(name, role, script_name, deployments, writeSetvarsSh=True, writeSetvarsPy=False)

Bases: muddled.pkg.PackageBuilder

Build an init script.

build_label(builder, label)

Install is the only one we care about ..

muddled.pkgs.initscripts.get_effective_env(builder, name, role, domain=None)

Retrieve the effective runtime environment for this initscripts package. Note that setting variables here will have no effect.

muddled.pkgs.initscripts.get_env(builder, name, role, domain=None)

Retrieve an environment to which you can make changes which will be reflected in the generated init scripts. The actual environment used will have extra values inserted from wildcarded environments - see get_effective_env() above.

muddled.pkgs.initscripts.medium(builder, name, roles, script_name, deployments=[], writeSetvarsSh=True, writeSetvarsPy=False)

Build an init script for the given roles.

muddled.pkgs.initscripts.setup_default_env(builder, env_store)

Set up the default environment for this initscript.

muddled.pkgs.initscripts.simple(builder, name, role, script_name, deployments=[], writeSetvarsSh=True, writeSetvarsPy=False)

Build an init script for the given role.

muddled.pkgs.linux_kernel

System Message: WARNING/2 (/home/docs/checkouts/readthedocs.org/user_builds/muddle/checkouts/latest/docs/muddle-package.txt, line 300)

autodoc: failed to import module u’muddled.pkgs.linux_kernel’; the following exception was raised: Traceback (most recent call last): File “/home/docs/checkouts/readthedocs.org/user_builds/muddle/envs/latest/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py”, line 560, in import_object __import__(self.modname) ImportError: No module named linux_kernel

muddled.pkgs.make

Some standard package implementations to cope with packages that use Make

class muddled.pkgs.make.ExpandingMakeBuilder(name, role, co_name, archive_file, archive_dir, makefile='Makefile.muddle')

Bases: muddled.pkgs.make.MakeBuilder

A MakeBuilder that first expands an archive file.

A MakeBuilder that first expands an archive file.

For package ‘name’ in role ‘role’, look in checkout ‘co_name’ for archive ‘archive_file’. Unpack that into $MUDDLE_OBJ, as ‘archive_dir’, with ‘obj/’ linked to it, and use ‘makefile’ to build it.

build_label(builder, label)

Build our label.

Cleverly, Richard didn’t define anything for MakeBuilder to do at the PreConfigure step, which means we can safely do whatever we need to do in this subclass...

unpack_archive(builder, label)
class muddled.pkgs.make.MakeBuilder(name, role, co, config=True, perRoleMakefiles=False, makefileName='Makefile.muddle', rewriteAutoconf=False, usesAutoconf=False, execRelPath=None)

Bases: muddled.pkg.PackageBuilder

Use make to build your package from the given checkout.

We assume that the makefile is smart enough to build in the object directory, since any other strategy (e.g. convolutions involving cp) will lead to dependency-based disaster.

Constructor for the make package.

build_label(builder, label)

Build the relevant label. We’ll assume that the checkout actually exists.

ensure_dirs(builder, label)

Make sure all the relevant directories exist.

muddled.pkgs.make.attach_env(builder, name, role, checkout, domain=None)

Write the environment which attaches MUDDLE_SRC to makefiles.

We retrieve the environment for package:<name>{<role>}/*, and set MUDDLE_SRC therein to the checkout path for ‘checkout:<checkout>’.

muddled.pkgs.make.deduce_makefile_name(makefile_name, per_role, role)

Deduce our actual muddle Makefile name.

‘makefile_name’ is the base name. If it is None, then we use DEFAULT_MAKEFILE_NAME.

If ‘per_role’ is true, and ‘role’ is not None, then we add the extension ‘.<role>’ to the end of the makefile name.

Abstracted here so that it can be used outside this module as well.

muddled.pkgs.make.expanding_package(builder, name, archive_dir, role, co_name, co_dir, makefile='Makefile.muddle', deps=None, archive_file=None, archive_ext='.tar.bz2')

Specify how to expand and build an archive file.

As normal, ‘name’ is the package name, ‘role’ is the role to build it in, ‘co_name’ is the name of the checkout, and ‘co_dir’ is the directory in which that lives.

We expect to unpack an archive

<co_dir>/<co_name>/<archive_dir><archive_ext>

into $(MUDDLE_OBJ)/<archive_dir>. (NB: if the archive file does not expand into a directory of the obvious name, you can specify the archive file name separately, using ‘archive_file’).

So, for instance, all of our X11 “stuff” lives in checkout “X11R7.5” which is put into directory “x11” – i.e., “src/X11/X11R7.5”.

That lets us keep stuff together in the repository, without leading to a great many packages that are of no direct interest to anyone else.

Within that we then have various muddle makefiles, and a set of .tar.bz archive files.

  1. The archive file expands into a directory called ‘archive_dir’
  2. It is assumed that the archive file is named ‘archive_dir’ + ‘archive_ext’. If this is not so, then specify ‘archive_file’ (and/or ‘archive_ext’) appropriately.

This function is used to say: take the named archive file, use package name ‘name’, unpack the archive file into $(MUDDLE_OBJ_OBJ), and build it using the named muddle makefile.

This allows various things to be build with the same makefile, which is useful for (for instance) X11 proto[type] archives.

Note that in $(MUDDLE_OBJ), ‘obj’ (i.e., $(MUDDLE_OBJ_OBJ)) will be a soft link to the expanded archive directory.

muddled.pkgs.make.medium(builder, name, roles, checkout, rev=None, branch=None, deps=None, dep_tag='preconfig', simpleCheckout=True, config=True, perRoleMakefiles=False, makefileName='Makefile.muddle', usesAutoconf=False, rewriteAutoconf=False, execRelPath=None)

Build a package controlled by make, in the given roles with the given dependencies in each role.

  • simpleCheckout - If True, register the checkout as simple checkout too.
  • dep_tag - The tag to depend on being installed before you’ll build.
  • perRoleMakefiles - If True, we run ‘make -f Makefile.<rolename>’ instead of just make.
muddled.pkgs.make.multilevel(builder, name, roles, co_dir=None, co_name=None, rev=None, branch=None, deps=None, dep_tag='preconfig', simpleCheckout=True, config=True, perRoleMakefiles=False, makefileName='Makefile.muddle', repo_relative=None, usesAutoconf=False, rewriteAutoconf=False, execRelPath=None)

Build a package controlled by make, in the given roles with the given dependencies in each role.

  • simpleCheckout - If True, register the checkout as simple checkout too.
  • dep_tag - The tag to depend on being installed before you’ll build.
  • perRoleMakefiles - If True, we run ‘make -f Makefile.<rolename>’ instead of just make.
muddled.pkgs.make.simple(builder, name, role, checkout, rev=None, branch=None, simpleCheckout=False, config=True, perRoleMakefiles=False, makefileName='Makefile.muddle', usesAutoconf=False, rewriteAutoconf=False, execRelPath=None)

Build a package controlled by make, called name with role role from the sources in checkout checkout.

  • simpleCheckout - If True, register the checkout too.
  • config - If True, we have make config. If false, we don’t.
  • perRoleMakefiles - If True, we run ‘make -f Makefile.<rolename>’ instead of just make.
  • usesAutoconf - If True, this package is given access to .la and .pc
    files from things it depends on.
  • rewriteAutoconf - If True, we will rewrite .la and .pc files in the output directory so that packages which use autoconf continue to depend correctly. Intended for use with the MUDDLE_PKGCONFIG_DIRS environment variable.
  • execRelPath - Where, relative to the object directory, do we find
    binaries for this package?
muddled.pkgs.make.single(builder, name, role, deps=None, usesAutoconf=False, rewriteAutoconf=False, execRelPath=None)

A simple make package with a single checkout named after the package and a single role.

muddled.pkgs.make.twolevel(builder, name, roles, co_dir=None, co_name=None, rev=None, branch=None, deps=None, dep_tag='preconfig', simpleCheckout=True, config=True, perRoleMakefiles=False, makefileName='Makefile.muddle', repo_relative=None, usesAutoconf=False, rewriteAutoconf=False, execRelPath=None)

Build a package controlled by make, in the given roles with the given dependencies in each role.

  • simpleCheckout - If True, register the checkout as simple checkout too.
  • dep_tag - The tag to depend on being installed before you’ll build.
  • perRoleMakefiles - If True, we run ‘make -f Makefile.<rolename>’ instead of just make.

muddled.pkgs.version

Write a version.xml file containing version information for the current build

class muddled.pkgs.version.VersionBuilder(name, role, filename, swname=None, version=None, build=None, withDate=True, withUser=True, withMachine=True)

Bases: muddled.pkg.PackageBuilder

Write a version number file

Constructor for the version package type

build_label(builder, label)

Build the version.xml file.

dir_name(builder)
erase_version_file(builder)

Erase the version file.

file_name(builder)
write_elem(f, elem, val)
write_version_file(builder)

Write the version file

muddled.pkgs.version.simple(builder, name, roles, filename='/version.xml', swname=None, version=None, build=None, withDate=True, withUser=True, withMachine=True)

muddled.resources

Not Python files:

  • c_env.c - boilerplate for accessing environments from C.
  • initscript.sh - a generic init script

muddled.vcs

muddled.vcs.bazaar

Muddle support for Bazaar.

Note that Bazaar does not support “branches” in the muddle sense. Bazaar itself uses the “bzr branch” command to make a clone of a repository. It does not (or did not at time of writing) support lightweight branching in the manner of git - i.e., separate branches stored within the same clone. Thus the “branch” argument of a Repository class is not supported for Bazaar.

Available Bazaar specific options are:

  • no_follow: In a build description that has set:

    builder.follow_build_desc_branch = True
    

    then a Bazaar repository can either:

    1. Specify a particular revision
    2. Choose a different repository location (presumably a bazaar “branch”), and set the no_follow option to True
    3. Continue using the original repository without setting a revision (almost certainly not sensible, but still), and set the no_follow option to True.

    If none of these are done, then muddle will fail with a complaint like:

    The build description wants checkouts to follow branch '<branch-name>',
    but checkout <co-name> uses VCS Bazaar for which we do not support branching.
    The build description should specify a revision for checkout <co-name>.
    
class muddled.vcs.bazaar.Bazaar

Bases: muddled.version_control.VersionControlSystem

Provide version control operations for Bazaar

add_files(files=None, verbose=True)

If files are given, add them, but do not commit.

Will be called in the actual checkout’s directory.

allows_relative_in_repo()
checkout(repo, co_leaf, options, verbose=True)

Checkout (clone) a given checkout.

Will be called in the parent directory of the checkout.

Expected to create a directory called <co_leaf> therein.

commit(repo, options, verbose=True)

Will be called in the actual checkout’s directory.

get_file_content(url, verbose=True)

Retrieve a file’s content via BZR.

get_vcs_special_files()
goto_revision(revision, branch=None, repo=None, verbose=False)

Make the specified revision current.

Note that this may leave the working data (the actual checkout directory) in an odd state, in which it is not sensible to commit, depending on the VCS and the revision.

Will be called in the actual checkout’s directory.

Raises GiveUp if there is no such revision, or if a branch is given.

init_directory(verbose=True)

If the directory does not appear to have had ‘<vcs> init’ run in it, then do so first.

Will be called in the actual checkout’s directory.

merge(other_repo, options, verbose=True)

Merge ‘other_repo’ into the local repository and working tree,

‘bzr merge’ will not (by default) merge if there are uncommitted changes in the destination (i.e., local) tree. This is what we want.

Will be called in the actual checkout’s directory.

pull(repo, options, upstream=None, verbose=True)

Pull changes, but don’t do a merge.

Will be called in the actual checkout’s directory.

push(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

reparent(co_leaf, remote_repo, options, force=False, verbose=True)

Re-associate the local repository with its original remote repository,

  • ‘co_leaf’ should be the name of this checkout directory, for use in messages reporting what we are doing. Note that we are called already in that directory, though.
  • ‘remote_repo’ is the repository we would like to associate it with.

bzr info is your friend for finding out if the checkout is already associated with a remote repository. The “parent branch” is used for pulling and merging (and is what we set). If present, the “push branch” is used for pushing.

If force is true, we set “parent branch”, and delete “push branch” (so it will default to the “parent branch”).

If force is false, we only set “parent branch”, and then only if it is not set.

The actual information is held in <checkout-dir>/.bzr/branch/branch.conf, which is a .INI file.

revision_to_checkout(repo, co_leaf, options, force=False, before=None, verbose=True)

Determine a revision id for this checkout, usable to check it out again.

  • ‘co_leaf’ should be the name of this checkout directory, for use in messages reporting what we are doing. Note that we are called already in that directory, though.

If ‘force’ is true, then if we can’t get one from bzr, and it seems “reasonable” to do so, use the original revision from the muddle depend file (if it is not HEAD).

If ‘before’ is given, it should be a string describing a date/time, and the revision id chosen will be the last revision at or before that date/time.

Note

This depends upon what the VCS concerned actually supports. This feature is experimental. XXX NOT YET IMPLEMENTED XXX

‘bzr revno’ always returns a simple integer (or so I believe)

‘bzr version-info’ returns several lines, including:

revision-id: <something>
revno: <xxx>

where <xxx> is the same number as ‘bzr revno’, and <something> will be different depending on whether we’re “the same” as the far repository.

If the –check-clean flag is used, then there will also be a line of the form:

clean: True

indicating whether the source tree contains uncommitted changes (although not whether it is matching the far repository).

So ideally we would (1) grumble if not clean, and (2) grumble if our revision id was different than after the last push/pull/checkout

Well, ‘bzr missing’ should show unmerged/unpulled revisions between two branches, so if it ends “Branches are up to date” then that may be useful. Or no output with ‘-q’ if they’re OK. (needs to ignore stderr output, since I get that for mismatch in Bazaar network protocols)

status(repo, options=None, branch=None, verbose=False, quick=False)

Will be called in the actual checkout’s directory.

Return status text or None if there is no interesting status.

muddled.vcs.file

Muddle support for naive file copying.

class muddled.vcs.file.File

Bases: muddled.version_control.VersionControlSystem

Provide version control operations for simple file copying

add_files(files=None, verbose=True)

If files are given, add them, but do not commit.

Will be called in the actual checkout’s directory.

allows_relative_in_repo()
checkout(repo, co_leaf, options, verbose=True)

Clone a given checkout.

Will be called in the parent directory of the checkout.

Expected to create a directory called <co_leaf> therein.

commit(repo, options, verbose=True, quick=False)

Will be called in the actual checkout’s directory.

get_file_content(url, verbose=True)

Retrieve a file’s content via Subversion.

init_directory(verbose=True)

If the directory does not appear to have had ‘<vcs> init’ run in it, then do so first.

Will be called in the actual checkout’s directory.

merge(other_repo, options, verbose=True)

Merge ‘other_repo’ into the local repository and working tree,

Just copies everything again. This is an imperfect sort of “merge”.

pull(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

Just copies everything again.

push(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

We refuse to copy anything back to the original directory.

reparent(co_dir, remote_repo, options, force=False, verbose=True)
revision_to_checkout(repo, co_leaf, options, force=False, before=None, verbose=False)

Determine a revision id for this checkout, usable to check it out again.

status(repo, options, branch=None, quick=None)

Status is not supported for ‘file’.

Just report ‘nothing important has happened’.

muddled.vcs.git

Muddle suppport for Git.

TODO: The following needs rewriting after work for issue 225

  • muddle checkout

    Clones the appropriate checkout with git clone.

    If the build description (by whatever means) requires a branch, then git clone -b <branch> is used. If no branch is specified, we default to git clone -b master.

    If a shallow checkout is selected, then the --depth 1 switch is added to the clone command. Note that there are some restrictions on what can be done with a shallow checkout - git says:

    A shallow repository has a number of limitations (*you cannot clone or
    fetch from it, nor push from nor into it*), but is adequate if you are
    only interested in the recent history of a large project with a long
    history, and would want to send in fixes as patches.
    

    (the emphasis is mine).

    If a revision is requested, then git checkout is used to check it out.

    If a branch and a revision are requested, then muddle checks to see if cloning the branch gave the correct revision, and only does the git checkout if it did not. This avoids unnecessary detached HEADs,

    Note: the checking of the revision id is very simple, and assumes that the revision is specified as a full SHA1 string (it is compared with the output of git rev-parse HEAD).

  • muddle pull, muddle pull-upstream

    This does a git fetch followed by a fast-forwards git merge.

    If the build description specifies a particular revision, and the checkout is already at that revision, then an error will be reported, saying that.

    If there are any local changes uncommitted, or untracked files, then appropriate error messages will also be reported.

    If the build description specified a branch for this repository (by whatever means), then we will first go to that branch. If no branch was specified, we first go to “master”.

    The command checks that the remote is configured as such, then does git fetch. If a revision was specified, it then checks out that revision, otherwise it does git merge --ff-only, which will merge in the fetch if it doesn’t require human interaction.

  • muddle push, muddle push-upstream

    If the checkout is marked as “shallow’, or is on a detached HEAD, then an appropriate error message will be given.

    The command checks that the remote is configured as such, then does git push of the current branch. If the branch does not exist at the far end, it will be created.

  • muddle merge

    This is identical to “muddle pull”, except that instead of doing a fast-forward merge, it does a simple git merge, allowing human interaction if necessary.

  • muddle commit

    Simply runs git commit -a.

  • muddle status

    This first does git status --porcelain. If that does not return anything, it then determines the SHA1 for the local HEAD, and the SHA1 for the equivalent HEAD in the remote repository. If these are different (so presumably the remote repository is ahead of the local one), then it reports as much,

    Note that a normal git status does not talk to the remote repository, and is thus fast. If this command does talk over the network, it can be rather slower.

  • muddle reparent

    Re-associates the local repository’s “origin” with the remote repository indicated by the build description. It tries to detect if this is necessary first.

Available git specific options are:

  • shallow_checkout: If True, then only clone to a depth of 1 (i.e., pass the git switch “–depth 1”). If False, then no effect. The default is False.

    This is typically of use when cloning the Linux kernel (or some other large tree with a great deal of history), when one is not expecting to modify the checkout in any way in the future (i.e., neither to push it nor to pull it again).

    If ‘shallow_checkout’ is specified, then “muddle push” will refuse to do anything.

class muddled.vcs.git.Git

Bases: muddled.version_control.VersionControlSystem

Provide version control operations for Git

add_files(files=None, verbose=True)

If files are given, add them, but do not commit.

Will be called in the actual checkout’s directory.

allows_relative_in_repo()

TODO: Check that this is correct!

branch_exists(branch)

Is there a branch of this name?

Will be called in the actual checkout’s directory.

checkout(repo, co_leaf, options, verbose=True)

Clone a given checkout.

Will be called in the parent directory of the checkout.

Expected to create a directory called <co_leaf> therein.

commit(repo, options, verbose=True)

Will be called in the actual checkout’s directory.

Does ‘git commit -a’ - i.e., this implicitly does ‘git add’ for you. This is a contentious choice, and needs review.

create_branch(branch, verbose=False)

Create a branch of the given name.

Will be called in the actual checkout’s directory.

Also sets up the equivalent remote.

It is an error if the branch already exists, in which case a GiveUp exception will be raised.

get_current_branch()

Return the name of the current branch.

Will be called in the actual checkout’s directory.

Returns None if we are not on a branch (e.g., a detached HEAD)

get_vcs_special_files()
goto_branch(branch, verbose=False)

Make the named branch the current branch.

Will be called in the actual checkout’s directory.

It is an error if the branch does not exist, in which case a GiveUp exception will be raised.

goto_revision(revision, branch=None, repo=None, verbose=False)

Make the specified revision current.

Note that this may leave the working data (the actual checkout directory) in an odd state, in which it is not sensible to commit, depending on the VCS and the revision.

Will be called in the actual checkout’s directory.

If a branch name is given, we will go to that branch first, and see if we already got to the correct revision. Note that the check for this assumes that ‘revision’ is a full SHA1, so is a bit simplistic. If we don’t appear to be at the required revision, we’ll then go there as normal.

Raises GiveUp if there is no such revision, or no such branch.

init_directory(verbose=True)

If the directory does not appear to have had ‘<vcs> init’ run in it, then do so first.

Will be called in the actual checkout’s directory.

merge(repo, options, verbose=True)

Merge changes from ‘repo’ into the local repository and working tree.

Will be called in the actual checkout’s directory.

Broadly, does a ‘git fetch’ followed by a merge. Be aware that this last may require user interaction.

If a fast-forward merge is possible, then this identical to doing a “muddle pull”.

If the build description specifies a particular revision, then if it was already at that revision, nothing needs doing. Otherwise, the ‘git fetch’ is done and then the specified revision is checked out using ‘git checkout’.

pull(repo, options, upstream=None, verbose=True)

Pull changes from ‘repo’ into the local repository and working tree.

Will be called in the actual checkout’s directory.

Broadly, does a ‘git fetch’ followed by a fast-forward merge - so it will only merge if it is obvious how to do it.

If the build description specifies a particular revision, then if it was already at that revision, nothing needs doing. Otherwise, the ‘git fetch’ is done and then the specified revision is checked out using ‘git checkout’.

push(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

XXX Should we grumble if the ‘effective’ branch is not the same as XXX the branch that is currently checked out?

reparent(co_dir, remote_repo, options, force=False, verbose=True)

Re-associate the local repository with its original remote repository,

revision_to_checkout(repo, co_leaf, options, force=False, before=None, verbose=True)

Determine a revision id for this checkout, usable to check it out again.

  1. Document
  2. Review the code to see if we now support versions of git that would allow us to do this more sensibly

If ‘before’ is given, it should be a string describing a date/time, and the revision id chosen will be the last revision at or before that date/time.

NB: if ‘before’ is specified, ‘force’ is ignored.

XXX TODO: Needs reviewing given we’re using a later version of git now

status(repo, options, quick=False)

Will be called in the actual checkout’s directory.

Return status text or None if there is no interesting status.

supports_branching()
muddled.vcs.git.expand_revision(revision)

Given something that names a revision, return its full SHA1.

Raises GiveUp if the revision appears non-existent or ambiguous

muddled.vcs.git.git_supports_ff_only()

Does my git support –ff-only?

muddled.vcs.svn

Muddle support for Subversion

Note that Subversion does not support “branches” in the muddle sense. Subversion branches are handled by a different mechanism, and in a muddle sense are more like inner paths of a Repository. At the moment muddle does not provide any special support for Subversion branches.

Available Subversion specific options are:

  • no_follow: In a build description that has set:

    builder.follow_build_desc_branch = True
    

    then a Subversion repository can either:

    1. Specify a particular revision
    2. Choose a different repository location (presumably a subversion “branch”), and set the no_follow option to True
    3. Continue using the original repository without setting a revision (almost certainly not sensible, but still), and set the no_follow option to True.

    If none of these are done, then muddle will fail with a complaint like:

    The build description wants checkouts to follow branch '<branch-name>',
    but checkout <co-name> uses VCS Subversion for which we do not support branching.
    The build description should specify a revision for checkout <co-name>.
    
class muddled.vcs.svn.Subversion

Bases: muddled.version_control.VersionControlSystem

Provide version control operations for Subversion

add_files(files=None, verbose=True)

If files are given, add them, but do not commit.

Will be called in the actual checkout’s directory.

allows_relative_in_repo()
checkout(repo, co_leaf, options, verbose=True)

Clone a given checkout.

Will be called in the parent directory of the checkout.

Expected to create a directory called <co_leaf> therein.

commit(repo, options, verbose=True)

Will be called in the actual checkout’s directory.

This command does nothing, because Subversion does not have a local repository. Use ‘muddle push’ instead.

get_file_content(url, verbose=True)

Retrieve a file’s content via Subversion.

get_vcs_special_files()
goto_revision(revision, branch=None, repo=None, verbose=False)

Make the specified revision current.

Note that this may leave the working data (the actual checkout directory) in an odd state, in which it is not sensible to commit, depending on the VCS and the revision.

Will be called in the actual checkout’s directory.

Raises GiveUp if there is no such revision, or if branch is given.

init_directory(verbose=True)

If the directory does not appear to have had ‘<vcs> init’ run in it, then do so first.

Will be called in the actual checkout’s directory.

merge(other_repo, options, verbose=True)

Merge ‘other_repo’ into the local repository and working tree,

This runs Subversion’s “update” - its “merge” command does something different.

Will be called in the actual checkout’s directory.

must_pull_before_commit(options)

Subversion recommends doing ‘commit’ before “pull” (i.e., pull/update)

pull(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

This runs Subversion’s “update”, but only if no merging will be needed. That is, first it runs “svn status”, and if any lines contain a “C” (for Conflict) in columns 0, 1 or 6, then it will not perform the update.

(“svn help status” would call those columns 1, 2 and 7)
push(repo, options, upstream=None, verbose=True)

Will be called in the actual checkout’s directory.

This actually does a “svn commit”, i.e., committing to the remote repository (which is the only one subversion has).

reparent(co_dir, remote_repo, options, force=False, verbose=True)

Re-associate the local repository with its original remote repository,

Our subversion support does not provide this.

revision_to_checkout(repo, co_leaf, options, force=False, before=None, verbose=False)

Determine a revision id for this checkout, usable to check it out again.

Uses ‘svnversion’, which I believe is installed as standard with ‘svn’

For the moment, at lease, the ‘force’ argument is ignored (so the working copy must be be equivalent to the repository).

status(repo, options, quick=False)

Will be called in the actual checkout’s directory.

Return status text or None if there is no interesting status.