This chapter is the official repository for the collected wisdom of the
Buildbot hackers.

It contains some sparse documentation of the inner workings of Buildbot, but of
course, the final reference for that is the source itself.

More importantly, this chapter represents the official repository of all
agreed-on patterns for use in Buildbot.  In this case, the source is a
@i{terrible} reference, because much of it is old and crusty.  But we are
trying to do things the new, better way, and those new, better ways are
described here.

@menu
* Buildmaster Service Hierarchy::
* Utilities::
* The Database::
* Log File Format::
* Web Status::
* Obfuscating Passwords::
* Master-Slave API::
* Twisted Idioms::
* Buildbot Tests::
* Developer Class Index::
@end menu

@node Buildmaster Service Hierarchy
@section Buildmaster Service Hierarchy

Buildbot uses Twisted's service hierarchy heavily.  The hierarchy looks like
this:

@itemize

@item BuildMaster
@dvindex buildbot.master.BuildMaster

This is the top-level service.

@itemize

@item BotMaster
@dvindex buildbot.master.BotMaster

The BotMaster manages all of the slaves.  BuildSlave instances are added as
child services of the BotMaster.

@item ChangeManager
@dvindex buildbot.changes.manager.ChangeManager

The ChangeManager manages the active change sources, as well as the stream of
changes received from those sources.

@item SchedulerManager
@dvindex buildbot.schedulers.manager.SchedulerManager

The SchedulerManager manages the active schedulers and handles inter-scheduler
notifications.

@item IStatusReceiver implementations

Objects from the 'status' configuration key are attached directly to the
buildmaster. These classes should inherit from StatusReceiver or
StatusReceiverMultiService and include an 'implements(IStatusReceiver)' stanza.

@end itemize

@end itemize

@node Utilities
@section Utilities

@menu
* buildbot.util.collections::
* buildbot.util.eventual::
* buildbot.util.json::
@end menu

Several small utilities are available at the top-level @code{buildbot.util}
package.  As always, see the API documentation for more information.

@table @code

@item natualSort

This function sorts strings "naturally", with embedded numbers sorted
numerically.  This ordering is good for objects which might have a numeric
suffix, e.g., @code{winslave1}, @code{winslave2}, ..

@item formatInterval

This function will return a human-readable string describing a length of time,
given a number of seconds.

@item ComparableMixin

This mixin class adds comparability to a subclass.  Use it like this:

@example
class Widget(FactoryProduct, ComparableMixin):
    compare_attrs = [ 'radius', 'thickness' ]
    # ...
@end example

Any attributes not in @code{compare_attrs} will not be considered when
comparing objects.  This is particularly useful in implementing buildbot's
reconfig logic, where a simple comparison between the new and existing objects
can determine whether the new object should replace the existing object.

@item safeTranslate

This function will filter out some inappropriate characters for filenames; it
is suitable for adapting strings from the configuration for use as filenames.
It is not suitable for use with strings from untrusted sources.

@item LRUCache

This is a simple least-recently-used cache.  Its constructor takes a maximum
size.  When the cache grows beyond this size, the least-recently used items
will be automatically removed from the cache.  The class has @code{get} and
@code{add} methods, and can also be accessed via dictionary syntax
(@code{lru['id']}).

@end table

@node buildbot.util.collections
@subsection buildbot.util.collections

This package provides a few useful collection objects.

For compatibility, it provides a clone of the Python
@code{collections.defaultdict} for use in Python-2.4.  In later versions, this
is simply a reference to the built-in @code{defaultdict}, so buildbot code can
simply use @code{buildbot.util.collections.defaultdict} everywhere.

It also provides a @code{KeyedSets} class that can represent any numbers of
sets, keyed by name (or anything hashable, really).  The object is specially
tuned to contain many different keys over its lifetime without wasting memory.
See the docstring for more information.

@node buildbot.util.eventual
@subsection buildbot.util.eventual

This package provides a simple way to say "please do this later":

@example
from buildbot.util.eventual import eventually
def do_what_I_say(what, where):
    # ...
eventually(do_what_I_say, "clean up", "your bedroom")
@end example

The package defines "later" as "next time the reactor has control", so this is
a good way to avoid long loops that block other activity in the reactor.
Callables given to @code{eventually} are guaranteed to be called in the same
order as the calls to @code{eventually}.  Any errors from the callable are
logged, but will not affect other callables.

If you need a deferred that will fire "later", use @code{fireEventually}.  This
function returns a deferred that will not errback.

@node buildbot.util.json
@subsection buildbot.util.json

This package is just an import of the best available JSON module.  Use it
instead of a more complex conditional import of @code{simplejson} or
@code{json}.

@node The Database
@section The Database

@dvindex buildbot.db.dbspec.DBSpec
@dvindex buildbot.db.connector.DBConnector

Buildbot stores its state in a database, using the Python DBAPI to access it.

A database is specified with the @code{buildbot.db.dbspec.DBSpec} class, which
encapsulates all of the parameters necessary to create a DBAPI connection.  The
DBSpec class can create either a single synchronous connection, or a twisted
adbapi connection pool.

Most uses of the database in Buildbot are done through the
@code{buildbot.db.connector.DBConnector} class, which wraps the DBAPI to
provide purpose-specific functions.

The database schema is managed by a special class, described in the next
section.

@menu
* Database Schema::
* Changing the Schema::
@end menu

@node Database Schema
@subsection Database Schema

@dvindex buildbot.db.schema.DBSchemaManager

The SQL for the database schema is available in
@code{buildbot/db/schema/tables.sql}.  However, note that this file is not used
for new installations or upgrades of the Buildbot database.

Instead, the @code{buildbot.db.schema.DBSchemaManager} handles this task.  The
operation of this class centers around a linear sequence of database versions.
Versions start at 0, which is the old pickle-file format.  The manager has
methods to query the version of the database, and the current version from the
source code.  It also has an @code{upgrade} method which will upgrade the
database to the latest version.  This operation is currently irreversible.

There is no operation to "install" the latest schema.  Instead, a fresh install
of buildbot begins with an (empty) version-0 database, and upgrades to the
current version.  This trades a bit of efficiency at install time for
assurances that the upgrade code is well-tested.

@node Changing the Schema
@subsection Changing the Schema

To make a change to the database schema, follow these steps:

@enumerate

@item

Increment @code{CURRENT_VERSION} in @code{buildbot/db/schema/manager.py} by
one.  This is your new version number.

@item

Create @file{buildbot/db/schema/vN.py}, where N is your version number, by
copying the previous script and stripping it down.  This script should define a
subclass of @code{buildbot.db.schema.base.Updater} named @code{Updater}. 

The class must define the method @code{upgrade}, which takes no arguments.  It
should upgrade the database from the previous version to your version,
including incrementing the number in the @code{VERSION} table, probably with an
@code{UPDATE} query.

Consult the API documentation for the base class for information on the
attributes that are available.

@item

Edit @file{buildbot/test/unit/test_db_schema_master.py}.  If your upgrade
involves moving data from the basedir into the database proper, then edit
@code{fill_basedir} to add some test data.

Add code to @code{assertDatabaseOKEmpty} to check that your upgrade works on an
empty database.

Add code to @code{assertDatabaseOKFull} to check that your upgrade works on a
database with pre-existing data.  Do this even if your changes do not move any
data from the basedir.

Run the tests to find the bugs you introduced in step 2.

@item

Increment the version number in the @code{test_get_current_version} test in the
same file.  Only do this after you've finished the previous step - a failure of
this test is a good reminder that testing isn't done yet.

@item

Updated the version number in @file{buildbot/db/schema/tables.sql}, too.

@item

Finally, make the corresponding changes to @file{buildbot/db/schema/tables.sql}.

@end enumerate

@node Log File Format
@section Log File Format
@dvindex buildbot.status.builder.LogFile

The master currently stores each logfile in a single file, which may have a
standard compression applied.

The format is a special case of the netstrings protocol - see
@url{http://cr.yp.to/proto/netstrings.txt}.  The text in each netstring
consists of a one-digit channel identifier followed by the data from that
channel.

The formatting is implemented in the LogFile class in
@file{buildbot/status/builder.py}, and in particular by the @code{merge}
method.

@node Web Status
@section Web Status

@menu
* Jinja Web Templates::
* Web Authorization Framework::
@end menu

@node Jinja Web Templates
@subsection Jinja Web Templates

Buildbot uses Jinja2 to render its web interface.  The authoritative source for
this templating engine is @uref{http://jinja.pocoo.org/2/documentation/, its
own documentation}, of course, but a few notes are in order for those who are
making only minor modifications.

@heading Whitespace

Jinja directives are enclosed in @code{@{% .. %@}}, and sometimes also have
dashes.  These dashes strip whitespace in the output.  For example:

@example
@{% for entry in entries %@}
  <li>@{@{ entry @}@}</li>
@{% endfor %@}
@end example

will produce output with too much whitespace:

@example

  <li>pigs</li>


  <li>cows</li>

@end example

But adding the dashes will collapse that whitespace completely:

@example
@{% for entry in entries -%@}
  <li>@{@{ entry @}@}</li>
@{%- endfor %@}
@end example

yields

@example
<li>pigs</li><li>cows</li>
@end example

@node Web Authorization Framework
@subsection Web Authorization Framework

Whenever any part of the web framework wants to perform some action on the
buildmaster, it should check the user's authorization first.

Always check authorization twice: once to decide whether to show the option to
the user (link, button, form, whatever); and once before actually performing
the action.

To check whether to display the option, you'll usually want to pass an authz
object to the Jinja template in your HtmlResource subclass:

@example
    def content(self, req, cxt):
        # ...
        cxt['authz'] = self.getAuthz(req)
        template = ...
        return template.render(**cxt)
@end example

and then determine whether to advertise the action in th template:

@example
@{@{ if authz.advertiseAction('myNewTrick') @}@}
  <form action="@{@{ myNewTrick_url @}@}"> ...
@{@{ endif @}@}
@end example

Actions can optionally require authentication, so use @code{needAuthForm} to
determine whether to require a 'username' and 'passwd' field in the generated
form.  These fields are usually generated by the @code{auth()} form:

@example
@{% if authz.needAuthForm('myNewTrick') %@}
  @{@{ auth() @}@}
@{% endif %@}
@end example

Once the POST request comes in, it's time to check authorization again.
This usually looks something like

@example
if not self.getAuthz(req).actionAllowed('myNewTrick', req, someExtraArg):
    return Redirect(path_to_authfail(req))
@end example

The @code{someExtraArg} is optional (it's handled with @code{*args}, so you can
have several if you want), and is given to the user's authorization function.
For example, a build-related action should pass the build status, so that the
user's authorization function could ensure that devs can only operate on their
own builds.

The available actions are listed in @pxref{WebStatus Configuration Parameters}.

@node Obfuscating Passwords
@section Obfuscating Passwords
@dvindex buildslave.util.Obfuscated

It's often necessary to pass passwords to commands on the slave, but it's no
fun to see those passwords appear for everyone else in the build log.  The
@code{Obfuscated} class can help here.  Instantiate it with a real string and a
fake string that should appear in logfiles.  You can then use the
@code{Obfuscated.get_real} and @code{Obfuscated.get_fake} static methods to
convert a list of command words to the real or fake equivalent.

The @code{RunProcess} implementation in the buildslave will apply these methods
automatically, so just feed it a list of strings and @code{Obfuscated} objects.

@node Master-Slave API
@section Master-Slave API

This section is a (very incomplete) description of the master-slave interface.
The interface is based on Twisted's Perspective Broker.

@heading Connection

The slave connects to the master, using the parameters supplied to
@code{buildslave create-slave}.  It uses a reconnecting process with an
exponential backoff, and will automatically reconnect on disconnection.

@dvindex buildslave.bot.Bot

Once connected, the slave authenticates with the Twisted Cred (newcred)
mechanism, using the username and password supplied to @code{buildslave
create-slave}.  The "mind" is the slave bot instance (class
@code{buildslave.bot.Bot}).

@dvindex buildbot.master.Dispatcher
@dvindex buildbot.buildslave.BuildSlave

On the master side, the realm is implemented by
@code{buildbot.master.Dispatcher}, which examines the username of incoming
avatar requests.  There are special cases for @code{change}, @code{debug}, and
@code{statusClient}, which are not discussed here.  For all other usernames,
the botmaster is consulted, and if a slave with that name is configured, its
@code{buildbot.buildslave.BuildSlave} instance is returned as the perspective.

@heading Build Slaves

At this point, the master-side BuildSlave object has a pointer to the remote,
slave-side Bot object in @code{self.slave}, and the slave-side Bot object has a
reference to the master-side BuildSlave object in @code{self.perspective}.

@subheading Bot methods

The slave-side object has the following remote methods:

@table @code
@item remote_getCommands
Returns a list of @code{(name, version)} for all commands the slave recognizes

@item remote_setBuilderList
Given a list of builders and their build directories, ensures that those
builders, and only those builders, are running.  This can be called after the
initial connection is established, with a new list, to add or remove builders.

This method returns a dictionary of @code{SlaveBuilder} objects - see below

@item remote_print
Adds a message to the slave logfile

@item remote_getSlaveInfo
Returns the contents of the slave's @file{info/} directory

@item remote_getVersion
Returns the slave's version

@end table

@subheading BuildSlave methods

The master-side object has the following method:

@table @code
@item perspective_keepalive
Does nothing - used to keep traffic flowing over the TCP connection

@end table

@heading Slave Builders

@dvindex buildslave.bot.SlaveBuilder
@dvindex buildbot.process.builder.Builder

Each build slave has a set of builders which can run on it.  These are represented
by distinct classes on the master and slave, just like the BuildSlave and Bot objects
described above.

On the slave side, builders are represented as instances of the
@code{buildslave.bot.SlaveBuilder} class.  On the master side, they are
represented by the @code{buildbot.process.builder.SlaveBuilder} class.  The
following will refer to these as the slave-side and master-side SlaveBuilder
classes.  Each object keeps a reference to its opposite in @code{self.remote}.

@subheading slave-side SlaveBuilder methods

@table @code
@item remote_setMaster
Provides a reference to the master-side SlaveBuilder

@item remote_print
Adds a message to the slave logfile; used to check round-trip connectivity

@item remote_startBuild
Indicates that a build is about to start, and that any subsequent
commands are part of that build

@item remote_startCommand
Invokes a command on the slave side

@item remote_interruptCommand
Interrupts the currently-running command

@item remote_shutdown
Shuts down the slave cleanly

@end table

@subheading master-side SlaveBuilder methods

The master side does not have any remotely-callable methods.

@heading Setup

After the initial connection and trading of a mind (Bot) for an avatar
(BuildSlave), the master calls the Bot's @code{setBuilderList} method to set up
the proper slave builders on the slave side.  This method returns a reference to
each of the new slave-side SlaveBuilder objects.  Each of these is handed to the
corresponding master-side SlaveBuilder object.  This immediately calls the remote
@code{setMaster} method, then the @code{print} method.

@heading Pinging

To ping a remote SlaveBuilder, the master calls the @code{print} method.

@heading Building

When a build starts, the msater calls the slave's @code{startBuild} method.
Each BuildStep instance will subsequently call the @code{startCommand} method,
passing a reference to itself as the @code{stepRef} parameter.  The
@code{startCommand} method returns immediately, and the end of the command is
signalled with a call to a method on the master-side BuildStep object.

@subheading master-side BuildStep methods

@table @code
@item remote_update
Update information about the running command.  See below for the format.

@item remote_complete
Signal that the command is complete, either successfully or with a Twisted failure.

@end table

Updates from the slave are a list of individual update elements.  Each update
element is, in turn, a list of the form @code{[data, 0]} where the 0 is present
for historical reasons.  The data is a dictionary, with keys describing the
contents, e.g., @code{header}, @code{stdout}, or the name of a logfile.  If the
key is @code{rc}, then the value is the exit status of the command.  No further
updates should be sent after an @code{rc}.

@node Twisted Idioms
@section Twisted Idioms

@menu
* Helpful Twisted Classes::
@end menu

@node Helpful Twisted Classes
@subsection Helpful Twisted Classes

Twisted has some useful, but little-known classes.  They are listed here with
brief descriptions, but you should consult the API documentation or source code
for the full details.

@itemize

@item twisted.internet.task.LoopingCall

Calls an asynchronous function repeatedly at set intervals.

@item twisted.application.internet.TimerService

Similar to @code{t.i.t.LoopingCall}, but implemented as a service that will
automatically start an dstop the function calls when the service is started and
stopped.

@end itemize

@node Buildbot Tests
@section Buildbot Tests

@menu
* Toward Better Buildbot Tests::
* Keeping State in Tests::
* Better Debugging through Monkeypatching::
@end menu

@node Toward Better Buildbot Tests
@subsection Toward Better Buildbot Tests

In general, we are trying to ensure that new tests are @i{good}.  So what makes a good test?

@heading Independent of Time

Tests that depend on wall time will fail. As a bonus, they run very slowly. Do
not use @code{reactor.callLater} to wait "long enough" for something to happen.

For testing things that themselves depend on time, consider using
@code{twisted.internet.tasks.Clock}.  This may mean passing a clock instance to
the code under test, and propagating that instance as necessary to ensure that
all of the code using @code{callLater} uses it.  Refactoring code for
testability is difficult, but wortwhile.

For testing things that do not depend on time, but for which you cannot detect
the "end" of an operation: add a way to detect the end of the operation!

@heading Clean Code

Make your tests readable. This is no place to skimp on comments! Others will
attempt to learn about the expected behavior of your class by reading the
tests. As a side note, if you use a @code{Deferred} chain in your test, write
the callbacks as nested functions, rather than using object methods with funny
names:

@example
def testSomething(self):
  d = doThisFirst()
  def andThisNext(res):
    pass # ...
  d.addCallback(andThisNext)
  return d
@end example

This isolates the entire test into one indented block. It is OK to add methods
for common functionality, but give them real names and explain in detail what
they do.

@heading Good Name

Your test module should be named after the package or class it tests, replacing
@code{.} with @code{_} and omitting the buildbot_. For example,
@code{test_status_web_authz_Authz.py} tests the Authz class in
@code{buildbot/status/web/authz.py}. Modules with only one class, or a few
trivial classes, can be tested in a single test module. For more complex
situations, prefer to use multiple test modules.

Test method names should follow the pattern test_METHOD_CONDITION where METHOD
is the method being tested, and CONDITION is the condition under which it's
tested. Since we can't always test a single method, this is not a hard-and-fast
rule.

@heading Assert Only One Thing

Each test should have a single assertion. This may require a little bit of work
to get several related pieces of information into a single Python object for
comparison. The problem with multiple assertions is that, if the first
assertion fails, the remainder are not tested.  The test results then do not
tell the entire story.

If you need to make two unrelated assertions, you should be running two tests.

@heading Use Mocks and Stubs

Mocks assert that they are called correctly. Stubs provide a predictable base
on which to run the code under test. See
@url{http://en.wikipedia.org/wiki/Mock_object,Mock Object} and
@url{http://en.wikipedia.org/wiki/Method_stub,Method Stub}.

Mock objects can be constructed easily using the aptly-named
@url{http://www.voidspace.org.uk/python/mock/,mock} module, which is a
requirement for Buildbot's tests.

One of the difficulties with Buildbot is that interfaces are unstable and
poorly documented, which makes it difficult to design stubs.  A common
repository for stubs, however, will allow any interface changes to be reflected
in only one place in the test code.

@heading Small Tests

The shorter each test is, the better. Test as little code as possible in each test.

It is fine, and in fact encouraged, to write the code under test in such a way
as to facilitate this. As an illustrative example, if you are testing a new
Step subclass, but your tests require instantiating a BuildMaster, you're
probably doing something wrong! (Note that this rule is almost universally
violated in the existing buildbot tests).

This also applies to test modules.  Several short, easily-digested test modules
are preferred over a 1000-line monster.

@heading Isolation

Each test should be maximally independent of other tests. Do not leave files
laying around after your test has finished, and do not assume that some other
test has run beforehand. It's fine to use caching techniques to avoid repeated,
lengthy setup times.

@heading Be Correct

Tests should be as robust as possible, which at a basic level means using the
available frameworks correctly. All deferreds should have callbacks and be
chained properly. Error conditions should be checked properly. Race conditions
should not exist (see "Independent of Time", above).

@heading Be Helpful

Note that tests will pass most of the time, but the moment when they are most
useful is when they fail.

When the test fails, it should produce output that is helpful to the person
chasing it down. This is particularly important when the tests are run
remotely, in which case the person chasing down the bug does not have access to
the system on which the test fails. A test which fails sporadically with no
more information than "AssertionFailed?" is a prime candidate for deletion if
the error isn't obvious. Making the error obvious also includes adding comments
describing the ways a test might fail.

@heading Mixins

Do not define setUp and tearDown directly in a mixin. This is the path to
madness. Instead, define a @code{myMixinNameSetUp} and
@code{myMixinNameTearDown}, and call them explicitlyi from the subclass's
@code{setUp} and @code{tearDown}. This makes it perfectly clear what is being
set up and torn down from a simple analysis of the test case.

@node Keeping State in Tests
@subsection Keeping State in Tests

Python does not allow assignment to anything but the innermost local scope or
the global scope with the @code{global} keyword.  This presents a problem when
creating nested functions:

@example
def test_localVariable(self):
    cb_called = False
    def cb():
        cb_called = True
    cb()
    self.assertTrue(cb_called) # will fail!
@end example

The @code{cb_called = True} assigns to a @i{different variable} than
@code{cb_called = False}.  In production code, it's usually best to work around
such problems, but in tests this is often the clearest way to express the
behavior under test.

The solution is to change something in a common mutable object.  While a simple
list can serve as such a mutable object, this leads to code that is hard to
read.  Instead, use @code{State}:

@example
from buildbot.test.state import State

def test_localVariable(self):
    state = State(cb_called=False)
    def cb():
        state.cb_called = True
    cb()
    self.assertTrue(state.cb_called) # passes
@end example

This is almost as readable as the first example, but it actually works. 

@node Better Debugging through Monkeypatching
@subsection Better Debugging through Monkeypatching
@dvindex buildbot.test.util.monkeypatches

The module @code{buildbot.test.util.monkeypatches} contains a few
monkey-patches to Twisted that detect errors a bit better.  These patches
shouldn't affect correct behavior, so it's worthwhile including something like
this in the header of every test file:

@example
from buildbot.test.util.monkeypatches import monkeypatch
monkeypatch()
@end example

@node Developer Class Index
@section Developer Class Index
@printindex dv
