This chapter is the official repository for the collected wisdom of the
Buildbot hackers.

It contains some sparse documentation of the inner workings of Buildbot, but of
course, the final reference for that is the source itself.

More importantly, this chapter represents the official repository of all
agreed-on patterns for use in Buildbot.  In this case, the source is a
@i{terrible} reference, because much of it is old and crusty.  But we are
trying to do things the new, better way, and those new, better ways are
described here.

@menu
* Buildmaster Service Hierarchy::
* Utilities::
* The Database::
* The Event Loop::
* String Types::
* Subscription Interfaces::
* Web Status::
* Twisted Idioms::
* Buildbot Tests::
* Developer Class Index::
@end menu

@node Buildmaster Service Hierarchy
@section Buildmaster Service Hierarchy

Buildbot uses Twisted's service hierarchy heavily.  The hierarchy looks like
this:

@itemize

@item BuildMaster
@dvindex buildbot.master.BuildMaster

This is the top-level service.

@itemize

@item BotMaster
@dvindex buildbot.master.BotMaster

The BotMaster manages all of the slaves.  BuildSlave instances are added as
child services of the BotMaster.

@item ChangeManager
@dvindex buildbot.changes.manager.ChangeManager

The ChangeManager manages the active change sources, as well as the stream of
changes received from those sources.

@item SchedulerManager
@dvindex buildbot.schedulers.manager.SchedulerManager

The SchedulerManager manages the active schedulers and handles inter-scheduler
notifications.

@item IStatusReceiver implementations

Objects from the 'status' configuration key are attached directly to the
buildmaster. These classes should inherit from StatusReceiver or
StatusReceiverMultiService and include an 'implements(IStatusReceiver)' stanza.

@end itemize

@end itemize

@node Utilities
@section Utilities

@menu
* buildbot.util.collections::
* buildbot.util.eventual::
* buildbot.util.json::
@end menu

Several small utilities are available at the top-level @code{buildbot.util}
package.  As always, see the API documentation for more information.

@table @code

@item natualSort

This function sorts strings "naturally", with embedded numbers sorted
numerically.  This ordering is good for objects which might have a numeric
suffix, e.g., @code{winslave1}, @code{winslave2}, ..

@item formatInterval

This function will return a human-readable string describing a length of time,
given a number of seconds.

@item ComparableMixin

This mixin class adds comparability to a subclass.  Use it like this:

@example
class Widget(FactoryProduct, ComparableMixin):
    compare_attrs = [ 'radius', 'thickness' ]
    # ...
@end example

Any attributes not in @code{compare_attrs} will not be considered when
comparing objects.  This is particularly useful in implementing buildbot's
reconfig logic, where a simple comparison between the new and existing objects
can determine whether the new object should replace the existing object.

@item safeTranslate

This function will filter out some inappropriate characters for filenames; it
is suitable for adapting strings from the configuration for use as filenames.
It is not suitable for use with strings from untrusted sources.

@item LRUCache

This is a simple least-recently-used cache.  Its constructor takes a maximum
size.  When the cache grows beyond this size, the least-recently used items
will be automatically removed from the cache.  The class has @code{get} and
@code{add} methods, and can also be accessed via dictionary syntax
(@code{lru['id']}).

@end table

@node buildbot.util.collections
@subsection buildbot.util.collections

This package provides a few useful collection objects.

For compatibility, it provides a clone of the Python
@code{collections.defaultdict} for use in Python-2.4.  In later versions, this
is simply a reference to the built-in @code{defaultdict}, so buildbot code can
simply use @code{buildbot.util.collections.defaultdict} everywhere.

It also provides a @code{KeyedSets} class that can represent any numbers of
sets, keyed by name (or anything hashable, really).  The object is specially
tuned to contain many different keys over its lifetime without wasting memory.
See the docstring for more information.

@node buildbot.util.eventual
@subsection buildbot.util.eventual

This package provides a simple way to say "please do this later":

@example
from buildbot.util.eventual import eventually
def do_what_I_say(what, where):
    # ...
eventually(do_what_I_say, "clean up", "your bedroom")
@end example

The package defines "later" as "next time the reactor has control", so this is
a good way to avoid long loops that block other activity in the reactor.
Callables given to @code{eventually} are guaranteed to be called in the same
order as the calls to @code{eventually}.  Any errors from the callable are
logged, but will not affect other callables.

If you need a deferred that will fire "later", use @code{fireEventually}.  This
function returns a deferred that will not errback.

@node buildbot.util.json
@subsection buildbot.util.json

This package is just an import of the best available JSON module.  Use it
instead of a more complex conditional import of @code{simplejson} or
@code{json}.

@node The Database
@section The Database

@dvindex buildbot.db.dbspec.DBSpec
@dvindex buildbot.db.connector.DBConnector

Buildbot stores its state in a database, using the Python DBAPI to access it.

A database is specified with the @code{buildbot.db.dbspec.DBSpec} class, which
encapsulates all of the parameters necessary to create a DBAPI connection.  The
DBSpec class can create either a single synchronous connection, or a twisted
adbapi connection pool.

Most uses of the database in Buildbot are done through the
@code{buildbot.db.connector.DBConnector} class, which wraps the DBAPI to
provide purpose-specific functions.

The database schema is managed by a special class, described in the next
section.

@menu
* Database Schema::
* Changing the Schema::
@end menu

@node Database Schema
@subsection Database Schema

@dvindex buildbot.db.schema.DBSchemaManager

The SQL for the database schema is available in
@code{buildbot/db/schema/tables.sql}.  However, note that this file is not used
for new installations or upgrades of the Buildbot database.

Instead, the @code{buildbot.db.schema.DBSchemaManager} handles this task.  The
operation of this class centers around a linear sequence of database versions.
Versions start at 0, which is the old pickle-file format.  The manager has
methods to query the version of the database, and the current version from the
source code.  It also has an @code{upgrade} method which will upgrade the
database to the latest version.  This operation is currently irreversible.

There is no operation to "install" the latest schema.  Instead, a fresh install
of buildbot begins with an (empty) version-0 database, and upgrades to the
current version.  This trades a bit of efficiency at install time for
assurances that the upgrade code is well-tested.

@node Changing the Schema
@subsection Changing the Schema

To make a change to the database schema, follow these steps:

@enumerate

@item

Increment @code{CURRENT_VERSION} in @code{buildbot/db/schema/manager.py} by
one.  This is your new version number.

@item

Create @file{buildbot/db/schema/vN.py}, where N is your version number, by
copying the previous script and stripping it down.  This script should define a
subclass of @code{buildbot.db.schema.base.Updater} named @code{Updater}. 

The class must define the method @code{upgrade}, which takes no arguments.  It
should upgrade the database from the previous version to your version,
including incrementing the number in the @code{VERSION} table, probably with an
@code{UPDATE} query.

Consult the API documentation for the base class for information on the
attributes that are available.

@item

Edit @file{buildbot/test/unit/test_db_schema_master.py}.  If your upgrade
involves moving data from the basedir into the database proper, then edit
@code{fill_basedir} to add some test data.

Add code to @code{assertDatabaseOKEmpty} to check that your upgrade works on an
empty database.

Add code to @code{assertDatabaseOKFull} to check that your upgrade works on a
database with pre-existing data.  Do this even if your changes do not move any
data from the basedir.

Run the tests to find the bugs you introduced in step 2.

@item

Increment the version number in the @code{test_get_current_version} test in the
same file.  Only do this after you've finished the previous step - a failure of
this test is a good reminder that testing isn't done yet.

@item

Finally, make the corresponding changes to @file{buildbot/db/schema/tables.sql}.

@end enumerate

@node The Event Loop
@section The Event Loop

TODO

@node String Types
@section String Types

TODO

@node Subscription Interfaces
@section Subscription Interfaces

TODO use @code{buildbot.eventually}

@node Web Status
@section Web Status

@menu
* Jinja Web Templates::
* Web Authorization Framework::
@end menu

@node Jinja Web Templates
@subsection Jinja Web Templates

Buildbot uses Jinja2 to render its web interface.  The authoritative source for
this templating engine is @uref{http://jinja.pocoo.org/2/documentation/, its
own documentation}, of course, but a few notes are in order for those who are
making only minor modifications.

@heading Whitespace

Jinja directives are enclosed in @code{@{% .. %@}}, and sometimes also have
dashes.  These dashes strip whitespace in the output.  For example:

@example
@{% for entry in entries %@}
  <li>@{@{ entry @}@}</li>
@{% endfor %@}
@end example

will produce output with too much whitespace:

@example

  <li>pigs</li>


  <li>cows</li>

@end example

But adding the dashes will collapse that whitespace completely:

@example
@{% for entry in entries -%@}
  <li>@{@{ entry @}@}</li>
@{%- endfor %@}
@end example

yields

@example
<li>pigs</li><li>cows</li>
@end example

@node Web Authorization Framework
@subsection Web Authorization Framework

Whenever any part of the web framework wants to perform some action on the
buildmaster, it should check the user's authorization first.

Always check authorization twice: once to decide whether to show the option to
the user (link, button, form, whatever); and once before actually performing
the action.

To check whether to display the option, you'll usually want to pass an authz
object to the Jinja template in your HtmlResource subclass:

@example
    def content(self, req, cxt):
        # ...
        cxt['authz'] = self.getAuthz(req)
        template = ...
        return template.render(**cxt)
@end example

and then determine whether to advertise the action in th template:

@example
@{@{ if authz.advertiseAction('myNewTrick') @}@}
  <form action="@{@{ myNewTrick_url @}@}"> ...
@{@{ endif @}@}
@end example

Actions can optionally require authentication, so use @code{needAuthForm} to
determine whether to require a 'username' and 'passwd' field in the generated
form.  These fields are usually generated by the @code{auth()} form:

@example
@{% if authz.needAuthForm('myNewTrick') %@}
  @{@{ auth() @}@}
@{% endif %@}
@end example

Once the POST request comes in, it's time to check authorization again.
This usually looks something like

@example
if not self.getAuthz(req).actionAllowed('myNewTrick', req, someExtraArg):
    return Redirect(path_to_authfail(req))
@end example

The @code{someExtraArg} is optional (it's handled with @code{*args}, so you can
have several if you want), and is given to the user's authorization function.
For example, a build-related action should pass the build status, so that the
user's authorization function could ensure that devs can only operate on their
own builds.

The available actions are listed in @pxref{WebStatus Configuration Parameters}.

@node Twisted Idioms
@section Twisted Idioms

@menu
* Helpful Twisted Classes::
@end menu

@node Helpful Twisted Classes
@subsection Helpful Twisted Classes

Twisted has some useful, but little-known classes.  They are listed here with
brief descriptions, but you should consult the API documentation or source code
for the full details.

@itemize

@item twisted.internet.task.LoopingCall

Calls an asynchronous function repeatedly at set intervals.

@item twisted.application.internet.TimerService

Similar to @code{t.i.t.LoopingCall}, but implemented as a service that will
automatically start an dstop the function calls when the service is started and
stopped.

@end itemize

@node Buildbot Tests
@section Buildbot Tests

@menu
* Toward Better Buildbot Tests::
* Keeping State in Tests::
* Better Debugging through Monkeypatching::
@end menu

@node Toward Better Buildbot Tests
@subsection Toward Better Buildbot Tests

In general, we are trying to ensure that new tests are @i{good}.  So what makes a good test?

@heading Independent of Time

Tests that depend on wall time will fail. As a bonus, they run very slowly. Do
not use @code{reactor.callLater} to wait "long enough" for something to happen.

For testing things that themselves depend on time, consider using
@code{twisted.internet.tasks.Clock}.  This may mean passing a clock instance to
the code under test, and propagating that instance as necessary to ensure that
all of the code using @code{callLater} uses it.  Refactoring code for
testability is difficult, but wortwhile.

For testing things that do not depend on time, but for which you cannot detect
the "end" of an operation: add a way to detect the end of the operation!

@heading Clean Code

Make your tests readable. This is no place to skimp on comments! Others will
attempt to learn about the expected behavior of your class by reading the
tests. As a side note, if you use a @code{Deferred} chain in your test, write
the callbacks as nested functions, rather than using object methods with funny
names:

@example
def testSomething(self):
  d = doThisFirst()
  def andThisNext(res):
    pass # ...
  d.addCallback(andThisNext)
  return d
@end example

This isolates the entire test into one indented block. It is OK to add methods
for common functionality, but give them real names and explain in detail what
they do.

@heading Good Name

Your test module should be named after the package or class it tests, replacing
@code{.} with @code{_} and omitting the buildbot_. For example,
@code{test_status_web_authz_Authz.py} tests the Authz class in
@code{buildbot/status/web/authz.py}. Modules with only one class, or a few
trivial classes, can be tested in a single test module. For more complex
situations, prefer to use multiple test modules.

Test method names should follow the pattern test_METHOD_CONDITION where METHOD
is the method being tested, and CONDITION is the condition under which it's
tested. Since we can't always test a single method, this is not a hard-and-fast
rule.

@heading Assert Only One Thing

Each test should have a single assertion. This may require a little bit of work
to get several related pieces of information into a single Python object for
comparison. The problem with multiple assertions is that, if the first
assertion fails, the remainder are not tested.  The test results then do not
tell the entire story.

If you need to make two unrelated assertions, you should be running two tests.

@heading Use Mocks and Stubs

Mocks assert that they are called correctly. Stubs provide a predictable base
on which to run the code under test. See
@url{http://en.wikipedia.org/wiki/Mock_object,Mock Object} and
@url{http://en.wikipedia.org/wiki/Method_stub,Method Stub}.

One of the difficulties with Buildbot is that interfaces are unstable and
poorly documented, which makes it difficult to design stubs.  A common
repository for stubs, however, will allow any interface changes to be reflected
in only one place in the test code.

@heading Small Tests

The shorter each test is, the better. Test as little code as possible in each test.

It is fine, and in fact encouraged, to write the code under test in such a way
as to facilitate this. As an illustrative example, if you are testing a new
Step subclass, but your tests require instantiating a BuildMaster, you're
probably doing something wrong! (Note that this rule is almost universally
violated in the existing buildbot tests).

This also applies to test modules.  Several short, easily-digested test modules
are preferred over a 1000-line monster.

@heading Isolation

Each test should be maximally independent of other tests. Do not leave files
laying around after your test has finished, and do not assume that some other
test has run beforehand. It's fine to use caching techniques to avoid repeated,
lengthy setup times.

@heading Be Correct

Tests should be as robust as possible, which at a basic level means using the
available frameworks correctly. All deferreds should have callbacks and be
chained properly. Error conditions should be checked properly. Race conditions
should not exist (see "Independent of Time", above).

@heading Be Helpful

Note that tests will pass most of the time, but the moment when they are most
useful is when they fail.

When the test fails, it should produce output that is helpful to the person
chasing it down. This is particularly important when the tests are run
remotely, in which case the person chasing down the bug does not have access to
the system on which the test fails. A test which fails sporadically with no
more information than "AssertionFailed?" is a prime candidate for deletion if
the error isn't obvious. Making the error obvious also includes adding comments
describing the ways a test might fail.

@heading Mixins

Do not define setUp and tearDown directly in a mixin. This is the path to
madness. Instead, define a @code{myMixinNameSetUp} and
@code{myMixinNameTearDown}, and call them explicitlyi from the subclass's
@code{setUp} and @code{tearDown}. This makes it perfectly clear what is being
set up and torn down from a simple analysis of the test case.

@node Keeping State in Tests
@subsection Keeping State in Tests

Python does not allow assignment to anything but the innermost local scope or
the global scope with the @code{global} keyword.  This presents a problem when
creating nested functions:

@example
def test_localVariable(self):
    cb_called = False
    def cb():
        cb_called = True
    cb()
    self.assertTrue(cb_called) # will fail!
@end example

The @code{cb_called = True} assigns to a @i{different variable} than
@code{cb_called = False}.  In production code, it's usually best to work around
such problems, but in tests this is often the clearest way to express the
behavior under test.

The solution is to change something in a common mutable object.  While a simple
list can serve as such a mutable object, this leads to code that is hard to
read.  Instead, use @code{State}:

@example
from buildbot.test.state import State

def test_localVariable(self):
    state = State(cb_called=False)
    def cb():
        state.cb_called = True
    cb()
    self.assertTrue(state.cb_called) # passes
@end example

This is almost as readable as the first example, but it actually works. 

@node Better Debugging through Monkeypatching
@subsection Better Debugging through Monkeypatching
@dvindex buildbot.test.util.monkeypatches

The module @code{buildbot.test.util.monkeypatches} contains a few
monkey-patches to Twisted that detect errors a bit better.  These patches
shouldn't affect correct behavior, so it's worthwhile including something like
this in the header of every test file:

@example
from buildbot.test.util.monkeypatches import monkeypatch
monkeypatch()
@end example

@node Developer Class Index
@section Developer Class Index
@printindex dv
