testing pyblosxom -- a brain dump

In the pyblosxom git repository, there are two big things that have test
suites:

1. the PyBlosxom core code
2. the plugins that are in plugins/

These have two separate testing infrastructures, but both are based on
unittest which comes with Python.


PyBlosxom core code testing
===========================

These tests are located in Pyblosxom/tests/ and test the PyBlosxom core
functionality.

Tests are executed by::

   python setup.py test

This uses the test_suite parameter to setup which is in both distribute
and setuptools.

The test_suite is Pyblosxom.tests.testrunner.test_suite which is a
function that goes through all the files in Pyblosxom/tests/ and loads
tests in files where the filename starts with "test_" and ends in ".py".

There's a helper module which has a UnitTestBase derived from
unittest.TestCase which has some helper functions for building a blog
setup, tearing it down, building a Request object, and comparing two
dicts where one must be a subset the other.

Writing tests for the core is pretty easy:

1. create a file in Pyblosxom/tests/ with a filename that starts with
test_ and ends with .py.

2. import UnitTestBase from Pyblosxom.tests.helpers

3. create a subclass of UnitTestBase

4. write some tests using pretty standard unittest stuff

There are a bunch of existing examples in the directory.


testing plugins
===============

Plugins are a lot harder because we want to make sure they remain
modular and easy to install.  "Easy to install" really means that a
plugin is a single monolithic file.  I didn't want to have a ton of
testing code in the modules, so I wrote the test harness to allow for
plugins to have tests in the plugin as well as tests in a parallel tests
directory.

To run all the plugin tests, I do something like this::

    PYTHONPATH=. python plugins/testrunner.py


Tests in the plugin
-------------------

Testing inside a plugin is pretty limited because you can't take
advantage of the helper infrastructure--you're limited to what you can
load from Python and PyBlosxom core.

1. import unittest

2. create a class that extends unittest.TestCase

3. write your tests in that

4. create a function at the module level called get_test_suite that
returns the tests loaded from the test class(es)


Tests in the parallel directory
-------------------------------

Writing tests in the parallel directory has the benefit of being able to
use helper infrastructure, but the cost of being separate from the
actual plugin.  I think this is ok, but it's something that we have to
keep in mind when we make plugin fixes.

Tests are located in plugins/tests/ and are in a parallel directory
structure to the plugins in plugins/ .

For example, the test_acronyms.py test file is in
plugins/tests/text/test_acronyms.py and the plugin is in
plugins/text/acronyms.py .

Test files should import PluginTest from plugins.test.test_base .  This
class has a bunch of helper functions for dealing with request objects,
creating blog data, flavours, ...

1. import PluginTest from plugins.test.test_base

2. create a class that extends PluginTest

3. write your tests in that

4. edit plugins/tests/__init__.py and add a line to import things from
your new test module


Summary
=======

I think that's it.  It definitely seems more complicated than it is.
Generally speaking, the test infrastructure is ok, but not fantastically
amazing.  It'll evolve over time as we need it to do more.

Speaking of doing more, there are parts of PyBlosxom that could use
better testing.  Help on this front would be fantastic.

Having said that, adding tests for the code we're writing helps us a TON
in the quality department.  PyBlosxom 1.5 is ten times as good as
previous versions if only because we've got a better testing
infrastructure and we're testing plugin functionality.

Many thanks to Ryan Barret for the awesome work he did a while back.
The infrastructure we have now is based largely on his efforts.

/will

------------------------------------------------------------------------------