ServerZen.Net

Types of Tests

Unit tests

Testing done in as extremely minimal fashion as possible. Having some monstrous setup done before hand is not acceptable. These tests are done in a mind set by which the developer knows what the inner code looks like so he can test that certain inner code flags/values have been set after running the test.

Integration tests

Testing done very similar to unit tests (that is they’re all code based and don’t actually go through the browser interface) but require some integration to have been performed. In the case of Plone this often means having a complete plone site installed with all the trimmings. Basically these are unit tests on steriods. 95% of plone add-on developers write their tests these ways (the fastest way to see if a test is an integration test is to see if it uses PloneTestCase or ZopeTestCase … if it does, it’s an integration test). If a test requires a working portal instance, it’s an integration test.

Functional tests

Testing from as close to the real environment as possible, in most casee this means using something like selenium or testbrowser (I tend to use testbrowser). These tests never touch actual code api’s (other than to run the mock web browser).

An Example

Let’s say we have browser.py. Anyone familiar with the newer Zope 3 style way of coding applications is familar with the browser module. In this case, as expected, browser.py is giving us view classes that ultimately will get accessed via a web browser. Here’s how we would write different types of tests for that browser module.

But before we get into the actual tests, We need to begin by defining browser.py as follows:

from Products.Five.browser import BrowserView
from Products.CMFCore import utils as cmfutils

class SimpleView(BrowserView):
    """A simple view."""

    def nextval(self):
        portal = cmfutils.getToolByName(self.context, 'portal_url') \
                 .getPortalObject()
        current = getattr(portal, '_simpleview_count', 0)
        portal._simpleview_count = current + 1
        return portal._simpleview_count

    def __call__(self):
        return 'Retrieved %i' % self.nextval()

Also need configure.zcml:

<configure
  xmlns="http://namespaces.zope.org/zope"
  xmlns:browser="http://namespaces.zope.org/browser">

<browser:page
    name="simpleview"
    for="*"
    class=".browser.SimpleView"
    permission="zope.Public"
    />

</configure>

Unit test

In general no setup will be performed for this at all. Here the developer would simply import the browser module and instantiate the view classes using python code. And then with the view instance, test each of the methods. In general when any additional functionality is needed, it’s done in the form of mock objects.

Here’s what the test harness looks like (in this example, expected to live as tests/test_unit.py):

import unittest
from zope.testing import doctest

def test_suite():
    return unittest.TestSuite(doctest.DocFileSuite
                              ('unit-example.txt',
                               package='testingexample'))

Here’s the test (in doctest-style) as is expected to live in unit-example.txt:

First some mock objects.

    >>> class Mock(object):
    ...     def __init__(self, **kwargs):
    ...         for k, v in kwargs.items(): setattr(self, k, v)

    >>> portal = Mock()
    >>> context = Mock(portal_url=Mock(getPortalObject=lambda: portal))

We don't bother with request since we know the innards of our code and
the fact that it doesn't use the request for anything.

    >>> from testingexample.browser import SimpleView
    >>> view = SimpleView(context, None)
    >>> view.nextval()
    1
    >>> portal._simpleview_count
    1

And adjusting the private var manually will work as expected.

    >>> portal._simpleview_count = 50
    >>> view.nextval()
    51
    >>> portal._simpleview_count
    51

And then testing the string output of ``__call__``.

    >>> view()
    'Retrieved 52'
    >>> portal._simpleview_count
    52

The only way to see this break is if someone corrupted the _simpleview_count
value (which we should have to account for anyhow).

    >>> portal._simpleview_count = 'foobar'
    >>> view.nextval()
    Traceback (most recent call last):
    TypeError: cannot concatenate 'str' and 'int' objects

Integration test

Use a setUp that sets up a simple plone site and installs any plone add-ons we need. Use code like view = somecontentobj.restrictedTraverse(’@@someview’) to look up a view that is being created with browser.py and interact with that view component on an api level.

Here’s what the test harness looks like (in this example, expected to live as tests/test_integration.py):

import unittest
import testingexample
from Testing import ZopeTestCase
from Testing.ZopeTestCase.zopedoctest import ZopeDocFileSuite
from Products.PloneTestCase import PloneTestCase
from Products.PloneTestCase.layer import PloneSite
from Products.Five import zcml

PloneTestCase.setupPloneSite()

class MainTestCase(PloneTestCase.PloneTestCase):
    def afterSetUp(self):
        zcml.load_config('configure.zcml', testingexample)
        self.portal._simpleview_count = 0

def test_suite():
    suite = ZopeDocFileSuite('integration-example.txt',
                             package='testingexample',
                             test_class=MainTestCase)
    suite.layer = PloneSite

    return unittest.TestSuite((suite,))

And here’s the actual tests:

In these tests we expect that the portal object has already been setup
(ala ``PloneTestCase``) and is available as simply ``portal``.

    >>> portal
    <PloneSite at /plone>

Our first integration test just checks to make sure that we can actually
lookup the view by traversing.

    >>> view = portal.restrictedTraverse('@@simpleview')
    >>> view is not None
    True

Our view instance is already expected to have a working *context* and
*request* so we can continue as expected.

    >>> view.nextval()
    1
    >>> portal._simpleview_count
    1

And adjusting the private var manually will work as expected.

    >>> portal._simpleview_count = 50
    >>> view.nextval()
    51
    >>> portal._simpleview_count
    51

And then testing the string output of ``__call__``.

    >>> view()
    'Retrieved 52'
    >>> portal._simpleview_count
    52

The only way to see this break is if someone corrupted the _simpleview_count
value (which we should have to account for anyhow).

    >>> portal._simpleview_count = 'foobar'
    >>> view.nextval()
    Traceback (most recent call last):
    TypeError: cannot concatenate 'str' and 'int' objects

Functional test

Use a setUp that sets up a simple plone site and installs any plone add-ons we need. Instantiate a test browser instance (via zope.testbrowser) and mimick browser actions to “log into” the site and access whatever views were produced by browser.py.

Here’s what the test harness looks like (in this example, expected to live as tests/test_functional.py):

import unittest
import testingexample
from Testing import ZopeTestCase
from Testing.ZopeTestCase import FunctionalDocFileSuite
from Products.PloneTestCase import PloneTestCase
from Products.PloneTestCase.layer import PloneSite
from Products.Five import zcml

PloneTestCase.setupPloneSite()

class MainTestCase(PloneTestCase.PloneTestCase):
    def afterSetUp(self):
        zcml.load_config('configure.zcml', testingexample)
        self.portal._simpleview_count = 0

def test_suite():
    suite = FunctionalDocFileSuite('functional-example.txt',
                                   package='testingexample',
                                   test_class=MainTestCase)
    suite.layer = PloneSite

    return unittest.TestSuite((suite,))

And here’s the actual tests:

These tests are all about seeing and testing what the browser sees.  We
make no assumptions on the innards of the code -- pretending we have indeed
never seen the code itself.

First we need to setup a browser instance.

    >>> from Products.Five.testbrowser import Browser
    >>> browser = Browser()

Now we can start checking things out.  Really all we can test here now
is that the output to the browser has an integer that increments each time.

    >>> browser.open(portal.absolute_url()+'/@@simpleview')
    >>> browser.contents
    'Retrieved 1'

    >>> browser.open(portal.absolute_url()+'/@@simpleview')
    >>> browser.contents
    'Retrieved 2'

Seeing It In Use

All of the example code here can be found in the svn collective as an actual python package. Read the included README.txt to figure out how to set it up in your own zope instance. The package is available at: http://svn.plone.org/svn/collective/examples/testingexample/tags/rocky-blog-post-20070919/

The tests are meant to be run with:

$ ./bin/zopectl test -s testingexample

But don’t forget that when developing code you can save yourself a ton of time by maintaining 100% unit test coverage. Then you can run the unit tests as often as you want (at a very rapid speed) and only run the integration and/or functional tests at milestone intervals. To run them separately you would do:

$ ./bin/zopectl test -m testingexample.tests.test_unit
$ ./bin/zopectl test -m testingexample.tests.test_integration
$ ./bin/zopectl test -m testingexample.tests.test_functional

Conclusion

Testing is great. I’m not particularly advocating test driven development (it works for some people, but other people it does not). But it’s important to understand the differences between the different types of tests. The author suggests maintaining 100% unit test coverage and some (client-derived) acceptable amount of functional tests. Integration tests aren’t so important when you already have good unit and functional test coverage.