Archives

Friday April 3, 2009

Continuous Integration Support for Wikis

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.

-- Continuous Integration, Martin Fowler

In a way, wikis have been using Continuous Integration (CI) since their inception: new content, configuration settings, and even plugins are always being added to wikis. Unfortunately, no wikis that I know of have any formal support for CI. In particular, I don't see any test suites that help wiki administrators and users detect and correct broken content, configuration problems, and/or dysfunctional plugin combinations. Perhaps it's time to resolve this deficiency.

Overview

The agile software development (eg, Ruby on Rails) community has developed quite a bit of testing infrastructure for CI. This ranges from servers that continuously perform tests on the checked-in code base to editor plugins that tell developers about newly-detected problems. Collectively, this lets agile developers know when the "build" breaks, who broke it, etc.

The needs and practices of typical wiki users and administrators are somewhat different from those of agile software development teams. For example, it is relatively unusual for changes in a wiki page to affect other pages, much less the wiki installation as a whole. Nonetheless, most users would like to be informed when they create an invalid page and most administrators would like to know about configuration problems.

We could begin by testing the wiki software itself. Most "production" software includes a substantial test suite, composed of various types of tests (eg, unit, functional, integration, regression). Wiki software generally has this sort of thing, although there may be holes in the coverage (eg, inadequate tests for some plugins).

It can't hurt to run these sorts of tests on an installed wiki, in case the installed software has become dysfunctional. Given that patches, plugins, and system updates are commonly added to installed wikis, running a test suite on the installed software might well uncover problems.

However, this is only a small part of what can be done. Other testing possibilities include:

  • checking access control for pages and/or files
  • looking for broken references or invalid syntax in local content
  • looking for unreferenced local content
I'm using the term "local content" quite broadly, above. For example, I'd look for broken references (eg, content inclusions, links) and syntax errors in both the raw content (eg, CSS, pages, templates) and the generated HTML. Basically, anything that can be tested is fair game...

Implementation Notions

Some checking can certainly be done interactively. In fact, MediaWiki does some basic syntax checking when a page is saved. For example, it flags <ref> tags that won't be rendered because there's no <references/> section. A Wikipedia-specific variant might complain about any pages that lack a <references/> section.

Borrowing some ideas from other CI infrastructure (eg, CI plugins for Mozilla Firefox, TextMate), we could put a "status bar" at the top of the rendered page. So, when a page is saved or viewed in a particular mode, the user might see warning messages, etc.

It's not clear that all types of problems can be detected "on the fly", but it's certainly worth thinking about the possibility. In any case, the critical capability is to detect common problems and bring them to the attention of a human, so that they can be corrected.

Meanwhile, here are some notions about "batch" implementations. Some of the details are specific to TWiki and/or Foswiki (T/F). However, my guess is that similar notions could be applied to any wiki. Finally, please understand that none of the following has been implemented (though I'm pretty confident that most of it could be :-).

Architecture

The test suite can be constructed as a collection of scripts, supported by a simple database. The scripts would be run in several passes, some of which might involve parallel processing. The output might be recorded as data or log files, database records, and/or wiki pages.

Environment

The production server is the simplest environment to use: nothing has to be created, cloned, set up, or simulated. However, this approach has some drawbacks and limitations:

  • additional load on a production server
  • authentication issues may be a nuisance
  • environmental details may change unexpectedly
  • tests cannot safely modify "real" content
So, it is probably best to check configuration details on the production host(s), but use a separate host for at least some of the "dynamic" testing. In any case, we need to have a testing host whose behavior will only deviate from the production host(s) in predictable ways.

Pass 0 - Initialization

This pass ensures that conditions are in place for the CI run. So, it might check for configuration files, create or clear out directories, ensure that the test database exists and is accessible, and/or initialize log files.

If some of the scripts use a testing host, this pass will also ensure that a snapshot of the local content is in place. This is also the right time to pre-load "test content": pages which are expected to (mis-)behave in specified ways.

Pass 1 - Check Configuration

This pass walks through the local configuration, attempting to find broken or suspicious settings. This pass might, for example, look at:

  • access lists, group definitions, etc.
  • file ownerships and permissions
  • file presence (eg, missing, unexpected)
  • settings for skins, templates, etc.
  • software update levels
  • syntax of file (etc) names
  • syntax of support (eg, CSS) files
  • web server configuration
The pass should also harvest information needed by later passes. For example, it would be useful to set up database records for pages, skins, styles, templates, etc.

The first time this pass is run, a number of configuration problems can be expected to surface. These should be corrected or (if need be) worked around and the pass re-run until it completes without error.

Pass 2 - Check Raw Content

By default, T/F stores all of its persistent content in files (RCS files are used for version control). So, it's simple to walk the local file tree(s), checking files for syntax errors, broken references, etc. Again, the pass should harvest information (eg, references) that may be needed by later passes.

One of the things that makes this pass tricky is the fact that plugin-specific markup may not have consistent (or even well-defined) syntax. However, it should be possible to recognize the extent of a plugin's markup, even if the exact format cannot be checked.

Pass 3 - Traverse Web pages

Using a web spidering library (eg, Swish-e), it should be possible to traverse all of the web pages that the wiki can produce. Because we already know the names of the pages we expect to find, we can be certain we won't miss any unconnected "islands" of wiki pages (ie, disjoint sub-graphs in the site).

Again, we're looking for syntax errors, broken references, etc. However, we're looking at the generated HTML this time, so we may encounter structural problems created by the wiki software or our local configuration. As in Pass 1, we'll need to correct or work around these issues...

Pass 4 - Generate Reports

At this point, all of the data has been collected and stored in the test database. So, we just need to summarize it, format and save reports, etc. For example, we can compare the list of page names to the list of page references, generating a list of unreferenced pages.

Status and Prospects

This comparison of wiki software lists several dozen wikis. Each of these wikis has its own architecture, implementation and markup languages, etc. Some use flat files for persistent storage; others use databases. Crafting a "one size fits all" testing solution isn't going to be easy, given that we need to examine "internal" details such as raw pages and configuration settings.

However, it may be possible to share a common approach and parts of the test suite. For example, checking the syntax of CSS and HTML isn't a wiki-specific task. Also, much of the infrastructure (eg, base system, database, report generator) can be written in a wiki-agnostic manner.

Nonetheless, key parts of the infrastructure (and most of the tests) will need to be written or adapted for each wiki. Wiki developers are in the best position to adapt the infrastructure, because of their knowledge of internal details. Conversely, wiki administrators are in a good position to craft tests, because they can adapt them from "real world" problems.

The only way I can see any of this happening is through a "snowball effect". A basic test suite for a popular wiki (eg, TWiki/Foswiki, MediaWiki) could be developed and released by the core team and/or a large installation. This might induce other developers and/or administrators to add their own tests.

Competitive forces might then cause other wiki communities to devise similar suites. Although this is all highly speculative, similar scenarios have played out in the areas of scripting languages, web frameworks, etc. So, it's not impossible...

Thanks to Dan Dascalescu, Gene Dronek, Peter Thoeny, Sven Dowideit, and Vicki Brown for their reviews and comments. As usual, any errors and omissions in this essay are my own responsibility.

Continuous Integration Support for Wikis - posted at Fri, 03 Apr, 23:18 Pacific | «e» | TrackBack


Post a comment

Note: All comments are subject to approval. Spam will be deleted before anyone ever sees it. Excessive use of URLs may cause comments to be auto-junked. You have been warned.

Any posted comments will be viewable by all visitors. Please try to stay relevant ;-) If you simply want to say something to me, please send me email.