DailyJS

Let's Make a Test Framework

04 Nov 2010 | By Alex Young | Tags frameworks tutorials lmaf testing

Welcome to part 37 of Let’s Make a Framework, the ongoing series about building a JavaScript framework.

If you haven’t been following along, these articles are tagged with lmaf. The project we’re creating is called Turing.

Last week I was discussing how frameworks write and package their tests. Looking at other frameworks was interesting, and made me realise that using the CommonJS assert and test modules would be a good idea. However, I couldn’t find a suitable library.

I don’t like writing code for the sake of it, so typically I’d make do with something like QUnit. However, I realised that test frameworks are full of interesting code, in particular assertions. So rather than reusing something, let’s make our own CommonJS test framework for Turing that will run in Node, Rhino, and a browser.

Plus, if we create our own unit testing library, we can call it Turing Test!

Test Module

In Unit Testing 1.0, CommonJS defines how the test module should work. It’s essentially the same as most unit testing frameworks:

  • The test module provides a run method
  • This accepts an object
  • It will run any methods that start with but aren’t equal to test
  • Sub-objects will also run as sub-tests if their names match in the same way
  • The run method will catalog the results of the tests

That means tests can look as simple as this:

var assert = require('assert');

require('test').run({
  testEqual: function() {
    assert.equal(true, true, 'True should be true');
  },

  testAGroupOfThings: {
    testOK: function() {
      assert.ok("I'm OK!", "If you're OK you're OK, OK?");
    }
  }
});

Test Organisation

In real projects, tests are usually organised into several categorised files like this:

exports['test equal'] = function() {
  assert.equal(true, true, 'True should be true');
}

exports['test ok'] = function() {
  assert.ok(true, 'True should be OK');
}

Then a test runner can be written that requires each file and calls require('test').run() on it.

In a Browser

Running these tests in a browser will require a harness that can provide require and generate suitable HTML reports.

We don’t really need to use require at all, so tests could take this pattern:

if (typeof require !== 'undefined') {
  if (typeof require.paths !== 'undefined')
    require.paths.unshift('lib/');
  var assert = require('assert');
}

exports['test equal'] = function() {
  assert.equal(true, true, 'True should be true');
};

exports['test ok'] = function() {
  assert.ok(true, 'True should be OK');
};

if (typeof module !== 'undefined')
  if (module === require.main) 
    require('test').run(exports);

The last two lines are recommended by the CommonJS specification. It makes running single test files possible, like this:

node test/assertions.js

This is important because it’s useful to run individual test files during development. The test suit will be run less often, perhaps as part of a build process or automatically before deployment.

The only problem with this boilerplate code is it’ll have to appear in every file. We could write a test runner script that wraps this, but that would end up being a test runner… runner.

Basic Assertions

Let’s look at two very basic assertions. Writing assertions is actually what motivated me to write this tutorial — they contain a lot of interesting JavaScript code.

I’ve based these assertions on assert.js from Node, because it’s a very neat little library. These assertions should only really be required in the browser, because platforms like Node and Narwhal already provide them as part of their CommonJS libraries.

The way assertions generally work is to define a fail method that gets called with details about the assertion, and then raises an exception that the test runner will look for — in this case AssertionError. The test runner can then differentiate between failed assertions and other exceptions.

The signature of fail reveals how any assertion can be written:

function fail(actual, expected, message, operator, stackStartFunction) {
  throw new assert.AssertionError({
    message: message,
// etc.

Almost any assertion you could think up has an actual and expected value.

Messages can be used to make failures clearer. This is a common convention, yet I wrote unit tests in other languages for years before I realised how useful messages are.

In the case of assert.ok, the expected value is always true. This assertion just looks for truthy values, or as the spec says:

Pure assertion tests whether a value is truthy, as determined by !!guard

Which means assert.equal is very similar:

assert.equal = function(actual, expected, message) {
  if (actual != expected)
    fail(actual, expected, message, '==', assert.equal);  
};

The concept of equality is a little bit tricky, given that objects aren’t just simple values. The CommonJS specification addresses this with the assert.deepEqual assertion. We can look at this in a future tutorial.

The Test Runner

The test runner should:

  • Run each test
  • Gather results
  • Display results

Displaying results is the interesting part for browser testing. I’ve kept the code very simple for this part of the tutorial though, fancy styles can come later.

printMessage = (function() {
  if (typeof window !== 'undefined') {
    return function(message) {
      var li = document.createElement('li');
      li.innerHTML = message;
      document.getElementById('results').appendChild(li);
    }
  } else if (typeof console !== 'undefined') {
    return console.log;
  } else {
    return function() {};
  }
})();

The anonymous function is used to return a function we can reuse depending on the environment. It checks if window is available, and if so assumes the environment is a browser. The results are generated without relying on a JavaScript client-side framework, and Node will provide console.log for us.

The structure of the test runner collects fail, pass, and error counts. This allows it to display a set of results after running through all the tests. Exceptions will be displayed during run time. I haven’t really made much effort to display backtraces because it seemed too much for this tutorial, but I may come back to it later.

Gathering results just needs a Test class that can run tests and keep counters. Tests are run in an exception handler. This is pretty simple:

try {
  // TODO: Setup
  obj[testName]();
  this.passed += 1;
} catch (e) {
  if (e.name === 'AssertionError') {
    result.message = e.toString();
    logger.fail('Assertion failed in: ' + testName);
    showException(e);
    this.failed += 1;
  } else {
    logger.error('Error in: ' + testName);
    showException(e);
    this.errors += 1;
  }
} finally {
  // TODO: Teardown
}

Notice where I’ve marked those TODO comments. Setup and teardown methods are fairly common in JavaScript frameworks, so we could support those in the future.

As it turns out Node doesn’t currently have a test module, so the one I intended to be used by the browser will get used. That’s why I was conditionally messing around with require paths in the unit test boilerplate code.

Conclusion

I don’t actually intend turing-test to be a definitive test framework, it’s merely part of the Let’s Make a Framework tutorial series. However, because tests give a deep insight into the language, I hope this part of the series helps you sharpen up your JavaScript skills.

Our goals are modest: to write a test framework that facilitates browser and console unit tests which uses CommonJS. Yet writing test frameworks can reveal a lot about the language.

I wrote some proof-of-concept code and posted it under turing-test.js on GitHub. I only wrote it to make sure browsers wouldn’t explode on the code and destroy all lifeforms in a 10 mile radius1. As this part of Let’s Make a Framework continues hopefully it’ll become something more useful.

Resources

If you want to build the next awesome test framework, get a head start:

1 I’ve been playing Fallout: New Vegas


blog comments powered by Disqus