Managing Asynchronous Assertions
frameworks
tutorials
lmaf
testing
Let’s Make a Framework is an ongoing series about building a JavaScript framework from the ground up.
These articles are tagged with lmaf. The project we’re creating is called Turing. Documentation is available at turingjs.com.
The biggest problem I’ve had when writing tests for Turing is checking when asynchronous callbacks complete. For example, when testing the resource loading feature, I wrote tests with this pattern:
'test queue loading with no local scripts': function() {
$t.require([
'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js'
]).on('complete', function() {
assert.ok(true);
assert.done();
}).on('loaded', function(item) {
if (item.src === 'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js') {
assert.ok(jQuery, 'jQuery should be set');
}
});
}
What happens if loaded never fires? Well, the test would still pass. That’s a consequence of the assertion never being run.
One solution is to wrap the assert module with counters that count how many assertions have been called. Then at the start of a test we can write assert.expect(2); to say ‘raise an error if anything other than two assertions are run’.
That’s fine, but at this point Turing’s test framework always runs tests asynchronously. If the assert module kept a counter, other tests would increment that value too. The number of assertions would be the total number for all of the current suite’s tests, rather than the current test.
Counting Assertions
The initial solution I came up with was to wrap every assertion method with a function that incremented a counter. Tests take this pattern:
'test queue loading with no local scripts': function() {
var assertExpect = mixinExpect(assert);
assertExpect.expect(2);
$t.require([
'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js'
]).on('complete', function() {
assertExpect.ok(true);
assertExpect.done();
}).on('loaded', function(item) {
if (item.src === 'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js') {
assertExpect.ok(jQuery, 'jQuery should be set');
}
});
}
Each test gets its own instance of the assert module.
The code to do this is slightly clumsy, because I aliased the old methods using a __ prefix:
function mixinExpect(m) {
var m2 = {}, method;
for (method in m) {
if (m.hasOwnProperty(method)) {
m2['__' + method] = m[method];
(function(methodName) {
m2[methodName] = function() {
m2.mixinExpectAssertionCount++;
m2['__' + methodName].apply(m2, arguments);
};
}(method));
}
}
m2.expect = function(count) {
m2.mixinExpectAssertionCount = 0;
m2.mixinExpectExpected = count;
};
m2.done = function() {
if (m2.mixinExpectAssertionCount !== m2.mixinExpectExpected) {
throw('Expected assertion count was not found, expected: ' + m2.mixinExpectExpected + ', got: ' + m2.mixinExpectAssertionCount);
}
};
return m2;
}
It does the job, but is there a more elegant way that doesn’t require changing the assert module?
Expectations
I got this idea when I reviewed Tim Caswell’s Step library:
'test queue loading with no local scripts': function() {
var expect = new Expect();
expect.add('jQuery was set');
expect.add('loaded fired');
$t.require([
'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js'
]).on('complete', function() {
assert.ok(true);
expect.fulfill('loaded fired');
expect.done();
}).on('loaded', function(item) {
if (item.src === 'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js') {
expect.fulfill('jQuery was set');
assert.ok(jQuery, 'jQuery should be set');
}
});
}
Each test creates an instance of Expect and adds a list of expectations. As the expectations are fulfilled, it marks down this fact, then at the end of the test when expect.done() is called it can see if any expectations were unfulfilled.
I made a quick implementation and it’s only a few lines of code:
function Expect() {
this.expectations = {};
}
Expect.prototype = {
add: function(expectation) {
this.expectations[expectation] = false;
},
fulfill: function(expectation) {
this.expectations[expectation] = true;
},
done: function() {
for (var expectation in this.expectations) {
if (!this.expectations[expectation]) {
throw('Expected assertion was fulfilled , expected: ' + expectation);
}
}
}
};
Conclusion
Working with asynchronous tests can quickly get confusing — sometimes tests appear to be passing but aren’t actually running as expected. The CommonJS Unit Testing spec actually says the following:
The assertions defined above will not be to everyone’s taste style wise (or infact behaviour wise.) Authors are free to define their own assertion methods in new modules as desired.
So adding a assert.expect method to your own assertion module implementations is perfectly acceptable.
This week’s code can be found in the ajax tests in commit dc3b41f.
