Node 0.4.10

Node 0.4.10 has been released. This version mainly contains bug fixes:

• #394 Fix Buffer drops last null character in UTF-8
• #829 Backport r8577 from V8 (Ben Noordhuis)
• #877 Don’t wait for HTTP Agent socket pool to establish connections
• #915 Find kqueue on FreeBSD correctly (Brett Kiefer)
• #1085 HTTP: Fix race in abort/dispatch code (Stefan Rusu)
• #1274 debugger improvement (Yoshihiro Kikuchi)
• #1291 Properly respond to HEAD during end(body) hot path (Reid Burke)
• #1304 TLS: Fix race in abort/connection code (Stefan Rusu)
• #1360 Allow _ in url hostnames
• Revert 37d529f8 – unbreaks debugger command parsing.
• Bring back global execScript (it was removed from v8)

Geocoder

Geocoder (GitHub: wyattdanger / geocoder, License, npm: geocoder) by Stephen Wyatt Bush is a module for using Google’s geocoding and reverse geocoding services:

var geocoder = require('geocoder');

geocoder.geocode("Atlanta, GA", function(data) {
  // do stuff with data
});

// Reverse Geocoding
geocoder.reverseGeocode(33.7489, -84.3789, function(data) {
  // do stuff with data
});

He’s planning to add error handling support, although I’d have expected the callback signature to be err, data rather than data, err.

Bunker

Bunker (npm: bunker, License: MIT/X11) by James Halliday generates code coverage reports using the author’s burrito library.

There’s a simple example that demonstrates the project’s event-based API:

var bunker = require('bunker');
var b = bunker('var x = 0; for (var i = 0; i < 30; i++) { x++ }');

var counts = {};

b.on('node', function (node) {
    if (!counts[node.id]) {
        counts[node.id] = { times : 0, node : node };
    }
    counts[node.id].times ++;
});

b.run();

Object.keys(counts).forEach(function (key) {
    var count = counts[key];
    console.log(count.times + ' : ' + count.node.source());
});

This is actually a native JavaScript parser based on AST data generated by UglifyJS. Bunker doesn’t currently provide a generalised script for analysing your project’s test coverage; for something like that TJ Holowaychuk’s node-jscoverage can work in conjunction with a test framework like Expresso.

Chosen

Chosen (GitHub: harvesthq / chosen, License: MIT) by Patrick Filler for Harvest is a nice little library for enhancing select boxes. Like many other similar plugins, this one hides standard selects and provides an alternative using modern HTML and CSS. That means it’s possible to add behaviour that standard selects don’t support without losing support when JavaScript is turned off.

The basic usage involves adding the plugin’s source to your project and then applying the chzn-select class to select boxes.

With other similar plugins I’ve found some browsers like Internet Explorer can suffer from a very noticeable visual hiccup when the selects are rendered. I haven’t yet tested this one with a complex page with lots of selects, but I’d be interested to see how well it performs.

jQuery.bulletGraph

jQuery.bulletGraph (GitHub: johdax / jQuery.bulletGraph) by johdax helps render bullet graphs, created by Stephen Few, which are “linear and no-frills” graphs for “rich display of data in a small space”.

// bulletgraph with custom ranges & range labels
$('select#metric2').bulletGraph({
  width: 200,
  height: 20,
  ranges: ['0%', '20%', '70%', '100%'],
  rangesLabels: ['low', 'medium', 'high'],
  sliderOptions: {
    disabled: true
  }
});

It’s always good to see graph libraries based on serious technical design instead of whatever combination of shadows and gradients is currently fashionable, so I’d bookmark this if you’re looking for ways to present data visually.

ARIA Hackathon Summary

The jQuery UI Blog has a post detailing the ARIA Hackathon which was an event designed to get jQuery UI developers and ARIA experts together to work on jQuery UI accessibility. It sounds like the jQuery UI keyboard accessibility is being improved.

If you’re interested in jQuery UI and ARIA, I found the jQuery UI Accessibility Hackathon wiki page. If you want to read up on ARIA, there are a lot of resources at the WAI-ARIA Overview.

About SocketStream

SocketStream (License: MIT, npm: socketstream) is a full stack framework for building single-page applications. It’s still under heavy development, but it already offers some extremely useful features:

• WebSockets
• Client-side rendering
• HTTPS support
• Built-in authentication and user model

Usage

SocketStream apps can be generated with a command-line client, so it’s a good idea to install it globally with npm:

npm install -g socketstream
socketstream new dailyjs-test
cd dailyjs-test
socketstream start

Running this skeleton app will show a welcome page with a little WebSocket chat demo.

Structure

SocketStream is built with CoffeeScript and includes a Cakefile which is the CoffeeScript version of a good old fashioned Makefile. The core project is really contributor/hacker territory rather than something users of the framework are likely to use, but that’s where I started when checking out the project. Anyway, I’m baffled by this particular Cakefile because it refers to tests that don’t seem to exist (it’s looking for a spec/ folder but there isn’t one).

The SocketStream binary script is integral to working with SocketStream projects. This concerned me because I often have multiple projects that use different versions of a framework. However, fortunately it’s possible to get around this by running npm install socketstream inside a SocketStream project’s directory, then running node_modules/socketstream/bin/socketstream start to start it up.

While looking around this level of the project I noticed the package.json file is extremely specific about dependency versions:

"dependencies": {
  "coffee-script":  "= 1.1.1",
  "socket.io":      "= 0.6.18",
  "redis":          "= 0.6.6",
  "hiredis":        "= 0.1.12",
  "node-static":    "= 0.5.6",
// ...

I recommend doing this in your own projects, simply because I can guarantee that people will break APIs given half a chance. Only use more relaxed versions if a library is extremely simple and is unlikely to change its public API.

The SocketStream binary script itself is only 11 lines of code. I like short binary files because it’s usually an indication that everything is broken up into modules and therefore easy to test.

Command line arguments are parsed using argsparser. I typically parse arguments with this pattern:

var args = process.argv.slice(2)
  , path = '.';

while (args.length) {
  var arg = args.shift();
  switch (arg) {
    case '-h':
    case '--help':
      console.log(usage);
      process.exit(0);
    case '-v':
    case '--version':
      console.log('Version 1.0');
      process.exit(0);
    case '--server':
      useServer = true;
      break;
    default:
        path = arg;
  }
}

I copied this pattern from TJ Holowaychuk’s projects, and as you can see it’s pretty short and sweet. I’m not sure what improvements argsparser adds to this, so further investigation may be necessary. This might seem like a minor point, but so many Node projects ship with binaries these days that I think it’s worth thinking about.

Back to SocketStream. Once the arguments are parsed, CoffeeScript is loaded, and then the arguments are passed to main.coffee.

main.coffee

Right off the bat I noticed the generous amount of comments and consistent code formatting in this project. Although CoffeeScript can sometimes feel a bit like a wall of text (without those extra braces everywhere), it’s fairly easy to follow this project’s code. The command processing is done with a simple case statement, and heredocs are used to print out long sections of text (I’m a big fan of heredocs).

In exports.init the SS global, which is effectively used to communicate between client and server:

The key to using SocketStream is the ‘SS’ global variable which can be called anywhere within your server or client-side code.

This file also loads helpers.js which I presume is JavaScript rather than CoffeeScript because it’s shared between the client and server-side code. It provides a lot of useful methods, but extends native objects. Obviously this saves typing, but I think it would be better to avoid doing this unless the additions are ECMAScript shims. Also, this file mixes tabs and spaces — is it really that hard to set your editor to use spaces like everyone else? Clearly it’s just one dude here who’s adding these phantom tabs in random parts of the project. As well as phantom tabs there’s also phantom whitespace before line endings, so I assume someone is using TextMate.

The fact this file has the following comment raises a red flag:

Be very careful about introducing new helpers as they may break external third-party client-side libraries.

Elsewhere in main.coffee, I saw something that I keep finding myself doing: loading the modules used throughout the project in one global so they don’t need to be required all the time:

  externalLibs: ->
    [
      ['coffee',    'coffee-script']
      ['io',        'socket.io']
      ['static',    'node-static']
      ['jade',      'jade']
      ['stylus',    'stylus']
      ['uglifyjs',  'uglify-js']
      ['redis',     'redis']
      ['semver',    'semver']
    ].forEach (lib) ->
      npm_name = lib[1]
      try
        version = SS.internal.package_json.dependencies[lib[1]].substring(2)
      catch e
        throw new Error("Unable to find #{npm_name} within the package.json dependencies")
      try        
        SS.libs[lib[0]] = require("#{npm_name}@#{version}")
      catch e
        try
          SS.libs[lib[0]] = require("#{npm_name}")
        catch e
          throw new Error("Unable to start SocketStream as we're missing #{npm_name}.\nPlease install with 'npm install #{npm_name}'. Please also check the version number if you're using a version of npm below 1.0")

I usually just do this with something like winston — anything that might be instantiated and shared. Node caches modules, so I’m not sure why something like Jade needs to be loaded this way. Elsewhere we see code like SS.libs.jade.renderFile which seems needlessly wordy for something that’s obviously a third-party module.

App Generation

At the moment apps are just generated by copying an example app. I expect this will be expanded in the future. The script checks to ensure the directory isn’t already taken, then runs something that should look familiar to anyone who’s done a lot of Node filesystem work:

exports.recursiveCopy = (source, destination) ->
  files = fs.readdirSync source

  # iterates over current directory
  files.forEach (file) ->
    unless file.match /^\./

      sourcePath = path.join source, file
      destinationPath = path.join destination, file

      stats = fs.statSync sourcePath
      if stats.isDirectory()
        fs.mkdirSync destinationPath, 0755
        exports.recursiveCopy sourcePath, destinationPath
      else
        exports.copyFile sourcePath, destinationPath

logger.coffee

I was surprised to find logging in the project, since winston seems so popular lately, but the authors have left a note saying logging is a work in progress:

NOTE: Let’s be honest. This idea sucks. It seemed like a good idea at the time, but I’m sure we can a lot better

The output of the logs looks relatively useful, and the current structure looks like it was designed to help transition to i18n in the future. There’s also a comment about an exception handling system elsewhere, so hopefully the authors will add an app-level option to turn off logging and just collect exceptions (have you ever seen the gigabytes of useless logs generated by Rails apps?)

Server

The server, session, and request files are mostly custom solutions for SocketStream. This is heavily based around Socket.IO. The server code is split between HTTP, HTTPS, and “API” code. SocketStream treats persistent HTTP connections differently to traditional stateless HTTP requests, and this is reflected by allowing applications to include optional HTTP APIs. These are mounted at a URL, the default is /api.

The server code also includes a form of keepalive, referred to as “heartbeat”. This helps maintain a count of how many users are online. I wondered if this whole user tracking code could instead be built like a plugin, rather than inside the main server code. It seems like it would be desirable to add an EventEmitter (or perhaps the pubsub system) to the server to allow external code, such as user management, bind to various events and process them accordingly.

Example Apps

The SocketStream Chat demo includes a deployment script that’s easy to follow and potentially reuse for your own projects. I think this is great, because many frameworks rarely give much guidance to this side of the project life cycle.

Benchmarks

SocketStream’s documentation has this to say:

[Features] … Crazy fast! Starts up instantly. No HTTP handshaking/headers/routing to slow down every request

That makes sense, but where are the benchmarks? I only found one benchmark in the HTTP middleware code. If you’re going to market your project as “crazy fast” I want to see real benchmarks! Look at Jade, right there at the top-level there’s a folder called benchmarks/. If you’re serious about speed be scientific about it.

Conclusion

The authors of SocketStream write great CoffeeScript, and it’s an extremely ambitious project. I’d love to spend some more time looking over the code to fully appreciate it, but sadly I’m out of time for this Code Review. So far they’ve put a lot of effort into the project, and judging by their code comments the project is heading in a positive direction.

In summary:

• Ship tests (where are the SocketStream tests)?
• Could we get more benchmarks, perhaps something that can be run from the command line?
• Selecting a house style for code formatting might help collaborators work together better. It’s consistent but there’s definitely a rogue tabber out there
• Is it really a good idea to load every module and pass it around the project?
• Is extending native objects in the helpers module the best approach?

Something else I thought about while writing and researching this article was this: is building something as fundamental as a framework in CoffeeScript a good idea? Please don’t beat me up about this point, I mention it for the prospect of an interesting discussion!

godzi-webgl

godzi-webgl (License: LGPL v3) by Glenn from Pelican Mapping is an SDK for building WebGL maps. The ReadyMap Web SDK supports 2D and 3D maps — the 2D maps look like the familiar maps we’ve seen on the web before, while the 3D maps are rendered on a globe that can be rotated and populated with geographical data.

There’s a list of ReadyMap SDK demos that includes many examples of the possibilities the current implementation offers.

The company’s website at pelicanmapping.com has more details behind the SDK and associated services:

Pelican Mapping is the creator and maintainer of the osgEarth and ReadyMap SDK open source projects. These are powerful, “on-demand” 3D terrain engines capable of providing an interactive, 3D globe populated with a variety of local or web-based data sources.

paper.js

paper.js (GitHub: paperjs / paper.js, License) by Juerg Lehni and Jonathan Puckey is a “vector graphics scripting framework” that uses Canvas. It’s designed to be mostly compatible with Scriptographer, which is a scripting environment for Adobe Illustrator.

It comes packed with features:

• Simple scene graph
• Paths and segments
• Mouse and keyboard handling
• Bitmap colour processing
• Selection outlines
• Vector geometry abstraction
• Mathematical operations

One thing I like about the API is the ability to use arrays or simple objects:

var rect = new Rectangle({ x: 10, y: 20, width: 100, height: 200 });
console.log(rect); // { x: 10, y: 20, width: 100, height: 200 }

// Define the size as an array containing [width, height]:
rect.size = [200, 300];
console.log(rect); // { x: 10, y: 20, width: 200, height: 300 }

There’s a good set of paper.js tutorials which cover everything from paths and geometry to animations and rasters.

WebGL Tron Tank on Liquid Galaxy WebGL Demo

SceneJS Tron Tank is a WebGL demo from SceneJS, except it’s running on 5 tiled screens:

This is a hack to see if WebSocket/WebGL is viable for achieving view synchronisation on immersive multi-machine visualisation rigs. The test rig is a 5 screen curved tile-wall at the University of Western Sydney based on Google’s Liquid Galaxy.

Last week I demonstrated a simple event class implementation. Some readers pointed out weaknesses in that implementation, so I’ve written fixes and regression tests.

Listener Removal

Richard pointed out that listeners that remove themselves might cause unexpected side effects:

Take care when removing listeners! If the listener itself calls remove, all the listeners added after it are moved up one in the array. You may cause one of the elements to not be called. i.e., the element that gets moved into the position of the removed listener.

To put it concisely: removing listeners mutates the same array that’s used to fire events.

If you’re still puzzled, consider this test:

'test listener removal in Emitter': function() {
  var Emitter = turing.events.Emitter,
      emitter = new Emitter(),
      i = 0,
      j = 0;
  
  function add() {
    i++;
    assert.ok(emitter.removeListener('add', add));
  }

  // This should end up being called twice
  function add2() {
    j++;
  }

  emitter.on('add', add);
  emitter.on('add', add2);

  assert.ok(emitter.emit('add'));
  assert.ok(!emitter.removeListener('add', add));
  assert.ok(emitter.emit('add'));
  assert.equal(2, j);
}

There are two counters that get incremented when the same event, 'add', fires: i and j. The add listener removes itself, so it should only fire once. I test this by making sure it can’t be removed after it fires:

assert.ok(!emitter.removeListener('add', add));

Then I fire the same event again, which should still trigger add2. Before fixing this, the test will display an error like this:

✕ Assertion failed in: test listener removal in Emitter
AssertionError: 1 == 2

But how do we fix it? Well, the clue is in my description of the problem: the underlying event listener array is being mutated. The method that calls listeners, emit, needs to work with a copy of the array.

How do we copy an array? An easy way is to use Array.prototype.slice, and this is what Node’s events.js code does.

The fixed code doesn’t look too different from the original:

emit: function(eventName) {
  var fired = false;
  if (eventName in this.events === false) return fired;

  var list = this.events[eventName].slice();

  for (var i = 0; i < list.length; i++) {
    list[i].apply(this, Array.prototype.slice.call(arguments, 1));
    fired = true;
  }
  
  return fired;
}

Deleting Items from Arrays

Cameron Knight had this to say:

For your array remove, you should just use array.splice(i, 1); If you read the comments on Resig’s blog, even he makes this conclusion.

I’ve been avoiding splice() for a long time because I once hit a browser support issue with it. Unfortunately, I’ve forgotten the exact reasons why. I decided to forge ahead and replace the complicated removeListenerAt method:

// Before:
removeListenerAt: function(eventName, i) {
  var array = this.events[eventName],
      rest = array.slice(i + 1);
  array.length = i;
  array.push.apply(array, rest);
  this.events[eventName] = array;
}

// After:
removeListenerAt: function(eventName, i) {
  this.events[eventName].splice(i, 1);
}

I tested this in recent versions of Chrome, Firefox, and IE6 and above, and it worked well. Whatever made me avoid splice seems invalid now.

Conclusion

Thanks for your comments, this event handling code is definitely looking a lot sharper now!

The latest commit was 1a41b80.

New Node Branding

In Evolving the Node.js Brand, Emily Tanaka-Delgado of Joyent discusses the reasoning behind the new Node website and logo. The new logo initially attracted a mixed response, but I’m sure it’ll grow on us over time. Emily writes:

To echo Node’s evolutionary nature, we have refreshed the identity to help mark an exciting time for developers.

So perhaps it’s fair to say this iteration of Node’s logo is the difficult second album?

Node Bindings for Hadoop Hive

node-hive (npm: node-hive) by Jae is a set of bindings for Hadoop Hive, a data warehouse system for Hadoop:

var hive = require('node-hive').for({ server:"hive.myserver" });

hive.fetch("SELECT * FROM my_table", function(err, data) {
  data.each(function(record) {
    console.log(record);
  });
});

Each query runs in its own connection. There’s currently no connection pooling, but the author notes that most users run a small number of long running hive queries so it should work well enough for most people for now.

node-googl

node-googl (npm: goo.gl) is a small client for shortening URLs using Google’s service:

goo.gl www.google.com
goo.gl http://goo.gl/fbsS
goo.gl www.google.com http://goo.gl/fbsS nba.com

It can be used as a module as well:

var googl = require('goo.gl');

// Shorten a long url and output the result
googl.shorten('http://www.google.com/', function (shortUrl) {
    console.log(shortUrl);
});

Cuepoint

Cuepoint (GitHub: owainlewis / Cuepoint-JS) by Owain Lewis is a small plugin for adding subtitles to HTML5 video:

cuepoint.init({ 5: 'Hello World' });
cuepoint.play();

It’s also possible to skip to points within the video, thereby displaying specific subtitles:

cuepoint.setTime(60)

Boombox.js

Boombox.js (GitHub: cgcardona / boombox.js) by Carlos Cardona is another library that makes dealing with the HTML5 audio tag easier.

It actually expects the buttons given in the author’s example, which on the one hand makes it easy to distribute themes, but on the other is a little bit restrictive. It might work well in cases where you want to quickly add HTML5 audio capabilities to a site, but for more customised UIs Buzz may be more suitable.

jqMessageBar

jqMessageBar (GitHub: mindscratch / jqMessageBar, License: MIT) by Craig Wickesser helps manage a global message bar that displays errors or information. I’m always writing my own versions of this, so I’ve been looking around for open source ones:

$('body').messagebar('addMessage', 'hello world', 'info');

Craig has even included tests with this plugin, written using the Jasmine BDD library.

I usually select code review subjects based on what I use in my own projects. However, this week Mihai Bojin sent in a project: jLL (JavaScript Library Loader). This is the first user-submitted code review — why not send in your own projects?

About jLL

jLL (License: MIT) by Mihai Bojin is a lightweight script loader. It can either load scripts asynchronously or in predetermined order.

Usage

Scripts can be loaded by pushing them onto a queue with a callback:

var deferrer = jLL.staticDeferrer();

deferrer.push('jquery-1.6.1.js', function() {
  // Done
});

deferrer.run();

It’s also possible to load when the page is ready, using deferrer.runOnLoad(). To load jQuery then several other libraries, the following pattern can be used:

var deferrer = jLL.staticDeferrer();

deferrer.push('jquery-1.6.1.js', function() {
  var asyncLoader = jLL.buildScriptDeferrer();
  
  // Push a number of scripts to be loaded in order
  asyncLoader.push([
    ['js/jquery.roundabout-1.0.min.js'],
    ['js/jquery.roundabout-shapes-1.1.js'],
    
    // after final script is loaded attach onReady event
    ['js/contact.js', function() {
      // Ready
    }]
  ]);

  asyncLoader.run();
});

deferrer.runOnLoad();

Structure

The project is distributed with the source file, a minified version, a big set of examples, and some tests. It’s really great to see tests distributed with a project like this, so I’ll take a deeper look at those later.

The main source file itself includes solid documentation, which makes it very easy to navigate the code. The self-executing anonymous function pattern is used yet again, this time passing in window and document:

(function (window, document) {
	"use strict";

  var jLL,	
		head,
		ScriptTag,
		ScriptDeferrer;

  // ... snip

	// attach an instance of the jLL object to the window
	window.jLL = jLL;
}(window, document));

Mihai has placed everything inside the anonymous function in strict mode. This is something that’s starting to crop up more and more in the client-side community.

Next, ScriptTag and ScriptDeferrer are defined as simple prototype classes. Most of the public methods are added to this inside the constructor, which means they’re “privileged” in that they can see the private members of the class:

ScriptDeferrer = function () {
  var queue,
    running,
    loadFromQueue,
    raceFromQueue;

  // ... snip

  this.push = function (src, onload, check, checkTrigger) {
    var i, 
      tag,
      allowContinue;

Now push can access queue without exposing queue as a public property. In general, Mihai is careful about what’s being exposed throughout this project.

Other parts of the library, like script tag rendering, are nicely encapsulated by suitable methods.

Tests

The tests are written with QUnit so they’ll run in a browser. The tests are organised nicely using modules, grouped according to the main classes in the project: ScriptTag and ScriptDeferrer.

A helper file is used to keep a lot of useful mocking functions outside of the main test file. Most of these are simple functions that are passed to jLL as callbacks:

var mockCheckEventTrue = function() {
  ok(true, "Mocking check and returning true");
  return true;
}

test("Check passes and loads object", 2, function() {
	var script = jLL.buildScriptTag('mock_load/mock_add_2.js');
	stop();
	script.addCheck(mockCheckEventTrue);
	script.addEvent(mockOnloadEvent);
	script.addEvent(start);
	script.render();
});

Style

I think my tabstop was set to something that made this project’s spacing look weird, so I had to tinker with that. It’s probably worth remembering that if you like to line up var declarations, then mixing tabs and spaces defeats the point of using tabs if people aren’t using the same tab spacing as you:

// What you want:
var a,
    b,
    c,;

// What someone might see with a different tabstop:
var a,
      b,
      c;

// Or:
var a,
  b,
  c;

Mihai is another fan of placing literals first in conditionals:

if ("object" !== typeof src[i] || -1 === src[i].constructor.toString().indexOf("Array")) {
}

Notice he’s also carefully using the right comparison operators.

Conclusion

This project would definitely benefit from a simple website with some easy to follow examples and automatically generated API documentation. Also, the minified version is distributed with no reproducible build script. Including a simple Makefile is a good idea for this type of project, just so the author or contributors are less likely to forget to minify the latest version.

These might seem like high-level project-management issues, but it’s worth keeping these things in mind when a project starts to mature and the forks/contributions start rolling in.

If this was my project I’d consider a simplified chained API, and removing the window onload functionality:

jLL
  .push('script1.js')
  .push('script2.js');

I’d also consider removing .run, and add a method to the chain that can handle waiting for callbacks:

jLL
  .push('script1.js')
  .wait()
  .push('script2.js');

However, that line of thinking ends up being suspiciously similar to LABjs, and I expect the author has his own reasons for making it different.

d3.js

Data-Driven Documents (GitHub: mbostock / d3, License) by Michael Bostock is a library for manipulating HTML based on data. It can be used to visualise data using HTML or SVG, and supports interactivity and animations.

This relatively simple graph example demonstrates some basic visualisation and animation possibilities. This choropleth example illustrates some of d3’s features in a more concise manner, and this Voronoi diagram is just fun to play with!

The author has written detailed documentation and tutorials for d3. It also includes a guide for people who have worked with Protovis (also by Mike Bostock).

log.io

Log.io (GitHub: NarrativeScience / Log.io, License: Apache 2.0) by Mike Smathers and Narrative Science is log viewer written with Node. It can be used to monitor logs from multiple machines, generating data effectively instantly using inotify then pushing them to the browser with socket.io.

There’s a live demo running on logio.org:8998.

Although the authors have a lot planned, the project is already formidable. Best of all, it’s fast:

Log.io has no persistence layer. Harvesters are informed of file changes via inotify, and log messages hop from harvester to server to web client via socket.io. The result is fast; internal benchmarks have exceeded 5000 messages/sec.

Buzz

Buzz (GitHub: jaysalvat / buzz, License: MIT) by Jay Salvat is a library that makes working with the audio element easier. It has a chainable API:

var mySound = new buzz.sound( '/sounds/myfile', {
  formats: [ 'ogg', 'mp3', 'acc' ]
});

mySound.play()
  .fadeIn()
  .loop()
  .bind('timeupdate', function() {
    var timer = buzz.toTimer(this.getTime());
    document.getElementById('timer').innerHTML = timer;
  });

Recently I’ve been working with HTML5 audio and it’s not exactly as straightforward as it should be. Buzz looks like it does some of the things I found awkward — for some reason most browsers seem to say maybe for canPlayType('/audio/aac') which confused me when I knew they wanted vorbis. Buzz makes this easier by accepting groups of files for each supported format then testing to see which the browser can play, using the right jiggery pokery behind the scenes.

Last week I looked at how Node supports event handling with EventEmitter. This tutorial explores building a simple event manager that can be used for custom event handling.

Event Emitter API

This is the API I settled on in the previous part:

• addListener(event, fn), aliased as on: Register an event handler
• emit(event, [arg1], [arg2], [...]): Emit an event with optional arguments
• removeListener(event, fn): Remove the handler
• removeAllListeners(event): Remove all handlers for the specified event

A skeleton prototype class will look like this:

  Emitter = function() {
    this.events = {};
  };

  Emitter.prototype = {
    /**
     * Adds a listener.  Multiple can be added per eventName.  Aliased as `on`.
     *
     * @param {String} eventName The name of the event
     * @param {Function} handler A callback
     */
    addListener: function(eventName, handler) {
    },

    /**
     * Triggers all matching listeners.
     *
     * @param {String} eventName The name of the event
     * @returns {Boolean} `true` if an event fired
     */
    emit: function(eventName) {
    },

    /**
     * Removes all matching listeners.
     *
     * @param {String} eventName The name of the event
     * @returns {Boolean} `true` if an event was removed
     */
    removeAllListeners: function(eventName) {
    },

    /**
     * Removes a listener based on the handler function.
     *
     * @param {String} eventName The name of the event
     * @param {Function} handler The handler function to remove
     * @returns {Boolean} `true` if an event was removed
     */
    removeListener: function(eventName, handler) {
    }
  };

  Emitter.prototype.on = Emitter.prototype.addListener;

I’ve added code comments in at this stage because I wanted to think about the return values and parameters. I wasn’t sure if most of these methods should return anything at all, but I decided returning some kind of status would make testing a lot easier.

Unit Test

Adding listeners and emitting events can be tested together:

exports.testEventEmitter = {
  'test addListener': function() {
    var Emitter = turing.events.Emitter,
        emitter = new Emitter(),
        i = 0;
    
    emitter.on('eventName', function() {
      i++;
    });

    emitter.on('testFired', function() {
      assert.equal(i, 1);
    });

    assert.ok(emitter.emit('eventName'));
    assert.ok(emitter.emit('testFired'));
  },

A counter is used to determine if an event has fired, and each method’s return value is tested.

Removing all listeners is another simple case:

  'test removeAllListeners': function() {
    var Emitter = turing.events.Emitter,
        emitter = new Emitter();
    
    emitter.on('event1', function() {});
    emitter.on('event2', function() {});

    emitter.removeAllListeners('event1');
  
    assert.ok(!emitter.emit('event1'));
    assert.ok(emitter.emit('event2'));
  },

Removing an event then trying to emit returns false, which helps make these tests fairly readable.

Removing a specific listener is a little bit more complicated. Multiple handlers can be added per event name, so we need to ensure this is managed correctly. You’ll also notice I use functions instead of anonymous functions for these tests, that’s just how removeListener has to work.

  'test removeListener': function() {
    var Emitter = turing.events.Emitter,
        emitter = new Emitter(),
        i = 0;
    
    function add() {
      i++;
    }

    function nothing() {
    }

    emitter.on('add', add);
    emitter.on('add', nothing);
    emitter.on('testFired', function() {
      assert.equal(i, 1);
    });

    assert.ok(emitter.emit('add'));
    assert.ok(emitter.removeListener('add', add));
    assert.ok(emitter.emit('add'));
    assert.ok(emitter.removeListener('add', nothing));
    assert.ok(!emitter.removeListener('add', nothing));
    assert.ok(!emitter.emit('add'));
    assert.ok(emitter.emit('testFired'));
  }
};

Notice how the boolean return values are useful here as well.

Implementation

I’ve removed the comments from these examples to make this more readable in this tutorial.

Adding a listener is pretty easy. We have a simple JavaScript Object containing events indexed by name. Each item is an array of event handler functions.

Emitter.prototype = {
  addListener: function(eventName, handler) {
    if (eventName in this.events === false)
      this.events[eventName] = [];

    this.events[eventName].push(handler);
  },

If you’re confused about the eventName in this.events syntax it’s just a simple test to see if the property is present in the internal list of events. Just think about it like this:

'a' in { a: 1, b: 2 }
// true

'c' in { a: 1, b: 2 }
// false

Emitting an event uses this to avoid adding an extra layer of indentation, and then loops through each matching handler and runs it:

  emit: function(eventName) {
    var fired = false;
    if (eventName in this.events === false) return fired;

    for (var i = 0; i < this.events[eventName].length; i++) {
      this.events[eventName][i].apply(this, Array.prototype.slice.call(arguments, 1));
      fired = true;
    }
    return fired;
  },

Removing all listeners is fairly simple too, I just use delete to remove the property from the list of events:

  removeAllListeners: function(eventName) {
    if (eventName in this.events === false) return false;
    
    delete this.events[eventName];
    return true;
  },

Removing single events is probably the hardest part. For a start, we need to remember to pass in the original function. It’s often more convenient to use an anonymous callback for this style of API, so this usually confuses people at first.

Then the handler has to actually be removed from the array. I’d usually use slice and indexOf, but as I’m building this with browsers in mind I’ve used a helper method called removeListenerAt which is really just JavaScript Array Remove by John Resig.

This method chops an array into two parts based on the index of the item we want to remove, found using a for loop. The original array is split one position after the item we want to remove. Then the original array’s length is shortened, removing the item we don’t need. Finally, the two halves are joined back together using push.

  removeListenerAt: function(eventName, i) {
    var array = this.events[eventName],
        rest = array.slice(i + 1);
    array.length = i;
    array.push.apply(array, rest);
    this.events[eventName] = array;
  },

  removeListener: function(eventName, handler) {
    if (eventName in this.events === false) return false;

    for (var i = 0; i < this.events[eventName].length; i++) {
      if (this.events[eventName][i] == handler) {
        this.removeListenerAt(eventName, i);
        return true;
      }
    }

    return false;
  }
};

Conclusion

I hope this demonstrates how simple event handling really is. I’ve avoided some of the optimisations from EventEmitter in Node to keep the code easy to follow.

The full version of this code can be found in commit 1e9f11d of turing.js.

References

• delete
• JavaScript Array Remove
• Node’s EventEmitter

Node 0.4.9, 0.5.0

Node 0.4.9 and Node 0.5.0 have been released. The 0.4.9 updates include SSL and crypto fixes, performance improvements for util.inherits, a V8 engine update, and module loading changes to better support nested project structures.

Node 0.5.0 is where things start to get exciting for Windows users. There’s a (non-default) libuv backend to support IOCP in Windows. In terms of deprecations, only http.cat is mentioned. There’s a huge amount of fixes and improvements elsewhere, but one thing I noticed was AMD Compatibility:

Node’s modules have access to a function named define, which may be used to specify the module’s return value. This is not necessary in node programs, but is present in the node API in order to provide compatibility with module loaders that use the Asynchronous Module Definition pattern.

node-secure

node-secure (npm: node-secure) by David de Rosier is a module that protects globals from being overridden that programs may depend on for safe execution: undefined, NaN, Infinity, eval, isNaN.

This module also adds event handling to eval:

var secure = require('node-secure');
secure.on('eval', function(caller) {
  console.log('Evel executed in following function:', caller);
});

The project’s README contains a lot more examples, including David’s reasoning behind the features.

Kue

Kue (License: MIT, npm: kue) by TJ Holowaychuk and LearnBoost is a priority job queue backed by redis. Queues and jobs are created like this:

var kue = require('kue')
  , jobs = kue.createQueue();

jobs.create('email', {
    title: 'welcome email for tj'
  , to: 'tj@learnboost.com'
  , template: 'welcome-email'
}).save();

Then elsewhere they can be processed:

jobs.process('email', function(job, done){
  email(job.to, done);
});

The job object also has job.progress(frames, totalFrames) for displaying progress during long-running jobs.

Kue also comes with a little Express app for managing jobs:

var kue = require('kue');
kue.app.listen(3000);

jQuery 1.6.2

jQuery 1.6.2 has been released. This maintenance release includes fixes for attributes in IE, a memory leak, css() fixes, and a whole slew of other improvements. I was glad to see this fix for the logic of animation callbacks.

DropKick

DropKick (GitHub: JamieLottering / DropKick, License) is a new custom select box plugin. Like other related plugins, this one hides the original select element and proxies events and value changes. It comes with some styles that look nice and modern.

One major issue I’ve had with similar plugins is the time it takes to render the custom controls. This is typically very fast until a remote script or image blocks the loading of the assets required by the dropdown. I don’t see anything that specifically addresses this issue, but the DropKick demo site fades in selects when everything is ready. This is preferable to native selects that suddenly transform into custom ones.

dotjs

dotjs (GitHub: defunkt / dotjs, License) by Chris Wanstrath is a little ruby script for Mac OS and Chrome that loads JavaScript files from ~/.js. Adding a file called ~/.js/google.com.js will load custom JavaScript when viewing google.com.

Each script gets jQuery 1.6 and ~/.js/default.js will always be loaded if it’s available.

This is similar to GreaseMonkey, except it’s even easier to write scripts and test them. I think it’s likely that someone might fork the project and rewrite the Ruby part in Node, just to make sure they can.

I was recently looking for something in the source to Express, and suddenly it struck me that this would make a great candidate for a code review. Let’s take a look at how Express is put together.

About Express

Express (GitHub: visionmedia / express, License: MIT, npm: express) by TJ Holowaychuk is an extremely popular web framework. It’s fast, simple, and easy to learn, and even better — it’s actively maintained! Strolling through npm looking at web frameworks is a little bit like a graveyard of half-finished ideas, but Express has received much love from its author, contributors, and the Node community.

Usage

Express actually comes with a command-line app that can be used to generate a basic app. I usually just write my own, it’s very easy:

var express = require('express')
  , app = express.createServer();

app.get('/', function(req, res) {
  res.send('Hello from DailyJS');
});

app.listen(3000);

Structure

The main lib/express.js starts off by loading Connect. Express is built on the Connect library which provides a number of features we’d typically expect from basic web framework: http and https servers, sessions, cookies, HTTP auth, static asset management, query string parsing, and more.

The http and https servers in Express wrap the ones from Connect to provide some Express-specific features, like routing and middleware. Connect’s servers are in turn derived from Node’s.

The rest of the file loads other pieces of key functionality and exposes them using exports as required.

Moving on to lib/http.js, we can see that an Express “app” is really a little server. When a HTTPServer is instantiated, it runs app.init which sets up simple objects for things like the cache, settings, and routes, then sets defaults. Some of this default behaviour is based on the environment, so for example, the production environment gets caching enabled.

The initialisation code also heavily reuses features we typically associate with the public Express API. The use method allows Express to be extended with middleware, and it returns this so it’s chainable. In fact, chaining is an interesting point because most methods in the server code return this. I don’t usually see this being used in Express apps.

This code also shows how routing really works. The whole server is heavily dependent on the router, which is set up in app.init:

this.routes = new Router(this);

This Router object represents a collection of routes, and it also tracks URL parameters so route param pre-conditions can be handled (this is a very useful feature).

Routes

The route library is loaded like this:

var router = require('./router');

This loads router/index.js (see Folders as Modules) which contains collection-related routing methods. Routes can be added, removed, searched, and actually dispatched.

The core route finding method is Router.prototype.find. There are convenience functions for this, namely Router.prototype.lookup and Router.prototype.match. The naming makes sense here because lookup matches text and match uses regular expressions.

Results are built using a class called Collection which is based on Array, but with some custom initialization and a remove method. The body of find just runs a callback over each route using a simple for loop.

There’s also a Route class. Routes are made up of a HTTP method, path, and some additional options. The options can include a parameter for route strictness:

- strict: enable strict matching for trailing slashes

Path strings are turned into regular expressions using normalize(). This uses some nicely managed regular expression grouping to extract parameters, formats, and other URL options.

One detail about route handling is HTTP verbs are delegated out in the HTTP server like this:

methods.forEach(function(method){
  app[method] = function(path){
    if (1 == arguments.length) return this.routes.lookup(method, path);
    var args = [method].concat(toArray(arguments));
    if (!this.__usedRouter) this.use(this.router);
    return this.routes._route.apply(this.routes, args);
  }
});

During initialisation a getter is set up for this.router:

this.routes = new Router(this);
this.__defineGetter__('router', function(){
  this.__usedRouter = true;
  return self.routes.middleware;
});

Going back to lib/router/index.js:

function Router(app) {
  var self = this;
  this.app = app;
  this.routes = {};
  this.params = {};
  this._params = [];

  this.middleware = function(req, res, next){
    self._dispatch(req, res, next);
  };

We can see that self.routes.middleware will call the dispatcher.

Views

Moving on to views… lib/view/view.js provides a simple interface to views that helps manage paths and rendering engines. Partial template support is even more surprisingly simple, with lib/view/partial.js containing just one function that lib/view.js uses to determine which view to render.

Request and Response

The req and res objects, ubiquitous in Express apps, wrap around http.IncomingMessage.prototype and http.ServerResponse.prototype which ultimately comes from Node’s http module via Connect.

The res.send method does a lot of the work for generating responses. The data passed in is checked using switch (typeof body) and this.header('Content-Type'). Good old JSON.stringify is used to generate JSON responses.

Style

Throughout Express TJ places literal values first in if statements:

if ('string' != typeof route) {
  middleware = route, route = '/';
}

I assume this is because accidentally typing if ('value' = a) will cause an exception to be thrown.

Another style observation I made (beyond those I’ve already explored in TJ’s Jade project) was a heavy reliance on native functionality. I expected to see a fairly heavy route collection management class, but TJ has simply built on Array.prototype.

This is also another project that includes lots of well formatted code comments, so it’s easy to understand the author’s intent in the more abstract corners of the project.

Conclusion

I hope this tour of the Express source has given you a feel of how it’s put together, and also what it takes to build a web framework with Node.

I invite you to take a look around yourself! If you’re looking for help when developing your own Node apps, Express contains a lot of pointers. It’s also worth looking at how these underlying mechanisms work if you’re convinced you’ve exhausted the Express Guide documentation.

pdf.js: Rendering PDF with HTML5 and JavaScript

In pdf.js: Rendering PDF with HTML5 and JavaScript Andreas Gal discusses a project to determine if HTML5 is complete enough to render PDFs. The GitHub pdf.js project contains progress so far, and it looks like Andreas is updating it regularly.

There’s also another related post, Overview of pdf.js guts, which explains how it works:

pdf.js (currently) parses raw arrays of bytes into streams of PDF “bytecode”, compiles the bytecode into JavaScript programs, then executes the programs. Yes, it’s a sort of PDF JIT. The side effect of those programs is to draw on an HTML5 Canvas.

It sounds like a fascinating project, particularly the PDF stream parsing.

VisualSearch.js

VisualSearch.js (GitHub: documentcloud / visualsearch, License) from DocumentCloud is a novel take on search design. It allows field-level options to be specified using dynamic inline completion, using simple icons and drop-down menus.

Voice Debugger

Voice Debugger (GitHub: ferrante / Voice-Debugger) by Damian Wielgosik is a little HTML5 experiment to catch JavaScript errors and read them out. A set of standard errors are saved as WAV files, and then played with an audio tag.

I’ve been doing some pretty heavy HTML audio work lately, so I found this extremely amusing…

Way back at the start of this series I created a basic DOM events implementation, in Let’s Make a Framework: Events. Dealing with browser events is mostly a case of providing a simple API and patching browser quirks. As we saw, various frameworks implement this differently. In particular, jQuery has an almost completely custom event handling system, which allows it to support lots of useful features in almost every browser.

When building modern web applications we often need to deal with events that don’t apply to DOM elements. Sometimes we create abstract objects and want to emit or listen for events. This is becoming more common with the growing popularity of Node and its EventEmitter class.

Let’s add something like EventEmitter to Turing, so developers can make use of customised event handling for their own objects.

EventEmitter

The EventEmitter class can be found in lib/events.js in Node and there’s also documentation in events.EventEmitter.

It’s actually very easy to use EventEmitter. I think a lot of people find it confusing at first, because they hear about “evented JavaScript” and think Node’s little EventEmitter is a magical tool that makes everything fast.

In reality, it’s basically this:

var EventEmitter = require('events').EventEmitter,
    bomb = new EventEmitter();

bomb.on('explode', function() {
  console.log('BOOM!');
});

bomb.emit('explode');

This is similar to what CommonJS Events/A describes.

Structure

In EventEmitter, event handlers are stored in an object that contains arrays of events indexed by the event names. Some additional complexity comes from optimisations. For example, one optimisation is for the case where there’s only one listener:

EventEmitter.prototype.addListener = function(type, listener) {
  // ...
  if (!this._events[type]) {
    // Optimize the case of one listener. Don't need the extra array object.
    this._events[type] = listener;
  } else if (isArray(this._events[type])) {

    // If we've already got an array, just append.
    this._events[type].push(listener);

Then, when an event is emitted:

EventEmitter.prototype.emit = function(type) {
  if (!this._events) return false;
  var handler = this._events[type];
  if (!handler) return false;

  if (typeof handler == 'function') {
    // The optimised case
    switch (arguments.length) {
      // fast cases
      case 1:
        handler.call(this);
      // ... snip

  } else if (isArray(handler)) {
    // An array of handlers has been added
    var listeners = handler.slice();
    for (var i = 0, l = listeners.length; i < l; i++) {
      listeners[i].apply(this, args);
    }
    return true;

API Design

Let’s follow the basics of Node’s API:

• addListener(event, fn), aliased as on: Register an event handler
• emit(event, [arg1], [arg2], [...]): Emit an event with optional arguments
• removeListener(event, fn): Remove the handler
• removeAllListeners(event): Removes all handlers for the event

These methods should provide enough functionality to do some cool stuff.

Conclusion

Whether we’re using events in DOM programming, EventEmitter in Node, or even similar techniques in other languages (I keep finding myself working with NSNotification in Objective-C), events are an incredibly useful paradigm.

Next week I’ll start building our own events class!

References

• Let’s Make a Framework: Events
• Node’s lib/events.js
• Node’s events.EventEmitter documentation.
• CommonJS Events/A

Porting Node to Windows

Building Node in Windows has gone from “use Cygwin”, to serious thought starting in liboio (renamed to libuv), and now this: Porting Node to Windows With Microsoft’s Help.

As you may have heard in a talk we gave earlier this year, we have started the undertaking of a native port to Windows – targeting the high-performance IOCP API.

There are good reasons why a serious port will be far superior to kludging builds with Cygwin, most notably IO concurrency. This initial announcement mentions Windows Azure support, Microsoft’s application hosting environment. Hopefully this means Windows developers will get great performance on Microsoft’s server platforms and on local development machines.

socketstream

socketstream (License: MIT, npm: socketstream) is a new and ambitious web framework for building single-page applications. Rather than focusing on pretty graphics, the socketstream team has focused on ridiculous performance and bandwidth efficiency, using Socket.IO, pub/sub, Redis, jQuery, Jade, Stylus, and more of our favourite libraries.

Applications are generated using a script, and the resulting template app is a little bit like a Rails project (except it starts up instantly). The template app also has user management baked in:

As almost all web applications have users which need to sign in and out, we have built the concept of a ‘current user’ into the core of SocketStream. This not only makes life easier for developers, but is vital to the correct functioning of the pub/sub system, authenticating API requests…

The examples are in CoffeeScript, but the documentation mentions using good old fashioned JavaScript is fine too.

There’s already an interesting tutorial for socketstream by Addy Osmani: Building Real-time CoffeeScript Web Applications With SocketStream.

EventEmitter2

EventEmitter2 (License: MIT, npm: eventemitter2) by hij1nx is an implementation of the EventEmitter class found in Node. It introduces lots of useful features, including namespaces and wildcards and browser compatibility.

The project comes with tests, so presumably the author has some solid benchmarks to back up his performance claim:

As good or better performance for emission and listener registration as Node.js core EventEmitter

FitText

Somehow I missed this companion project to the popular Lettering.js, but in case you also missed it… FitText (GitHub: davatron5000 / FitText.js) by Dave Rupert and Paravel resizes text to fit the parent element. It makes creating fluid magazine-like layouts possible, just by using appropriate CSS and $('#responsive_headline').fitText();.

There’s also an interesting comment in the readme about window.resize:

If you oppose window.resize(), it’s worth mentioning that @chriscoyier created a fork of FitText using a debounced resize method.

The original post about debouncing is here: Debouncing Javascript Methods.

gmap3

gmap3 (GitHub: jbdemonte / gmap3) by Jean-Baptiste Demonte makes working with Google Maps a lot simpler, leveraging jQuery’s succinctness rather nicely:

$('#example').gmap3();

It also takes parameters in the form of an array of objects. Each object has an action:

$('#example').gmap3(
  { action: 'init',
    options: {
      center: [46.578498, 2.457275],
      zoom: 5
    }
  },
  { action: 'addMarker',
    latLng: [48.8620722, 2.352047]
  },
  { action: 'addMarker',
    latLng: [48.8620722, 2.352047]
  }
);

Awesomemarkup

Awesomemarkup (GitHub: clint-tseng / awesomemarkup, License: WTFPL) by Clint Tseng is a framework-agnostic library for generating markup. When used with jQuery, it’ll add $.tag:

// Instead of doing this:
var markup = '<div class="content ' + article.contentType +
  '" style="display:' + (article.visible ? 'block' :
  'none') + '">' + '<h2>' + article.title + '</h2>' +
  '</div>';

// Awesomemarkup makes this possible:
var markup = $.tag({
  _: 'div',
  class: [ 'content', article.contentType ],
  style: {
    display: { i: article.visible, t: 'block', e: 'none' }
  },
  contents: {
    _: 'h2',
    contents: article.title
  }
});

Clint has also added CommonJS support, which means Awesomemarkup can be used in Node too.

Underscore provides functional programming tools for JavaScript. Let’s take a look inside to see how key functions are implemented.

About Underscore

Underscore (GitHub: documentcloud / underscore, License, npm: underscore) from DocumentCloud provides a rich set of functions that help when dealing with arrays and objects. Many of these functions are available in other languages, and some can even be found natively in JavaScript depending on the browser or interpreter.

Underscore’s API is accessed through an object rather than changing Array or Object. That means it’s easy to drop into a project without affecting existing code.

Usage

The canonical example of Underscore is the each function:

_.each([1, 2, 3], function(num) {
  alert(num);
});

The map function is similar, but allows the list to be changed:

_.map([1, 2, 3], function(num) {
  return num * 3;
});

// Returns [3, 6, 9]

I really like reduce:

_.reduce([1, 2, 3], function(memo, num) { return memo + num; }, 0);
// Returns 6

Underscore quickly became popular for client-side programming, but an increasing number of Node packages depend on it.

Structure

Like most JavaScript modules, Underscore is distributed in an anonymous wrapper:

(function() {
  // Establish the root object, `window` in the browser, or `global` on the server.
  var root = this;

  // Save the previous value of the `_` variable.
  var previousUnderscore = root._;

  // Establish the object that gets returned to break out of a loop iteration.
  var breaker = {};

  // Save bytes in the minified (but not gzipped) version:
  var ArrayProto = Array.prototype, ObjProto = Object.prototype, FuncProto = Function.prototype;
  
  // ... snip
})();

The author (Jeremy Ashkenas) has written detailed comments through the project.

Underscore will attempt to use native JavaScript methods where available. We see this intention straight away at around line 35:

// All **ECMAScript 5** native function implementations that we hope to use
// are declared here.
var
  nativeForEach      = ArrayProto.forEach,
  nativeMap          = ArrayProto.map,
  nativeReduce       = ArrayProto.reduce,
  nativeReduceRight  = ArrayProto.reduceRight,
  nativeFilter       = ArrayProto.filter,
  nativeEvery        = ArrayProto.every,
  nativeSome         = ArrayProto.some,
  nativeIndexOf      = ArrayProto.indexOf,
  nativeLastIndexOf  = ArrayProto.lastIndexOf,
  nativeIsArray      = Array.isArray,
  nativeKeys         = Object.keys,
  nativeBind         = FuncProto.bind;

The most important function in Underscore is each, because so many other methods are built on it. Because it delegates to ECMAScript 5’s native forEach it has to be compatible. It’s also defined in the each variable, which makes using it throughout the rest of the file more straightforward (and save bytes):

var each = _.each = _.forEach = function(obj, iterator, context) {
  if (obj == null) return;
  if (nativeForEach && obj.forEach === nativeForEach) {
    obj.forEach(iterator, context);
  } else if (_.isNumber(obj.length)) {
    for (var i = 0, l = obj.length; i < l; i++) {
      if (i in obj && iterator.call(context, obj[i], i, obj) === breaker) return;
    }
  } else {
    for (var key in obj) {
      if (hasOwnProperty.call(obj, key)) {
        if (iterator.call(context, obj[key], key, obj) === breaker) return;
      }
    }
  }
};

A guard against null values is used, then nativeForEach is tested. If this isn’t available and the object looks like an array, a simple for loop is used to iterate over each value and run the callback. If the object looks like an Object, a for... in loop is used instead.

However, because the nativeForEach && obj.forEach === nativeForEach test is used, nativeForEach will only ever be used for arrays. The Mozilla Developer Network has some guidance on implementing forEach-compatible methods, in the Array forEach documentation:

if (!Array.prototype.forEach)
{
  Array.prototype.forEach = function(fun /*, thisp */)
  {
    "use strict";

    if (this === void 0 || this === null)
      throw new TypeError();

    var t = Object(this);
    var len = t.length >>> 0;
    if (typeof fun !== "function")
      throw new TypeError();

    var thisp = arguments[1];
    for (var i = 0; i < len; i++)
    {
      if (i in t)
        fun.call(thisp, t[i], i, t);
    }
  };
}

Since forEach is defined on Array.prototype, Underscore has to provide its own functionality for objects. That means this will work as expected:

_.each({ a: '1', b: '2' }, function(a, b) { console.log(a, b) });

Building on each

The map function works in a very similar way, testing for native support then calling each if required:

_.map = function(obj, iterator, context) {
  var results = [];
  if (obj == null) return results;
  if (nativeMap && obj.map === nativeMap) return obj.map(iterator, context);
  each(obj, function(value, index, list) {
    results[results.length] = iterator.call(context, value, index, list);
  });
  return results;
};

If the native method can be used, the results are returned right away. Otherwise, the results are collected up and returned by using each.

The context parameter is used throughout Underscore. This is used by call to bind the callback. The first parameter of call is “thisArg”:

If thisArg is null or undefined, this will be the global object. Otherwise, this will be equal to Object(thisArg)

The MDN documentation on call has more information and examples.

Tests

The tests are written with QUnit, so they can be run in a browser very easily: Underscore tests.

Conclusion

Underscore’s source reads almost like JavaScript documentation for the methods it implements. And it’s nice to see that a library that can give so many productivity gains is actually simple and easy to follow.

Do you fancy building your own functional programming library? Start with each and see how far you can get!

Web developers, like all developers, need good 3D support, and — as with Flash and Silverlight — browser implementers will need to be careful and thoughtful about how to expose that functionality securely on all operating systems.

- Mike Shaver

Last week, Microsoft voiced concerns over WebGL security on the Microsoft Security Research & Defense blog. Since then prominent Mozilla and Microsoft employees have joined the discussion. The press tried to make this sound like an exciting backlash from Mozilla, but from what I’ve read the response was well-tempered.

Avi Bar-Zeev had this to say, in Why Microsoft and Internet Explorer need WebGL:

Operating systems and security mitigation are what Microsoft is known for. It’s our bread and butter. Why would we run away from that challenge with such an alarmist attitude of “shut it off, shut it off, it might hurt me!”

And:

WebGL will be running on my PC and yours, one way or another. Microsoft will need to deal with it.

Elsewhere, Mozilla VP of Engineering Mike Shaver wrote A Three-Dimensional Platform, in which he states:

Microsoft’s concern that a technology be able to pass their security review process is reasonable, and similar matters were the subject of a large proportion of the discussions leading to WebGL’s standardization; I also suspect that whatever hardening they applied to the low-level D3D API wrapped by Silverlight 3D can be applied to a Microsoft WebGL implementation as well.

However, the Mozilla Security Blog does contain this interesting bug report: WebGL graphics memory stealing issue

This issue allows attackers to capture screen shots of private or confidential information.

Although deliciously ironic, this is based on further research by James Forshaw. He wrote the post that was the epicentre for the WebGL security debate: WebGL – A New Dimension for Browser Exploitation.

Over the last three weeks I’ve been looking at how web frameworks implement promises, and I’ve built a Promise class along the way.

• Promises: Part 1
• Promises: Part 2
• Promises: Part 3

Ajax Integration

After researching promises, jQuery’s API stood out as a solid example of using them to do common tasks. In particular, jQuery.ajax is Deferred-compatible, which means it transparently provides promise-related methods:

$.get('/example').then(
  function() { alert('Success'); },
  function() { alert('Failure'); }
);

We should be able to recreate this with our turing.Promise class, even though it’s extremely simple. There are a few things to keep in mind:

• When implementing functions that support turing.Promise, they should return an object with a method that has the same signature as then(success, failure)
• I designed Turing to be modular, so turing.Promise may not be available
• The callee will return a promise that can be set with success/failure functions that are subsequently run later inside the original callee. It’s not as mind-bending as it sounds!

Tests

I wrote a little Express app to test turing.net. I’ve updated it to work with Express 2.×. This should provide a useful test for promises:

'test promises': function() {
  $t.get('/get-test').then(
    function(r) { assert.equal('{"key":"value"}', r.responseText); },
    function(r) { assert.ok(false); }
  );
}

The value in the assertion, '{"key":"value"}', is just something I set the Express app to return. These tests can be run by changing directory to test/functional then running node ajax.js and visiting http://localhost:3000.

Implementation

The beginning of the net module’s internal ajax function needs to be changed to instantiate a Promise if it’s available:

function ajax(url, options) {
  var request = xhr(),
      promise;
      
  if (turing.Promise) {
    promise = new turing.Promise();
  }

Now we’ve got one, what do we do with it? Well, we need to make it accessible to the outside by changing whatever ajax() returns to include a then method. This is the bare minimum.

This function returns an XMLHttpRequest object (or IE’s various equivalents). I just added a then method to this, but it would probably be better to return something else and keep the request object around internally.

request.then = function() {
  if (promise) promise.then.apply(promise, arguments);
};

I made then just defer to the current promise object, and used apply to call it with arguments. If any beginners find this puzzling, give arguments and apply a read on MDN. These language features are extremely useful for making flexible APIs.

The final thing to do is actually call the success or failure functions set up by then. They just need to go where the original callbacks get executed. Recall that our Promise class uses resolve(value) and reject(value) to handle the resolution of a promise:

  function respondToReadyState(readyState) {
    if (request.readyState == 4) {
      if (request.getResponseHeader('content-type') === 'application/json')
        request.responseJSON = net.parseJSON(request.responseText);

      if (successfulRequest(request)) {
        if (options.success) options.success(request);

        // HERE:
        if (promise) promise.resolve(request);
      } else {
        if (options.error) options.error(request);

        // HERE:
        if (promise) promise.reject(request);
      }
    }
  }

Conclusion

That wasn’t too painful, was it? I think there may be a good argument for making Promise part of Turing’s core module, particularly as it’s so short and useful. By following this example, promises could be used throughout the framework to enhance the API.

The latest code update was commit 0d0bfac.