Milkshake

Milkshake (GitHub: gattis / milkshake, License: LGPL) by Matt Gattis is a WebGL audio visualiser that uses SoundCloud’s API to play audio. It’ll play lots of tracks and cycle through tonnes of visualisations based on MilkDrop.

I particularly like the way the visualiser presets are JSON files, and how everything is client-side — this thing can be uploaded to GitHub Pages or Dropbox and shared easily.

The choice of tracks really made me want to get some Girl Talk out (Feed the Animals is a great way to start the weekend!)

An Introduction to WebGL

In An Introduction to WebGL, Luz Caballero explains the basics behind WebGL, including: browser support, the rendering pipeline, WebGL libraries, and example code using PhiloGL.

MapsGL

MapsGL from Google is a new way to view Google Maps using WebGL. To use it, load up Google Maps in the latest Chrome or Firefox 8+ and look for the “What to try something new?” button on the bottom left-hand-side:

MapsGL uses new technology called WebGL (Web-based Graphics Library) to enhance the Google Maps experience. WebGL brings 3D graphics to your browser without the need to install additional software. This allows us to provide seamless transitions between various levels of imagery and different map views.

I really like the 45 degree view mode, but I could only get it working around Rome for some reason. It’s also slightly weird that when MapsGL is enabled the Google Earth option still requires a plugin. However, it does feel extremely slick, particularly zooming right in which automatically transitions to street view mode.

Previous parts:

• Part 1: Introduction, library review
• Part 2: Script Insertion

HTML5 Asynchronous Loading Support

In the last part I created this simple API for loading scripts asynchronously:

$t.require('/load-me.js', function() {
  // Loaded
});

The next step is to look at how to handle execution order control. The script element gets two new attributes in HTML5: async and defer. Technically defer was present in HTML4:

When set, this boolean attribute provides a hint to the user agent that the script is not going to generate any document content (e.g., no “document.write” in javascript) and thus, the user agent can continue parsing and rendering.

From: HTML4 Scripts

That means there are the following possible states:

• async: Execute the script asynchronously as soon as it is available
• defer: Execute the script when the page has finished parsing, thereby allowing other scripts to download and execute
• async and defer: Use async if available, else legacy browsers will fall back to defer (from: Using HTML 5 for performance improvements)
• If neither are present, the script is fetched and executed immediately before the page has finished parsing

Out of interest, I tried setting these attributes on the generated script elements, and it didn’t cause IE6 to break. I suspect this should really use feature detection to be safe.

In order to support the baseline HTML5 API, I added some options to the require method:

$t.require('/load-me.js', { async: true, defer: true }, function() {
  assert.equal(loadMeDone, 1);
});

And this just sets the properties as you’d expect:

function require(scriptSrc, options, fn) {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src = scriptSrc;

  if (options.async) {
    script.async = options.async;
  }

  if (options.defer) {
    script.defer = options.defer;
  }

Execution Order and Preloading

Now require works as asynchronously as possible, have we solved the problem of asynchronous resource loading? Not by a long shot. Although execution order could now be controlled through callbacks, this wouldn’t do what we really want to do:

// This is wrong:
$t.require('/library-a.js', function() {
  $t.require('/library-b.js', function() {
    $t.require('/app-code.js', function() {
      // Done!
    });
  });
});

Why is this wrong? It fails to distinguish between loading and executing scripts. Libraries like LABjs utilise multiple strategies for managing script execution order. Going back to Loading Scripts Without Blocking by Steve Souders, we can find six techniques for downloading scripts without blocking. Steve even includes a table to compare the properties of each of these approaches, and it shows that only three techniques can be used to control execution order.

There are more techniques available — many libraries use object or img to preload scripts. A script element is used once the script has loaded, effectively rerequesting the script when it’s required, and therefore hitting the cache. This preloading approach is fairly widely used, but has to account for a lot of browser quirks. In some cases it can cause scripts to be loaded twice.

XMLHttpRequest Loading

LABjs has a whole load of code for managing loading scripts with XMLHttpRequest. Why? Well, it makes preloading possible and avoids loading scripts twice. However, it can only be used to load local scripts due to the same origin policy.

Dynamic Script Execution Order

In Dynamic Script Execution Order, an extension to the behaviour of the async attribute is considered. This proposal is known as async=false:

If a parser-inserted script element has the `async` attribute present, but its value is exactly “false” (or any capitalization thereof), the script element should behave EXACTLY as if no `async` attribute were present.

This would allow our script loader to set scriptTag.async = false when execution order is important, else scriptTag.async = true could be used to load it and run it whenever possible.

Conclusion

Despite script loading libraries like LABjs and RequireJS existing for a few years, the problem still hasn’t been completely solved, and we still need to support legacy browsers. Simply loading non-blocking JavaScript by inserting script tags is possible, but controlling execution order requires an inordinate amount of effort.

If you’ve ever wondered why LABjs is around 500 lines of code, then I hope you can now appreciate the lengths the author has gone to!

The HTML5 support I added can be found in commit c007251.

References

• HTML4 Script Tags, defer
• Using HTML 5 for performance improvements
• Comments on ControlJS part 1: async loading
• Dynamic Script Execution Order

Node Cluster

I saw a Tweet in which Ryan Dahl mentioned this commit to Node: introduce node cluster, 87339a2. It adds a command line option to run a script using multiple processes. The obvious criticism of this is it’s possible to build this at a higher level, but I agree with Daniel Ennis’ comment on the commit:

While it may seem high level, I’d much rather have an officially supported method of doing it that’s maintained […]

Node 0.5.9 and 0.6

Node 0.5.9 has been released, and Node 0.6 is getting close. This means 0.6 will be the new stable branch, and Ryan recommended testing modules against 0.5.9.

I apologize that it’s taken so long to reach a new stable release – porting to Windows took longer than expected. In the future we will try to keep dev/stable cycles to 6 weeks.

I thought this was a very humble statement given the herculean effort that’s been put into Windows support. I’m looking forward to 0.6!

More French Node Blogs

I recently featured a French Node blog in the Node Roundup, and reader Sebastien Chopin sent in two more:

• Sebastien Chopin’s Node Blog — covers topics suitable for beginners, and other JavaScript tricks
• Nicolas Chambrier’s Node Blog — suitable for more advanced programmers

Notificon

Notificon (GitHub: makeable / Notificon, License: BSD) by Matt Williams dynamically changes a site’s favicon to include a notification value. This works by generating link elements with data URIs using a canvas:

var changeFavicon = function changeFavicon(canvas) {
  var link = document.createElement('link');
  link.type = 'image/x-icon';
  link.rel = 'icon notificon';
  link.href = canvas.toDataURL("image/png");
  removeNotificon();
  document.getElementsByTagName('head')[0].appendChild(link);
};

It’s a cool hack, and it made me wonder about Internet Explorer support. IE8 apparently supports link elements with base64 encoded data, so perhaps something like ExplorerCanvas could be used to get broader browser support?

stickySectionHeaders

stickySectionHeaders (GitHub: polarblau / stickySectionHeaders, License: GPL and MIT) by Polarblau is a jQuery plugin that can help create iOS-style sticky headers in a list. All that’s needed is a simple call:

$('#sticky-list').stickySectionHeaders({
  stickyClass     : 'sticky',
  headlineSelector: 'strong'
});

The author has provided some iOS-style CSS as well.

jQuery.suggest

jQuery.suggest (GitHub: polarblau / suggest, License: GPL and MIT) also by Polarblau is an autocomplete plugin that works a little bit like native elements, with greyed out text.

While it’s true there’s a lot of autocomplete plugins out there, how many of them are written this cleanly and come with tests?

smartTruncation

Finally, smartTruncation (GitHub: polarblau / smarttruncation, License: GPL and MIT) is another plugin by Polarblau. This one truncates text within its parent element. It can also truncate from the centre: “Hello World” becomes “Hel…rld”.

Text truncation in client-side code is a source of fascination for me because I’ve attempted it a few times without a huge amount of success. Polarblau’s algorithm creates a cache of font size values and then adds characters to an element until it has filled the desired size. It also takes file name extensions into account.

It’s pretty easy to use:

$('.files li').smartTruncation({
  'protectExtensions' : true // "myimagefile.jpg" -> "myimagef...jpg"
});

It’s worth noting that each of these plugins by Polarblau is clearly licensed, includes tests, and has its own project web site (hosted by GitHub). This is how you distribute plugins!

Mozilla Secure Coding Guidelines

Mozilla’s WebAppSec/Secure Coding Guidelines is a set of coding guidelines for developing secure applications. There’s a lot information about securing application layer communications, but there’s also some JavaScript-specific advice. JavaScript input validation is considered, along with preventing XSS attacks, and uploads as a JavaScript-based XSS attack vector.

Mozilla also introduced Aurora 9 recently, which includes a JavaScript interface for Do Not Track, and the addition of type inference.

Raphaël 2.0

Dmitry Baranovskiy has released Raphaël 2.0 (GitHub: DmitryBaranovskiy / raphael). Dmitry wrote a post on February 10th about the planned features for Raphaël 2.0. The GitHub history indicates that this version has a new VRML version, and the project has been split up into three files: raphael.svg.js, raphael.vml.js, and raphael.core.js.

If you want to figure out the other changes, either look through Raphaël’s documentation or try to read more of the history on GitHub.

cryptico.js

cryptico.js (Google Code: cryptico, License: New BSD License) is a public key cryptography library that can generate RSA key pairs, encrypt and decrypt messages.

Keys can be generated with cryptico.generateRSAKey(passPhrase, 1024), and messages can be encrypted with cryptico.encrypt(message, publicKeyString).

The cryptico documentation includes notes on the library’s implementation:

A hash is generated of the user’s passphrase using the SHA256 algorithm found at webtoolkit.info. This hash is used to seed David Bau’s seedable random number generator. A (seeded) random RSA key is generated with Tom Wu’s RSA key generator with 3 as a hard-coded public exponent.

W3C Game Development Course

W3C recently announced a new online course, Game Development in HTML5, all about creating browser-based multiplayer games. The course aims to review the state of JavaScript game development, explore how new HTML5 elements can be used for building games, and even covers deploying games to app stores.

The tickets are €225, but there are a limited number of seats available at the early bird rate of €145. Registration is here: W3C Training: Game Development in HTML5.

dyn.js

dyn.js (GitHub: dynjs / dyn.js) is an invokedynamic-based JavaScript implementation. Why is this interesting and potentially big news? Well, this quote from an interview with Douglas Campos sums it up:

Rhino is an truly piece of art – back when Rhino was born, the JVM was not cool as today, so they really made miracles getting performance by hand-crafting almost all aspects of their runtime. So dyn.js is kinda like we were starting Rhino again in 2011 – with a much powerful JVM and not worrying about backward compatibility.

If you’d like to read more about this subject, Charles Oliver Nutter wrote a phenomenal explanation back in 2008 called A First Taste of InvokeDynamic.

ShapeWright

ShapeWright is an experimental “mass customization” platform that uses WebGL for visualisation. One of the experiments, Ship generates 3D spaceships and allows VRML object downloads and purchasing of 3D prints.

Is there really not a JavaScript/WebGL multiplayer Elite clone yet?

JavaScript is weird. There’s the obvious weirdness of prototypal inheritance, == vs. ===; but those are just surface issues. One feature that boggles non-JavaScript programmers is space optimizations. JavaScript is unique in this fact. Much of our insane syntax flows from the fact that the code goes over the wire. Space saving optimizations are not just acceptable — they are encouraged.

Think about that for a moment. How long has it been since programmers have had to worry about the size of their code? Front-end developers do this all the time, but to anyone else it’s crazy.

At Yahoo I got really good at space optimizations. There were a bundle of internal documents you could turn to that would offer tricks, and rules for making your code smaller. The importance placed on these syntax rules turned code reviews into a checking off process. Every time we passed the list we patted ourselves on the back. We thought it was the best use of our time, and we plodded on, writing the same code over and over. It was prettier, smaller, and well documented, but there wasn’t a ton of original thought that went into it. At least on my part there wasn’t. I am embarrassed to say this now, but I felt like I was at my JavaScript zenith during this time. Oh man, was I wrong.

I don’t think like that anymore. My rubric for good code is how well others can readily understand it. I no longer consider something useful until someone else has successfully modified it. And still, the finish line keeps speeding away from me. There is always a way to be better.

When I moved from a large place like Yahoo, where I rarely had other kinds of engineers looking at my code, my code habits had to change. Other kinds of engineers — especially more polyglot engineers — were looking at my code. I found I had to explain myself constantly.

Their acceptance of my crazily optimised code was tepid. Over time, the craziness started to wear on me. Every time another programmer had to interact with my code I had to explain everything to them. I felt like a baboon pounding my chest then leaning over and saying “see, it’s simple, right?” — it wasn’t! And I had to admit, the blame did not lie with the language or the environment. I am sure that I had become lax in my syntax, and my code wasn’t as clean as it used to be. When there is no one to call you on your habits, you slide a little.

I was the only front-end developer for a while, but even that wasn’t the worst part. The biggest failure was forcing other kinds of engineers to accept the craziness, and not taking their code habits into consideration. I thought that was how it has always been done, and will forever be done. My coworkers enlightened me. In a small but diverse team, shared code habits might be the most important key to understandable code.

With shared code habits in mind, we set about mixing and matching a front-end system that was more understandable to everyone. I had to learn how others would see the front-end system. We went about encapsulating things in a way that would make more sense to others. Some engineers think of JavaScript as something that needs to disappear, like in GWT, but our reformulation was more like a blending of technology that kept the good parts of JavaScript.

During the transition I began to realize the reasons other programmers reel away from front-end code. Engineers who are fully capable of writing code for the web will be turned off by the syntax. It might even stop them from writing web code. Then, even if they do write some code, they switch off their analytical brains when using something they don’t like. They’ll just do what they need to do to get the job done. You won’t get them to help you make the code better. In the end you took someone who could have been a net positive on code quality, and created a net negative on code quality. And for what; smaller code?

One of the best things we did was to switch to intermediaries. Uglify.js is a great example of an intermediary code processor. We have begun to use a number of these tools. Basically, you can use something like Uglify.js or CoffeScript to bridge the gap of understanding. You can turn something that people found ugly, into something beautiful. And then you start to switch the editorial part of their brain on.

None of this means that I don’t care about optimizing code, or keeping things small. This is important for everyone, and especially so for high-traffic websites like Yahoo. It is not a means to an end. It’s that 90% of my checklist has been replaced by code compressors like Uglify.js. I don’t know what I would have done at Yahoo code reviews if we started using Uglify.js. It would have been a lot of latte sipping, and crying as the company crumbled around me.

We should have used better tools, and spent more of our time learning better ways to code.

An Easy Way to Build Scalable Network Programs

An Easy Way to Build Scalable Network Programs is a short essay by Ryan Dahl, that helps address some recent criticism of Node:

Node has a clear purpose: provide an easy way to build scalable network programs. It is not a tool for every problem. Do not write a ray tracer with Node. Do not write a web browser with Node. Do however reach for Node if tasked with writing a DNS server, DHCP server, or even a video encoding server.

I’ve noticed several high-profile Node developers say similar things recently, even before the community at large was attacked by certain readers of sites like Hacker News and Reddit. What summed it up for me was this tweet by TJ Holowaychuk:

I love how rails people keep coming to node, expecting a rails-like framework. Why not just use…. rails?

It’s nice to have options, and we have enough high quality high-level programming languages and libraries to keep everyone happy. Be nice!

tap

tap (License: MIT, npm: tap) by Isaac Z. Schlueter is essentially a collection of packages that forms a TAP-compliant test framework. Presumably the reason Isaac is interested in this is to encourage Node developers to generate machine-consumable test output for use with npm’s npat option.

Using the tap package as a test framework looks a lot like other Node test frameworks — there’s an object passed to each test that can be used to run assertions and call a method that denotes the test has finished (useful for asynchronous testing). I noticed that it uses a slightly different approach to the CommonJS Unit Testing module, but this might be because Isaac’s examples demonstrate testing inline rather than through a test runner.

The best thing about tap is the way it’s been split up into lots of smaller packages. There are currently 8 packages, and each one is based around either generating or consuming TAP streams. That means it’s possible to generate TAP output from existing frameworks, or they could even be sewn together in new ways to form your own Frankenstein’s monster of test frameworks.

Gittyup

Gittyup (npm: gittyup) by Craig Condon is a deployment library aimed at Node. It supports rollback, testing before deployment, and making slugs of apps. It’s based around JavaScript files rather than a configuration language or DSL:

// Deploy from a git repository
gittyup.app('myApp').checkout('git@github.com:crcn/gittyup-test.git', function(err, result) {
});

// Or from a local path
gittyup.app('myApp').checkout('/some/local/path', function(err, result) {

});

jQuery 1.7 Beta 1

jQuery 1.7 beta 1 has been released, and this version looks significant with a new events API and major underlying event code changes to improve Internet Explorer support.

The new events API is pretty simple on the surface: $.on and $.off have been added, which attach or remove events. However, the intention behind this is to effectively combine the existing APIs for .bind, .live. and .delegate. The rationale behind this is to remove some “surprising interactions” caused when mixing bind and live events. This is fully explained on the jQuery blog’s 1.7 announcement.

Sensible DateTime

Sensible DateTime (GitHub: crossbreeze / jquery-sensible-datetime, License: MIT) by Jaewoong Kim is a plugin for formatting ISO times. Formatting strings are supported, as well as natural language relative times — for example, “2 minutes ago” rather than a date and time.

To use it, call $('.datetime').sensible() on an element with either a datetime or title attribute, and make sure it contains a ISO8601 datetime string.

The last time I solved this problem I’m fairly sure I used an ancient scrap of code by John Resig. This library looks neatly written and has some fixes to support older browsers like IE6, so I’m going to use it the next time I’m dealing with client-side dates.

The Slim Milo Affair

The Slim Milo Affair is a tutorial with jQuery source that details the creation of a Pablo Ferro-inspired animation. The author wanted to employ a similar effect to Ferro’s animation used by the Thomas Crown Affair title sequence.

The basic concept of the library is that you have a single DOM element that you want to animate with various “stages”. Each stage consists of a main image shown in full, a set of horizontal and vertical bars, and a set of sub images put into the “slots” defined by the bars.

It’s interesting to read through the reasoning behind this type of animation design, because we often use animation libraries without much thought to how they actually work.

Given that I had never attended a tech conference in Italy, I didn’t know what to expect, so was very impressed with the experience. The venue was a picturesque conference and accommodation facility run by the church in the northern city of Brescia. The city itself was also amazing, given its proximity to the mountains, old architecture, wonderful cafes & food and almost a complete lack of tourists . The event kicked off with a dinner the night before, which although a great opportunity to talk to the local developer community, was perhaps missed by some who still needed to polish their slides for the next day. The day itself was extremely well organized, ran smoothly and even included catering despite being a volunteer effort organized by the local web developer community and very reasonably priced at only ten euros.

A wide range of themes were covered. All of the talks were in English except for the keynote. I found the talks on Socket.IO by NKO winner Arnout, Cloud9 talk by Sergi and pubsub.io talk by Ian to be particularly interesting. I also tried to stir up a bit of trouble with a talk on node-fibers and my common-node project.

The full list of recorded talks and slides is online. A number of workshops also took place alongside the talks. The time in between talks was an opportunity to make new acquaintances. Unfortunately due to the conference taking place on Saturday, there was no official program after the conference.

Overall, I found the event to be very very good and, given the opportunity, will be the first to visit the place again in a year – or maybe even sooner!

enchant.js

enchant.js (GitHub: wise9 / enchant.js, License: MIT or GPL) from Ubiquitous Entertainment Inc. is a new game engine that uses HTML5. It supports events, sprites, sounds, scenes, maps, and a whole host of other game-management classes. It also supports controls for touchscreen devices.

The game engine itself is designed around events, so from what I can tell it’s mostly asynchronous. There’s a plugin system for extending the engine itself, and some sample images are included which includes some font sprites.

Source Engine Levels in WebGL

In Source Engine Levels in WebGL by Brandon Jones, Valve’s Source Engine BSP format is discussed in the context of WebGL. Brandon demonstrated his code at onGameStart using a level from Team Fortress 2, but he hasn’t uploaded it due to licensing concerns.

The post contains a lot of details about Valve’s level file format, and how rendering complex commercial 3D game data relates to WebGL:

I was told once that one of the big reasons that WebGL doesn’t have instanced rendering is because it’s hard to make use of it in real world scenarios, so very few games do. I’d like to present my counter example, and ask nicely for the people making decisions to reconsider!

Ender

Ender (GitHub: ender-js / Ender, License: MIT, npm: ender) by Dustin Diaz and Jacob Thornton is a package manager for front-end JavaScript. It has a command-line interface that allows scripts to be packaged and minimised, which can then be loaded like CommonJS modules in a browser.

This builds ender.js and ender.min.js files that contain the specified modules:

ender build jquery underscore backbone

Then in my client-side scripts I can require libraries when they’re needed:

var _ = require('underscore')
  , _.each([1, 2, 3], alert)

// Access methods augmented directly to the $
$.ready(function() {
  $([1, 2, null, 3])
    .filter(function(item) { return item; })
    .each(alert)
})

The ender.js file adds this line to each module:

$.ender(module.exports);

From what I understand of Ender’s source, it actually wraps the entire client-side source in its own closure:

!function () {
  var module = { exports: {} }, exports = module.exports;

  // Original source follows
  (function() {
  })();

  provide('module name here, underscore or jquery or whatever', module.exports);
  $.ender(module.exports);
}();

It seems like an interesting solution to managing client-side dependencies as our projects become more complicated. I’ve actually found myself including client-side modules in my package.json, and then exposing their files to the client.

The authors claim that Ender encourages people to use smaller modules rather than monolithic libraries, and I think this is a good idea. Particularly now that strong, tightly focused libraries are becoming more commonplace.

Last week in Asynchronous Resource Loading I introduced some popular script loading libraries, and discussed how recent web standards aim to give us more control over how scripts are loaded. This week I want to explain how to programatically load scripts asynchronously in a way that supports legacy browsers.

Resource loading can be split into three main concerns:

1. Controlling how things are loaded
2. Providing an API
3. Supporting legacy browsers

Each script loader that I looked at seemed to place a different emphasis on each of these areas. RequireJS has a lot of code for its module-based API; LABjs has a simpler chained API which is a little more focused on the first problem.

Script Insertion

In Loading Scripts Without Blocking by Steve Souders, various non-blocking loading techniques are compared. Steve summarises with the following

In many situations, the Script DOM Element is a good choice. It works in all browsers, doesn’t have any cross-site scripting restrictions, is fairly simple to implement, and is well understood. The one catch is that it doesn’t preserve execution order across all browsers.

Inserting script tags is the approach most libraries use for basic asynchronous script loading. Preserving order is problematic and requires the use of several different solutions to accomplish it.

The most basic script insertion technique looks like this:

(function(global) {
  var appendTo = document.head || document.getElementsByTagName('head');

  function require(scriptSrc, fn) {
    var script = document.createElement('script');

    script.type = 'text/javascript';
    script.src = scriptSrc;
    script.onload = fn;

    appendTo.insertBefore(script, appendTo.firstChild);
  }

  turing.require = require;
}(window));

The handling of appendTo is based on LAB.src.js. To create a suitable unit test, I’ve used the Turing Ajax test harness. The test itself just needs to ensure JavaScript gets loaded and executed as expected:

'test require': function() {
  $t.require('/load-me.js', function() {
    assert.equal(loadMeDone, 1);
  });
}

The load-me.js script sets a global, loadMeDone, and the assertion tests for this. The script’s onload method will be set to this callback.

Internet Explorer Support

Internet Explorer provides an onreadystatechange property instead of onload. That means the script element’s readyState has to be checked.

(function(global) {
  var appendTo = document.head || document.getElementsByTagName('head');

  function require(scriptSrc, fn) {
    var script = document.createElement('script');
    script.type = 'text/javascript';
    script.src = scriptSrc;

    script.onload = script.onreadystatechange = function() {
      if (!script.readyState || (script.readyState === 'complete' || script.readyState === 'loaded')) {
        script.onload = script.onreadystatechange = null;
        fn();
        appendTo.removeChild(script);
      }
    };

    appendTo.insertBefore(script, appendTo.firstChild);
  }

  turing.require = require;
}(window));

This creates an onload/onreadystatechange callback that should work in both Internet Explorer and standards-compliant browsers. However, running the test produces an error raised on the insertBefore line that inserts the script tag. After looking at LAB.src.js I realised it was because the document isn’t yet ready, which means we need a way of deferring until appendTo.firstChild is available.

The easiest way to do this is by using setTimeout:

(function(global) {
  var appendTo = document.head || document.getElementsByTagName('head');

  function require(scriptSrc, fn) {
    var script = document.createElement('script');
    script.type = 'text/javascript';
    script.src = scriptSrc;

    script.onload = script.onreadystatechange = function() {
      if (!script.readyState || (script.readyState === 'complete' || script.readyState === 'loaded')) {
        script.onload = script.onreadystatechange = null;
        fn();
        appendTo.removeChild(script);
      }
    };

    appendTo.insertBefore(script, appendTo.firstChild);
  }

  turing.require = function(scriptSrc, fn) {
    setTimeout(function() {
      if ('item' in appendTo) {
        if (!appendTo[0]) {
          return setTimeout(arguments.callee, 25);
        }

        appendTo = appendTo[0];
      }

      require(scriptSrc, fn);
    });
  };
}(window));

The callback will keep calling itself and returning until appendTo[0] is available. At this point execution will continue to call the original require function.

Conclusion

This example only illustrates loading scripts by inserting script tags. There’s no way to control the order the scripts are executed, and there isn’t any HTML5 support.

This code can be found in commit 2450ee3.

References

• LAB.src.js
• Loading Scripts Without Blocking by Steve Souders

libuv Status Report

Ryan Dahl posted a status report for libuv to the Node blog, and described some of the background to libuv and reviewed the features implemented so far. This includes non-blocking TCP sockets (using IOCP in Windows), non-blocking named pipes, UDP, timers, child process spawning, asynchronous DNS, asynchronous file APIs, and more. They’re currently working on file system events (using inotify, kqueue, and event ports), a VT100 TTY, and socket sharing between processes.

Ryan also linked to some interesting projects built with libuv, and this includes Mozilla’s Rust which is billed as a low-level concurrent language.

node-sql

node-sql (npm: sql) by Brian Carlson (the author of node-postgres, better known as pg) builds strings of SQL using a simple API. It uses the kind of chained API that suits this kind of problem:

var sql = require('sql')
  , user
  , query;

user = sql.define({
  name: 'user',
  columns: ['id', 'email', 'lastLogin']
});

query = user.select(user.id, user.email)
            .from(user)
            .where(user.lastLogin.lt(new Date()));

It’s still a major work in progress, but I think it could lead to some extremely interesting database APIs in the future.

Zipper

Zipper (GitHub: rubenv / zipper, npm: zipper, License: BSD) by Ruben Vermeersch is a zip file creator with an extremely easy API:

var zipper = require('zipper').Zipper
  , zipfile;

zipfile = new zipper('file.zip');
zipfile.addFile('image.jpg', 'path/image.jpg', function(err) {
});

Jade and Stylus Updates

Jade and Stylus have been updated to 0.16.0. Jade’s grammar has been extended to include literal support for if, unless, while, until, and it also supports assignment. This means that rather than using JavaScript, it’s now possible to use declarative statements:

// Old:
- if (!(foo && bar))
 p stuff

// New:
unless foo && bar
  p stuff

This demonstrates the unless keyword, and the new Jade grammar.

The other major feature is template inheritance. The keywords extends and block allow templates to be inherited from, and “blocks” of markup grouped together and overridden as required.

jQuery Mockjax

jQuery Mockjax (GitHub: appendto / jquery-mockjax, License: MIT or GPL) by Jonathan Sharp is a library for mocking jQuery’s network requests. This can be useful when writing tests. It can mock the same data types that jQuery supports: text, HTML, JSON, JSONP, Script, and XML.

Mocking a request looks a lot like the ajax API:

$.mockjax({
  url: '/restful/fortune'
, responseTime: 750
, responseText: { status: 'success' }
});

Mockjax modifies jQuery using $.extend({ ajax: function() { // ... } }); — by passing one argument to $.extend the jQuery object itself is modified. The original ajax module is kept around internally and used to simulate network requests by changing jQuery’s xhr function.

As shown above, request latency can be simulated using a parameter. There are other advanced configuration options as well, including contentType and isTimeout.

ImageMapster

ImageMapster (GitHub: jamietre / ImageMapster, License: MIT) by James Treworgy is a plugin that makes it easier to manipulate HTML image maps. That means interesting effects can be created using image maps and more modern HTML/CSS/JavaScript techniques, while still falling over to the basic functionality in older browsers. The ImageMapster demos page has a lot of examples — the “Vegetable Tray” example shows off some of the basic features.

By calling $('img').mapster(options), ImageMapster will attempt to bind to each image that has an associated map. An area within the image map can be selected with $('area').mapster('select'). The project has lots of options and features, but fortunately the ImageMapster documentation is thorough.

JavaScript Face Detection

This isn’t a jQuery project, but some extremely interesting client-side coding: JavaScript Face Detection + Canvas + Video = HTML5 Glasses! The author, Wes Bos, has written a tutorial and the provided source (wesbos / HTML5-Face-Detection) for some interesting HTML5 Canvas and JavaScript that uses the Core Computer Vision Library.

The CCV JavaScript library uses Web Worker to parallelise computation, but Wes said:

In the CCV examples, they provide a web worker example so we could do this asynchronously, but in my tests it was significantly slower.

The HTML5 Glasses demo that comes with this tutorial doesn’t run in real time (on my computer it updates every 380-400ms), but the tracking library seems to work very well.

I really like the pg PostgreSQL library by Brian Carlson, and considering the amount of attention we’ve given to Redis and MongoDB on DailyJS I thought it was time to give relational databases some coverage again.

Heroku is one of many services that supports Node. This tutorial will demonstrate how easy it is to get a simple Express and pg app running.

This tutorial is based on the following documentation:

• Getting Started With Node.js on Heroku/Cedar
• Heroku Database FAQ
• pg’s Documentation
• pg’s Example App

The files for this tutorial can be found here: alexyoung / dailyjs-heroku-postgres.

Getting Started

An account at Heroku is required first. Next, install the Heroku client:

• Heroku for Mac OS X
• Heroku for Windows
• Ubuntu Heroku installation instructions
• For other operating systems, use the Heroku client tarball

Once that’s installed, try typing heroku help in a terminal to see what the command-client client can do. Heroku obviously realised that us developers prefer using the command-line to a GUI — although some basic management features are available through Heroku’s web interface, almost everything is handled from the command-line tool.

Authentication is requried before progressing:

heroku login

I had to tell Heroku about my public SSH key too:

heroku keys:add ~/.ssh/id_rsa.pub

Module Installation

Heroku wisely supports npm, so our app begins with a package.json:

{ 
  "name": "dailyjs-heroku-postgres"
, "version": "0.0.1"
, "dependencies": {
    "express": "2.4.5"
  , "pg": "0.5.7"
  }
}

PostgreSQL Setup

Heroku uses environmental variables to supply database connection parameters. This is simply process.env.DATABASE_URL for PostgreSQL. Connecting to the database is as simple as this:

var pg = require('pg').native
  , connectionString = process.env.DATABASE_URL || 'postgres://localhost:5432/dailyjs'
  , client
  , query;

client = new pg.Client(connectionString);
client.connect();
query = client.query('SELECT * FROM mytable');
query.on('end', function() { client.end(); });

Notice how pg uses events — I’ve called client.end() so this script will exit gracefully when it’s finished. If you’ve got PostgreSQL installed locally you could try experimenting with this script.

Schema

There’s a few ways to change the database schema on Heroku. I’ve made a little schema creation script:

var pg = require('pg').native
  , connectionString = process.env.DATABASE_URL || 'postgres://localhost:5432/dailyjs'
  , client
  , query;

client = new pg.Client(connectionString);
client.connect();
query = client.query('CREATE TABLE visits (date date)');
query.on('end', function() { client.end(); });

I’ll explain how to run this on Heroku later.

Another option would be to use a library like node-migrate by TJ Holowaychuk. I haven’t actually used this before, but it seems like a sensible way to keep local schemas in sync as developers work on a project.

Typing heroku help pg shows the commands available for PostgreSQL, and this includes heroku pg:psql which can be used to open a remote connection to a dedicated database. This won’t be allowed for a shared database, but could be used to modify the schema.

Example App

Now we’ve got a package.json, we just need an app to run. Create a file called web.js that starts like this:

var express = require('express')
  , app = express.createServer(express.logger())
  , pg = require('pg').native
  , connectionString = process.env.DATABASE_URL || 'postgres://localhost:5432/dailyjs'
  , start = new Date()
  , port = process.env.PORT || 3000
  , client;

Notice how I use Heroku’s environmental variable for the database connection string and server port, or defaults for development purposes.

Now we can add the code required to connect to the database:

client = new pg.Client(connectionString);
client.connect();

A single Express route should suffice for this tutorial:

app.get('/', function(req, res) {
  var date = new Date();

  client.query('INSERT INTO visits(date) VALUES($1)', [date]);

  query = client.query('SELECT COUNT(date) AS count FROM visits WHERE date = $1', [date]);
  query.on('row', function(result) {
    console.log(result);

    if (!result) {
      return res.send('No data found');
    } else {
      res.send('Visits today: ' + result.count);
    }
  });
});

And we better start the app too:

app.listen(port, function() {
  console.log('Listening on:', port);
});

Procfile

The last thing we need is a file that tells Heroku what our main script is called. Create a file called Procfile:

web: node web.js

Deploying

Heroku uses Git for deployment, so set up a repo:

git init
git add .
git commit -m 'First commit'

Then run this command which creates a remote app on the service with a random name:

heroku create --stack cedar

It’ll give you the URL, but your app isn’t quite ready yet.

Now push the repo to make the magic happen:

git push heroku master

And tell Heroku you want to use a database:

heroku addons:add shared-database

And finally… run the schema creation script:

heroku run node schema.js

Hopefully you now have a little Node and PostgreSQL app running on Heroku!

If anything went wrong, Heroku’s documentation is excellent, and you can download my sample source here: alexyoung / dailyjs-heroku-postgres.

WebGL Examples

I found this collection of WebGL examples at ibiblio.org that includes a few things I haven’t seen before, particularly Cycleblob which is an interesting take on the Tron light cycles concept.

Backbone Forms

Backbone Forms by Charles Davison is a form framework for Backbone.js with some advanced features like nested forms, and it even comes with CSS for styling fields.

This library makes it possible to automatically generate forms based on model instances:

var form = new Backbone.Form({
  model: users.get(userId)
}).render();

If a model isn’t available, a Backbone.Form can be instantiated using data and schema properties:

new Backbone.Form({
  data: { id: 123, name: 'Rod Kimble', password: 'cool beans' },
  schema: {
    id:       { type: 'Number' },
    name:     {},
    password: { type: 'Password' }
  }
}).render();

The source is documented and the README has enough examples to get started.

Third-Party JavaScript

Third-Party JavaScript by Ben Vinegar and Anton Kovalyov (both from Disqus) is a book about the art of third-party scripting — writing JavaScript intended to be executed on a remote site. It’s aimed at intermediate developers, and covers distributing and loading applications, server communication, cross-domain iframe messaging, rendering HTML and CSS, authentication, security, and performance. Considering the authors work at Disqus it seems like a potentially essential book on the subject.

The book will be published by Manning, and an “early access” edition can be purchased from $35.99. Three chapters are available so far, and the example source code is on GitHub at benvinegar / thirdpartyjs-code. The first chapter is available for free: Introduction to Third-Party JavaScript (PDF).

To pre-order the book, visit Third-Party JavaScript at Manning’s site.

The use of asynchronous script loaders has become very popular over the last few years. Rather than bundling assets into monolithic files to improve performance, it’s possible to coax browsers to load several files at once. There are different approaches to this, but in general JavaScript APIs are used rather than script elements.

I want to look at how asynchronous script loaders are built, what HTML5 provides, and then build a new asynchronous script loader as an advanced client-side tutorial. This tutorial series will be split over several weeks.

Popular Libraries

There are a huge amount of asynchronous script loaders out there. The diversity is largely due to the complexity of the problem space — different situations can be optimised, and browser support requirements play a huge role in the internal design of these libraries.

The first library I found and used actively was LABjs (GitHub: getify / LABjs, License: MIT) by Kyle Simpson. To understand what LABjs does, consider the following HTML:

<script src="jquery.js"></script>
<script src="jquery-ui.js"></script>
<script src="my-script.js"></script>

In the HTML4 spec, it says:

All SCRIPT elements are evaluated in order as the document is loaded

That means if a framework like jQuery is specified first, then subsequent scripts can safely depend on it being loaded. The downside of this, however, is downloading each of these scripts will block the loading of other page components like images and stylesheets. This can create a noticeable visual delay when the page is loaded.

Using LABjs can solve this problem:

<script type="text/javascript">
   $LAB
   .script("jquery.js").wait()
   .script("jquery-ui.js").wait()
   .script("my-script.js");
</script>

Here, every script will be loaded in parallel. The wait() method will cause a execution to delay until the script has finished loading and has been parsed.

Technically, code within the application’s client-side JavaScript could call $LAB.script() whenever a dependency is needed. This means a single page app could have a lower footprint until certain functionality is required.

Another popular library is RequireJS (GitHub: jrburke / requirejs, License: new BSD and MIT) by James Burke. RequireJS is very different to LABjs, with a large feature set:

• Plugins
• Server-side bundling and optimisation
• Module support
• Remote service dependencies (JSONP)
• Script order control (through a plugin)
• Web Worker support

The basic usage looks a bit like LABjs:

<script src="scripts/require.js"></script>
<script>
require(['script1.js', 'script2.js'], function(someModule) {
  // This optional callback runs when all dependencies are loaded
});
</script>

By embracing concepts like CommonJS Modules and the Asynchronous Module Definition API, RequireJS offers a way to not only avoid blocking when loading scripts, but also add structure to projects.

Yet another approach used by Modernizr and yepnope.js is to use a test to see if a script should be loaded:

yepnope({
  test: window.JSON,
  nope: 'json2.js',
  complete: function () {
    var data = window.JSON.parse('{ "json" : "string" }');
  }
});

The power here, is that you were able to parse a JSON string, but if your browser already had built in support for JSON parsing, it didn’t require you to load anything extra!

HTML5

JavaScript-powered asynchronous loading isn’t the end of the story. In HTML5, the script element has been updated to support async and defer attributes. This is already available in recent WebKit-based browsers and Firefox 3.6.

These attributes effectively let us do what LABjs offers:

• If async is present, the script will be executed asynchronously, as soon as it is available
• If defer is present, the script is executed when the page has finished parsing
• If neither are present, the script is fetched and executed immediately before the page has finished parsing

From Mozilla’s documentation on the script element:

In older browsers that don’t support the async attribute, parser-inserted scripts block the parser; script-inserted scripts execute asynchronously in IE and WebKit, but synchronously in Opera and pre-4.0 Firefox.

Tony Gentilcore compared this approach to JavaScript libraries on the Surfin’ Safari Blog:

There are many clever techniques for working around this performance bottleneck, but they all involve extra code and browser-specific hacks. Instead, scripts which do not require synchronous execution can now be marked as either async or defer.

AMD

As well as HTML5, asynchronous script loading is addressed by the Asynchronous Module Definition API. To understand why this is useful, consider CommonJS Modules:

var fn = require('my-module').fn;
fn();

How would this work in a browser? Assuming 'my-module' is loaded through script tags, it would be difficult to control execution and parsing. With AMD it’s possible to do this instead:

define('my-module', ['require', 'exports', 'mydep'], function (require, exports, mydep) {
  exports.fn = function() {
    return 'hello from my-module:fn';
  }
});

References

• JavaScript and HTML Script Tags by Douglas Crockford
• LABjs
• HTML5 script element specification
• Mozilla script element documentation
• Running scripts in WebKit

Celeri

Celeri by Craig Condon is a command line library with lots of useful features:

• Argument parsing
• History
• Progress bars
• Busy spinner
• Masked password input
• Confirmation and prompts
• Tables

The API is based around events:

var celeri = require('celeri');

celeri.on('hello :name', function(data) {
  console.log('Hello %s!', data.name); 
});

celeri.open();
celeri.parse(process.argv);

There are examples available on GitHub that demonstrate the main features of the library. The API is a lot different to Commander.js which I posted about recently, but the feature set is slightly different. However, the Commander.js authors have also written several other userful CLI libraries (which TJ links to in the Commander.js documentation).

Strata

Strata (GitHub: mjijackson / strata, License: MIT, npm: strata) is a Node web framework inspired by WSGI/Rack instead of the popular Connect middleware framework. Strata includes a specification that can be enforced by using the strata.lint middleware.

On the surface Strata looks like Express, but it’s actually quite different:

var strata = require('strata')
  , redirect = strata.redirect
  , app = new strata.Builder;

app.use(strata.commonLogger);
app.use(strata.contentType);
app.use(strata.contentLength);
app.use(strata.sessionCookie);

app.get('/', function(env, callback) {
  callback(200, {}, 'Friends, help! A guinea pig tricked me!');
});

app.get('/redir', function(env, callback) {
  env.session = {}; 
  redirect(env, callback, '/');
});

module.exports = app;

There’s a lively debate about the Strata framework at Hacker News.

Node ACL

Node ACL (License: MIT, npm: acl) by Manuel Astudillo is an ACL implementation using Redis. It includes support for users, roles, hierarchies, resources, and Express middleware.

Granting administrator privileges might look like this in your application:

acl = new acl(redisClient);
acl.allow('admin', ['blogs', 'forums'], '*');

Express routes and resources can make use of acl.middleware():

app.put('/route', acl.middleware(), function(req, res, next) {
});

jquery-expander

The Expander Plugin (License: MIT and GPL) by Karl Swedberg can collapse part of an element’s content and add ‘read more’ links. Truncation is based on characters, and this can be changed along with many other things when invoking the plugin:

$(selector).expander({
  slicePoint: 100
, preserveWords: true
, expandText: 'read more'
, expandPrefix: '… '
});

Marco Polo

Marco Polo (License: MIT, demo) by Justin Stayton is an autocomplete plugin with some interesting features:

• Prevents unnecessary requests using a cache
• Automatic selection, in a similar fashion to select inputs
• Require selections
• Supports large text through overlabel support
• Easy styling
• Event-based, with public access

These features are documented in the Marco Polo wiki. This simple example demonstrates how results can be obtained and formatted:

$('#userSearch').marcoPolo({
  url: '/users/search',
  formatItem: function(data, $item) {
    return data.first_name + ' ' + data.last_name;
  },
  onSelect: function(data, $item) {
    window.location = data.profile_url;
  }
});

Manifest

Manifest (License: MIT, examples) also by Justin Stayton enhances text inputs so multiple items can be selected and edited. The API is very consistent with Marco Polo, and Manifest can take a marcoPolo object if autocomplete functionality is required.

$('#recipients').manifest({
  marcoPolo: {
    url: '/contacts/search',
    formatItem: function(data) {
      return '"' + data.name + '" (' + data.email + ')';
    }
  }
});

The author also points out how these plugins differ from the popular Chosen plugin:

If you want to make a select element with a lot of options more user-friendly, use Chosen. If you can’t reasonably list out every possible option (like all users in a system), or you need to allow arbitrary values (like new tags), use Manifest.

CacheProvider

Felipe Abreu sent us a client-side caching library called CacheProvider (License: MIT). He hasn’t yet written a README, but the most of the methods in the CacheProvider source are nicely commented so it’s easy to follow how it works.

It’s a continuation of the JavaScript Cache Provider class by Dustin Diaz. Like Diaz’s class, CacheProvider will use localStorage if available.

A few weeks ago I was working on the deceptively simple problem of writing a custom events library for the Framework series in Custom Events. The EventEmitter2 module is an interesting alternative to Node’s EventEmitter, which adds a few novel features to this problem space.

About EventEmitter2

EventEmitter2 (License: MIT, npm: eventemitter2) by hij1nx is an alternative to EventEmitter that adds several unique features:

• Namespaces
• Wildcards
• A many method, which is similar to once
• Browser compatibility
• Performance improvements over EventEmitter

Usage

Usage is basically the same as EventEmitter. The constructor takes a configuration object, where the namespace delimiter can be changed:

var server = EventEmitter2({
    wildcard: true
  , delimiter: '::'
  , maxListeners: 20
});

The many method sounds a little bit confusing at first, but all it does is fires an event several times and then removes it:

server.many('quad hello', 4, function() {
  console.log('hello');
});

Structure

EventEmitter2 is distributed as a Node module with a detailed package.json and tests. The main library file, lib/eventemitter2.js contains all of the source in a self-executing anonymous function so browsers can be catered for:

;!function(exports, undefined) {
  // source goes here

  exports.EventEmitter2 = EventEmitter;
}(typeof exports === 'undefined' ? window : exports);

An inArray method is also defined for browser support:

var isArray = Array.isArray ? Array.isArray : function _isArray(obj) {
  return Object.prototype.toString.call(obj) === "[object Array]";
};

Other than that the library’s overall structure should look familiar to anyone who’s heavily used EventEmitter.

Configuration

I thought it was interesting how configuration is removed from the constructor:

function configure(conf) {

  if (conf) {
    this.wildcard = conf.wildcard;
    this.delimiter = conf.delimiter || '.';

    if (this.wildcard) {
      this.listenerTree = new Object;
    }
  }
}

function EventEmitter(conf) {
  this._events = new Object;
  configure.call(this, conf);
}

Presumably this is to differentiate between constructor initialisation and setting configuration value defaults.

Similarities to EventEmitter

I noticed a few familiar techniques from EventEmitter, like this optimisation from when events are added:

if (!this._events[type]) {
  // Optimize the case of one listener. Don't need the extra array object.
  this._events[type] = listener;
}

And the listener leak protection is the same too:

// Check for listener leak
if (!this._events[type].warned) {

var m;
if (this._events.maxListeners !== undefined) {
  m = this._events.maxListeners;
} else {
  m = defaultMaxListeners;
}

if (m && m > 0 && this._events[type].length > m) {

  this._events[type].warned = true;
  console.error('(node) warning: possible EventEmitter memory ' +
                'leak detected. %d listeners added. ' +
                'Use emitter.setMaxListeners() to increase limit.',
                this._events[type].length);
  console.trace();
}

Wildcard Support

When wildcard support is enabled, two methods are used: searchListenerTree and growListenerTree. EventEmitter2 uses an object to hold “branches” for namespaces so they can be recursively searched.

When adding an event, growListenerTree is called with the event name and the listener function. The event name is split based on the configured delimiter. Each part of the event name is removed from an array, and a tree is built up as required. The leaves are the listener functions.

Just like the EventEmitter optimisation, if there’s only one handler it’s stored as a function, else an array of functions is used. The growListenerTree function will return true, but the return value is never used.

Tests

The tests include a benchmark to compare against EventEmitter, and this uses the Benchmark.js library that we seem to keep referencing on DailyJS lately. The rest of the tests are split into “simple” and “wildcardEvents”, because EventEmitter2 uses very different code if wildcards are required.

These tests use good ol’ nodeunit, and use test.expect to ensure the correct number of assertions are run per-test.

Conclusion

Comparing EventEmitter2 to the original EventEmitter is interesting, because it shows how adding a seemingly simple feature like namespaces requires a very different approach.

The original thread on the nodejs group where hij1nx came up with EventEmitter2 is here: Namespaced EventEmitters?, and you can see the comments that influenced the fundamental separation of simple and wildcard/namespaced events.