Node 0.6.4

Node 0.6.4 has been released. One interesting thing I noticed was this:

sys: deprecate module some more, print stack trace if NODE_DEBUG=sys

It can be a little bit frustrating when working with dependencies that still use this module, so at least these stack traces will make it easier to find out who to blame.

Node Web Development

Node Web Development (Price: $22.49, Ebook: $12.74, Publisher: Packt) by David Herron assumes rudimentary JavaScript knowledge and introduces Node, covering the basic theory behind asynchronous and event-driven development, performance, installation, CPU utilisation, modules, Express, EventEmitter, SQLite3, and even Mongoose.

If you’re a web developer struggling to understand Node, then Node Web Development may help you get up to speed quickly. For more information, take a look at the Node Web Development table of contents and Chapter 3: Node Modules (PDF).

Node.js in Action

Node.js in Action (Price: $39.99, Ebook: $31.99, Publisher: Manning) by Mike Cantelon and TJ Holowaychuk is currently available as part of Manning’s Early Access Program, which means you can subscribe to read the book as chapters are finished.

The book is split into three parts, which cover the fundamentals, web development, and “going further with Node” which appears to cover “stuff that isn’t web development”. The first chapter is actually a well-tempered argument on why we need Node, so it’s worth considering for boss-convincing ammunition. The second chapter explains how to install Node for Windows, Mac OS X, and Linux, then goes on to make some fun scripts like a little TCP/IP server accessed with telnet.

The authors waste no time diving right into asynchronous development in chapter 3. This chapter is a solid treatment of asynchronous programming, with EventEmitter taking centre stage, and coverage of control flow techniques and tools.

Node.js in Action’s estimated publishing date is April 2012, with 300 pages in the finished version. It’s currently around 130 pages, so solid progress has been made so far. Chapter 1: Why the Web needs Node (PDF) is available as a free sample chapter.

read

read (npm: read) by Isaac Z. Schlueter is billed as “read(1) for Node”, and is intended to read lines from the standard input.

var read = require('read');

read({ prompt: 'echo: ' }, function(err, res) {
  if (err) {
    console.error(err);
  } else {
    console.log(res);
  }
});

Documentation is available in the project’s README. Isaac notes that care must be taken to manage process.stdin in certain versions of Node:

In node 0.6.0 through 0.6.5, you must explicitly call process.stdin.destroy() or process.exit() when you know that your program is done reading, or else it will keep the event loop running forever.

XJST

XJST (GitHub: veged / xjst, npm: xjst) by Sergey Berezhnoy is a Node and browser-compatible DSL for creating data transformations. Suggested applications include HTTP request routing, template engines, and parsers.

The authors have written some XJST examples to help people get the hang of the language. To use XJST, an input file must be created using XJST’s DSL. It reads like JavaScript, and XJST will compile it into JavaScript. The simple.xjst example is a good one for getting started.

If you’re writing a parser of some kind but feel put off by something like Jison, then XJST might help you get started without too much effort.

The jQuery plugins site is currently down. The intention is to replace it with a better site. However, is a replacement even necessary? When researching our weekly jQuery Roundup articles I rarely use the jQuery plugins site.

Fortunately, there are alternatives, and they’re arguably more friendly to front-end developers.

Hosting Code

Guess what? I don’t want to download a zip file from your website.

Most plugins I encounter are found through GitHub. With a free GitHub account, open source projects get version control, issue tracking, and a wiki. Another useful service for plugin developers is GitHub Pages which can be used to host site for your plugin, including any documentation and code examples. GitHub is essentially a complete service for plugin developers.

And believe it or not, GitHub recently announced a client-side editor in Edit like an Ace. Even if you’re a less technically-minded client-side developer, you can now edit code through GitHub without even learning how git works. Although, GitHub has solid documentation, I’d still recommend learning the basics.

If a web-based editor appeals to you, then why not give Cloud9 IDE a try? It can integrate with GitHub, and allows free public project creation which is perfect for open source projects. Old fashioned Vim/Emacs die-hard like me might be suspicious of services like Cloud9, but if it works for you then use it!

There are GitHub alternatives, including Bitbucket and Google Code.

Issue Tracking

Allow people to get help when things go wrong. Writing client-side code is an art, and it’s likely that no matter how well-tested your plugin is, it’ll have a few bugs.

If you’re looking for issue tracking outside of what the version control services offer, then Lighthouse is a good choice because it allows free open source issue tracking. I’ve also noticed open source developers using Pivotal Tracker.

Promotion

There are curated jQuery plugin lists out there. The jQuery Handpicked Plugin Repository is one such list, which features a plugin demo browser. Writing to the authors of such lists, or sites like DailyJS, is a perfectly acceptable way to promote your plugin. In fact, if you don’t send me your plugins, it’s unlikely that I’ll ever find them!

Writers like myself will look at several indicators to determine if our readership will be interested in a given plugin, and if it’s a new plugin that’s poorly packaged then I’m unlikely to cover it. I also quickly download plugins and try them out, but a demo page is preferred because it saves me the effort of downloading a plugin and creating a suitable mock HTML page to run it.

Another useful way to host demos is to use jsFiddle. It’s a convenient way of bundling and running HTML, JavaScript, and CSS. A project README file that links to a demo on jsFiddle is perfectly acceptable.

It’s also possible that you could write a tutorial for your plugin and syndicate it. Lots of blogs and dedicated tutorial sites are looking for content like this. Editors generally prefer it if the content is unique, so write to your favourite blogs with a tutorial pitch before committing to writing anything. If you don’t mind putting the effort in and not getting published, it only takes a few minutes to set up a free blog and post your own tutorials and news.

Open source projects were traditionally promoted through receptive mailing lists. This is still common in some circles — I often look for [announce] posts on popular JavaScript mailing lists. The equivalent for jQuery is the Using jQuery Forum, according to the jQuery site:

Using jQuery: […] if you’ve built a site that uses jQuery, or would like to announce a new plugin, this is the place to do it.

Discovery

Beyond curated plugin lists and blogs, a good way to discover new plugins is social bookmarking sites. The pinboard.in/t:jquery link is the first autocomplete item in my browser address bar when I type ‘pi’. The reason for this is Pinboard has a good signal to noise ratio.

Example: Chosen

Chosen (GitHub: harvesthq / chosen, License: MIT) is a solid example of a beautifully presented and well-marketed jQuery plugin. It hits everything on my checklist:

• Includes a README with useful information about jQuery version support and how to contribute
• Tracks issues through GitHub
• Has a great site on GitHub Pages with full demos, example code, FAQs, credits
• Includes a license
• Includes the build scripts the authors use to generate releases

I think the only thing that’s missing here is unit tests, unless they’re hidden somewhere in the source that I didn’t expect.

Echo JS

Echo JS (GitHub: antirez / lamernews) is a social news site that focuses on JavaScript. I’ve already found some interesting libraries through it, and I basically read about JavaScript every day so it gets my vote.

It’s built with “Lamer News” which is a Ruby-based Hacker News implementation. Can’t someone write them a slick real-time Node version with the cutting edge server-side and client-side libraries that we all know and love?

If you prefer Twitter for consuming news, there’s also @echojs. It’s currently getting around five posts a day which seems promising. Judging by the Twitter account, Echo JS was launched on December 1st, so hopefully posts will increase in regularity rather than petering out.

brid.js

I’m not sure what to make of brid.js (License: MIT, Demo: brid.js demo). It’s a small Lisp implementation that uses JSON instead of S-expressions. On one hand, it’s a fun and interesting project, and on the other it seems like affront against nature.

This is a Hello World example in brid.js:

["println", ["quote", "hello world"]]

Adding numbers looks closer to Lisp:

["+", 1, 1]
// Lisp: (+ 1 1)

Lisp’s parenthesized syntax is analogous to the JSON representation, but the quoting required to create valid JSON is a little bit hard to get the hang of.

If you’re interested in parsing in JavaScript, then you might enjoy looking at the source. We’ve covered Jison on DailyJS before, but context-free grammar isn’t the only way to build parsers, and hand-coded parsers like this are just as useful to learn from.

ql.io

ql.io (GitHub: ql-io / ql.io, License: Apache 2.0) from eBay is a “declarative, evented, data-retrieval and aggregation gateway for HTTP APIs”. It has a DSL inspired by JSON and SQL for declaring API calls and their interdependencies.

The code itself is built on Node, and is comprised of several packages:

• ql.io-engine: This is the core ql.io engine
• ql.io-ecv: Checks servers are responding with JSON
• ql.io-mon: Statistics on the ql.io runtime
• ql.io-uri-template: A small URI template processor
• ql.io-app: An app template for building sites using ql.io
• ql.io-compiler: A PEG.js-based parser with a compiler (ql.peg contains the main grammar)
• ql.io-console: Provides the HTTP interface for ql.io
• ql.io-mutable-uri A utility for manipulating URIs

This is a diverse and complicated project, employing some interesting techniques. I’m particularly intrigued by the use of PEGs (parsing expression grammar) over context-free grammar, as Jison seems like the easiest way to write parsers in JavaScript at the moment. PEG.js (GitHub: dmajda / pegjs, npm: pegjs) is available for Node and browsers.

Announcing ql.io at the eBay Tech Blog has more details on the design and architecture of the project.

Ender

The core Ender build tool has been updated to version 0.8 for the release of Node 0.6. A minor change was required to properly support the new version of Node; anyone using Node 0.6 should upgrade Ender with npm update ender -g.

Qwery

Qwery (NPM / Ender: qwery), by Dustin Diaz, is Ender’s default selector engine, it’s small and very fast and you get it if you build with The Jeesh, Ender’s official starter-pack.

Qwery has seen a flurry of activity recently, culminating in a bump to version 3.1. Qwery now has much improved support for older browsers and is better able to handle queries on detached fragments, queries across frames and other quirky features normally found in the likes of Sizzle while remaining super quick and small.

Qwery has no external dependencies and can be used as a stand-alone tool as well as Ender’s selector engine.

Sel

Sel (NPM / Ender: sel), by Andrew McCollum, is an alternative selector engine for Qwery. Beautifully written in CoffeeScript, Sel aims to be fast, small and clean. Some fixes for IE8 bring Sel up to version 0.5.4.

Sel has a single dependency, Andrew’s ES5-Basic (NPM / Ender: es5-basic), a shim to make older browsers more ECMAScript 5 compatible. It can be used as a stand-alone tool as well as Ender’s selector engine.

Reqwest

Reqwest (NPM / Ender: reqwest), by Dustin Diaz, is the module most Ender users include for AJAX functionality.

The recent release of version 0.3.3 brings improved JSONP support and largely rewritten serialization functionality, including a new .toQueryString() method.

Forms, groups or individual elements can be serialized in a few different ways:

$(form).serialize()
// returns query string -> x=y&...
$(form).serialize({type:'map'})
// returns an object representation -> { x: y, ... }
$(form).serialize({type:'array'})
// returns array name/value pairs -> [ { name: x, value: y}, ... ]
$(form).serializeArray()
// for backward compatibility
$.toQueryString({
    foo: 'bar'
  , baz: 'thunk'
})
// returns query string -> foo=bar&baz=thunk

Leading to more interesting uses:

$('#myform input[name=myradios]').serialize({type:'map'})['myradios']
// get the selected value
$('input[type=text],#specialthing').serialize()
// turn any arbitrary set of form elements into a query string

Reqwest has no external dependencies and can be used as a stand-alone tool as well as within an Ender build (giving you $.ajax() and the serialization methods).

Ender Twitter Bootstrap

Ender Twitter Bootstrap (NPM / Ender: ender-twitter-bootstrap) is a newly released, unofficial port of the JavaScript components from Twitter’s wildly popular Bootstrap (currently the third most watched project on GitHub).

The Ender port aims to provide support for Bootstrap without the need to compile jQuery in your Ender build. A jQuery-free live copy of the Bootstrap JavaScript page (original, jQuery version here) demonstrates Modals, Dropdown, ScrollSpy, Buttons, Tabs, Twipsy, Popover and Alerts.

Ender Twitter Bootstrap depends on The Jeesh (Qwery, Bonzo, Bean, domReady) plus Bowser.

CalEnder

CalEnder (NPM / Ender: calender) is a basic, themeable calendar datepicker for Ender. After surveying the vast JavaScript datepicker landscape, Dustin Diaz decided that yet another one was required, but this time, just for Ender.

CalEnder depends on The Jeesh (Qwery, Bonzo, Bean, domReady).

It’s time for the JavaScript Developer Survey, 2011! 2009’s survey had 151 responses, then 2010 completely demolished that with 1218. Please share this post on your social networks and news sites to help us get a wide range of JavaScript developers represented in the survey.

This survey will close on the 15th of December. The results will be shared with the community through DailyJS, and are also visible after submitting the survey.

The survey has now closed. Get the results here: JavaScript Developer Survey 2011 Results.

Previous Surveys

• 2009
• 2010

Pushover

Pushover (License: MIT/X11, npm: pushover) by the prolific James Halliday can serve git repositories over HTTP. James says this makes it easy to build customised git push-based deployment logic.

James’ example instantiates a server like this:

var pushover = require('pushover');
var repos = pushover(__dirname + '/repos');

repos.on('push', function (repo) {
    console.log('received a push to ' + repo);
});

repos.listen(7000);

// Run with `node example/simple.js`

And then after pushing something using git push http://localhost:7000/beep master, the server will display received a push to beep.

This works by forking the command-line git binary with ('child_process').spawn, and wrapping it with an object that descends from EventEmitter which can handle HTTP requests. It’s surprisingly simple given the possibilities it creates.

Police

Police (License: MIT/X11, npm: police) by Pavan Kumar Sunkara helps discover outdated modules in GitHub repositories based on a project’s package.json. After authenticating with GitHub, police repo or police user/repo can be used to generate a report that details missing package.json properties and old module versions.

When I tested Police out I noticed the .policeconf file gets created with permissions set to 644, so if you’re going to use this on a shared server you might want to change the permissions as it contains a GitHub authentication token.

Raynos’ Blog

When I wrote New Node Web Frameworks: Derby, Flatiron I asked people to send in web apps they’ve built using a Node web framework. Raynos sent in his blog, Raynos.org (GitHub: Raynos / raynos-blog) which appears to be based on a generic blog engine the author has previously written and released through NPM.

The blog engine is based on CouchDB, and includes some unit tests. Raynos has decided to split this Express app into routes, controllers, models, and view models. Templates are powered by the author’s trinity templating system.

Smartupdater

Smartupdater (GitHub: vkiryukhin / Smartupdater, License: MIT and GPL) by Vadim Kiryukhin is a plugin that specialises in Ajax polling, originally featured on DailyJS last year in jQuery Rolling and Polling. It has recently been updated to version 4.0 (beta).

The API has been updated to reduce the number of functions the plugin adds to jQuery, so rather than $('#myObject').smartupdaterStop(), $('#myObject').smartupdater("stop") can be used instead.

vkTemplate

Vadim Kiryukhin has also released vkTemplate (GitHub: vkiryukhin / vkTemplate, License: MIT and GPL). This is a small (less than 1.5kb) templating plugin that’s designed to be used with Ajax. Given URLs for a template and date, a template can be rendered like this:

$('#subcontainer').vkTemplate('/user.tmpl', '/users/1.json');

// JSON can be passed as well
$('#subcontainer').vkTemplate('/user.tmpl', { name: 'Kurt', email: 'kurt@thepope.org' });

Handpicked jQuery Plugins

The Handpicked jQuery Plugins Repo is a list of jQuery plugins curated by David Higgins. It’s actually a repository on GitHub, and he invites people to fork it here: higgo / handpicked.jquery.plugins.repo.

David has also made a site where the plugins can be browsed, at iwantaneff.in/repo/. And, if you’re not comfortable with git he’s even uploaded the plugins as files that can be downloaded from iwantaneff.in/repo/browser/.

Part 2

If you are using a script loader, or even if you are just putting your scripts at the bottom of your DOM, you have a problem. In some cases your scripts will load after the user has clicked on something that requires a JavaScript function to handle the click. It’s possible you have a pure HTML version, but if the user has JavaScript enabled then we want to use it, even if the JavaScript hasn’t loaded yet. There needs to be a way of handling events before all of the assets have finished loading.

There are a number of ways we could do this, but it’s helpful to look at a working implementation. This is going to be a two part series. First, we are going to look at how Flickr does this. In the second part, we’ll take the code that Flickr uses and extract the main features so that anyone can use the code.

In all of the code that Flickr loads before the body this is the part that matters to us: Isolated actionQueue Code.

Let’s walk through this. F is the global Flickr object that contains all of Flickr’s JavaScript. Not only is this a common pattern among frontend developers, this technique is widely taught, and accepted at Yahoo. By keeping all of your site’s functionality in one global object you are not polluting the global namespace. It also is a great way to organize your code.

If a user is able to interact with the site before the scripts have finished loading, you still want to handle the user’s click somehow. It may not need to be completely handled, but the interface should be responsive. Code may run in the time between when the click event is fired, and when the module that can handle the event is loaded. Finally, it should be possible to run code after everything has loaded.

Using Flickr’s terminology, there are two phases: interim and cleanup. Interim code will be run right away, as soon as the user clicks something. Cleanup code will run as soon as the rest of the code has finished loading.

Flickr calls this module the actionQueue. Given that we now understand the basics of how the actionQueue works, the first step is to register a click_queue handler. The fav button is a good example of this.

How the Fav Button is Supposed to Work

In the un-faved state the star is black and white:

The active state is a colour star:

There are two things that need to happen when a person clicks on the favorite button.

1. The button needs to change state to purple
2. An Ajax call needs to be made to register the state change

The visual state change is as simple as adding a class to the element. Very little code is required for that. On the other hand, the Ajax operation is going to require a lot more code. So, to use actionQueue an action should be registered so that in the interim the visual state can be changed, and in the cleanup stage the Ajax operation will be performed.

The advantage of this approach is immediate feedback. Once the module has loaded then the Ajax operation can be performed. And, this is exactly how Flickr implemented this functionality: Flickr actionQueue Fav Handler.

The register function only handles potential actions, and not actions that have actually been triggered. That is what the queue_click function is for. The queue_click function can be called when an actual event has happened. In the case of the fav button, this will be a click event.

Looking at the HTML for the button shows that the queue_click function is just called from the onclick handler. In turn the function will return false if the page is still waiting for code to load, thus canceling the browser’s default operation: Flickr Favorite Button.

Now, this is where things get awesome. I am a fan of killing sacred cows. One of those in the front-end community is to not use onclick attributes – it’s certainly become dogma for many. Clearly Flickr doesn’t care about this. I don’t know the details of why Flickr is using onclick, but I have to assume it’s because it’s the lowest common denominator, and it works.

And, that’s it. In this manner Flickr is able to handle all of us who are able to click faster than JavaScript is able to load. After breaking down this technique I wondered if there might be an easier way to handle actions before they loaded.

Conclusion

Now that we have seen how Flickr handles this problem, how can we use the techniques in our own code? In part 2, I’ll explore Flickr’s code in detail, and demonstrate how it can be used with any site.

UtahJS Conference

UtahJS announced a conference as part of the Silicon Slopes conference which will be held on March 29-31, 2012. They have issued a call For papers, so if you’re interested in speaking fill out the form to apply.

This is going to be big. Think of hundreds of people, all in the same place, learning about JavaScript and HTML5!

Talks should be 45 minutes with 5 minutes for questions.

jQuery Conference 2012: UK

Query Conference 2012: UK will be held in Oxford on the 10th of February. Tickets are on sale at jQuery UK 2012 tickets with prices starting at £160 — lots of options are available, including access to selected workshops.

Speakers currently include Paul Irish, Todd Parker, and Addy Osmani.

The original announcement on the jQuery blog is here: jQuery Conference 2012: United Kingdom Announced.

DevCon5

DevCon5 will be held on December 7-8 in Santa Clara, California. Tickets are $595. The conference is focused on HTML5, and there are two talks that focus on WebGL. The Khronos Group posted Why You Need to be at DevCon5 which details the two WebGL talks:

• Introduction to WebCL and WebGL by Neil Trevett from Nvidia
• Hands on with WebGL by Zhenyao Mo, from Google Chrome’s GPU team

When I started writing Turing I said I wanted to keep the modules decoupled so parts of the framework could be used in isolation. Most popular web frameworks package a large chunk of core functionality that includes support for everything from a selector engine to event handling and ajax.

I think it’s fair to say that this convention has now changed, and people want to be able to cherry pick parts of a framework alongside other libraries. And what’s more, this can’t just be addressed by a build process because some projects might load scripts with a RequireJS style library, or by concatenating and minifying all of their client-side JavaScript using a server-side framework.

With that in mind, can Turing truly be considered modular? If it is indeed modular, then this should be possible:

<!DOCTYPE html>
<html>
  <head><title>Let's Make a Framework: Modularity</title></head>
  <body>
    <div id="test"><p>Test Content</p></div>

    <script src="turing-test/lib/assert.js" type="text/javascript"></script>
    <script src="../turing.core.js" type="text/javascript"></script>
    <script src="../turing.dom.js" type="text/javascript"></script>
    <script type="text/javascript">
assert.equal(turing('#test p').html(), 'Test Content');

turing('#test')
  .css({ backgroundColor: 'green', color: 'white' })
  .find('p')
  .html('Test Passed');
    </script>
  </body>
</html>

I tried this little test and it worked. Each Turing module is split into Immediately-Invoked Function Expressions (IIFE) that depend on the existence of a ‘turing’ global variable:

// turing.core.js
(function(global) {
  if (global.turing) {
    throw new Error('turing has already been defined');
  } else {
    global.turing = turing;
    if (typeof exports !== 'undefined') {
      exports.turing = turing;
    }
  }
}(typeof window === 'undefined' ? this : window));

// turing.dom.js
(function() {
  var dom = {};

  turing.dom = dom;
}());

This pattern isn’t ideal however, it’s simply what I used to keep the initial tutorials focused on their specific subjects. A better solution would be to embrace the Asynchronous Module Definition (AMD). Rather than trying to deal with CommonJS modules in browsers, or by using a framework-specific solution, embracing AMD yields several advantages.

Supporting AMD

I’ve already demonstrated how client-side projects support CommonJS, and supporting AMD projects is very similar. Some well-known projects already support it, and the RequireJS site has a great post with a list of these projects here: Why AMD?.

jQuery does it like this:

(function( jQuery ) {

// Expose jQuery to the global object
window.jQuery = window.$ = jQuery;

if ( typeof define === "function" && define.amd && define.amd.jQuery ) {
	define( "jquery", [], function () { return jQuery; } );
}

})( jQuery );

Let’s break that down to see how it works. If define exists, then set up a module named “jquery” with no dependencies. AMD is based around the module pattern, so a function is used as define‘s third parameter that returns the jQuery function. I’ve edited the original file for the purpose of publication, but the original (exports.js) has a lengthy comment explaining just about everything here. For example, the module is registered as “jquery” because AMD derives module names from file names, and jQuery’s file name is usually lowercase.

Conclusion

From a client-side developer’s perspective, AMD is a more friendly way to manage JavaScript modules when compared to CommonJS Modules. It helps us to avoid creating unnecessary global variables, and helps support client-side script loading libraries.

It may even be worth considering structuring entire frameworks around it, so modularity goes right into the core of the project. As the interest grows in projects like jquip, it seems obvious that there is a need for more granular client-side frameworks, and AMD gives us the tools to make that happen in an open and reusable way. If you take a look at Dojo source, you’ll see that it already works this way. This was documented in Asynchronous Modules Come to Dojo 1.6, which explains AMD’s implications for a large project like Dojo.

References

• jquip
• Ben Alman’s Immediately-Invoked Function Expression
• Why AMD? from RequireJS
• Asynchronous Module Definition
• CommonJS Modules/1.1
• jQuery’s exports.js
• Asynchronous Modules Come to Dojo 1.6

Mocha

Mocha (GitHub: visionmedia / mocha, License: MIT, npm: mocha) by TJ Holowaychuk is a new test framework that works both in Node and browsers. It runs tests serially, making it intuitive, and includes a huge collection of reporters including a TAP reporter. It even comes with a sane setup/teardown implementation, allowing code to be run before and after each test or the whole suite.

Like other test frameworks, Mocha uses an asynchronous “done” function. However, calling it is optional for synchronous code, and it can even accept an error to improve test reporting:

describe('User', function(){
  describe('#save()', function(){
    it('should save without error', function(done){
      var user = new User('Luna');
      user.save(function(err){
        done(err);
      })
    })
  })
})

TJ’s example shows that for methods that include an error as the first argument, then this will be automatically passed to done:

describe('User', function(){
  describe('#save()', function(){
    it('should save without error', function(done){
      var user = new User('Luna');
      user.save(done);
    })
  })
})

Mocha can work with any assertion library, which allows alternative assertion libraries like should.js to be used. It’s even possible to use BDD style tests describe() / it() or TDD: suite() / test().

TJ considers Mocha to be Expresso successor, and it does seem to address a lot of things that were missing from Expresso.

Banzai

Banzai (npm: banzai) by Pedro Teixeira is a document processing framework (or an ETL) that uses the concept of pipelines to represent state and perform multiple operations on a document. On the surface, a pipeline looks like a combination of an event-based API and a state machine:

pipeline
  .on('initial', initialHandler, {
      next: 'order received email sent'
  })
  .on('order received email sent', orderEmailSentHandler, {
      priority: 2
    , condition: allItemsAvailable
    , next: 'items available'
  })
  .on('order received email sent', confirmationEmailSentHandler, {
      priority: 1
    , next: 'items not available'
  })
  .on('items not available', itemsNotAvailableHandler)
  .on('items available', itemsAvailableHandler, {
    next: 'order placed'
  })
  .on('order placed', orderPlacedHandler, {
    next: 'order placed email sent'
  });

Each pipeline is backed by a document store, and this is designed to be extensible. The first document store to be implemented is banzai-couchdb-store, but document stores only require a minimum of two functions to work with Banzai (load(docId, callback) and save(doc, callback)) so shouldn’t be too difficult to write support for more databases.

Pedro’s documentation includes a lot more details on pipelines, state handling, state stores, and work queues. Banzai seems like a useful tool for working with format conversion, but Pedro also notes it’s been used with Amazon Mechanical Turk.

Node Inotify

Node Inotify (License: MIT, npm: inotify) by Camilo Aguilar is an inotify API for Node and Linux. This library has been kicking around for over a year now, but the author and several contributors have been actively working on it.

The README includes a detailed code example, but the API is fairly straightforward: by using an instance of Inotify, addWatch() can be used to watch files or directories for changes. The callback fired when the file system changes receives an event which includes a mask that details what kind of change occurred.

jQuery 1.7.1

jQuery 1.7.1 has been released, soon after 1.7. This version is a maintenance release with bug fixes and some code tidying. There are also some performance tweaks — for example, ticket #10733 replaces $.each in the CSS module with regular for loops.

jquip

jquip by Demis Bellot and Jey Balachandran is an attempt to modularise jQuery to make it easier to pick the parts of jQuery that are actually required. Although jQuery’s developers are addressing size issues, outlined in last week’s blog post Building a Slimmer jQuery, jquip splits jQuery into a core library and a set of plugins. The plugins include modules like css, documentReady, and ajax.

Although jQuery’s source is already broken up into modules, jquip uses $.addPlugin('docready', function($) { which is found in src/jquip.js:

$.addPlugin = function (meta, fn)
{
  var name = typeof meta == "string" ? meta : meta['name'];
  fn = typeof meta == "function" ? meta : fn;
  if (typeof fn != "function") throw "Plugin fn required";
  if (name && fn) plugins[name] = fn;
  fn($);
};

The authors claim that jquip.js minified and zipped is 13% of the size of the equivalent jQuery download — I tested this against jQuery 1.7.1 and I got the same result. It’s worth remembering that the authors don’t expect to replace jQuery with jquip, they’re just trying to inspire a change in direction.

jQuery Mobile 1.0

jQuery Mobile 1.0 has been released. This version is 30% faster than RC2, and includes some new jQuery Mobile documentation.

jQuery Mobile has a platform support rating system where each major mobile platform is given a grade. Apple iOS 3.2-5.0, Android 2.1-2.3, Android Honeycomb, Windows Phone 7-7.5, Kindle Fire, and plenty of other popular platforms have been tested on a variety of devices and awarded the A grade.

Derby

Derby (GitHub: codeparty / derby, License: MIT, npm: derby), other than being the city closest to where I grew up, is a new web framework for Node by Nate Smith and Brian Noguchi. It features a data synchronisation library called Racer that synchronises data between browsers, servers, and the database. This is one of the few libraries to truly realise the goal of running the same code on clients and servers. This data synchronisation layer is far from simple — it even includes features to help resolve conflicts.

Derby isn’t just a solution for data syncing, however. It also renders templates, and handles view bindings. Derby uses a single page application structure, so the same routes are used on both the client and the server.

Applications are created using a command-line tool, much like other popular web frameworks:

derby new dailyjs

Derby’s template language may take some getting used to, but it’s very close to mustache. It also includes HTML extensions to make it easier to bind models and views without lots of client-side “glue” code:

<info:>
  <div id=info>
    <!--
      Mustache syntax is used for conditional blocks, except that the
      name need not be repeated in the closing tag or if-else blocks
    -->
    ((^connected))
      ((#canConnect))
        <!-- Leading space is removed, and trailing space is maintained -->
        Offline 
        ((#_showReconnect))&ndash; <a x-bind=click:connect>Reconnect</a>((/))
      ((^))
        Unable to reconnect &ndash; <a x-bind=click:reload>Reload</a>
      ((/))
    ((/))
  </div>

In terms of models, Racer uses Redis for publish/subscribe and to store a journal of transactions; MongoDB is used as the data store. Racer is currently structured so that there are database adapters, so hopefully it shouldn’t be too hard to add support for more databases. It seems like the Redis dependency is more central to the design of Racer.

The motivation behind Derby is to offer flexibility while removing the requirement for what the authors call “glue code”. If you’ve ever worked with Express, Backbone, jQuery, and Mongoose, you’ve probably found yourself writing a lot of the same code to bind things together between projects. As an alternative, Derby is both interesting and opinionated — it seems likely to appeal to a certain type of developer rather than for certain projects.

Flatiron

Flatiron (GitHub: flatiron / flatiron, License: MIT, npm: flatiron) from Nodejitsu is a full-stack framework but it’s actually very loosely coupled. Each library in the framework can be used in isolation, and it’s entirely possible to cherry pick parts of it.

The libraries that make up Flatiron are as follows:

• Routing: Director
• Middleware: Union
• Templating: Plates
• ODM: Resourceful
• Composition: Broadway

The routing library, Director, works in both browsers and Node with no external dependencies. Objects are used to represent the relationship between URLs and methods. Union, the routing middleware, is simple but compatible with Connect middleware.

Plates also works in browsers and Node, but it’s slightly different to other template libraries. All templates are actually valid HTML, with no special characters for value insertion. The relationship between tags and values is defined through object literals that are passed to Plates.bind:

Plates.bind(
  // The first parameter is the HTML template
  '<span></span><img id="bar" class="foo bazz" src=""/>',

  // The second parameter is key/value pairs of data
  { 'bazz': 'Hello, World' },

  // The third parameter defines how values map to tags
  { 'bazz': ['class', 'src'] }
);

I thought ODM stood for Object Data Manager, or something to do with object databases, but in this context it means “Object-Document Mapper”. What’s nice about this library is it uses simple prototype objects. Properties can be defined and validated. It also has a chainable property definition API:

var resourceful = require('resourceful')
  , Creature = resourceful.define('creature');

Creature.property('legs', Number)
  .required()
  .minimum(0)
  .maximum(8)
  .assert(function (val) { return val % 2 === 0 });

Judging by the source, only CouchDB is supported right now, but like Racer this library has been designed with a view to adding support for more databases in the future.

The Broadway library provides a simple base for application-like objects that can be extended through plugins. Each app inherits from EventEmitter2.

Conclusion

Like a lot of new Node web frameworks, Derby capitalises on the relationship between WebSockets, server-side JavaScript, and data storage to make something extremely attractive to both server-side and client-side developers. Conversely, Flatiron doesn’t mandate anything, and the example that ships with Flatiron itself is a command-line application.

Derby has addressed a complaint many people have when working with client-side technologies like Backbone, so I’d like to see how it gets used in the future. Send in your Derby, Flatiron, Express, or other apps for review in the weekly Node Roundup!

ThreeNodes.js

ThreeNodes.js is an attempt to create a vvvv clone. It provides a library of modular nodes that can be used to generate animated graphics. There’s a ThreeNodes.js demo where you can try it out.

It’s actually distributed as a Node app built using socket.io, connect, and watch. The UI currently takes a little bit of getting used to, but it worked well enough for me to make a few little experiments without reading any documentation.

WebGL GPU Landscaping and Erosion

In WebGL GPU Landscaping and Erosion, Florian Boesch discusses a project he created that was inspired by From Dust to create a real-time landscaping and erosion simulation using WebGL.

The post discusses some of the algorithms used to create the erosion effects, and there’s an awesome demo called Craftscape (GitHub: pyalot / craftscape) which has been featured as a Chrome Experiment.

DHTMLX Touch

There’s now a formidable array of mobile web frameworks for those looking to create mobile web experiences that look close to native apps. One I haven’t mentioned before on DailyJS is DHTMLX Touch (License: GPL), which has just seen its 1.0 release. It features lots of UI widgets, and a suite of developer tools including a skin builder and IDE integration.

There are a few demos available at DHTMLX Touch – Live Demos and the DHTMLX Touch 1.0 Released blog post details the major features of the library.

Recently at work we were discussing a new trend in web application development, whereby markup is rendered on both the server and client. The Google+ team talked about this openly in this post by Mark Knichel, with particular focus given to performance. Why does rendering content on both the server and the client improve performance?

This is clearly something that people seem to think is a good idea. Some templating languages, like mustache, are available in many different languages, making it easy to use them on both the client and server. Now, if you’re a front-end developer by trade, you might be more comfortable with rendering everything on the client. That doesn’t mean that it’s faster; it’s more of a comfort issue.

So, community, my questions for you are as follows:

• Is Google+’s approach of rendering markup on the server really faster than transmitting data?
• Are there ideological or philosophical arguments for delivering data or markup?
• Is it appropriate to mix both approaches?

Node 6.1

Node 0.6.1 was released a few days ago, which adds improvements to documenation, cluster, and includes a host of bug fixes.

Scaffoldit

Scaffoldit (npm: scaffoldit) by Craig Condon is a scaffolding library that makes it easy to take user input and display data using templates. Templates can be written using mustache, and it has command-line support through Craig’s celeri library.

One of the examples reminds me of skeleton app generators that come with frameworks like Express:

scaffoldit({
  input: {
    projectName: 'What is your project name?'
  },

  build: function(ops, next) {
    scaffoldit.fromDir(ops, '/path/to/input/dir', '/path/to/output/dir');
  },

  complete: function() {
    //...
  }
});

Also worth noting is Tim Caswell’s Step library is used to simplify the internal asynchronous code.

Mixture

Mixture (License: MIT, npm: mixture) is a “heterogeneous cluster task manager” by Daniel Shaw. It can be used to programmatically create a diverse set of forked tasks using a straightforward API:

var mix = require('mixture').mix()
  , ioPort = 8880
  , nodeId = 0
  , i
  , workers = [];

// announce data server
mix.task('announce').fork('data.js');

// socket.io instances
var socketio = mix.task('socket.io', { filename: 'app.js' })

for (i = 0; i < count; i++) {
  ioPort++;
  nodeId++;
  workers.push(socketio.fork({ args: [ioPort, nodeId] }));
}

// At some point, workers[n].kill() could be called

Daniel has written an example app that uses the library, called Stock Quote Stream.

Building a Slimmer jQuery

Building a Slimmer jQuery is a post on the official jQuery blog that discusses guidelines for feature deprecation, as the importance of small file sizes for mobile delivery is becoming more predominant. This doesn’t mean blanked requirements like IE6 support will be removed, even though reducing browser support is a feature of competing micro-frameworks aimed at mobile devices. It does, however, mean more features will move to plugins in the future:

Features that can be implemented via plugins, special events, attribute hooks, or other jQuery extension mechanisms are likely to stay outside core for now.

jQuery Fundamentals

jQuery Fundamentals by Rebecca Murphey (License: CC BY-SA 3.0) is a freely available book all about jQuery, with a primer on JavaScript, and coverage of everything from core features like events to plugin authorship.

It’s an extremely thorough tome on the subject, with contributions from such developers as James Padolsey and Paul Irish. The example code is even available on GitHub under rmurphey / jqfundamentals.

Stalactite

Stalactite (GitHub: jonobr1 / stalactite, License: Apache Version 2.0) by Jono Brandel is a “lightweight element packing” plugin. It will attempt to lazily stack images, videos, and even iframes into a container.

Basic usage is simply $('#container').stalactite();, but there are options for controlling animations, and a callback can be run on completion.

When writing asynchronous code it quickly becomes apparent that certain patterns are more natural than others. These patterns are fairly generic, so many people have started to rely on control flow libraries rather than reimplementing the same patterns.

There are a lot of control flow libraries for Node, and they all solve the same problems in subtly different ways. Some are more accurate than others with regard to established nomenclature, particularly when it comes to design patterns that were already popular in other languages.

People usually discover the need for control flow libraries when running something like this:

files.forEach(function(file) {
  fs.readFile(file, function() {
    // Callback
  });
});

A set of files is going to be read with an asynchronous callback. That’s great, but what happens when we need to determine when every callback has finished?

One solution is to use a counter:

files.forEach(function(file) {
  fs.readFile(file, function() {
    // Callback
    fileCount--;
    if (fileCount === 0) {
      events.emit('complete');
    }
  });
});

events.on('complete', function() {
  // All files have been processed
});

However, adding a few more levels of asynchronous dependencies will make this hard to manage. This is where libraries step in to provide another layer of abstraction.

One such library is Tim Caswell’s Step, which is probably the first control flow library I saw:

Step(
  function readSelf() {
    fs.readFile(__filename, this);
  },
  function capitalize(err, text) {
    if (err) throw err;
    return text.toUpperCase();
  },
  function showIt(err, newText) {
    if (err) throw err;
    console.log(newText);
  }
);

There are dozens of other, similar libraries available through npm. Which one should you use? I’ve made a table of the popular ones to try to get a handle on the situation. Post your favourites in the comments and I’ll take a look at them.

Backbone.ioBind

Backbone.ioBind (GitHub: logicalparadox / backbone.iobind, License: MIT) by Jake Luer binds socket.io events to Backbone model and collection events, and also adds a drop in replacement for Backbone.sync that uses socket.io. This makes building applications that sync across multiple browser instances much easier.

There’s an obligatory example to-do list app, with instructions for installation here: Backbone.ioBind Example App.

Using ioBind in models and controllers is straightforward enough:

// Example Model.extend
urlRoot: 'todo',
initialize: function() {
  _.bindAll(this, 'serverChange', 'serverDelete', 'modelCleanup');
  this.ioBind('update', window.socket, this.serverChange, this);
  this.ioBind('delete', window.socket, this.serverDelete, this);
}

// Example Collection.exend
url: 'todos',
initialize: function() {
  _.bindAll(this, 'serverCreate', 'collectionCleanup');
  this.ioBind('create', window.socket, this.serverCreate, this);
}

Jake notes that Backbone.ioBind works well with Backbone.ModelBinding, which is another useful Backbone library for data binding.

One Millionth Tower

One Millionth Tower is a WebGL and HTML5-powered documentary about high-rise apartments, with one particular neighbourhood explored in detail:

You see them all over the world. More than a billion of us live in highrises. But most of these low- and middle-income buildings are now aging and falling into disrepair. Could life in the global highrise be different?

The technology used to drive the documentary is available on GitHub here: m4robb / one-millionth-tower.

Popcorn.js

Popcorn.js (GitHub: webmademovies / popcorn-js, License: MIT) from Mozilla is a HTML5 media framework for creating timeline-based web presentations. This means events on a video’s timeline can coincide with images, text, or rich data from sources like social networks. A list of Popcorn.js plugins demonstrates some of the current possibilities.

I’ve reviewed similar frameworks before on DailyJS, but this must be the most advanced and polished implementation I’ve seen so far.

The biggest problem I’ve had when writing tests for Turing is checking when asynchronous callbacks complete. For example, when testing the resource loading feature, I wrote tests with this pattern:

'test queue loading with no local scripts': function() {
  $t.require([
    'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js'
  ]).on('complete', function() {
    assert.ok(true);
    assert.done();
  }).on('loaded', function(item) {
    if (item.src === 'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js') {
      assert.ok(jQuery, 'jQuery should be set');
    }
  });
}  

What happens if loaded never fires? Well, the test would still pass. That’s a consequence of the assertion never being run.

One solution is to wrap the assert module with counters that count how many assertions have been called. Then at the start of a test we can write assert.expect(2); to say ‘raise an error if anything other than two assertions are run’.

That’s fine, but at this point Turing’s test framework always runs tests asynchronously. If the assert module kept a counter, other tests would increment that value too. The number of assertions would be the total number for all of the current suite’s tests, rather than the current test.

Counting Assertions

The initial solution I came up with was to wrap every assertion method with a function that incremented a counter. Tests take this pattern:

'test queue loading with no local scripts': function() {
  var assertExpect = mixinExpect(assert);
  assertExpect.expect(2);

  $t.require([
    'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js'
  ]).on('complete', function() {
    assertExpect.ok(true);
    assertExpect.done();
  }).on('loaded', function(item) {
    if (item.src === 'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js') {
      assertExpect.ok(jQuery, 'jQuery should be set');
    }
  });
}

Each test gets its own instance of the assert module.

The code to do this is slightly clumsy, because I aliased the old methods using a __ prefix:

function mixinExpect(m) {
  var m2 = {}, method;

  for (method in m) {
    if (m.hasOwnProperty(method)) {
      m2['__' + method] = m[method];
      (function(methodName) {
        m2[methodName] = function() {
          m2.mixinExpectAssertionCount++;
          m2['__' + methodName].apply(m2, arguments);
        };
      }(method));
    }
  }

  m2.expect = function(count) {
    m2.mixinExpectAssertionCount = 0;
    m2.mixinExpectExpected = count;
  };

  m2.done = function() {
    if (m2.mixinExpectAssertionCount !== m2.mixinExpectExpected) {
      throw('Expected assertion count was not found, expected: ' + m2.mixinExpectExpected + ', got: ' + m2.mixinExpectAssertionCount); 
    }
  };

  return m2;
}

It does the job, but is there a more elegant way that doesn’t require changing the assert module?

Expectations

I got this idea when I reviewed Tim Caswell’s Step library:

'test queue loading with no local scripts': function() {
  var expect = new Expect();
  expect.add('jQuery was set');
  expect.add('loaded fired');

  $t.require([
    'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js'
  ]).on('complete', function() {
    assert.ok(true);
    expect.fulfill('loaded fired');
    expect.done();
  }).on('loaded', function(item) {
    if (item.src === 'https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js') {
      expect.fulfill('jQuery was set');
      assert.ok(jQuery, 'jQuery should be set');
    }
  });
}

Each test creates an instance of Expect and adds a list of expectations. As the expectations are fulfilled, it marks down this fact, then at the end of the test when expect.done() is called it can see if any expectations were unfulfilled.

I made a quick implementation and it’s only a few lines of code:

function Expect() {
  this.expectations = {};
}

Expect.prototype = {
  add: function(expectation) {
    this.expectations[expectation] = false;
  },

  fulfill: function(expectation) {
    this.expectations[expectation] = true;
  },

  done: function() {
    for (var expectation in this.expectations) {
      if (!this.expectations[expectation]) {
        throw('Expected assertion was fulfilled , expected: ' + expectation); 
      }
    }
  }
};

Conclusion

Working with asynchronous tests can quickly get confusing — sometimes tests appear to be passing but aren’t actually running as expected. The CommonJS Unit Testing spec actually says the following:

The assertions defined above will not be to everyone’s taste style wise (or infact behaviour wise.) Authors are free to define their own assertion methods in new modules as desired.

So adding a assert.expect method to your own assertion module implementations is perfectly acceptable.

This week’s code can be found in the ajax tests in commit dc3b41f.