DailyJS: A JavaScript Blog

jQuery Roundup

07 Dec 2010 | By Alex Young | Comments |

Welcome to the jQuery roundup 35. You can send your plugins and articles in for review through our contact form or @dailyjs.

Head JS

Head JS by Tero Piirainen (GitHub) is a parallel script loader. The project has the byline “the only script in your <head>”. It also includes some other features:

onReady handling based on when scripts have been loaded with head.ready
CSS modernizer and HTML5 backwards compatibility
Dynamic CSS loading based on conditions
Browser detection
JavaScript feature detection

Which makes this library a flexible meta-library that sits above your favourite frameworks like jQuery.

The story behind Head JS is interesting because the project was being quietly developed by the author… that is until it was discovered on GitHub and apparently went viral:

This project was never announced. git push and it was all viral.

Unlike most projects out there, the Piirainen made a good site with detailed documentation early on, so it just goes to show how packaging your projects can help them succeed. Dumping jQuery plugins on free hosting with a poor website and little or no documentation isn’t a good way to get your work noticed.

TumblrPostMap

TumblrPostMap by Cláudio Gil (demo, documentation) is another well-documented project that aims to enable Tumblr blogs to show graphs with as little work as possible. It looks at posts tagged with geo for markup like this:

<p class="geo">Uluru, NT, Austrália: 
    <span class="latitude">-25.345</span>, <span class="longitude">131.036111</span>
</p>

I like the fact he’s using the Geo microformat to structure the markup. The Tumblr demo blog doesn’t seem to work right now, I’m not sure if this is due to Tumblr’s recent downtime or the fact it’s being actively worked on. Anyway, kudos for the documentation!

Fixed Center

Fixed Center (demo) by David Tang is a simple plugin that centres a div on a page. It’s the kind of thing a client wants you to do that you’d rather not have to spend much time on. He recently updated it with performance enhancements, so it feels fairly lightweight.

Packaging Your Plugins

I’ve said it many times before, but please package your jQuery projects with the following:

A README with basic information
A license
Documentation
Tests, however limited
A corresponding website and demo

GitHub even hosts pages for free, so you’ve got no excuse for not having a site or documentation. If you’re going to spend time building a plugin, why not package it in a way that makes it easy for us to use? Even better, your work will be more likely to be featured on sites like this!

Node Tutorial Part 5

06 Dec 2010 | By Alex Young | Comments |

server node tutorials lmawa nodepad

Welcome to part 5 of Let’s Make a Web App, a tutorial series about building a web app with Node. This series will walk you through the major areas you’ll need to face when building your own applications. These tutorials are tagged with lmawa.

Previous tutorials:

Part 1: Introduction
Part 2: Installation and Skeleton App, source code commit: 4ea936b
Part 3: RESTful Methods and Testing, source code commit: 39e66cb
Part 4: Templates, Partials, Creating and Editing Documents, source code commit: f66fdb

Before starting this tutorial remember to start up a mongo daemon if your computer doesn’t run one automatically.

Authentication

We’ve built a serviceable app, but it’s not that useful without some kind of authentication system. Most production systems and client projects require authentication, and even though there are interesting initiatives like OpenID and OAuth, most commercial projects are likely to have their own login system.

This is usually achieved with a session:

A user fills out a form with their username and password
The password is encrypted with a hashing algorithm and salt
This value is compared with the user’s record in the database
If it matches, a session key is generated that identifies the user

We need the following things to manage users and sessions:

Users in the database
Sessions that can store the logged-in user ID
Password encryption
A way of restricting access to routes that require a logged-in user

Sessions in Express

Express relies on Connect’s session middleware, which is backed by a data storage mechanism. There’s a memory-based store, and third-party stores including connect-redis and connect-mongodb. An alternative option is cookie-sessions which stores session data inside the user’s cookie.

A session can be configured like this:

app.use(express.cookieDecoder());
app.use(express.session());

It’s very important to place these configuration options in the right place during configuration, else the session variable won’t appear in request objects. I’ve put it after bodyDecoder and before methodOverride. Refer to the full source on GitHub to see this in context.

Now our HTTP responders will have access to req.session:

app.get('/item', function(req, res) {
  req.session.message = 'Hello World';
});

MongoDB Sessions

Install connect-mongodb with npm install connect-mongodb.

connect-mongodb works like any other session store. During application configuration we need to specify our connection details:

app.configure('development', function() {
  app.set('db-uri', 'mongodb://localhost/nodepad-development');
});

var db = mongoose.connect(app.set('db-uri'));

function mongoStoreConnectionArgs() {
  return { dbname: db.db.databaseName,
           host: db.db.serverConfig.host,
           port: db.db.serverConfig.port,
           username: db.uri.username,
           password: db.uri.password };
}

app.use(express.session({
  store: mongoStore(mongoStoreConnectionArgs())
}));

Most of this code wouldn’t be required if the API authors could decide on a standard format for connection options. I’ve written a function that extracts the connection details from Mongoose. In this example, db holds a Mongoose connection instance. Mongoose expects connection details to be provided through URIs, which I like because it’s easy to remember the format. I’ve stored environment-specific connection strings using app.set.

When writing Express applications it’s a good idea to use app.set('name', 'value'). Just remember that app.set('name') is used to access the setting, rather than app.get.

Running db.sessions.find() in the mongo console will now return any sessions that have been created.

Access Control Middleware

Express provides an elegant way of restricting access to logged-in users. When HTTP handlers are defined, an optional route middleware parameter can be specified:

function loadUser(req, res, next) {
  if (req.session.user_id) {
    User.findById(req.session.user_id, function(user) {
      if (user) {
        req.currentUser = user;
        next();
      } else {
        res.redirect('/sessions/new');
      }
    });
  } else {
    res.redirect('/sessions/new');
  }
}

app.get('/documents.:format?', loadUser, function(req, res) {
  // ...
});

Now every route that requires a logged-in user can be handled by adding loadUser to the route. The middleware itself gets the usual route parameters, and also next — this can be used to run the route handler based on arbitrary logic. In our project a user is loaded using a user_id in the session; if the user cannot be found next simply isn’t called and the browser is redirected to the login screen.

RESTful Session Modelling

I’ve modelled sessions in a similar way to documents. There are new, create, and delete routes:

// Sessions
app.get('/sessions/new', function(req, res) {
  res.render('sessions/new.jade', {
    locals: { user: new User() }
  });
});

app.post('/sessions', function(req, res) {
  // Find the user and set the currentUser session variable
});

app.del('/sessions', loadUser, function(req, res) {
  // Remove the session
  if (req.session) {
    req.session.destroy(function() {});
  }
  res.redirect('/sessions/new');
});

User Model

The User model is more complicated than the Document model because it has to contain authentication-related code. The strategy I’ve used, which you’ve probably seen before in OO web frameworks, is:

Passwords are stored in an encrypted format with a salt
Authentication can be performed by comparing an encrypted plain text password with the password in the database for a given user
A ‘virtual’ password property exposes a plain text password for the convenience of registration and login forms
This property has a setter which automatically encrypts the password before saving
A unique index is used to ensure there’s only one user for each email address

The password encryption uses Node’s standard crypto library:

var crypto = require('crypto');

mongoose.model('User', {
  methods: {
    encryptPassword: function(password) {
      return crypto.createHmac('sha1', this.salt).update(password).digest('hex');
    }
  }
});

encryptPassword is an instance method which returns a sha1-hashed password with a salt. The salt is generated before encrypting the password in the password setter:

mongoose.model('User', {
  // ...

  setters: {
    password: function(password) {
      this._password = password;
      this.salt = this.makeSalt();
      this.hashed_password = this.encryptPassword(password);
    }
  },

  methods: {
    authenticate: function(plainText) {
      return this.encryptPassword(plainText) === this.hashed_password;
    },

    makeSalt: function() {
      return Math.round((new Date().valueOf() * Math.random())) + '';
    },

    // ...

The salt could be anything you like, I’ve generated a fairly random string here.

Saving Users and Registration

Mongoose makes it easy to do things when records are saved by overriding the save method:

mongoose.model('User', {
  // ...
  methods: {
    // ...

    save: function(okFn, failedFn) {
      if (this.isValid()) {
        this.__super__(okFn);
      } else {
        failedFn();
      }
    }

    // ...

I’ve overridden save to allow for a failed save method. This makes the failed registration handling simpler:

app.post('/users.:format?', function(req, res) {
  var user = new User(req.body.user);

  function userSaved() {
    switch (req.params.format) {
      case 'json':
        res.send(user.__doc);
      break;

      default:
        req.session.user_id = user.id;
        res.redirect('/documents');
    }
  }

  function userSaveFailed() {
    res.render('users/new.jade', {
      locals: { user: user }
    });
  }

  user.save(userSaved, userSaveFailed);
});

No error messages are displayed right now; I’ll get to that in another tutorial.

Even though this validation is pretty dumb, the index is critical to this application:

mongoose.model('User', {
  // ...

  indexes: [
    [{ email: 1 }, { unique: true }]
  ],

  // ...
});

This will prevent duplicate users from being saved. The format is the same as MongoDB’s ensureIndex.

Conclusion

As of commit 03fe9b2 we now have:

MongoDB sessions
User model, with support for sha1 password encryption
Routing middleware for controlling access to documents
User registration and login
Session management

I’ve updated the Jade templates to include a basic login form.

There are a few things missing from this version though:

Documents don’t take into account the owner
The tests don’t work properly because I’m having trouble figuring out session handling in Expresso tests

I’ll get to these things in future parts of this tutorial series.

Prototype Update, Kaffeine, js-signals, Zepto, rain.js

03 Dec 2010 | By Alex Young | Comments |

zepto prototype languages graphics

Prototype 1.7

Prototype 1.7 is out, and activity on 1.7.0.1 has begun already. This is the version of Prototype that adds Sizzle, $().on, Element.Layout, and ES5 compliance.

Kaffeine

Kaffeine (License) by Jonah Fox progressively enhances JavaScript while remaining backwards compatible. The documentation has examples of the bang syntax, @ alias for this, implicit brackets and more.

That means code that looks like this:

Edge::add = (nick, name, complete, opts) {
  opts ||= {}
  @client.select 15
  user = User.find! {id: nick}
  puzzle = Puzzle.find! {name: name}
  err, data = client.set! "u:#{user}:p:#{puzzle}"
  complete()
}

Will turn into this:

Edge.prototype.add = function(nick, name, complete, opts) {
  opts = opts || {}
  this.client.select(15)
  User.find({id: nick}, function(user) {
    Puzzle.find({name: name}, function(puzzle) {
      client.set(("u:" + (user) + ":p:" + (puzzle)), function(err, data) {
        return complete()
      })
    })
  })
}

The author has outlined his philosophy for the project:

Progressive enchancement
Concentrate on small useful feature set
Hackable, modular, extendable and testable
Whitespace is not significant

In case you’re wondering, the parser tokenizes input and eval’s the result.

js-signals

js-signals (MIT License) by Miller Medeiros is an event/messaging system for JavaScript:

A Signal is similar to an Event Emitter/Dispatcher or a Pub/Sub system, the main difference is that each event type has it’s own controller and doesn’t rely on strings to broadcast/subscribe to events. To know more about the differences and advantages/disadvantages of each approach check the Wiki page.

One of the claimed advantages of this project is to favor composition over inheritance. This idea appeals to me and the examples make it seem simple enough to get started with fairly quickly (should you need a pub/sub-like messaging system for your next project).

Zepto 0.2

Zepto 0.2 by Thomas Fuchs is out. This version adds support for Backbone.js which seems like a good fit for the WebKit iOS-app style projects people are building with it.

Rain.js

Rain.js (MIT License) by Julio Cesar Ody generates a great rain effect, there’s a demo here. As you might have noticed I’m a sucker for pretty graphical tricks. If you’ve got any scene-style JavaScript hacks please send them in!

Let's Make a Framework: Free eBook

02 Dec 2010 | By Alex Young | Comments |

frameworks tutorials lmaf

eBook

I’ve collected and edited the Let’s Make a Framework articles into a book that suitable for e-readers. Consider this a Christmas present!

Note: Remember that this book is based on progress up to commit 09d2c3. As the framework changes the book might refer to obsolete parts of the framework, so keep this in mind if referring to the latest version of turing.js. Older commits are available in turing.js’s history from GitHub.

Recap

Last week we finished building a CommonJS-based test framework. Next we need to replace the framework’s tests with tests written using this framework. First I’d like to review the progress we’ve made so far on the project.

If you’re new to the series or feeling lost after the last few parts, here’s a summary of every part so far. Feel free to dive in at any point that interests you.

Introduction
1. Introduction to the Series
2. Library Architecture
Object Oriented JavaScript
1. Classes, Inheritance, Extend
2. Class Creation In Depth
Functional Programming
1. Introduction, Iterators, Performance Concerns
2. More Functional Methods
Selector Engine
Events
Aliasing and Packing
1. jQuery-Style Aliasing, Packing and Minification
Ajax
1. Introduction, History, Request Objects, Sending Requests, Popular APIs
2. Cross-Domain Requests, Implementations in the Wild, API Design
Animations
Touch
1. Supporting Touchscreen Devices, Orientation
2. Events, State
Chained APIs
Test Framework

Rewriting the tests to work with turing-test.js
Revised packaging solution, JsLint builds
CSS API
Pseudo-selectors
Better DOM manipulation
Bug fixes and browser support

Node Roundup

01 Dec 2010 | By Alex Young | Comments |

node server scraping services testing

Welcome to the Node Roundup. Send in your apps and libraries using our contact form or @dailyjs.

Node.io

node.io (MIT License) by Chris O’Hara is a distributed data scraping and processing framework. It uses node-htmlparser, node-soupselect, and multi-node to provide flexible and friendly tools for building scrapers.

There’s a node.io command-line script that runs jobs. Jobs look a bit like standard CommonJS modules:

var nodeio = require('node.io'),
    options = {},
    methods = {};

exports.job = new nodeio.Job(options, methods);

The methods variable contains run and fail methods that will be run on each worker. Jobs can be linked together through STDIN and STDOUT.

I actually have a freelance client for whom I write a lot of scrapers (permission is given by the scraped sites), and I kept thinking Node would be a good choice for scraping. In particular writing selectors with a jQuery-like API makes scraping pretty easy. I haven’t looked at how deep the HTTP manipulation can get; I’ve needed access to cookies and the full range of HTTP methods to scrape some sites.

JsApp.US

JsApp.US by Matthew Francis-Landau (sent by @jefkoslowski) is a hosting platform for Node apps. It’s possible to run an app without registering, but user accounts can be created for features like sharing apps or accessing a virtual file system.

There’s a database API, and some sample apps.

I think the project might be open sourced at some point, because the GitHub repository that contains the wiki says “When the source becomes public, this is where it will be”.

Should

should.js (MIT License) by that stalwart JavaScript hacker TJ Holowaychuk is a test framework agnostic assertion library for Node. The syntax is possibly inspired by Thoughtbot’s shoulda, but it’s a little bit different:

var user = {
    name: 'tj'
  , pets: ['tobi', 'loki', 'jane', 'bandit']
};

user.should.have.property('name', 'tj');
user.should.have.property('pets').with.lengthOf(4)

It sounds like TJ has a lot of pets!

As you might have realised, this library extends Object with a getter. By using a single getter on Object, TJ has managed to cut down a lot of line noise that would typically be created by using function calls. This might put you off, but he addresses this decision in his README:

OMG IT EXTENDS OBJECT???!?!@ Yes, yes it does, with a single getter should, and no it wont break your code, because it does this properly with a non-enumerable property.

If any of this seems confusing, look at the documentation for defineProperty and read through lib/should.js:

Object.defineProperty(Object.prototype, 'should', {
  set: function(){},
  get: function(){
    return new Assertion(this);
  }
});

jQuery Roundup

30 Nov 2010 | By Alex Young | Comments |

jquery plugins graphics events

Welcome to the jQuery roundup 34. You can send your plugins and articles in for review through our contact form or @dailyjs.

London Ajax Meetup

The next London Ajax Meetup will be on Tuesday December 14th. The talks are on YUI, ExtJS, jQuery UI, and more.

If you’ve got a jQuery-related event planned, let me know!

jQuery UI Updates

This month saw the release of jQuery UI 1.8.6 which adds support for jQuery 1.4.3 and IE 9, as well as a whole bunch of bug fixes.

jQuery UI 1.9 Milestone 3 was also released, which adds a new Spinner widget:

A spinner is a simple widget that allows users to increment or decrement the current text box value without having to input it manually. Increments do not have to be whole numbers — they can be set to decimal values (0.1) or large increments (5) for each click.

The Accordion API has also been redesigned. It removes some options in favour of alternatives, and it makes the API a little bit cleaner and more consistent.

Slides

Slides Logo

Slides (GitHub: nathansearles / Slides, Apache license) is another slideshow plugin for jQuery. Like most modern sliding image plugins, this one uses simple markup to represent the gallery, then uses a simple call, $('#element').slides() to set everything up.

Slides has some useful features like image preloading and pagination, and a boatload of customisation options for controlling things like transition effects.

Snowfall Plugin

It’s snowing in the UK and I noticed this animated Snowfall Plugin by Jason Brown, so I thought I’d include it to be festive. The snowflakes are actually divs, imagine that!

JavaScript Developer Survey

29 Nov 2010 | By Alex Young | Comments |

community surveys

Last year I ran the DailyJS JavaScript Developer Survey (results). It produced some interesting results that are useful to both open source and commercial developers, so I’d like to run it again.

The results of this survey will be shared with the community on DailyJS, and are also visible after submitting the survey.

No personal details will be collected in this survey.

The survey is now closed.

A Simple Linked Data and JavaScript Tutorial

26 Nov 2010 | By Ric Roberts | Comments |

linked-data tutorial

Disclaimer: This article by Ric is a short tutorial on how to consume Linked Data using JavaScript. Some of the examples use Ric’s company’s new Linked Data publishing platform, PublishMyData.

Introduction

Mention Linked Data or RDF and many developers run screaming. The truth is that Linked Data is really rather simple (and in many cases you don’t even need to use RDF if you don’t want to). In this tutorial we’re going to use jQuery to request some data from a Linked Data service, and then display the results in a table and on a map. You can see the final result of the tutorial here (just view source to see all the code).

Writing the SPARQL Query

We’ll need to run some SPARQL to get the data we’re interested in (SPARQL is a query language analogous to SQL for relational databases). The following query selects the names and locations (northing and easting) of all the secondary schools in the City of Manchester (district ‘00BN’).

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

SELECT ?name ?northing ?easting WHERE {
  ?school <http://education.data.gov.uk/def/school/districtAdministrative> <http://statistics.data.gov.uk/id/local-authority-district/00BN> .
  ?school rdfs:label ?name .
  ?school <http://education.data.gov.uk/def/school/phaseOfEducation> <http://education.data.gov.uk/def/school/PhaseOfEducation_Secondary> .
  ?school <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/northing> ?northing .
  ?school <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/easting> ?easting .
}

Constructing the URL to call against the endpoint

The SPARQL endpoint we’re using expects requests in the following format:

http://sparql.publishmydata.com?query=URL-ENCODED-SPARQL&output=FORMAT

We could have used the encodeURI JavaScript function to encode the SPARQL, but I just copied and pasted the url from the browser’s address bar after clicking on the “JSON” link under the results on this page, where I originally wrote the SPARQL.

{
  "head": {
    "vars": [ "name" , "northing" , "easting" ]
  } ,
  "results": {
    "bindings": [
      {
        "name": { "type": "literal" , "value": "Parrs Wood High School" } ,
        "northing": { "datatype": "http://www.w3.org/2001/XMLSchema#integer" , "type": "typed-literal" , "value": "390094" } ,
        "easting": { "datatype": "http://www.w3.org/2001/XMLSchema#integer" , "type": "typed-literal" , "value": "385464" }
      },
      ...
    ]
  }
}

The array in head.vars contains the variables that we selected in the SPARQL query, and results.bindings contains one object per row of results.

Using jQuery to call the URL

We’ll just call the query against the endpoint using jQuery’s ajax function (but you could use any other framework if you prefer). Notice that we use JSONP data type to avoid CORS issues.

 
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.js"></script>
<script type="text/javascript">
  //<![CDATA[
  var queryUrl = "http://publishmydata.com/sparql.json?q=PREFIX+rdfs%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3E%0D%0A%0D%0ASELECT+%3Fname+%3Fnorthing+%3Feasting+WHERE+%7B%0D%0A%0D%0A++%3Fschool+%3Chttp%3A%2F%2Feducation.data.gov.uk%2Fdef%2Fschool%2FdistrictAdministrative%3E+%3Chttp%3A%2F%2Fstatistics.data.gov.uk%2Fid%2Flocal-authority-district%2F00BN%3E+.+%0D%0A%0D%0A++%3Fschool+rdfs%3Alabel+%3Fname+.%0D%0A%0D%0A++%3Fschool+%3Chttp%3A%2F%2Feducation.data.gov.uk%2Fdef%2Fschool%2FphaseOfEducation%3E+%3Chttp%3A%2F%2Feducation.data.gov.uk%2Fdef%2Fschool%2FPhaseOfEducation_Secondary%3E+.%0D%0A%0D%0A++%3Fschool+%3Chttp%3A%2F%2Fdata.ordnancesurvey.co.uk%2Fontology%2Fspatialrelations%2Fnorthing%3E+%3Fnorthing+.%0D%0A%0D%0A++%3Fschool+%3Chttp%3A%2F%2Fdata.ordnancesurvey.co.uk%2Fontology%2Fspatialrelations%2Feasting%3E+%3Feasting+.%0D%0A%7D";

  $.ajax({
    dataType: "jsonp",  
    url: queryUrl
  });             
  //]]>
</script>

Making an HTML table from the results

Let’s create a table in the html:

            
<table id="results"></table>

… and add a success callback, to populate it with the results of the SPARQL.

$.ajax({
  dataType: "jsonp",
  url: queryUrl
  success: function(data) {    
    // get the table element
    var table = $("#results");              

    // get the sparql variables from the 'head' of the data.
    var headerVars = data.head.vars; 

    // using the vars, make some table headers and add them to the table;
    var trHeaders = getTableHeaders(headerVars);
    table.append(trHeaders);  

    // grab the actual results from the data.                                          
    var bindings = data.results.bindings;

    // for each result, make a table row and add it to the table.
    for(rowIdx in bindings){
      table.append(getTableRow(headerVars, bindings[rowIdx]));
    } 
  }
});

In the code above, I’ve used a few simple helper functions which just loop through the data from the JSON, and make table rows and cells from it.

function getTableHeaders(headerVars) {
  var trHeaders = $("<tr></tr>");
  for(var i in headerVars) {
    trHeaders.append( $("<th>" + headerVars[i] + "</th>") );
  }
  return trHeaders;
}

function getTableRow(headerVars, rowData) {
  var tr = $("<tr></tr>");
  for(var i in headerVars) {
    tr.append(getTableCell(headerVars[i], rowData));
  }
  return tr;
}

function getTableCell(fieldName, rowData) {
  var td = $("<td></td>");
  var fieldData = rowData[fieldName];
  td.html(fieldData["value"]);
  return td;
}

Plotting the schools on a map

Now things start to get interesting. To draw the map, we’ll need to include a couple of external scripts:

<!-- Google maps api-->
<script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"> </script>

<!-- Ordnance survey Lat-long converter (from: http://mapki.com/wiki/Tools:Snippets) -->
<script type="text/javascript" src="http://www.bdcc.co.uk/LatLngToOSGB.js"></script>

The LatLngToOSGB.js script is for converting northings/eastings into latitude/longitudes. Frustratingly, Google uses the World Geodetic System, for lat/long but the NEtoLL function from that library returns values using the Ordnance Survey National Grid. So we’ll need to do a 2-step process to convert them if we want accurate positions for the map points. The functions below wrap up that conversion for you:

// extra conversion functions: Google uses WGS84 whereas the National Grid uses OSGB36
// (thanks to: http://www.roxburgh.net/projects/googlemaps/)

// Returns object with attributes lat and lon
function ENtoLL84(easting, northing) {
  var vlatlng = NEtoLL(easting, northing);
  return OGBToWGS84(vlatlng.lat, vlatlng.lon, 0);
}

// Returns object with attributes east and north
function LL84toEN(lat, lon) {
  var vlatlon = WGS84ToOGB(lat, lon, 0);
  return LLtoNE(vlatlon.lat, vlatlon.lon);
}

Now we’ve got that out of the way, lets make a div for the map to live in:

<div id="map_canvas" style="width:500px; height:500px;"></div>

… and a function to plot the schools’ locations:

function drawMap(data) {
  // make a map, centred on Manchester
  var manchesterLatLon = new google.maps.LatLng(53.480110, -2.237940);
  var myOptions = {
    zoom: 11,
    center: manchesterLatLon,
    mapTypeId: google.maps.MapTypeId.ROADMAP
  };
  var map = new google.maps.Map(document.getElementById("map_canvas"), myOptions);
 
  // add a marker for each school 
  var bindings = data.results.bindings; 
  for (var i in bindings) {
  
    var schoolData = data.results.bindings[i];
   
    var northing = schoolData["northing"]["value"];
    var easting = schoolData["easting"]["value"];
    var name = schoolData["name"]["value"];
  
    // call our 2-step conversion
    var latLng = ENtoLL84(easting, northing);

    // make a marker with the label as the school name
    var marker = new google.maps.Marker({
      position: new google.maps.LatLng( latLng.lat, latLng.lon ),
      map: map,
      title: name
    });
  } 
}

Now all that’s left to do is to add the call to drawMap to the ajax success callback.

$.ajax({
  url: queryUrl
  success: function(data) {
  
    // ... existing code here
  
    drawMap(data);
  }
});

You should now have something that looks a bit like this page.

Useful links

☠ Let's Make a Test Framework

25 Nov 2010 | By Alex Young | Comments |

frameworks tutorials lmaf testing

Welcome to part 40 of Let’s Make a Framework, the ongoing series about building a JavaScript framework.

If you haven’t been following along, these articles are tagged with lmaf. The project we’re creating is called Turing.

Last week we continued building turing-test.js, a unit testing framework. The idea behind this framework is to build something that follows the CommonJS specifications and works in modern browsers. This will help us test Turing more effectively.

Previous parts:

Part 36 – An introduction to how JavaScript frameworks handle testing
Part 37 – A review of CommonJS test specifications, and examples of how to run assert module tests in a browser with a simple test runner
Part 38 and commit 9a4c33e – Assertions part 1
Part 39 and commit 8dd48b5 – Assertions part 2

Test Output Design

It’s time to make the test results easier to read. The test runner runs each method that is prefixed with test, as per the CommonJS specifications. I’d like it to display results a little bit like the jQuery test suit.

The test runner should list each test method that was run. Because we can write test method names with full text rather than a JavaScript function name, the output should be easy to follow:

exports['test strictEqual'] = function() {
  assert.strictEqual('1', '1', "'1' should be equal to '1'");
  assert.strictEqual(1, 1, '1 should be equal to 1');
};

The test runner could display this test as:

[OK] test strictEqual

Another thing to consider is the design of failed tests. The code we wrote way back at the start of this test framework detour already checks to see if exceptions have corresponding stack properties:

run: function(testName, obj) {
  var result = new Tests.Result(testName);

  function showException(e) {
    if (!!e.stack) {
      logger.display(e.stack);
    } else {
      logger.display(e);
    }
  }

We should see a full stack trace in the console, but browsers might not display it. At its most basic, failed tests should look like this:

✕ Assertion failed in: test strictEqual
  AssertionError: '1' should be equal to '1'
  In "===":
    Expected: 2
    Found: 1

The HTML output will require styles to preserve white space for stacktraces.

Colours and Prefixes

In HTML and consoles that support colour, red and green can be used to indicate pass or fail. We’ll also need another visual indicator for red/green colourblind people. I’ve opted to use HTML entities and UTF-8 symbols for these indicators:

Error is a skull and crossbones: ☠
Pass is a tick: ✓
Fail is a cross: ✕

This is purely superficial, I just thought readers might find it more interesting than text. There’s a switch statement in lib/test.js that converts the HTML entities to JavaScript UTF-8 codes for the console.

Colours are also converted, but this time based on the message type. The message type is used as a CSS class name, and is also used to determine the console colour:

function messageTypeToColor(messageType) {
  switch (messageType) {
    case 'pass':
      return '32';
    break;

    case 'fail':
      return '31';
    break;
  }

  return '';
}

// ...

var col    = colorize ? messageTypeToColor(messageType) : false;
  startCol = col ? '\033[' + col + 'm' : '',
  endCol   = col ? '\033[0m' : '',
console.log(startCol + (prefix ? htmlEntityToUTF(prefix) + ' ' : '') + message + endCol);

All of these functions are embedded within a closure, and they’re only evaluated if they’re needed, with this simple pattern:

printMessage = (function() {
  function htmlEntityToUTF(text) {
    // Removed for clarity
  }

  function messageTypeToColor(messageType) {
    // Removed for clarity
  }

  if (typeof window !== 'undefined') {
    return function(message, messageType, prefix) {
      // Display message with some simple DOM code
    }
  } else if (typeof console !== 'undefined') {
    return function(message, messageType, prefix) {
      // Display message with console.log()
    };
  } else {
    return function() {};
  }
})();

After this closure is evaluated, printMessage contains everything it needs to display messages. Then all that’s required is a helper logger object:

logger = {
  display: function(message, className, prefix) {
    printMessage(message, className || 'trace', prefix || '');
  },

  error: function(message) {
    this.display(message, 'error', '&#9760;');
  },

  pass: function(message) {
    this.display(message, 'pass', '&#10003;');
  },

  fail: function(message) {
    this.display(message, 'fail', '&#10005;');
  }
};

AssertionError toString

The AssertionError exceptions will need to carefully handle toString to ensure that exceptions are readable.

The way I like to think of failed assertions is they have a summary and extended details. The summary is basically the custom message supplied by the assertion invocation, and the details display the expected value, actual value, and the assertion operator (from lib/assert.js):

assert.AssertionError.prototype.summary = function() {
  return this.name + (this.message ? ': ' + this.message : '');
};

assert.AssertionError.prototype.details = function() {
  return 'In "' + this.operator + '":\n\tExpected: ' + this.expected + '\n\tFound: ' + this.actual;
};

assert.AssertionError.prototype.toString = function() {
  return this.summary() + '\n' + this.details();
};

I’ve based this approach on the test frameworks we looked at in part 36.

The Results

I’ve made a test fail on purpose here to illustrate the results. Console tests should look like this:

And a browser is very similar:

This version of turing-test.js is in commit 5a6cbf61.

References

CommonJS Unit Test 1.0

Paren Free

24 Nov 2010 | By Alex Young | Comments |

language

In Paren-Free Brendan Eich, creator of JavaScript, discusses the potential removal of parentheses around control structure heads:

if year > 2010 {
  syntax++
}

// i is a fresh let binding!
for i in iter {
  frob(i)
}

while lo <= hi {
  let mid = (lo + hi) / 2
}

// array comprehension
return [i * i for i in range(n)]

Eich discusses the difficulties of working in a committee, which gives us some interesting anecdotal details on the people and processes that shape the future of JavaScript.

The good news is that the committee agreed that some kind of meta-programmable iteration should be in the language.

The bad news is that the committee did something committees often do: try to compromise between divergent beliefs or subjective value theories.

His point about a change in lexical scope behaviour is interesting, because it would make one of JavaScript’s most confusing aspects more consistent:

We believe that future JS, the Harmony language, must include at least one incompatible change to runtime semantics: no more global object at the top of the scope chain. Instead, programmers would have lexical scope all the way up.

He also mentions some of the of the ideas that were implemented in Mozilla’s JavaScript 1.7, which many of us are now familiar with through writing server-side JavaScript. How many of you regularly use getters and setters, and array comprehensions now?

In fact, if the suggestions discussed here are ever formalised and accepted, we’re likely to be the first people to use them in earnest. Eich mentions Rhino and SpiderMonkey in his post, but he doesn’t point out how many of us work outside browsers. Of course, browsers are the primary concern, but I feel like the people in a position to take advantage of language changes sooner are server-side developers.

jQuery Roundup

23 Nov 2010 | By Alex Young | Comments |

jquery plugins graphics

Welcome to the jQuery roundup 33. You can send your plugins and articles in for review through our contact form or @dailyjs.

Face Detection

This jQuery face detection plugin by Jay Salvat can detect faces in images using a Canvas. The actual face detection code is by Liu Liu — the plugin provides a jQuery interface to it:

var coords = $('#myPicture').faceDetection();

There’s a demo if you’d like to try it out.

2D Transformations

2D Transformations by Grady provides a simple API to CSS3 transformations, and degrades to Microsoft’s matrix filter where appropriate. The Transform wiki has details on the client-side requirements and implementation, as well as code examples.

It can work with jQuery’s CSS API as well providing its own method:

$('#example').transform({rotate: '30deg'});
$('#example').css({rotate: '30deg'});

jQuery.splitter

I’ve made two web apps recently that use resizable panels, but unfortunately I found jQuery.splitter after I wrote my own code to do the same thing.

I found it easy to implement my own version of this, but Krikus’s version has a lot of features, like disabling text selection during dragging, and minimum and maximum sizes for the panels.

Node Tutorial Part 4

22 Nov 2010 | By Alex Young | Comments |

server node tutorials lmawa nodepad

Welcome to part 4 of Let’s Make a Web App, a tutorial series about building a web app with Node. This series will walk you through the major areas you’ll need to face when building your own applications. These tutorials are tagged with lmawa.

Previous tutorials:

Part 1: Introduction
Part 2: Installation and Skeleton App, source code commit: 4ea936b
Part 3: RESTful Methods and Testing, source code commit: 39e66cb

In this part I’ll add so much stuff you should get up and make a cup of tea first.

By the end of the tutorial (or right now if you check the code out from git), you’ll have something that looks like this:

Not all of the code is in this tutorial text: I’ve shortened a few code examples and haven’t included any CSS. It’s all available in the repository, so download it and open it up in your editor.

Updating Expresso

Expresso has been updated to 0.70. Upgrade yours with npm update expresso. The version we started out with is no-longer fully compatible with the documentation, particularly the beforeExit method handling.

Rendering Templates

The document list method (/documents) should render a list of documents that we can edit. To do this, add an appropriate render invocation:

res.render('documents/index.jade', {
  locals: { documents: documents }
});

… and a corresponding template:

ul
  - for (var d in documents)
    li= d.title

Remember that our templates are written with Express’s default language, Jade.

Jade

Using Jade is strange at first, but it’s actually pretty easy to get the hang of. Here are the key things to remember:

Indentations represent tag nesting
The equals sign means include a variable
Not equal means include a variable without escaping it first
The hyphen sign allows inclusion of JavaScript

Note that it’s generally a good idea to escape as much as possible to reduce the chance of XSS attacks.

Partials

Jade and Express make partials — little snippets of reusable templates — easy. Here’s the new document (views/documents/new.jade) template:

h2 New Document
form(method='post', action='/documents')
  !=partial('documents/fields', { locals: { d: d } })

The partial is rendered by calling partial(template file name, options). The output isn’t escaped because we’d just see HTML tags if it was — the user-defined fields within it will be escaped, so it’s still safe.

New and Edit Forms

Before creating the mind-blowingly awesome Ajax interface, let’s make some simple templates. Our REST API defined create and update methods, so we should make corresponding new and edit methods to provide a user interface.

I usually split forms like this into three templates. One is a reusable partial that contains the form fields. The other two are new and edit templates that contain the appropriate form code to wrap around the fields.

The new form was used as an example above. The edit form, views/documents/edit.jade looks like this:

h2 Edit Document
form(method='post', action='/documents/' + d.id)
  input(name='document[id]', value=d.id, type='hidden')
  input(name='_method', value='PUT', type='hidden')
  !=partial('documents/fields', { locals: { d: d } })

That’s the same as new, but with added hidden input fields. The _method field allows us to POST this form to a put route, which came from the RESTful API we designed last week.

The fields partial (views/partials/documents/fields.jade) is simple as well:

div
  label Title:
    input(name='document[title]', value=d.title || '')
div
  label Note:
    textarea(name='document[data]')
      =d.data || ''
div
  input(type='submit', value='Save')

By this point you should be getting a feel for Jade. I’m not a die-hard haml/Jade fan, but as you can see these examples have been very light on syntax.

New and Edit Back-end Methods

All the new and edit server-side methods do is load a document and render the forms:

app.get('/documents/:id.:format?/edit', function(req, res) {
  Document.findById(req.params.id, function(d) {
    res.render('documents/edit.jade', {
      locals: { d: d }
    });
  });
});

app.get('/documents/new', function(req, res) {
  res.render('documents/new.jade', {
    locals: { d: new Document() }
  });
});

The new method makes a blank Document to keep the form templates happy.

Mongo IDs

Did you notice that the templates refer to d.id? Mongoose makes _id fields by default, which are ObjectID data types. This doesn’t look too great on the web, so I made this getter method and added it to model.js:

getters: {
  id: function() {
    return this._id.toHexString();
  }
}

By using toHexString we get nice IDs like 4cd733fb20a558cee5000001.

Update and Delete

Both the update and destroy methods load the document first then call save or remove on it. The general pattern is as follows:

app.put('/documents/:id.:format?', function(req, res) {
  // Load the document
  Document.findById(req.body.document.id, function(d) {
    // Do something with it
    d.title = req.body.document.title;
    d.data = req.body.document.data;

    // Persist the changes
    d.save(function() {
      // Respond according to the request format
      switch (req.params.format) {
        case 'json':
          res.send(d.__doc);
         break;

         default:
          res.redirect('/documents');
      }
    });
  });
});

Delete is basically the same, except remove is called instead of save.

Delete JavaScript

One quirk of our API is we’re using Express’s del method, which means it’ll expect to see _method="delete" in the post parameters. Most frameworks implement this by using a bit of client-side JavaScript.

As I said in the first tutorial, we’re going to use jQuery. jQuery can be included by editing the layout.jade template to look like this:

!!!
html
  head
    title= 'Nodepad'
    link(rel='stylesheet', href='/stylesheets/style.css')
    script(type='text/javascript', src='https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js')
  body!= body
    script(type='text/javascript', src='/javascripts/application.js')

This also includes our JavaScript at the bottom. Express is already set up to serve static files that are in the public directory.

The client-side delete JavaScript works by:

Using confirm() to check if the user really wanted to delete the document
Dynamically inserting a form with a hidden input called _method with a value of delete
Submitting the form

Of course, this is simple with jQuery. I wrote the whole thing as one chained set of instructions:

$('.destroy').live('click', function(e) {
  e.preventDefault();
  if (confirm('Are you sure you want to delete that item?')) {
    var element = $(this),
        form = $('<form></form>');
    form
      .attr({
        method: 'POST',
        action: element.attr('href')
      })
      .hide()
      .append('<input type="hidden" />')
      .find('input')
      .attr({
        'name': '_method',
        'value': 'delete'
      })
      .end()
      .submit();
  }
});

It uses a live event delegate so we don’t litter our client-side HTML with inline JavaScript.

The Index Page

I’ve made the default action redirect to /documents, and the document index action do this:

h1 Your Documents

p
  a(class='button', href='/documents/new') + New Document

ul
  - for (var d in documents)
    li
      a(class='button', href='/documents/' + documents[d].id + '/edit') Edit
      a(class='button destroy', href='/documents/' + documents[d].id) Delete
      a(href='/documents/' + documents[d].id)
        =documents[d].title

This is an example of using an iterator in Jade. A preferable approach would be to use partial’s collection support, but it’s useful to demonstrate how control blocks within templates work in Jade.

Conclusion

As of commit f66fdb5 we now have a basic working notepad.

Check out the latest version of Nodepad and see what cool features you can add before I continue the series next week.

Processing.js 1.0, Higher Order JavaScript

19 Nov 2010 | By Alex Young | Comments |

graphics functional

Processing.js 1.0

Processing.js 1.0 has been released. Processing.js is a port of Processing, which is a programming language and environment for creating images, animations, interactive experiences, or just about anything that screams weird/cool art installation.

Version 1.0 adds optimisations, bug fixes, and better documentation. I’ve mentioned it a few times on this blog, but if you haven’t tried it out before I strongly recommend taking some time to try the demos and build some basic sketches. Who knows, maybe you’ll build the next Peggle in pure JavaScript?

Higher Order Javascript

Higher Order Javascript by Piers Cawley is a discussion about the application of higher order techniques to JavaScript. The bulk of his examples of CoffeeScript, and they touch on many areas I’ve tried to cover in our DailyJS tutorials over the last year:

Remember, higher order programming arises from applying the following principles: […] Functions can take functions as arguments, […] Functions can return new functions

It’s a very detailed tutorial, drawing on many influences. Definitely worth a readitlater/instapaper for the weekend!

This Just In: Awesome Node Libraries

I’ll cover these libraries next week in the Node roundup, but if you’ve got free time this weekend you might like to check out node.io and should.js.

Let's Make a Test Framework

18 Nov 2010 | By Alex Young | Comments |

frameworks tutorials lmaf testing

Welcome to part 39 of Let’s Make a Framework, the ongoing series about building a JavaScript framework.

If you haven’t been following along, these articles are tagged with lmaf. The project we’re creating is called Turing.

Last week we continued building turing-test.js, a basic unit testing framework. The idea behind this framework is to build something that follows the CommonJS specifications and works in modern browsers. This will help us test Turing more effectively.

The source code for last week’s tutorial can be found in commit 9a4c33e79cbeb0bf2a71. This week’s code is in commit 8dd48b5b6da1e26ab7f1.

Testing Exceptions

CommonJS Unit Test 1.0 defines assert.throws pretty briefly:

// Expected to throw an error:
assert.throws(block, Error_opt, message_opt);

The error and message arguments are optional. The error argument should check the type of the exception, and the message is an optional message.

When I was reading through Node’s assert.js I noticed they also define doesNotThrow. I debated the usefulness of this for a while, but I realised there are a lot of cases where testing for no exception is more meaningful than a simple assert.ok, and it should make the tests for the throw assertion easier to write.

If we’re going to implement that as well, we’ll need a generic throws() function with an option to determine if an exception was expected.

The basic throws function, called by the assertions, looks like this:

function throws(expected, block, error, message) {
  // Set up
  try {
    block();
  } catch (e) {
    actual = true;
    exception = e;
  }
  // Test outcome

If an error was expected and one has been passed, we can test the exception matches like this:

exception.constructor != error

In this case, error is expected to be an exception constructor like CustomException:

function CustomException() {
  this.message = 'Custom excpetion';
  this.name = 'CustomException';
}

assert.throws(function() {
  throw new CustomException();
}, CustomException);

Exception Assertion Pattern

The structure of the throws and doesNotThrow method looks like this:

Setup
Check if the error parameter is actually a message
Run the function to test and capture the results in a try/catch
If an exception was thrown and one wasn’t expected, fail
If an exception was not thrown and one was expected, fail
If an exception was thrown but doesn’t match the specific type, fail

Putting this together with the above snippets, we get something like the following:

function throws(expected, block, error, message) {
  var exception,
      actual,
      actual = false,
      operator = expected ? 'throws' : 'doesNotThrow';
      callee = expected ? assert.throws : assert.doesNotThrow;

  if (typeof error === 'string' && !message) {
    message = error;
    error = null;
  }

  message = message || '';

  try {
    block();
  } catch (e) {
    actual = true;
    exception = e;
  }

  if (expected && !actual) {
    fail((exception || Error), (error || Error), 'Exception was not thrown\n' + message, operator, callee); 
  } else if (!expected && actual) {
    fail((exception || Error), null, 'Unexpected exception was thrown\n' + message, operator, callee); 
  } else if (expected && actual && error && exception.constructor != error) {
    fail((exception || Error), null, 'Unexpected exception was thrown\n' + message, operator, callee); 
  }
};

assert.throws = function(block, error, message) {
  throws.apply(this, [true].concat(Array.prototype.slice.call(arguments)));
};

assert.doesNotThrow = function(block, error, message) {
  throws.apply(this, [false].concat(Array.prototype.slice.call(arguments)));
};

Please excuse the crudity of this model, I didn’t have time to build it to scale or to paint it.

Notice that we can use the expected result to determine the callee. That means the fail method will correctly track the operator and start stack function.

Tests

I promised tests of tests, so here they are. This is actually an interesting example because it illustrates how useful doesNotThrow is — we can use it to test the inverse without doing any hacking to simulate failed exceptions.

exports['test throws'] = function() {
  assert.throws(function() {
    throw 'This is an exception';
  });

  function CustomException() {
    this.message = 'Custom excpetion';
    this.name = 'CustomException';
  }

  assert.throws(function() {
    throw new CustomException();
  }, CustomException);

  assert.throws(function() {
    throw new CustomException();
  }, CustomException, 'This is an error');
};

exports['test doesNotThrow'] = function() {
  assert.doesNotThrow(function() {
    return true;
  }, 'this is a message');

  assert.throws(function() {
    throw 'This is an exception';
  }, 'this is a message');
};

The two and three argument versions are being tested here as well.

Object.keys

I noticed the code I ported from Node in last week’s tutorial included Object.keys. Some browsers don’t have that, so I wrote a little function to provide the functionality.

function objKeys(o) {
  var result = [];
  for (var name in o) {  
    if (o.hasOwnProperty(name))  
      result.push(name);  
  }  
  return result;  
}

Conclusion

That’s all of the assertions! We’ve discovered a few interesting things here, most notably that defining an inverse for assert.throws makes testing the assertions easier.

If you’re reading this in the future and you want to check out this version of the project, it’s commit 8dd48b5b6da1e26ab7f1.

References

Node Roundup

17 Nov 2010 | By Alex Young | Comments |

node server libraries events

Welcome to the Node Roundup. Send in your apps and libraries using our contact form or @dailyjs.

Node 0.2.5 and 0.3.1 Released

Node 0.2.5 and 0.3.1 have been released. In case you’re confused about 0.3.1, it’s considered unstable, so it might not work with all of your favourite libraries.

A new configure option, --profile, has been added to both branches, which enables gprof profiling. This can be used to benchmark Node using the gprof tool, which you probably have installed as part of your build environment.

Most of the other changes relate to playing better with the operating system, or network/buffering stability.

Node.js Camp

Node.js Camp is a Node workshop in San Francisco:

Line up of speakers and contributors includes Ryan Dahl, Isaac Schlueter, Tim Caswell, Guillermo Rauch, Paul Querna, Matt Ranney, Mikeal Rogers and more!

This looks like a pretty amazing event to me, and the tickets start at $35 (before 30th Nov)! They’re also planning a global tour, so I’ll let you know what else they announce.

Expose JavaScript Objects to the Web

I noticed this little Node/Connect script that exposes JavaScript objects to the web: gist: 700995

This script was created by Visnu Pitiyanuvath in response to an earlier Ruby/Rack script.

GlobTrie

GlobTrie by Rick Branson is a highly optimized Node script for pattern matching strings:

It breaks down the pattern matchers into pieces and organizes them in a Trie (prefix search tree) structure, allowing them to be searched in roughly logarithmic time, versus linear time for just an array of regular expressions, for instance.

The currently supported pattern matchers are limited, modelled after file path matching. However, the performance of this library destroys JavaScript regular expressions, according to the author’s benchmarks, so Branson could be onto something interesting.

jQuery Roundup

16 Nov 2010 | By Alex Young | Comments |

jquery plugins placeholders graphics

Welcome to the jQuery roundup 32. You can send your plugins and articles in for review through our contact form or @dailyjs.

jQuery 1.4.4

Have you noticed that jQuery has been moving like a juggernaut recently? 1.4.4 was released a few days ago, hot on the heels of 1.4.3, and they’re already talking about 1.5. This update adds a new animation method called .fadeToggle, and about 25 bug fixes.

I forgot to mention that return false has changed in jquery 1.4.3. This post by Neeraj Singh discusses the changing significance of returning false from event handlers in detail.

jQuery 1.5 Feature Nomination

I noticed this tweet by @rwaldron saying they’re looking for jQuery 1.5 feature ideas, and you can contribute using this Google spreadsheet. If you’ve got something to say, let them know!

Placeholder-jQuery-Plugin

I can’t keep track of the amount of jQuery placeholder plugins now. I was working on some code the other day for one of my web apps that uses the new HTML5 search input type, with the placeholder attribute. It’s pretty amazing in WebKit, and it really worked well with the app’s clean Mac-like design. Of course, no other browser wanted to play ball. That meant I had to code a quick JavaScript patch using Modernizr. I didn’t use a placeholder plugin just because I wanted to remember how laughably simple it is to do anyway.

So why would you bother with Placeholder-jQuery-Plugin, by Mathias Bynens? Well, it works in most browsers, including IE6. That’ll keep the corporate stooges happy. It also correctly identifies browsers that have partial placeholder support, like Safari 4. That means Bynens has raised the bar on this deceptively simple problem.

jSignature

jSignature by Brinley Ang, released under the MIT License, makes it possible to accept signatures in forms. Signatures in the literal sense; a little image that you draw. There’s a corresponding jSignature website with examples and demos.

Node Tutorial Part 3

15 Nov 2010 | By Alex Young | Comments |

server node tutorials lmawa nodepad

Welcome to part 3 of Let’s Make a Web App, a tutorial series about building a web app with Node. This series will walk you through the major areas you’ll need to face when building your own applications. These tutorials are tagged with lmawa.

Previous tutorials:

Part 1: Introduction
Part 2: Installation and Skeleton App, source code commit: 4ea936b4b426012528fc

In this part we will build on last week’s skeleton app. I already added a simple Document model, so let’s flesh that out a bit. These tutorials expect you to check out the code from the git repository, so visit nodepad to get it.

Logging

Let’s add some logging. Express has a logger, and it can be configured in the app.configure block. Just make sure you use it:

app.configure(function() {
  app.use(express.logger());
  // Last week's configure options go here
});

It’s usually a good idea to configure logging slightly differently, depending on environment. I’ve set it up the same way for now:

app.configure('development', function() {
  app.use(express.logger());
  app.use(express.errorHandler({ dumpExceptions: true, showStack: true })); 
});

app.configure('production', function() {
  app.use(express.logger());
  app.use(express.errorHandler()); 
});

API

We can model accessing documents over HTTP using a CRUD-based (Create, Read, Update and Delete) RESTful API:

GET /documents – Index method that returns a document list
POST /documents/ – Create a new document
GET /documents/:id – Returns a document
PUT /documents/:id – Update a document
DELETE /documents/:id – Delete a document

The HTTP verbs are important — notice that the index and create methods have the same URL, but respond differently depending on whether a HTTP GET or PUT is used. Express will route these to the appropriate methods.

HTTP Verbs are Significant

If you’re not used to working this way just remember that the HTTP verb is significant. For example, last week we defined this method:

app.get('/', function(req, res) {
  // Respond to GET for '/'
  // ...
});

If you write a form that posts to the same URL, Express will return an error because no route has been set up.

Also recall from last week that we added express.methodOverride to the configuration options. The reason for this was we can’t rely on browsers understanding HTTP verbs like DELETE, but we can use a convention to get around the problem — forms can use hidden variables that Express can interpret as a “real” HTTP method.

In some ways this approach to RESTful HTTP APIs might seem inelegant, but the advantage of using these conventions is a lot of web applications fit this approach.

CRUD Stub Reference

For reference, this is what a stubbed out CRUD set of routes looks like:

// List
app.get('/documents.:format', function(req, res) {
});

// Create 
app.post('/documents.:format?', function(req, res) {
});

// Read
app.get('/documents/:id.:format?', function(req, res) {
});

// Update
app.put('/documents/:id.:format?', function(req, res) {
});

// Delete
app.del('/documents/:id.:format?', function(req, res) {
});

Notice that Express uses del instead of delete.

Asynchronous Databases

Before we start writing each REST method, let’s look at an example: loading a list of documents. You’re probably used to working like this:

app.get('/documents', function(req, res) {
  var documents = Document.find().all();

  // Send the result as JSON
  res.send(documents);

We generally use database libraries asynchronously in Node. That means we need to do this:

app.get('/documents', function(req, res) {
  Document.find().all(function(documents) {
    // 'documents' will contain all of the documents returned by the query
    res.send(documents.map(function(d) {
      // Return a useful representation of the object that res.send() can send as JSON
      return d.__doc;
    }));
  });
});

The difference is a callback is used to access the results. This example isn’t particularly efficient, because it loads every single document into an array — it may be better to stream the results to the client as they become available.

Formats

I’d like to support HTML and JSON, where appropriate. The following pattern could be used:

// :format can be json or html
app.get('/documents.:format?', function(req, res) {
  // Some kind of Mongo query/update
  Document.find().all(function(documents) {
    switch (req.params.format) {
      // When json, generate suitable data
      case 'json':
        res.send(documents.map(function(d) {
          return d.__doc;
        }));
      break;

      // Else render a database template (this isn't ready yet)
      default:
        res.render('documents/index.jade');
    }
  });
});

This illustrates a load of core Express/Connect functionality: the routing string uses :format to detect if the client wants JSON or HTML. The question mark indicates that the format can be omitted.

Notice that this pattern wraps the database operation around the actual responder code. A similar pattern can be used for saving or deleting items.

Redirection

The create document method returns JSON documents, or redirects the client when HTML is requested:

app.post('/documents.:format?', function(req, res) {
  var document = new Document(req.body['document']);
  document.save(function() {
    switch (req.params.format) {
      case 'json':
        res.send(document.__doc);
       break;

       default:
        res.redirect('/documents');
    }
  });
});

This uses res.redirect to redirect the browser back to the document list. It could redirect to the edit form just as easily. We’ll take a closer look at this when we add the user interface.

Tests

I usually start building apps like this by testing the API. It’s easier to get a lot of this work done before embarking on client-side code. The first thing to do is add a database connection for test databases:

app.configure('test', function() {
  app.use(express.errorHandler({ dumpExceptions: true, showStack: true }));
  db = mongoose.connect('mongodb://localhost/nodepad-test');
});

Then in test/app.test.js, I’m forcing the test environment:

process.env.NODE_ENV = 'test';

That means the test database can be safely trashed.

The tests themselves take a bit of getting used to. Expresso tests work well for testing Express applications, but figuring out the finer details took serious source code reading and mailing list research.

Here’s a revealing example:

  'POST /documents.json': function(assert) {
    assert.response(app, {
        url: '/documents.json',
        method: 'POST',
        data: JSON.stringify({ document: { title: 'Test' } }),
        headers: { 'Content-Type': 'application/json' }
      }, {
        status: 200,
        headers: { 'Content-Type': 'application/json' }
      },

      function(res) {
        var document = JSON.parse(res.body);
        assert.equal('Test', document.title);
      });
  }

The name of the test, 'POST /documents.json', could be anything. The framework doesn’t actually parse these. The request is defined in the first set of parameters. In this case, I specify the Content-Type header. If these aren’t set to the appropriate type, the Connect middleware won’t be able to parse the data.

I specifically wrote a test for JSON and application/x-www-form-urlencoded because readers are likely to get stumped on this in their own code. Just remember that out of the box, Express doesn’t automatically deal with encoded form data, which is why we set up methodOverride in the configuration block.

Refer to commit 39e66cb to see these test examples.

Conclusion

You should now understand how to:

Stub CRUD methods with appropriate HTTP verbs in Express
Structure apps that can be tested using Express, Expresso, and Mongoose
Write simple Expresso tests

Next week I’ll finish off the document API methods and start adding some basic HTML templates. I intend to add a jQuery-based interface that will melt your faces off, but it’s best if we get the tests and API sorted out first.

References

Node Isn't...

12 Nov 2010 | By Alex Young | Comments |

node essays tutorials

There seem to be a lot of misconceptions about what Node is and why it’s worth using. This article should help clarify what Node is, particularly for newcomers who are interested in Node but haven’t seen much beyond the hype. In many ways it warrants the hype, and I’m enjoying working with it, but it’s not what many people think it is.

So, let’s take a look at what Node isn’t:

The Only WebSocket Solution

Node is a good solution for writing servers. It also works well with WebSockets. However, there’s no WebSocket support out of the box. True, there are great libraries for Node like Socket.IO-node, but Node isn’t necessarily the answer to WebSockets for current projects.

Somehow people have got Node and WebSockets inexorably linked in their minds. If you haven’t written much server-side JavaScript or Node before, and you’re thinking of using Node for a project that would benefit real-time communication or fast updates, you might be OK with your current platform. If you’re using PHP, Ruby, Python, Perl, or anything else, there are probably libraries that will help write low-latency server-side processes. People have been writing amazing code with Twisted and EventMachine for years.

A Magical Multi-Threaded Scalability Solution

Run a Node process and look at the process list in top. You’ll see that it uses one thread on one CPU. Node doesn’t magically scale code to use lots of threads and every processor.

Node’s homepage itself spells out why Node is good at concurrency without necessarily being concurrent in the typical sense of the word:

Node tells the operating system that it should be notified when a new connection is made, and then it goes to sleep. If someone new connects, then it executes the callback. Each connection is only a small heap allocation.

This is in contrast to today’s more common concurrency model where OS threads are employed. Thread-based networking is relatively inefficient and very difficult to use.

Furthermore, users of Node are free from worries of dead-locking the process—there are no locks.

This is good for a lot of the work we do when building web applications. This may not always be the ideal solution. Certain types of problems that benefit concurrency may better suit threads, because operating systems and modern processors can parallelize threads across multi-core processors.

It’s possible to take advantage of multi-core machines using projects like multi-node — this simply runs more than one Node instance so the operating system utilises each core.

One of the things that makes a lot of web applications perform poorly is IO. Using Node’s evented IO approach gets a lot of bang-for-buck out of hardware. I’ve written a few Objective-C projects that take advantage of threads for things that wouldn’t work as well in Node, but these instances are much rarer than times when evented IO would have helped improve performance.

Tim Caswell wrote a great post to the nodejs Google Group about this in Nodejs for Concurrency.

The Only Server-Side JavaScript Solution

Readers of DailyJS might be surprised that I point this out, but I keep meeting people who think Node is server-side JavaScript. There are other incredible projects out there, like RingoJS and Narwhal. RingoJS in particular evolved from Helma, which was already well-established. It also has some great features, like instant reloading.

Conclusion: How to Get Started with Node like a Pro

Before using Node for your next project, make sure it’s the best solution. It’s wonderful for a diverse range of applications, but make sure you’re clear about what its advantages are.

Let's Make a Test Framework

11 Nov 2010 | By Alex Young | Comments |

frameworks tutorials lmaf testing

Welcome to part 38 of Let’s Make a Framework, the ongoing series about building a JavaScript framework.

If you haven’t been following along, these articles are tagged with lmaf. The project we’re creating is called Turing.

Last week I started building turing-test.js — a basic unit testing framework. The idea behind this framework is to build something that follows the CommonJS specifications and works in modern browsers. This will help us test Turing more effectively.

Asserts

Since browsers don’t support CommonJS Unit Test 1.0, we starting building our own assert module. The architecture of our module is similar to many other unit testing frameworks — a fail function raises an exception whenever the actual value isn’t the same as the expected value.

The assertions defined by the specification are:

assert.ok(guard, message_opt)
Pure assertion tests whether a value is truthy.

assert.equal(actual, expected, message_opt)
Shallow, coercive equality with ==.

assert.notEqual(actual, expected, message_opt)
Tests if two objects are not equal with !=.

assert.deepEqual(actual, expected, message_opt)
Equivalence, as determined by ===, and special handling for dates, Object, Array.

assert.notDeepEqual(actual, expected, message_opt)
The inverse of the above.

assert.strictEqual(actual, expected, message_opt)
Tests strict equality, as determined by ===

assert.notStrictEqual(actual, expected, message_opt)
The inverse, using !==

assert.throws(block, Error_opt, message_opt)
Expects an exception.

For those of you who tl;dr’d that, the CommonJS assertion module defines methods for equality, deep equality, and strict equality. The difference between these is important, and if my tutorial fails to elucidate you please refer to Douglas Crockford’s JavaScript: The Good Parts.

Equality Assertions

I’ve written about the equality operators on this very blog before, but let’s recap. JavaScript offers two distinct sets of equality operators. As JSLint will tell you, === and !== are generally what you want. Crockford calls them good, because they work how we expect: if two values have the same type and value, === will result in true.

Meanwhile, == and != will coerce values if they’re not of the same type. This can lead to confusing behaviour, like 0 == '' equating to true.

That explains coercive equality and strict, but what about deep? The CommonJS specification defines it deep equality as:

All identical values are equivalent, as determined by ===
If the expected value is a Date object, the actual value is equivalent if it is also a Date object that refers to the same time
For pairs that do not both pass typeof value == "object", equivalence is determined by ==
For all other Object pairs, including Array objects, equivalence is determined by having the same number of owned properties (as verified with Object.prototype.hasOwnProperty.call), the same set of keys (although not necessarily the same order), equivalent values for every corresponding key, and an identical “prototype” property. Note: this accounts for both named and indexed properties on an Array.

Research

Let’s take a look at some equality code in the wild from an arbitrary JavaScript test framework, Jasmine:

jasmine.Env.prototype.equals_ = function(a, b, mismatchKeys, mismatchValues) {
  mismatchKeys = mismatchKeys || [];
  mismatchValues = mismatchValues || [];

  for (var i = 0; i < this.equalityTesters_.length; i++) {
    var equalityTester = this.equalityTesters_[i];
    var result = equalityTester(a, b, this, mismatchKeys, mismatchValues);
    if (result !== jasmine.undefined) return result;
  }

  if (a === b) return true;

  if (a === jasmine.undefined || a === null || b === jasmine.undefined || b === null) {
    return (a == jasmine.undefined && b == jasmine.undefined);
  }

  if (jasmine.isDomNode(a) && jasmine.isDomNode(b)) {
    return a === b;
  }

  if (a instanceof Date && b instanceof Date) {
    return a.getTime() == b.getTime();
  }

  if (a instanceof jasmine.Matchers.Any) {
    return a.matches(b);
  }

  if (b instanceof jasmine.Matchers.Any) {
    return b.matches(a);
  }

  if (jasmine.isString_(a) && jasmine.isString_(b)) {
    return (a == b);
  }

  if (jasmine.isNumber_(a) && jasmine.isNumber_(b)) {
    return (a == b);
  }

  if (typeof a === "object" && typeof b === "object") {
    return this.compareObjects_(a, b, mismatchKeys, mismatchValues);
  }

  //Straight check
  return (a === b);
};

There are some commonalities between this code and the CommonJS specification. For example, Date comparison using getTime. It deviates slightly by having specific handling for DOM nodes, and the type checks for strings or numbers.

Here’s what Node does in its assert module. The original source even has lines from the specification pasted in:

function _deepEqual(actual, expected) {
  // 7.1. All identical values are equivalent, as determined by ===.
  if (actual === expected) {
    return true;

  } else if (Buffer.isBuffer(actual) && Buffer.isBuffer(expected)) {
    if (actual.length != expected.length) return false;

    for (var i = 0; i < actual.length; i++) {
      if (actual[i] !== expected[i]) return false;
    }

    return true;

  // 7.2. If the expected value is a Date object, the actual value is
  // equivalent if it is also a Date object that refers to the same time.
  } else if (actual instanceof Date && expected instanceof Date) {
    return actual.getTime() === expected.getTime();

  // 7.3. Other pairs that do not both pass typeof value == "object",
  // equivalence is determined by ==.
  } else if (typeof actual != 'object' && typeof expected != 'object') {
    return actual == expected;

  // 7.4. For all other Object pairs, including Array objects, equivalence is
  // determined by having the same number of owned properties (as verified
  // with Object.prototype.hasOwnProperty.call), the same set of keys
  // (although not necessarily the same order), equivalent values for every
  // corresponding key, and an identical "prototype" property. Note: this
  // accounts for both named and indexed properties on Arrays.
  } else {
    return objEquiv(actual, expected);
  }
}

function isUndefinedOrNull (value) {
  return value === null || value === undefined;
}

function isArguments (object) {
  return Object.prototype.toString.call(object) == '[object Arguments]';
}

function objEquiv (a, b) {
  if (isUndefinedOrNull(a) || isUndefinedOrNull(b))
    return false;
  // an identical "prototype" property.
  if (a.prototype !== b.prototype) return false;
  //~~~I've managed to break Object.keys through screwy arguments passing.
  //   Converting to array solves the problem.
  if (isArguments(a)) {
    if (!isArguments(b)) {
      return false;
    }
    a = pSlice.call(a);
    b = pSlice.call(b);
    return _deepEqual(a, b);
  }
  try{
    var ka = Object.keys(a),
      kb = Object.keys(b),
      key, i;
  } catch (e) {//happens when one is a string literal and the other isn't
    return false;
  }
  // having the same number of owned properties (keys incorporates hasOwnProperty)
  if (ka.length != kb.length)
    return false;
  //the same set of keys (although not necessarily the same order),
  ka.sort();
  kb.sort();
  //~~~cheap key test
  for (i = ka.length - 1; i >= 0; i--) {
    if (ka[i] != kb[i])
      return false;
  }
  //equivalent values for every corresponding key, and
  //~~~possibly expensive deep test
  for (i = ka.length - 1; i >= 0; i--) {
    key = ka[i];
    if (!_deepEqual(a[key], b[key] ))
       return false;
  }
  return true;
}

Node’s code includes a few key techniques that are worth remembering:

A strict equality check is performed up front
getTime is used on the dates again
Array.prototype.slice.call is used to transform function arguments into arrays, so they can be tested like arrays
Key equivalence is tested to find differences in keys early before having to resort to another deep test on nested objects
_deepEqual is called recursively for nested objects

If you take a look at the Turing Test repository, I’ve basically used Node’s implementation with a few modifications.

Conclusion

This is actually the main body of assertions as defined by the CommonJS specification. There’s one more, assert.throws, which I’ll define next week, along with a look at testing the tests. If you’d like to look at the code for this tutorial, it’s in commit 9a4c33e79cbeb0bf2a71 on GitHub.

References

Node Roundup

10 Nov 2010 | By Alex Young | Comments |

node server licensing books graphics

Welcome to the Node Roundup. Send in your apps and libraries using our contact form or @dailyjs.

There’s quite a lot of commercial activity around Node lately. This post includes details on a book started by a Yahoo! employee in association with O’Reilly, open source contributions from LearnBoost, and Joyent hiring full-time Node developers.

A Node.js Book Project

Tom Hughes-Croucher, Technology Evangelist at Yahoo!, announced a Node book in association with O’Reilly Media, called Up and Running With Node.js. There’s a preview PDF available. For more information, see his blog post, Announcing a Node.js Book Project.

Server side HTML5 canvas API

LearnBoost are continuing to make some amazing open source contributions, this time in the form of node-canvas. This is a server-side Canvas API that you can use to generate images. That opens up a lot of possibilities given how many great JavaScript libraries generate images on Canvas, like graph libraries.

The library is available on GitHub / node-canvas and is MIT licensed. It uses Cairo to generate graphics.

Joyent and Node

In Joyent and Node Ryan Dahl talks about the growth of Node and building up a team of full-time development to better compete with other languages and platforms.

Some of the replies voiced concerns about the ownership of Node and relationship between the community and a partly commercial project. Micheil Smith made a good point that calmed some of the concerns about “Joyent” appearing on the contributor agreement:

The CLA has been in place for the last few months, all that has changed is now instead of transferring your IP and such to Ryan Dahl, you are now transferring them to Joyent, the text of the CLA is the same otherwise.

The debate is interesting and worth reading if you’re using Node for major projects. Overall it sounds positive to me, and I’m looking forward to seeing what happens over the next few months as Joyent invest more into it.

← previous

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 next →