The JavaScript blog.


libraries graphics modules graphs sql

Simple Modules, D3xter

Posted on .

Simple Modules

If you've got legacy JavaScript that's badly structured, then one early refactoring strategy that can help is to add modules. However, you might not be sure about the specific module system or framework that you'll eventually switch to, so what do you do?

George Mauer sent in simple-modules (GitHub: togakangaroo/simple-modules, License: MIT, Bower: simple-modules), a tiny module system that lets you define modules and then require them. There's an example that shows the main features, including a test for circular dependency detection.

George wrote an article about converting legacy code, which shows how to include Simple Modules on the page and then how to identify possible modules.



D3xter (GitHub: NathanEpstein/D3xter, License: MIT, Bower: d3xter) by Nathan Epstein is a D3.js-based library for plotting graphs. The functions in the library return SVG objects, so you can further customise them with the standard D3 API.

D3xter's built-in graphs should work well with minimal configuration, so if you're looking for a D3-friendly library then this one might be a good choice. For example, given arrays of co-ordinates, you can plot a line graph like this:

var lineGraph = xyPlot(index, y);  

The readme has an additional example that shows how to query the resulting object with D3 and modify the colours.


libraries node modules database sql

Alasql.js: In-memory SQL

Posted on .

I recently wrote about an in-memory database called LokiJS. Most in-memory databases are NoSQL, but there are also similar projects that use SQL. Andrey Gershun sent in Alasql.js (GitHub: agershun/alasql, License: MIT, npm: alasql), a relational database that runs in browsers and Node.

It's designed to be fast, and was created partly by using a Jison file that produces a parser for SQL statements. The author has made some benchmarks so you can compare it to other projects:

This project is different to WebSQL because it doesn't use the Storage API. It has an impressive set of features, including joins, group by, union, and subqueries. It even has partial support for transactions, so it's not a trivial implementation.

The API allows JavaScript objects to be mixed with SQL statements, so it's easy to create test data:

var alasql = require('alasql');

var db = new alasql.Database();

db.exec('CREATE TABLE test (one INT, two INT)');  
db.tables.test.data = [  
  { one: 3, two: 4 },
  { one: 5, two: 6 }

var res = db.exec('SELECT * FROM test ORDER BY two DESC');  

To make it fast, the authors have used various optimisation strategies:

  • Compiled queries are cached
  • Joined tables are pre-indexed
  • WHERE expressions are pre-filtered for joins

So in a way it has a query planner like more established databases.

The readme includes a list of similar projects, details on the tests (they use Mocha), and there's a to-do list where you can see what Andrey is planning.


databases node modules mongo sql mongoose

Node Roundup: Caterpillar, squel, mongoose-currency

Posted on .

You can send in your Node projects for review through our contact form.


Benjamin Lupton sent in Caterpillar (GitHub: bevry / caterpillar, License: MIT, npm: caterpillar), which is a logging system. It supports RFC-standard log levels, but the main reason I thought it was interesting is it's based around the streams2 API.

By piping a Caterpillar stream through a suitable instance of stream.Transform, you can do all kinds of cool things. For example, caterpillar-filter can filter out unwanted log levels, and caterpillar-human adds fancy colours.


I was impressed by Brian Carlson's sql module, and Ramesh Nair sent in squel (GitHub: hiddentao / squel, License: BSD, npm: squel) which is a similar project. This SQL builder module supports non-standard queries, and has good test coverage with Mocha.

Ramesh has included some client-side examples as well, which sounds dangerous but may find uses, perhaps by generating SQL fragments to be used by an API that safely escapes them, or for generating documentation examples.


mongoose-currency (GitHub: catalystmediastudios / mongoose-currency, License: MIT, npm: mongoose-currency) by Paul Smith adds currency support to Mongoose. Money values are stored as an integer that represents the lowest unit of currency (pence, cents). Input can be a string that contains a currency symbol, commas, and a decimal.

The Currency type works by stripping non-numerical characters. I'm not sure if this will work for regions where numbers use periods or spaces to separate groups of digits -- it seems like this module would require localisation support to safely support anything other than dollars.

Paul has included unit tests written with Mocha, so it could be extended to support localised representations of currencies.


databases node modules sql

The State of Node and Relational Databases

Posted on .

Recently I started work on a Node project that was built using Sequelize with MySQL. It was chosen to ease the transition from an earlier version written with Ruby on Rails. The original's ActiveRecord models mapped quite closely to their Sequelize equivalents, which got things started smoothly enough.

Although Sequelize had some API quirks that didn't feel very idiomatic alongside other Node code, the developers have hungrily accepted pull requests and it's emerging as a reasonable ORM solution. However, like many others in the Node community I feel uncomfortable with ORM.

Why? Well, some of us have learned how to use relational databases correctly. Joining an established project that uses ORM only to find there's no referential integrity or sensible indexes is to be expected these days, as programmers have moved away from caring about databases to application-level schemas. I've had my head down in MongoDB/Mongoose and Redis code for the last few years, but relational databases aren't going away any time soon so either programmers need to get the hang of them or we need better database modules.

This all prompted me to look at alternative solutions to relational databases in Node. First, I broke down the problem into separate areas:

  • Driver: The module that manages database connections, sends queries, and responds with data
  • Abstraction layer: Provide tools for escaping queries to avoid SQL injection attacks, and wrap multiple drivers so it's easy to port applications between MySQL/PostgreSQL/SQLite
  • Validator: Validates data against a schema prior to sending it to the database. Aids with the generation of human-readable error messages
  • Query generator: Generates SQL queries based on a more JavaScript-programmer-friendly API
  • Schema management: Keep schema up-to-date when fields are added or removed

Some projects won't need to support all of these areas -- you can mix and match them as needed. I prefer to create simple veneer-like "model" classes that wrap more low-level database operations. This works well in a web application where it can be make sense to decouple the HTTP layer from the database.

Database Driver

The mysql and pg modules are actively maintained, and are usually required by "abstraction layer" modules and ORM solutions.

A note about configuration: when it comes to connecting to the database, I strongly prefer modules that support connection URIs. It makes it a lot easier to deploy web applications to services like Heroku, because a single environmental variable can be set that contains the connection details for your production database.

Abstraction Layer

This level sits above the driver layer, and should offer lightweight JavaScript sugar. There are many examples of this, but a good one is transactions. Transactions are particularly useful in JavaScript because they can help create APIs that are less dependent on heavily nested callbacks. For example, it makes sense to model transactions as an EventEmitter descendent that allows operations to be pushed to an internal stack.

The author of the pg module, Brian Carlson, who occasionally stops by the #dailyjs IRC channel on Freenode, recently mentioned his new relational project that aims to provide a saner approach to ORM in Node. This module feels more like an abstraction layer API, but it's gunning to be a formidable new ORM solution.

There are some popular libraries that tackle the abstraction layer, including any-db and massive.


I usually find myself dealing with errors in web forms, so anything that makes error handling easier is an advantage. Validation and schemas are closely related, which is why ORM libraries usually combine them.

It's possible to treat them separately, and in the JavaScript community we have solutions based on or inspired by JSON Schema. The jayschema module by Nate Silva is one such project. It's really aimed at validating JSON, but it could be used to validate JavaScript objects spat out by a database driver.

Validator has some simple tools for validating data types, but it also has optional Express middleware that makes it easy to drop into a web application. Another similar project is conform by Oleg Korobenko.

Query Generator

The sql module by Brian Carlson is an SQL builder -- it has a chainable API that turns JavaScript into SQL:


He's using this to build the previously mentioned relational module as well.

Schema Management

Sequelize has an API for managing database migrations. It can migrate to a given version and back, and it can also "sync" a model's schema to the database (creating the table if it doesn't exist).

There are also dedicated migration modules, like db-migrate by Jeff Kunkle.


The Node community has created a rich set of modules for working with relational databases, and although there's a strong anti-ORM sentiment interesting projects like relational are appearing.

Although these modules don't address my concerns about the way in which ORM gets used with apathy towards best practices, it's promising to see lower-level modules that can be used as building blocks for more application-specific solutions.

All of this has come at a time when relational database projects are adapting, changing, and even growing in popularity despite the recent attention the NoSQL movement has been given. PostgreSQL is going from strength to strength, and Heroku provides it as default. MariaDB is a drop-in replacement for MySQL that has a non-blocking Node module. SQLite is probably technically growing in usage as it backs Core Data in iCloud applications -- Android developers also use SQLite.

Let other readers know how you deal with SQL-backed Node projects in the comments!


css node modules zip sql

Node Roundup: libuv Status, node-sql, Zipper, Jade and Stylus Updates

Posted on .

You can send your node modules and articles in for review through our [contact form](/contact.html) or [@dailyjs](http://twitter.com/dailyjs).

libuv Status Report

Ryan Dahl posted a status report for
to the
Node blog, and described some of the background to libuv and reviewed
the features implemented so far. This includes non-blocking TCP sockets
(using IOCP in Windows), non-blocking named pipes, UDP, timers, child process spawning, asynchronous DNS, asynchronous file APIs, and more.
They're currently working on file system events (using inotify, kqueue,
and event ports), a VT100 TTY, and socket sharing between processes.

Ryan also linked to some interesting projects built with libuv, and this
includes Mozilla's Rust which is billed as
a low-level concurrent language.


node-sql (npm: sql) by Brian Carlson (the author of
node-postgres, better known as pg) builds strings of SQL using a simple API. It uses
the kind of chained API that suits this kind of problem:

var sql = require('sql')
  , user
  , query;

user = sql.define({
  name: 'user',
  columns: ['id', 'email', 'lastLogin']

query = user.select(user.id, user.email)
            .where(user.lastLogin.lt(new Date()));

It's still a major work in progress, but I think it could lead to some
extremely interesting database APIs in the future.


Zipper (GitHub: rubenv / zipper, npm: zipper, License: BSD) by Ruben Vermeersch is a zip file creator with an extremely easy API:

var zipper = require('zipper').Zipper
  , zipfile;

zipfile = new zipper('file.zip');
zipfile.addFile('image.jpg', 'path/image.jpg', function(err) {

Jade and Stylus Updates

Jade and Stylus have been updated
to 0.16.0. Jade's grammar has been extended to include literal support
for if, unless, while,
until, and it also supports assignment. This means that
rather than using JavaScript, it's now possible to use declarative

// Old:
- if (!(foo && bar))
 p stuff

// New:
unless foo && bar
  p stuff

This demonstrates the unless keyword, and the new Jade

The other major feature is template inheritance. The keywords
extends and block allow templates to be
inherited from, and "blocks" of markup grouped together and overridden
as required.