NeDB: SQLite for Node

2013-08-01 00:00:00 +0100 by Alex R. Young

NeDB (GitHub: louischatriot / nedb, License: MIT, npm: nedb) by Louis Chatriot is datastore for Node that implements a subset of MongoDB's API. It has very modest dependencies (async, underscore, binary-search-tree, mkdirp), and can even be used in browsers:

As of v0.8.0, you can use NeDB in the browser! You can find it and its minified version in the repository, in the browser-version/out directory. You only need to require nedb.js or nedb.min.js in your HTML file and the global object NeDB can be used right away, with the same API as the server version.

The author has been enthusiastically benchmarking the project and is rather confident about its performance. And it really does look like MongoDB:

db.find({ satellites: { $lt: 'Amos' } }, function(err, docs) {
  // docs is empty since Phobos and Deimos are after Amos in lexicographical order

db.update({ system: 'solar' }, { $set: { system: 'solar system' } }, { multi: true }, function(err, numReplaced) {
  // numReplaced = 3
  // Field 'system' on Mars, Earth, Jupiter now has value 'solar system'

// Using a unique constraint with the index
db.ensureIndex({ fieldName: 'somefield', unique: true }, function(err) {

The persistence layer isn't required: databases can be in-memory if desired. To understand how it all works, first take a look at how each database operation is implemented. For example, Datastore.prototype._insert on lines 265 to 268 in datastore.js calls persistence.persistNewState. The persistNewState method is called for anything that changes data (insert, update, remove). The persistNewState method itself returns early if inMemoryOnly has been set, otherwise it appends UTF8-encoded serialized models to the file that backs the database.

Models are serialized using serialize in model.js. This uses JSON.stringify with a callback that maps undefined values to null, and checks if keys are valid using similar rules to MongoDB (you can't have keys that start with a dollar or contain a full-stop).

When data is retrieved from disk, async.waterfall is used, from Caolan McMahon's popular module:

Runs an array of functions in series, each passing their results to the next in the array. However, if any of the functions pass an error to the callback, the next function is not executed and the main callback is immediately called with the error.

Elsewhere the async module is used to queue commands, which ensures they're executed in the desired order. All commands are passed through an instance of the Executor class, including the loading of data from disk when the persistence layer is initialised.

By combining async and the native JSON encoder and parser, Louis has made a convincing yet lightweight MongoDB implementation. It ties together a lot of the skills required to work as a Node developer -- asynchronous I/O, and event-based programming. NeDB currently has 886 stars on GitHub, so it's clearly a popular project: I suggest taking a look at the source if you're interested in how people use modules like async.