DailyJS

DailyJS

The JavaScript blog.


Tagmongodb
Featured

libraries node modules mongodb

Node Roundup: Node 0.11.15, PublicSuffixList, Robe

Posted on .

Node 0.11.15

Node 0.11.15 is out. This is a substantial release that updates v8 to 3.28.73, npm to 2.1.6, and uv to 1.0.2. There are patches for the build system and core modules as well. I looked over the changes to the core modules and it looks like it's mostly bug fixes, but some modules (url and readline) have had performance improvements as well.

Meanwhile, io.js was updated to 1.0.3. This upgrades v8 from 3.31 to 4.1, which the changelog notes is not a major release because "4.1" is derived from Chrome 41. There's also better platform support, including Windows XP, Windows 2003, and improved FreeBSD support.

If you look through the commits in both projects it should be obvious that some commits are shared, but some are unique to each fork. There was a popular post on Hacker News about performance comparisons between Node and io.js, so it'll be interesting to see how the projects diverge and converge over the next few months.

PublicSuffixList

Matthias Thoemmes sent in PublicSuffixList (GitHub: cmtt/publicsuffixlist, License: MIT, npm: publicsuffixlist), a validator for domain names and top level domains. It uses data from Mozilla's publicsuffix.org project:

A "public suffix" is one under which Internet users can directly register names. Some examples of public suffixes are .com, .co.uk and pvt.k12.ma.us. The Public Suffix List is a list of all known public suffixes.

The module provides methods like validateTLD, validate(domainString), and you can also look up a domain to destructure it:

var result = psl.lookup('www.domain.com');

/*
result === {  
  domain: 'domain',
  tld: 'com',
  subdomain: 'www'
} */

Robe

Robe (GitHub: hiddentao/robe, License: MIT, npm: robe) by Ramesh Nair is a MongoDB ODM that uses ES6 generators. The author said he couldn't quite find the MongoDB API that he wanted, although Mongorito came close. So he used Monk to build Robe.

Because it supports generators you can yield results in the synchronous style:

var result = yield collection.findOne({  
  score: {
    $gt: 20
  },
}, {
  fields: ['name'],
  sort: { name: 1 },
  skip: 1,
});

It also supports lifecycle hooks, so you can run callbacks after or before data is inserted, removed, and so on.

Robe will validate data based on your schema, so running yield collection.insert() with an invalid record will cause an exception to be raised. Finally, it also supports streams, so you can use an event-based API to handle large sets of data.

Featured

libraries node modules database mongodb ai

Node Roundup: npm's Registry Architecture, Mongeese, Synaptic

Posted on .

npm's Registry Architecture

npm's blog has a post about the registry's roadmap for the second phase of its architectural development. This post has some interesting stats about the amount of traffic npm gets, and how the design is changing to make it scale up.

The new design is based around modular microservices, so it follows the trends employed by other large scale services.

We call this collection of changes "Registry 2". It will resolve some lingering inefficiency in the existing architecture and give us even more room to scale, but more importantly it will allow us to add a bunch of new features. We're very, very confident that it will work, because it's already in production: npm Enterprise already uses this architecture, and the Registry 2 project is largely a matter of taking this architecture and scaling it up.

The cool thing about this is it means npm will be able to offer private packages -- these are invisible packages that only your collaborators can see. The post points out that using lots of small packages is the "Node way", but npm doesn't currently help us achieve that due to the reliance on public packages.

The way I get around the lack of private packages is to host private modules on GitHub with an OAuth2 token in the URL. I find this approach better than Git submodules, but it lacks the ability to easily control what version of a package is included so it's not really suitable for deployments that I don't have complete control over.

But if you write code that is closed-source, or is just too specific to your own application to make a good public module, your options right now are full of friction and split-brain thinking. Why can't you just npm publish and npm install all the private packages in your app, the same way you can with public packages? People kept asking us to make it possible to do this, so we did.

This should be released in early 2015. You can register for the beta using a form in the blog post.

I don't know if private packages or "npm for teams" will be mean npm is going to offer subscription plans, but I expect to see subscriptions for some functionality next year. I don't have a problem with that, as a long paying GitHub subscriber, but the post doesn't seem to mention anything about paid plans. Of course, there is already npm Enterprise, but this is different from ad-hoc collaboration.

Mongeese

Mongeese (GitHub: zekenie / mongeese, License: ISC, npm: mongeese-array) by Zeke Nierenberg modifies Mongoose to make the Array class include extra async methods, including asyncEach, asyncMap, and asyncReduce. Here's an example of asyncMap:

var iterator = function(oneKitten, done) {  
  done(null, oneKitten.name)
};

someUser.kittens.asyncMap(iterator, function(err, kittenNames) {  
  // kitten names -> ['fluffy', 'cuddles', ...]
});

It works by using the async module and lodash. Because it modifies Mongoose it may cause issues with future API changes, but you may like it if you use a lot of arrays.

Synaptic

Synaptic (GitHub: cazala / synaptic, License: MIT, npm: synaptic) by Juan Cazala is a neural network library. There are some cool demos, like an image filter based on training a perceptron.

This library includes a few built-in architectures like multilayer perceptrons, multilayer long-short term memory networks (LSTM) or liquid state machines, and a trainer capable of training any given network, which includes built-in training tasks/tests like solving an XOR, completing a Distracted Sequence Recall task or an Embeded Reber Grammar test, so you can easily test and compare the performance of different architectures.

The readme explains what each of the module's classes do, to the extent that it reads like an introduction to neural networks. It also has some powerful features, like being able to export a network as a single JavaScript function, so you can use trained networks without any dependencies.

Featured

testing node modules npm mongodb

Node Roundup: mongo-select, asynctrace

Posted on .

mongo-select

mongo-select (GitHub: dschenkelman / mongo-select, License: MIT, npm: mongo-select) is a module for creating projections on results from mongodb.

You can specify the fields to include using select.include:

var select = require('mongo-select').select();  
var projection = select.include(['name', 'email', 'children.name']).make();

console.log(projection); // { 'name': false, 'email': false, 'children.name': false };  

There are also methods for excluding fields (exclude) and ignoring the _id field (noId). These methods can be chained:

select.noId()._exclude(['name', 'email', 'children.name']);  

Using it with MongoDB queries looks like this:

var select = require('mongo-select').select();  
var mongodb = require('mongodb');

var MongoClient = mongodb.MongoClient;

MongoClient.connect('mongodb://127.0.0.1:27017/test', function(err, db) {  
  if (err) throw err;

  var users = db.collection('users');

  users.find(
    {},
    select.include(['name', 'email']),
    function(err, result) {
      // code here, access to only result[i]._id, result[i].name and result[i].email
    }
  );
});

asynctrace

asynctrace (GitHub: Empeeric / asynctrace, License: MIT, npm: asynctrace) from Empeeric is a stack trace API based on AsyncListener. By requiring asynctrace, you'll get stack traces that include details on asynchronous boundaries.

It's built using the shimmer module, which is meant to help safely replace functions by wrapping them.

If you use Mocha then you can optionally use it by specifying it on the command line: mocha --require asynctrace. That means you don't need to modify your code to use the module during testing.

Featured

node modules bitcoin mongodb

Node Roundup: Nodemon 1.0, Prana, termcoin, node-bitwise-xor

Posted on .

Nodemon 1.0

Nodemon

I noticed Nodemon 1.0 was released this week (GitHub: remy / nodemon, License: MIT, npm: nodemon). This update includes local and global configuration files, execMap for mapping file extensions to programs, and some changes to the overall architecture of the project. You can now require Nodemon, and tests have been added.

Prana

Prana (GitHub: recidive / prana, License: MIT, npm: prana) by Henrique Recidive is a small framework for Node applications. Prana application objects are EventEmitters, and Prana "types" emit events as well.

It combines an ODM system with a plugin system, and currently persists data to memory or MongoDB. The author has included some examples which you can find in prana/examples, and one of them uses Express. The module's code itself has detailed comments, and the readme is solid too.

termcoin

termcoin (GitHub: chjj / termcoin, License: MIT, npm: termcoin) by Christopher Jeffrey is a terminal Bitcoin client with a command-line interface based on blessed. It requires bitcoind to work, and looks really cool in the screenshots.

node-bitwise-xor

Stanislas Marion sent in node-bitwise-xor (GitHub: czzarr / node-bitwise-xor, License: MIT, npm: bitwise-xor), a module for performing a bitwise XOR on two buffers or strings. It iterates over each element with ^, taking into account the length to ensure each item is changed.

Featured

frameworks node modules mongodb schema

Node Roundup: TJ Fontaine Takes Over, Conductance, Easymongo, Schema-Inspector

Posted on .

The Next Phase of Node.js

In The Next Phase of Node.js, Isaac Z. Schlueter has announced that TJ Fontaine is now the Node project lead:

Anyone who's been close to the core project knows that he's been effectively leading the project for a while now, so we're making it official. Effective immediately, TJ Fontaine is the Node.js project lead. I will remain a Node core committer, and expect to continue to contribute to the project in that role. My primary focus, however, will be npm.

Isaac also said he's founding npm, Inc., to allow him to focus on npm and develop related products:

I'll be sharing many more details soon about exactly how this is going to work, and what we'll be offering. For now, suffice it to say that everything currently free will remain free, and everything currently flaky will get less flaky. Pursuing new revenue is how we can keep providing the npm registry service in a long-term sustainable way, and it has to be done very carefully so that we don't damage what we've all built together.

Recently, there was an initiative by Nodejitsu to drive support and funding for npm, resulting in over $300,000 being raised. It'll be interesting to see how this all ties together over the next year.

Congratulations and good luck to TJ Fontaine!

Conductance

Conductance

Conductance (GitHub: onilabs / conductance, License: GPL, npm: conductance) from Oni Labs is a web application server with a UI toolkit and a module system that's compatible with client-side code.

It's built on Stratified JavaScript, by the same company, which adds new language primitives for block lambdas, destructuring data, arrow function syntax, and more.

Conductance already has a detailed tutorial, and there's an API guide as well.

Easymongo

Easymongo (GitHub: meritt / easymongo, License: MIT, npm: easymongo) by Alexey Simonenko is a wrapper around the native Node MongoDB driver. It has a clean, idiomatic API, and relies on plain old objects instead of models.

It has Mocha tests, and the API is documented in the readme. Basic use looks like this:

var options = {  
  dbname: 'test'
};

var mongo = new require('easymongo')(options);  
var users = mongo.collection('users');

var data = { name: 'Alexey', surname: 'Simonenko', url: 'http://simonenko.su' };  
users.save(data, function(error, results) {  
  // Returns a new document (array).
  console.log(results);
});

users.find({ name: 'Alexey' }, { limit: 1 }, function(error, results) {  
  // Always return array of documents.
  console.log(results);
});

Schema-Inspector

What do you do when you're using simple objects without an ORM layer? You use schemas to validate your user input! Schema-Inspector (GitHub: Atinux / schema-inspector, License: MIT, npm: schema-inspector, Bower: schema-inspector) by Sebastien Chopin is a JavaScript object validator that works in browsers and Node.

Given a suitable schema, you can validate objects like this: inspector.validate(schema, candidate). It can also be called asynchronously, which allows you to report issues with this.report inside functions in the schema.

It has tests written with Mocha, and a healthy amount of API documentation in the readme.