DailyJS

DailyJS

The JavaScript blog.


Tagfs
Featured

node modules fs streams

Node Roundup: 0.10.15, IntervalStream, StreamToMongo, fileswap-stream

Posted on .

You can send in your Node projects for review through our contact form.

0.10.15

Node 0.10.15 was released last week, which quickly followed 0.10.14. The newer release adds a fix for process.getuid, to address an issue on Mac OS X:

This commit should unbreak npm on OS X - it's hitting the new 'uid must be an unsigned int' check when installing as e.g. user 'nobody' (which has an UID of -2 in /etc/passwd or 4294967294 when cast to an uid_t.)

Version 0.10.14 fixed bugs in os, url, and upgraded npm and uv.

IntervalStream

node-interval-stream (GitHub: czzarr / node-interval-stream, License: MIT, npm: interval-stream) by Stanislas Marion is a simple module that provides a Transform stream that triggers events based on an interval. The author's example combines IntervalStream with request to display the results of a large download every 2 seconds:

var request = require('request');  
var IntervalStream  = require('interval-stream');  
var is = new IntervalStream(2000); // emit every 2 seconds

request('http://example.com/large_data_set.json')  
  .pipe(is)
  .pipe(process.stdout);

StreamToMongo

StreamToMongo (GitHub: czzarr / node-stream-to-mongo, License: MIT, npm: stream-to-mongo), also by Stanislas Marion, allows data to be streamed to MongoDB. This could be used to stream JSON data directly into a database. The example in the readme uses the npm registry, effectively allowing you to create a structured local cache in Mongo of all the module metadata on npm.

fileswap-stream

Finally, fileswap-stream (GitHub: bpostlethwaite / fileswap-stream, License: MIT, npm: fileswap-stream) by Ben Postlethwaite allows underlying file resources to be swapped. This might be useful if you're streaming data to log files, and want to split the files:

Write to a writable file-stream that swaps out its underlying file resources according to swapper and naming functions. This can be used for a persistent log or data stream - just stream to it 24/7 and let it swap out to new files whenever you trigger it to.

Featured

testing node modules ETL linux fs

Node Roundup: Mocha, Banzai, Inotify

Posted on .

You can send your Node modules and articles in for review through our [contact form](/contact.html) or [@dailyjs](http://twitter.com/dailyjs).

Mocha

Mocha (GitHub: visionmedia / mocha, License: MIT, npm:
mocha) by TJ Holowaychuk is a new test framework that works both in Node and browsers. It runs tests serially, making it intuitive, and
includes a huge collection of reporters including a TAP reporter. It
even comes with a sane setup/teardown implementation, allowing code to
be run before and after each test or the whole suite.

Like other test frameworks, Mocha uses an asynchronous "done" function.
However, calling it is optional for synchronous code, and it can even
accept an error to improve test reporting:

describe('User', function(){
  describe('#save()', function(){
    it('should save without error', function(done){
      var user = new User('Luna');
      user.save(function(err){
        done(err);
      })
    })
  })
})

TJ's example shows that for methods that include an error as the first
argument, then this will be automatically passed to done:

describe('User', function(){
  describe('#save()', function(){
    it('should save without error', function(done){
      var user = new User('Luna');
      user.save(done);
    })
  })
})

Mocha can work with any assertion library, which allows alternative
assertion libraries like
should.js to be used. It's even possible to use BDD style tests describe() / it() or
TDD: suite() / test().

TJ considers Mocha to be
Expresso successor, and it does seem to address a lot of things that were missing from Expresso.

Banzai

Banzai (npm: banzai) by Pedro Teixeira is a document processing framework (or an
ETL)
that uses the concept of pipelines to represent state and perform
multiple operations on a document. On the surface, a pipeline looks like
a combination of an event-based API and a state machine:

pipeline
  .on('initial', initialHandler, {
      next: 'order received email sent'
  })
  .on('order received email sent', orderEmailSentHandler, {
      priority: 2
    , condition: allItemsAvailable
    , next: 'items available'
  })
  .on('order received email sent', confirmationEmailSentHandler, {
      priority: 1
    , next: 'items not available'
  })
  .on('items not available', itemsNotAvailableHandler)
  .on('items available', itemsAvailableHandler, {
    next: 'order placed'
  })
  .on('order placed', orderPlacedHandler, {
    next: 'order placed email sent'
  });

Each pipeline is backed by a document store, and this is designed to be
extensible. The first document store to be implemented is
banzai-couchdb-store, but document stores only require a minimum of two functions to work with
Banzai (load(docId, callback) and save(doc,
callback)
) so shouldn't be too difficult to write support for
more databases.

Pedro's documentation includes a lot more details on pipelines, state
handling, state stores, and work queues. Banzai seems like a useful tool
for working with format conversion, but Pedro also notes it's been used
with Amazon Mechanical Turk.

Node Inotify

Node Inotify (License: MIT, npm: inotify) by Camilo Aguilar is an
inotify API for Node and Linux. This library has been kicking around for over a
year now, but the author and several contributors have been actively
working on it.

The README includes a detailed code example, but the API is fairly
straightforward: by using an instance of Inotify,
addWatch() can be used to watch files or directories for
changes. The callback fired when the file system changes receives an
event which includes a mask that details what kind of change occurred.