The JavaScript blog.


frameworks nodejs graphics

Mastering Node, Pragmatic CSS3, Wink

Posted on .

I was at the dConstruct conference on Friday. If you'd like to read
about it, take a look at my summary (lovingly crafted, complete with
photos): dConstruct 2010.

Mastering Node

You might remember TJ Holowaychuk from other JavaScript projects like
Express and Jade. He's back again with a book called Mastering Node.
It's a good start, but he isn't planning on writing much more:

Mastering node is an open source eBook by node hackers for node hackers. I started this as a side project and realized that I don't have time :) so go nuts, download it, build it, fork it, extend it and share it.

Despite the fact he says he doesn't have time to finish it, his last
update was 4 days ago and there's content on installing node, modules,
events, and other core subjects.

The content is all in Markdown, and he uses a simple Makefile with
command line tools to generate HTML, PDF, and e-reader formats. Which is
a pretty cool way to write a book -- I've always been a LaTeX man myself
but this seems like a much more straightforward way of generating all
those formats (I'm a Kindle fan so generating ePub/Mobi is important to

Why not check it out and contribute?

Pragmatic CSS3

Pragmatic CSS3 - Webit-Transform Matrix3d Tutorial is a tutorial that
demystifies CSS3 matrix transforms. It does a good job as well. Complete
source is available on GitHub, under 9elements /


Jérôme Giraud got in touch to talk about
Wink, a mobile JavaScript framework. The site has downloads, previews, demos, and even some tutorials.

Wink is targeting cutting-edge WebKit features, they're particularly
proud of their 3D
multitouch support.

I recently covered the basics of touchscreen device support in our
Let's Make A Framework series, and it sparked an interesting discussion about gesture support. Apple have
specifically targeted gestures as well as simple touch events, but there
was some confusion about the best way of supporting them. Hopefully Wink
is looking into this area.

I took a cursory look over the framework, and they appear to have
namespaced the framework like other modern JavaScirpt libraries.


nodejs interviews

Ryan Dahl Interview: Part 2

Posted on .

This interview was conducted by [Oleg Podsechin](http://twitter.com/olegpodsechin) with [Ryan Dahl](http://tinyclouds.org/) on the 8th of July, shortly after Ryan's talk in Cologne. Oleg is a JavaScript enthusiast who runs Ionsquare Ltd, an IT consultancy.

OP: So on the topic of CommonJS, are you following any of the APIs or
any of the discussions on the list?

RD: Yeah, sure

OP: And which ones are you most interested in?

RD: CommonJS has some good specs and some less good specs. Some specs are rather prescriptive without any implementation - which I find
wrong. I do like the idea of having a common server-side javascript
interface - I just think it’s going to take some time to experiment with
different APIs. A binary spec is quite important because JavaScript
currently lacks a way of dealing with raw binary in any reasonable way.
The module spec is good, the assert spec is good, the others are

OP: What about the package spec?

RD: Oh yeah, it also looks good. I don’t want to work on a package system, so I’m not following it super closely, but I think that there’s
a lot of good ideas there.

OP: As a user you must use a package management system. Which ones do
you use?

RD: I’m playing around with NPM. It’s OK, kind of buggy, but you can use it.

OP: So with regards to packages, obviously there’s some stuff that’s
going into the core of Node, but external packages, like XML parsing,
are there any packages that you think are important that aren’t there

RD: There needs to be a better MySQL solution, libmysql_client, the library that comes with MySQL is blocking so that is not a solution.
There are other solutions, but they seem kind of buggy. A lot of people
use MySQL and it would be a hindrance for them if they couldn’t access
that easily. That’s one.\
For a long time I was lusting after a good JavaScript HTML parser, but
it seems that has solved. I also wanted a DOM implementation and it
seems like that’s been solved too. I would really like a way to access
Cassandra, which uses Thrift - that’s not been done yet.

OP: There aren’t really any decent JavaScript Thrift libraries

RD: Thrift is a piece of crap but unfortunately some projects are using it so we’ve got to interface with it. Some sort of Thrift binding
would be good.

OP: I think in the next release they’re looking to have a RESTful

RD: I’ve heard they’re introducing an interface based on Avro, a new message serialization RPC thing, but I’m not sure how good the Avro
support is. Avro seems a lot better than Thrift so just binding to Avro
would be the best way go for talking to Casandra - I don’t know.

Being able to connect to databases is important for users. If it’s not
there, then it’s a total roadblock for a lot of people. So, MySQL is a
major one.

OP: A slight aside, but a big trend, aside from server side JavaScript
is non relational databases and Node seems like the perfect glue, if you
will, to connect these different data stores together. CouchDB guys are
using it for that purpose. What are your thoughts, can you see an
opportunity there?

RD: Exactly, Node perfectly fills the proxy and authentication layer, between the storage backend and client. So yeah, I think it’s a
good sort of glue and I agree with the CouchDB philosophy that the bulk
of the application can kind of sit in the database. All the hard stuff
can be back in Couch and Node can just proxy data back and forth.

OP: So talking about the packages you’d like in Node, moving into the
core and looking at the way the project is being built, what are your
thoughts on project leadership in open source projects? What do you
think is the right way to do it, which things shouldn’t you do? What’s
your personal approach? Because you used to post little challenges for
people to get them excited and get them contributing a little bit. Can
you talk more about that?

RD: I have a strong arm in the project. I’m the only committer and I dictate how things go and I think that’s a good approach for Node at
this stage. At some point, hopefully, Node will grow up and we’ll a
committee that decides on things. But at the moment having somebody
that’s dedicated to the project and who will make sure that any changes
that go in will be maintained is important. Part of that roll is not
accepting changes that I can’t maintain myself, and so it means
rejecting a lot of good code - just because it doesn’t fit into my
contrained idea of what “Node core” is. There are users who would have
contributed, for example, package manager code, but it’s not something I
have time to maintain.

Another part of leading this project is getting people involved by very
explicitly suggesting to people what needs to be done. I’m sending a lot
of I emails to people saying “hey, you should give me a patch on this
thing, that would be very helpful”.

OP: Nudging them a little bit in the right direction ...

RD: Yeah.

OP: So how big of a role has GitHub played in this? And git and the
social coding element of it?

RD: GitHub is great - it’s best feature is the ability to have web links source code - at a particular commit, with a specific section
highlighted. Linking to source code like that really improves
communication in email and on IRC. That’s probably the best feature of

OP: Issue tracker?

RD: The issue tracker I use, which is OK, but it could be better. Generally, GitHub could be doing more by hosting a mailing lists. Google
Groups sucks.

OP: Moving on, with regards to commitment to the project, you’re saying
that you’re fully behind (it and so on and so forth,) so you’re
currently employed by Joyent? and working 100% on Node?

RD: Yeah - Node and projects based on Node. It’s great.

OP: I guess the question is more about the commercial nature of Node and
commercialization of Node. Clearly Joyent have an interest in it, being
a hosting company, but do you see an ecosystem of businesses emerging
around Node at some point and if so what types of businesses are these
likely to be?

RD: One obvious thing is hosting of applications in an simple way like Heroku is doing. Node opens the door to independent contractors
making little real-time websites for people -- so there’s that

OP: You don’t have an interest in building a service on top of Node?
Rather you wish to maintain the core project?

RD: I work for Joyent, so I work on products for them, but my main interest is making Node perform well and make users happy.

OP: So the last couple of questions are a bit more abstract. The first
one is about the asynchronous nature of Node. Do you see event driven
webapps becoming more prevalent in the future? Not only Node, but
asynchronous webapps in general.

RD: Yeah, definitely. Not waiting for a database is a big win in terms of performance - the amount of bagage associated with each TCP
stream is just much smaller. We need that for real-time applications
where many mostly-idle connections are being held. But even for normal
request response websites I think we’ll see more asynchronous setups
just because of the performance wins - even if it’s necessary. It’s
clear that asynchronous servers perform better in almost every way, it’s
difficult to ignore that.

OP: I guess writing callbacks and indentation in JavaScript is one
hurdle towards asynchronous programming, but JavaScript is much easier
for doing such stuff than other languages.

RD: There are of course efficient green thread and coroutine implementations which allow you to write asynchronous code in a
synchronous looking way - Eventlet for example. I’m not convinced that’s
the right approach, I think it’s a leaky abstraction. There’s no
abstraction with callbacks - it’s a rather direct translation from the
interface the operating system gives.

OP: Have you tried other languages that compile to JavaScript, like
CoffeeScript, they do callbacks and deferred and stuff like that.

RD: CoffeeScript is cute. I’m not convinced by CoffeeScript’s deferred thing. I haven’t used it but it seems maybe that it will
confuse the users.

OP: Do you not like the idea of using JavaScript as an intermediary
language? If you’re going to be writing stuff for JavaScript, you should
write it in JavaScript?

RD: I mean nobody would choose JavaScript if they had a choose, there are a lot of things wrong with it, but it’s an important language
and it’s set in stone by its ties to the browser.

OP: So all these tools, like debugging ...

RD: CoffeeScript is beautiful but it makes programming more difficult. If there was more toolage around CoffeeScript, like a
debugger which translated line numbers from compiled code to CoffeScript
lines, it will be interesting. For myself, Node is already buggy enough,
another layer hurts rather than helps. The deferred concept is
interesting, basically when you put in a deferred keyword before a
function call, the rest of the current callback is put into a callback
as the last parameter to the deferred call. Wonder how that’s going to
work out - it seems too simplisitic. It’s kind of cute that you still
have the same programming model. I mean, it’s not the same as what’s
happening for coroutines or green threads, there’s still only one
execution stack. Who knows, maybe CoffeeScript’s deferred keyword will
end up working out well, I’m skeptical though.

OP: So, last question. Which is sort of two questions rolled into one
really. At the last talk you gave the other day you mention that your
view of a program is that it’s a set of inputs of data from various
sources, somehow transforms that data and forwards it on. Can you
elaborate on that a bit?

RD: Yeah, I think most of the programs, or a large part of the programs that we write, are just proxies of some form or another. We
proxy data from a database to a web browser, but maybe run it through a
template first and put some HTML around or do some sort of logic with
it. But largely, we’re just passing data from one place to the other.
It’s important that Node is setup to pass data from one place to the
other efficiently and with proper data throttling. So that when data is
coming in too quickly from the database, that you can stop that incoming
flow. Suppose it’s over a TCP connection, you can just stop reading from
that data source and not fill up your memory with the whole response.
Start sending out the first part of the template that you’re sending to
the web browser and then pull in more data from the DB. You know, it
must properly shuffle the data through the process without blowing up
the memory if one side is slower than the other. You shouldn’t have to
pull down the entire table, put into a template and then send it out. It
should just be able to flow through your system, so creating an
environment where it’s easy to setup these flows in the proper way is
important. We’re not there yet, but that’s kind of my vision of what
Node will be. Lots of shuffling of data from one file descriptor to the
next, without having to buffer a ton of data.

OP: So in a way you can look at different Node instances talking to each
other, forming a graph with directed edges between the different nodes?
Is that where the name Node comes from? How did you come up with the

RD: I used the name “Node” because I envision it as one part of a larger program. A program is not a process, a program is a database plus
an application plus a load balancer and Node is one node of that. It’s
not necessarily a bunch of NodeJS instances but a couple Node.js
instances plus some other things.

OP: Sounds good! Thank you for taking the time to chat.


nodejs interviews

Ryan Dahl Interview: Part 1

Posted on .

This interview was conducted by [Oleg Podsechin](http://twitter.com/olegpodsechin) with [Ryan Dahl](http://tinyclouds.org/) on the 8th of July, shortly after Ryan's talk in Cologne. Oleg is a JavaScript enthusiast who runs Ionsquare Ltd, an IT consultancy.

OP: The first question is an introduction really. How did you arrive at
Node? Did you have experience with JavaScript before? When did you get
started with JavaScript? And also event driven software?

RD: I was a contractor and I was doing various little C projects usually involving server and event driven software and I realized that I
was doing this same code over and over. C is a nice language to work in,
but I wanted something I could script in the same way that I was
programming these servers.

OP: Had you done any front end stuff in JavaScript?

RD: A little. I used to work a lot with Ruby on Rails - so I’d often be dealing with front-end code. Back then I wrote a little Ruby web
server called Ebb that was meant to be a faster Mongrel. That code was
the starting point for Node.

OP: Ebb was mostly in C right? So you went from writing it in Ruby, then
writing it in C and now you’re sort of ending up writing it in

RD: Right. So what originally was Ruby turned into C. For a while I toyed with the idea of having a small web server C library - but it’s
hard to get anything done in C. One day I had this epiphany: “JavaScript
is exactly the language that I’m looking for here.” That happened
shortly after V8 being released.

OP: You’ve said that there are two languages that will always be around:
C and JavaScript. So, what are your thoughts on JavaScript as a general
purposed programming language?

RD: JavaScript has certain characteristics that make it very different than other dynamic languages, namely that it has no concept of
threads. Its model of concurrency is completely based around events.
This makes it rather different than other general purpose dynamic
programming languages like Ruby and Python. At least for certain classes
of problems, I’ve found JavaScript to be much easier to program in--for
example when writing an IRC server.

OP: How do you see the future of JavaScript? Do you see JavaScript
becoming increasingly more prevalent, not only on servers but also on

RD: JavaScript is already doing a great job describing GUIs. I think with a familiar browser-like API JavaScript could also make a good
desktop application language.

OP: JavaScript is quite unstructured, so people just copy paste
JavaScript code ...

RD: Yeah, not having a module system doesn’t help. JavaScript really encourages people to dump everything into global variables. That’s a
real detractor for JavaScript but in the end better practices can
overcome that sort of thing.

OP: So, did you follow the whole discussion around EcmaScript 4 and
EcmaScript 5?

RD: I like Crockford’s opinion that the language should be kept simple. One of the best things about JavaScript was its simplicity. It
didn’t have many predefined ideas about how to do stuff - particularly
for I/O. Although EcmaScript 4 didn’t define any I/O, it did define a
lot. It did make a lot of breaking changes. That said, I wish EcmaScript
5 did have a few more features.

OP: Any particular ones in mind?

RD: What’s this called? Destructive assignment? If you have an array on the right and a list of variables on the left, and they can be define
that way. That would be nice to have.

OP: That’s included in Rhino, but not in V8

OP: So let’s move on to Node itself. what is the most difficult design
decision you made with regards to the project?

RD: Something that was very hard for me was ... my original idea was that it was going to be purely non-blocking system and I’ve backed out
of that a big in the module system and a few other areas. In the browser
load JavaScript from a script tag is non-blocking. You don’t really know
when the scripts are completely evaluated until an onLoad callback is
made. Originally Node was similar. You would load a bunch of module
files and you wouldn’t know that they were fully interpreted, fully
evaluated, until a “loaded” event was . This made things a bit
complicated. You couldn’t just do “require” and start using that stuff
right below it, you had to wait for the callback to do that.

OP: The hello world app would have one more indentation.

RD: Right.

OP: But it’s funny because people say that one of the benefits
JavaScript offers is that you can use it in the browser as well. You can
run the same validation logic on the server and browser, but the
CommonJS module spec doesn’t work within the browser, so there are these
efforts to try and make frameworks with asynchronous module loading.

RD: Right, so in terms of difficult design decisions, I wanted Node to be browser-like. Maybe it didn’t use the same methods but the same
structures could be ported easily, aliasing methods to the browser ones.
Originally Node achieved that--it was totally browser-like. Originally,
it even had a ‘window’ object. I slowly backed off that API as it became
clear it wasn’t necessary to have the server-side environment be exactly
the same. So I went with the CommonJS module system which was rather
reasonable; the CommonJS people had put a lot of thought into it and I
didn’t really want to worry about modules so much. So yeah, require is
blocking and there are some other minor things that are blocking in
Node. Generally this pragmatic approach of being non-blocking 99% of the
time, but allowing a few synchronous operations here and there has
worked out well. It probably doesn’t matter for a server-side program if
you load modules synchronously.

Part 2 of this interview will be posted tomorrow (Wednesday 11th August).



Yahoo! and Node

Posted on .

In Multi-Core HTTP Server with
Peter Griess (Principal Engineer, Yahoo! Mail) discusses using Node at
Yahoo! Mail. He discusses threading, and solutions like
multi-node. He also demonstrates using node-v0.1.100 with
node-webworker (which he wrote).

One interesting point in his post is about the quality of modules
written in C+. Given the paradigm shift between writing solid
JavaScript and efficient, portable and *safe
C, modules aren't always
reliable. Hopefully the community will attract more hardened C*+
developers to help improve popular modules.

It's great to see a company like Yahoo! taking an interesting in Node,
and I hope they end up using it in their architecture. Peter's already
created some useful projects (at
github/pgriess), and this would be great ammunition to use if you're trying to sell your boss on using Node.


libraries server nodejs

nStore, multi-node

Posted on .


nStore (MIT License) by Tim Caswell is a simple key/value store for Node. It uses an append-only
file format and writes changes to disk, so data should be safe if your
application's process crashes, and you can back it up with standard
tools like rsync.

nStore can be installed with npm.

To use a store, it has to be created first:

var users = nStore('data/users.db')

The syntax for creating documents requires a key, values, and then an
error callback. Loading documents is similar.

users.save('alex', { name: 'Alex', admin: false }, function(err) {
  if (err) { throw err; }

users.get('alex', function(err, document, meta) {
    if (err) { throw err; }

For examples on streaming, filtering, and searching results, see the


Multi-node by Kris Zyp can launch multiple Node processes to concurrently serve requests. The
following example creates a HTTP server, then 4 concurrent instances of

var http = require('http'),

server = http.createServer(function(request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'})

nodes = require('multi-node').listen({
  port: 80, 
  nodes: 4
}, server)

Multi-node can also be used for inter-process communication, which can
be used to share state across servers.