Nodejitsu

Save time managing and deploying your node.js app. Code faster with jitsu and npm

Introducing Winstond

About the author

Name
Location
Worldwide
nodejitsu nodejitsu

Logging seems like it should be simple enough, but when you're dealing with multiple machines, multiple nodes, and multiple apps which all need to log in some kind of organized fashion, things can get complicated very quickly.
To make things more interesting, not only do these things need to log, but they need to manage and access the logs as well.


A year and a half ago, Winston was created in an attempt to unify all
potential logging transports. While winston serves its job well, another
project needed to be added to the equation to help solve the problem mentioned above.

Winston's success has been defined by it's ability to instantiate loggers and add multiple different transports to each one.

var winston = require('winston');

//
// Create a winston Logger object.
//
var logger = new winston.Logger;

//
// Add the File transport, to log to a
// file or any kind of stream.
//
logger.add(winston.transports.File, {  
  filename: __dirname + '/my-log.log'
});

//
// Use our logger.
//
logger.log('info', 'Hello world!');

I'm sure it would be easy enough to whip up your own logger which simply creates and writes to a file stream, but, obviously, there is more to winston than just a single transport. The portability is key, along with the extensibility of any transport imagineable.

While the logger above is great, and winston carries all the extensibility in the world, you're still stuck with one logger inside one process. Kind of like an in-process database: it's great, and simple, until it becomes its own obstacle, and gets in its own way.

One day you're using sqlite, and everything just works, until you need massive amounts of webscale; direct intravenous injections of pure webscale. All of the sudden, in-process doesn't cut it, and you need a server. This is what led to the creation of winstond.

winstond

Winstond is an attempt to solve this problem. As the name implies, it is a configurable server, built on top of winston, using all of winston's core methods. This means it acts exactly like a logger, and uses the same transports winston does.

A winstond server doesn't look much different from winston itself. The only difference between a winstond instance and a winston Logger is the ability to act as a server using one of two different backends. Right now, that includes HTTP and NsSocket, although more future backends aren't out of the question.

Creating a winstond nssocket server and talking to it should be simple enough.

winstond (our server):

//
// Instantiate a winstond server using the nssocket backend
//
var server = winstond.nssocket.createServer({  
  services: ['collect', 'query', 'stream'],
  port: 9003
});

//
// Have our server log to a file
//
server.add(winstond.transports.File, {  
  filename: __dirname + '/foo.log'
});

//
// Test our server real quick
//
server.log('info', 'test', {}, function() {  
  server.query(function(err, results) {
    console.log(results[0]);
  });
});

//
// Listen on specified port
//
server.listen();  

The HTTP backend would look similar. The differences between the two, aside from the HTTP backend using a non-bidirectional protocol, is that the HTTP backend uses json-rpc and the Http transport to communicate with winstond.

winston (our client):

var logger = new winston.Logger;

//
// Specifiy nssocket for the winstond server, which is using the nssocket backend.
//
logger.add(winston.transports.Nssocket, {  
  host: 'localhost',
  port: 9003
});

//
// Send a log to the winstond server
//
logger.log('info', 'hello world!', {  
  foo: 1
}, function() {
  //
  // Query the logs
  // We should see the log we just added
  // (assuming it meets the criteria below).
  //
  logger.query({ start: 10, rows: 50 }, function(err, results) {
    if (err) throw err;
    console.log(results);
  });

  //
  // Stream our one log
  // We should see the single log we added above
  //
  logger.stream({ start: 0 }).on('log', function(log) {
    console.log(log);
  });
});

Using the Nssocket transport, winstond can handle the querying and streaming
on the client's behalf.


While it may not appear to be terribly practical at first, the fact that a winstond server is still just a winston Logger instance at heart has some interesting implications, such as creating the possibility of adding the nssocket transport to a winstond server, and have it talk to another winstond server.

//
// Instantiate a winstond server using the nssocket backend
//
var server = winstond.nssocket.createServer({  
  services: ['collect', 'query', 'stream'],
  port: 9003
});

//
// Talk to another winstond server
//
server.add(require('winston-nssocket').Nssocket, {  
  host: 'localhost',
  port: 9002
});

//
// Listen
//
server.listen();

//
// Use a logger
//
var logger = new winston.Logger;

//
// Connect to the new winstond nssocket server
//
logger.add(require('winston-nssocket').Nssocket, {  
  host: 'localhost',
  port: 9003
});

logger.log('info', 'Testing!', {});  

winston

While winstond is a thin layer on top of winston, maybe the the most interesting things this entails are the changes to winston itself.

The creation of winstond has some implications. Not only do you need this server over an in-process model, you also need to access and manage your data. This is very helpful for a winstond server, but it's also just helpful for winston being used standalone.

Most every winston transport has now been equipped with the ability to stream logs back to winston, as well as to query logs, using Loggly-like query options.

Depending on the transport's natural abilities to do this, (e.g. redis might have a harder time querying than couch), every future transport should include a stream() and query() method. If a transport does not support these methods, the transport will be ignored when a logger needs to query or stream.

Querying

logger.query({  
  start: 10,
  rows: 10
}, function(err, results) {
  if (err) throw err;
  console.log(results);
});

Query results will return an aggregate object, containing all results from each transport. This can be avoided by specifying a transport name in the options.

Streaming

var logger = new winston.Logger;

logger.add(winston.transports.File, {  
  filename: __dirname + '/foo.log'
});

//
// Create a stream from our loggers transports.
// Start at the end of the stream.
//
var stream = logger.stream({  
  start: -1
});

stream.on('log', function(log) {  
  console.log(log);
});

stream.on('error', function(err) {  
  console.error(err);
  stream.destroy();
});

logger.log('info', 'You should see me in the stream!');

This essentially tail -f's the log file we setup using the File transport. The stream will stream in every transport possible, and because it cannot produce an aggregate object like the query method can, a transports property will be added to each log object.

Underneath the Surface

So how is something like streaming implemented? In Mongo, we can use a
tailable cursor. In Redis, we can use the built-in pub/sub capabilities. As mentioned above, for files, we can implement our own tail -f, which polls the file with constant read(2) calls, and in Couch, we can use Couch's _changes notification functionality.

Each transport usually has some ability to stream, or query, that can be
tapped into. Winston will now unite all of this functionality.

Unfortunately, some transports might not carry any built-in streaming
(or querying) functionality in any way, in which case, the transport will have to resort to more primitive and less elegant measures, like polling. Don't get the impression that winston makes the choice of transport irrelevant, because it doesn't. It only makes things cleaner. You still probably want to decide between files, couch, redis, mongo, etc., depending.

Any future transports for winston would be smart to add query and stream methods, but it is not necessary.

Solutions

At Nodejitsu, winstond will be used as the basis for our logging server.
This will allow users to easily stream logs from their apps rather than
essentially polling for them by hand.

Conclusion

So what would be the difference between having all of your apps log to a shared Couch server, as opposed to passing all logs through a winstond server first? Winstond can handle multiple transports, it can talk to other winstond servers, all traffic can be handled at a single configurable, hookable, administrable endpoint, and only this endpoint has to be configured the way you want it. Every other process can mindlessly talk to the winstond server, using the nssocket or http transport.

If you have more than a few apps to deal with, and want to unify their logs in a useful way, winstond might do the trick as far as consolidating these logs.