Logging seems like it should be simple enough, but when you're dealing with multiple machines, multiple nodes, and multiple apps which all need to log in some kind of organized fashion, things can get complicated very quickly. To make things more interesting, not only do these things need to log, but they need to manage and access the logs as well.
A year and a half ago, Winston was created in an attempt to unify all potential logging transports. While winston serves its job well, another project needed to be added to the equation to help solve the problem mentioned above.
Winston's success has been defined by it's ability to instantiate loggers and add multiple different transports to each one.
I'm sure it would be easy enough to whip up your own logger which simply creates and writes to a file stream, but, obviously, there is more to winston than just a single transport. The portability is key, along with the extensibility of any transport imagineable.
While the logger above is great, and winston carries all the extensibility in the world, you're still stuck with one logger inside one process. Kind of like an in-process database: it's great, and simple, until it becomes its own obstacle, and gets in its own way.
One day you're using sqlite, and everything just works, until you need massive amounts of webscale; direct intravenous injections of pure webscale. All of the sudden, in-process doesn't cut it, and you need a server. This is what led to the creation of winstond.
Winstond is an attempt to solve this problem. As the name implies, it is a configurable server, built on top of winston, using all of winston's core methods. This means it acts exactly like a logger, and uses the same transports winston does.
A winstond server doesn't look much different from winston itself. The only
difference between a winstond instance and a winston
Logger is the ability
to act as a server using one of two different backends. Right now, that includes
HTTP and NsSocket, although more future backends aren't out of the
Creating a winstond nssocket server and talking to it should be simple enough.
winstond (our server):
The HTTP backend would look similar. The differences between the two, aside from
the HTTP backend using a non-bidirectional protocol, is that the HTTP backend
uses json-rpc and the
Http transport to communicate with winstond.
winston (our client):
Using the Nssocket transport, winstond can handle the querying and streaming on the client's behalf.
While it may not appear to be terribly practical at first, the fact that a winstond server is still just a winston Logger instance at heart has some interesting implications, such as creating the possibility of adding the nssocket transport to a winstond server, and have it talk to another winstond server.
While winstond is a thin layer on top of winston, maybe the the most interesting things this entails are the changes to winston itself.
The creation of winstond has some implications. Not only do you need this server over an in-process model, you also need to access and manage your data. This is very helpful for a winstond server, but it's also just helpful for winston being used standalone.
Most every winston transport has now been equipped with the ability to stream logs back to winston, as well as to query logs, using Loggly-like query options.
Depending on the transport's natural abilities to do this, (e.g. redis might
have a harder time querying than couch), every future transport should include
query() method. If a transport does not support these
methods, the transport will be ignored when a logger needs to query or stream.
Query results will return an aggregate object, containing all results from each transport. This can be avoided by specifying a transport name in the options.
tail -f's the log file we setup using the
The stream will stream in every transport possible, and because it cannot
produce an aggregate object like the
query method can, a
property will be added to each log object.
Underneath the Surface
So how is something like streaming implemented? In Mongo, we can use a
tailable cursor. In Redis, we can use the
built-in pub/sub capabilities. As mentioned above, for files, we can
implement our own
tail -f, which polls the file with constant
and in Couch, we can use Couch's
_changes notification functionality.
Each transport usually has some ability to stream, or query, that can be
tapped into. Winston will now unite all of this functionality.
Unfortunately, some transports might not carry any built-in streaming (or querying) functionality in any way, in which case, the transport will have to resort to more primitive and less elegant measures, like polling. Don't get the impression that winston makes the choice of transport irrelevant, because it doesn't. It only makes things cleaner. You still probably want to decide between files, couch, redis, mongo, etc., depending.
Any future transports for winston would be smart to add
methods, but it is not necessary.
At Nodejitsu, winstond will be used as the basis for our logging server. This will allow users to easily stream logs from their apps rather than essentially polling for them by hand.
So what would be the difference between having all of your apps log to a shared
Couch server, as opposed to passing all logs through a winstond server first?
Winstond can handle multiple transports, it can talk to other winstond servers,
all traffic can be handled at a single configurable, hookable, administrable
endpoint, and only this endpoint has to be configured the way you want it.
Every other process can mindlessly talk to the winstond server, using the
If you have more than a few apps to deal with, and want to unify their logs in a useful way, winstond might do the trick as far as consolidating these logs.