Nodejitsu

Save time managing and deploying your node.js app. Code faster with jitsu and npm

Getting Started with Flatiron HTTP

About the author

Name
Location
Worldwide
nodejitsu nodejitsu

Lately, I've noticed quite a few people asking to see examples of flatiron apps in #nodejitsu. Now, there are actually quite a few of our projects that use flatiron extensively. Unfortunately, there are a few problems, chief among them that no two flatiron projects do quite the same things or are organized the same ways.

Now, this isn't necessarily a problem when it comes to building tools. In fact, I think it's a great demonstration of flatiron's flexibility. Each project can use exactly the libraries and organization that make sense for their applications, even for apps as different as haibu and jitsu. It can be confusing for newcomers, though, that are used to frameworks that enforce stronger conventions, or those that are trying to figure out how to get from "hello world" to something more profound.

I thought what would be helpful for flatiron newcomers is an in-depth tutorial, which iterates on a "hello world" flatiron server (like those in the project's examples) into a relatively complex webservice that uses some of flatiron's more complex functionality.

In this tutorial, I'm going to create a webservice for browserify, a popular client-side code framework for node.js. All of the code is on github, and every section of this tutorial corresponds to a code commit. In early parts of this tutorial I will show all the code, but as it gets longer I will typically explain the code with links to the relevant files/commits on github.

Flatiron HTTP Hello World

To get started, let's write a simple "hello world" app using flatiron, in ./app.js:

// Require flatiron and grab the app object.
var flatiron = require('flatiron'),
    app = flatiron.app;

// Use the http plugin. This makes flatiron act as an http server with a
// router on `app.router`.
app.use(flatiron.plugins.http);

// Route handler for http GET on the root path
app.router.get('/', function () {
  // The request and response objects are attached to `this`.
  var req = this.req,
      res = this.res;

  // Flatiron comes with a logging object already attached!
  app.log.info('Saying hello!');

  // Handle the response as you would normally.
  res.writeHead(200, { 'content-type': 'text/plain'});
  res.end('hello!');
});

// Start the server!
app.start(8080, function (err) {
  if (err) {
    // This would be a server initialization error. If we have one of these,
    // the server is probably not going to work.
    throw err;
  }

  // Log the listening address/port of the app.
  var addr = app.server.address();
  app.log.info('Listening on http://' + addr.address + ':' + addr.port);
});

Writing a basic flatiron server is pretty simple. First, we require flatiron and grab the 'app' object. Then, we assign a route handler to '/' which responds to request with the text, 'hello!'. This server also takes advantage of flatiron's built-in logging to print information to the screen.

Here's the accompanying ./package.json:

{
  "name": "flatiron-helloworld",
  "version": "0.0.0",
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0"
  }
}

The most important thing to notice here is that union is also a dependency. Union is what's called a "peer dependency". This means that flatiron doesn't itself list union as a dependency, but the http plugin expects to be able to require it. This is because flatiron has uses outside of http servers where it doesn't make sense to install union.

Luckily, if you forget, flatiron will return a reasonable error message:

$ node app.js 
flatiron.plugins.http requires the `union` module from npm
install using `npm install union`.
Trace: 
    at Object.<anonymous> (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/node_modules/flatiron/lib/flatiron/plugins/http.js:25:11)
    at Module._compile (module.js:441:26)
    at Object..js (module.js:459:10)
    at Module.load (module.js:348:31)
    at Function._load (module.js:308:12)
    at Module.require (module.js:354:17)
    at require (module.js:370:17)
    at Object.http (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/node_modules/flatiron/node_modules/broadway/node_modules/utile/lib/index.js:261:14)
    at Object.<anonymous> (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/app.js:7:25)
    at Module._compile (module.js:441:26)

Once flatiron and union are both installed, of course, our app works as expected:

$ node app.js 
info: Listening on http://0.0.0.0:8080

Now we can make a request to our server and receive a kind message:

$ curl localhost:8080
hello!

Furthermore, we can see evidence of our request in the server's logs:

$ node app.js 
info: Listening on http://0.0.0.0:8080
info: Saying hello!

"Hello World" With A Twist: Configuration and POST

Let's make our "Hello World" app slightly more sophisticated:

var path = require('path');

var flatiron = require('flatiron'),
    app = flatiron.app;

// Set up app.config to use ./config.json to get and set configuration settings.
app.config.file({ file: path.join(__dirname, 'config.json') });

app.use(flatiron.plugins.http);

// This router syntax allows you to define multiple handlers for one path based
// on http method.
app.router.path('/', function () {

  // This is the same functionality as previously.
  this.get(function () {
    this.res.writeHead(200, { 'content-type': 'text/plain' });
    this.res.end('hello!');
  });

  // Now, when you post a body to the server, it will reply with a JSON
  // representation of the same body.
  this.post(function () {
    this.res.json(200, this.req.body);
  });
});

// Now we're using app.config to set the port, with a default of 8080.
app.start(app.config.get('port') || 8080, function (err) {
  if (err) {
    throw err;
  }

  var addr = app.server.address();
  app.log.info('Listening on http://' + addr.address + ':' + addr.port);
});

The accompanying package.json has a few minor changes as well:

{
  "name": "browserify-cdn",
  "version": "0.0.0",
  "scripts": {
    "start": "node ./app.js"
  }
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0"
  }
}

Besides a name change, our app also now has a "scripts.start" field. This means two things:

  • 'npm start' will start your app.
  • 'jitsu deploy' will successfully start your app.

Note that this app also has configuration now, using app.config! We're using this to load configuration settings from ./config.json:

{
  "port": "3600"
}

So, if you run this server and GET /, the response is the same as before:

$ curl localhost:3600
hello!

However, if we POST some data:

$ curl -X POST -d 'foo=foo&bar=bar' localhost:3600
{"foo":"foo","bar":"bar"}

We can also post JSON if we remember to set the content-type header properly:

$ curl -X POST -d '{ "foo": "foo", "bar": "bar" }' -H 'Content-Type: application/json' localhost:3600
{"foo":"foo","bar":"bar"}

This all Just Works is because flatiron has built-in tools for buffering and parsing request bodies, making the development of JSON APIs dead easy.

Now we're ready to do something a bit more interesting.

Basic Browserifying Server & Separation of Concerns 101

Now we're going to make our app browserify posted code snippets. Once it's done, the interface should work like so:

$ curl -X POST -d 'var traverse = require("traverse");' localhost:3600
var require = function (file, cwd) {
    var resolved = require.resolve(file, cwd || '/');
    var mod = require.modules[resolved];
    if (!mod) throw new Error(
        'Failed to resolve module ' + file + ', tried ' + resolved
    );
    var res = mod._cached ? mod._cached : mod();
    return res;
}

require.paths = [];
require.modules = {};
require.extensions = [".js",".coffee"];

require._core = {
    'assert': true,
    'events': true,
    'fs': true,

**snip**

I didn't include the entire bundle here, but the general idea is this: You post browserify-able javascript code to the service, and it returns the browserified bundle.

By this point, our app is getting big enough that we want to break functionality up into different pieces. The way I approach this is like so:

  1. Write a library (or a few) that implement the basic pieces of functionality you need.
  2. Call that library from ./app.js, only handling http details in the router.

The idea here is to separate the concerns so that there isn't routing logic in the bundler or bundling logic in the router. In larger apps, this approach gets broken down further into large-scale design patterns like MVC. In this case our app is pretty small though, so simply separating the bundling into its own library will suffice.

The bundler, which lives in ./bundler.js for this iteration, is a constructor that inherits from EventEmitter2:

var EventEmitter2 = require('eventemitter2').EventEmitter2,
    browserify = require('browserify'),
    detective = require('detective'),
    npm = require('npm'),
    crypto = require('crypto'),
    util = require('utile');

// I wrote the Bundler as a constructor with prototype methods.
// I find that it's a good fit for stateful problems.
var Bundler = module.exports = function (opts) {

  // This lets you create a new Bundler without the 'new' keyword.
  if (!(this instanceof Bundler)) {
    return new Bundler;
  }

  var bundler = this;

  // Set the bundler's persistent options here. Also handle defaults.
  bundler.options = opts || {};
  bundler.options.npm = bundler.options.npm || {};

  // Overrides falsy values such as `undefined`
  if (bundler.options.cache !== false) {
    bundler.options.cache = true;
  }

  // Bundler inherits from an EE2 with wildcards and the :: delimiter.
  EventEmitter2.call(this, {
    wildcard: true,
    delimiter: '::',
    maxListeners: 0
  })

  // Bundler requires a loaded npm in order to work properly.
  // The rest of the code in the constructor sets this up and emits events
  // to signal when it's ready.
  bundler.ready = false;

  npm.load({}, function (err) {
    if (err) {
      // What would be really cool is if I checked to see if there was an error
      // event listener or not.
      bundler.emit('error', err);
    }

    bundler.ready = true;
    bundler.emit('bundler::ready');
  });
};

util.inherits(Bundler, EventEmitter2);

The bulk of the constructor handles setting default options for our object and configuring the EventEmitter2 aspects of the object. The interesting piece here calls npm.load on the npm singleton object, emits events when it succeeds or fails, and sets a this.ready flag. This allows us to create our object and then listen for when we may use it.

Next, we'll define the prototype method that actually bundles our code:

// The 'bundle' method is what actually attempts to bundle your project.
Bundler.prototype.bundle = function (src, cb) {
  var bundler = this,
      modules = detective(src);

  // We used 'detective' to get a list of needed modules, so that we can make
  // sure they're installed before trying to browserify.
  npm.commands.install(modules, function (err) {
    if (err) {
      cb(err);
    }

    var bundle;

    // Attempt to browserify the passed-in javascript source
    try {
      bundle = browserify(this.options)
        .addEntry('index.js', { body: src || '' })
        .bundle();

      // Hit the callback with our complete bundle object.
      cb(null, {
        src: src,
        md5: crypto.createHash('md5').update(src).digest('base64'),
        bundle: bundle
      });
    }
    catch (err) {
      cb(err);
    }
  });
};

This method handles making sure that required modules are installed, and then attempts to browserify the code. It then calls back with an object representing the code bundle.

Finally, let's add .attach and .init methods, which will let our new module act as a broadway plugin:

// Bundler can be used as a broadway plugin.
Bundler.attach = function (opts) {
  this.bundler = new Bundler(opts);
}

// The major win we get from making Bundler attachable is that we can integrate
// its initialization step with our app.
Bundler.init = function (done) {
  var npm = 'npm'.red.inverse;

  if (this.bundler.ready) {
    console.log('init: %s is ready.', npm);
    return done();
  } else {
    console.log('init: Waiting for %s.', npm);
    this.bundler.once('bundler::ready', function () {
      console.log('init: %s is ready.', npm);
      return done();
    });
  }
}

Now that the bundler library is done, all we have to do is integrate it with our ./app.js.

First, let's talk about how the bundler was integrated into the app. After requiring it, we use bundler like so:

// This is a new library used for running browserify bundle jobs for us.
var bundler = require('./bundler');


// Note that 'bundler' is written to work as a flatiron plugin.
app.use(flatiron.plugins.http);
app.use(bundler);

This creates a property called app.bundler, and adds an initialization step so that when we call app.start npm is also loaded.

Later, we use app.bundler in our POST handler like so:

this.post(function () {

  // If the request body doesn't have the property we expect, it's assumed
  // to be raw javascript. Note that the raw unparsed body is buffered into
  // req.chunks (as a Buffer).
  var req = this.req,
      res = this.res,
      js = req.body.js ? req.body.js : req.chunks.toString();

  // 'app.bundler' was created when we attached 'bundler'.
  app.bundler.bundle(js, function (err, data) {
    if (err) {
      return res.json(500, {
        success: false,
        reason: err.message || 'unknown'
      });
    }
    res.writeHead(200, { 'content-type': 'text/javascript' });
    res.end(data.bundle);
  });
});

Note that most of the code here is about figuring out what the user wants to browserify and how to respond in return, and that all the bundling takes place in a single call.

Finally, here's the ./package.json for completeness:

{
  "name": "browserify-cdn",
  "version": "0.0.0",
  "scripts": {
    "start": "node ./app.js"
  },
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0",
    "utile": "0.0.10",
    "eventemitter2": "0.4.x",
    "browserify": "1.10.6",
    "detective": "0.1.x",
    "npm": "1.1.x"
  }
}

The only important changes are in the dependencies field, since we now have a few more of them!

Finally, we can run our service with npm start, and when a user runs the curl command from earlier (curl -X POST -d 'var traverse = require("traverse");' localhost:3600) they will receive a bundle and our logs will look like so:

$ npm start
npm info it worked if it ends with ok
npm info using npm@1.1.14
npm info using node@v0.6.14
npm info prestart browserify-cdn@0.0.0
npm info start browserify-cdn@0.0.0

> browserify-cdn@0.0.0 start /home/josh/dev/nodejitsu/flatiron-tutorial
> node ./app.js

init: Waiting for npm.
init: npm is ready.
info: Browserify-CDN Listening on http://0.0.0.0:3600
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse
npm info into /home/josh/dev/nodejitsu/flatiron-tutorial traverse@0.6.1
npm info installOne traverse@0.6.1
npm info unbuild /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info preuninstall traverse@0.6.1
npm info uninstall traverse@0.6.1
npm info postuninstall traverse@0.6.1
npm info preinstall traverse@0.6.1
npm info build /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info linkStuff traverse@0.6.1
npm info install traverse@0.6.1
npm info postinstall traverse@0.6.1
traverse@0.6.1 ./node_modules/traverse

Congratulations, you now have a working browserify webservice!

Add Caching with Resourceful

So, our browserify service works, but there's a problem: Our browserify webservice has to bundle the project every time, even if it's done it before! We can minimize this with caching.

To facilitate this, I reorganized the project a little bit:

$ ls
app.js  config.json  lib  node_modules  package.json  post
$ ls -R ./lib
./lib:
bundler  common.js

./lib/bundler:
bundler.js  cache.js  index.js

As you can see, I moved bundler.js into ./lib/bundler and added a cache.js and an index.js.

You'll also notice that I removed config.json from the git repository. This is because it now includes credentials to a couchdb. Mine looks like this:

{
  "port": "3600",
  "couch": {
    "https": true,
    "uri": "**********.iriscouch.com/browserify",
    "port": "6984",
    "auth": {
      "username": "**********",
      "password": "**********"
    }
  }
}

In order to add caching, the approach I took was to create a new constructor which inherits from Bundler, called CachingBundler, in ./lib/bundler/cache.js. This new constructor contains a new prototype method, .get, which checks our cache and only calls .bundle if required. This sort of inheritance pattern is appropriate for cases where your new object is "an x that also does y"---for example, Bundler is an EE2 that also bundles, and CachingBundler is a Bundler that also caches.

Our new library starts with a long list of requires:

var Bundler = require('./bundler'),
    common = require('../common'),
    resourceful = require('resourceful'),
    errs = require('errs'),
    colors = require('colors'),
    utile = require('utile');

Bundler and common are internal libraries, while resourceful, errs and utile are all flatiron components. This caching layer uses resourceful to interact with a storage backend (in this case, couchdb). Resourceful is an "Object Document Mapper," or ODM, which means that it internally maps javascript objects and prototypes with documents in a document store like couch. This saves developers the step of wrapping their couchdb http calls in a custom abstraction, and is typically used to put the "model" in MVC.

Here's what creating a resource with resourceful looks like:

// Cached bundle resources.
var CachedBundle = resourceful.define('cachedBundle', function () {
  this.string('md5');
  this.string('source');
  this.string('bundle');
});

This resource is quite simple. It's called a 'cachedBundle', and is expected to at least have three strings (md5, source and bundle).

Now, we can create our constructor:

// A version of the bundler that caches. Use `.get(text)`.
var CachingBundler = module.exports = function (opts) {
  // This lets you create a new caching Bundler without the 'new' keyword.
  if (!(this instanceof CachingBundler)) {
    return new CachingBundler;
  }

  opts = opts || {};
  Bundler.call(this, opts);
  // Set up resourceful to use couchdb.
  resourceful.use('couchdb', opts);
}

utile.inherits(CachingBundler, Bundler);

Generally, this constructor just repeats the actions of the original Bundler constructor and then inherits from it. However, it also sets up resourceful to use our couchdb, using the options passed into it.

The meat of our new functionality is in the get prototype method:

CachingBundler.prototype.get = function (src, cb) {
  var self = this,
      md5 = common.md5(src);

  // This logging function simply adds some extra text to the message and
  // then emits it with the event 'log::lvl' (ex: 'log::info').
  function log (lvl, msg) {
    var restOf = [].slice.call(arguments, 2),
        data = utile.format.apply(
          null,
          [ '(md5 `%s`) '.cyan + msg, md5 ].concat(restOf)
        );

    self.emit('log::' + lvl, data);
  }

  log('info', 'Trying to find bundle...');

  // Check to see if the file already exists in the cache, based on md5 hash of
  // the source text. This logic accounts for the possibility of hash collisions
  // and multiple versions of the same bundle.
  CachedBundle.find({ md5: md5 }, function (err, bundles) {
    if (err) {
      return cb(realError(err));
    }

    var matches;

    if (!bundles.length) {
      // No cached bundles found. We should bundle.
      log('warn', 'Not found.');
      return bundle(src);
    }

    // Log the document id's for found bundles. If there is more than one
    // bundle, it means there was either an md5 hash collision or multiple
    // documents for the same bundle exist.
    log('info', 'Found bundle(s): %j', bundles.map(function (doc) {
      return doc._id;
    }));

    // Try to find one that has the same text.
    matches = bundles.filter(function (bundle) {
      return bundle.source == src;
    });

    // Any answer other than 1 means an actual md5 hash collison.
    switch (matches.length) {
      case 1:
        log('info', 'Match found.');
        finish(null, matches[0]);
        break;
      case 0:
        log('info', 'Hash collision with existing document(s) %j', matches);
        bundle(src);
        break;
      default:
        log('Multiple documents describing this document exist.');
        log('Showing the first one.');
        finish(null, matches[0]);
        break;
    }
  });

  // This function is called when we need to bundle. It addition to logging, it
  // also attempts to cache the resulting bundle.
  function bundle (src) {
    log('info', 'Bundling.');
    self.bundle(src, function (err, doc) {
      if (err) {
        return cb(err);
      }

      // Here, we create a new CachedBundle, which maps to our data store.
      var bundle = new CachedBundle(doc);

      // This is how you save resources with resourceful.
      log('info', 'Caching bundle for next time...')
      CachedBundle.save(bundle, function (err, bundle) {
        if (err) {
          log('error', 'Error while caching bundle');
        }
        return finish(realError(err), bundle);
      });

    });
  }

  // This is pretty much the last step, whether the doc came from the cache
  // or from a bundling.
  function finish (err, bundle) {
    if (err) {
      return cb(err);
    }

    log('info', 'Returning bundle.');
    cb(null, bundle);
  }

};

Most of the logic you can see involves grabbing the document, and making sure it's the right one, and logging the behavior using the EventEmitter2 api (remember, Bundler inherited from EventEmitter2). Interaction with our datastore is simplified to the creation of a resource type, a call to .find and a call to .save.

Like Bundler, our caching bundler also has .attach and .init methods for integration with flatiron:

// Attach the caching bundler, passing in our config options for the couch.
// Also emits logging events to the app.
CachingBundler.attach = function (options) {
  var app = this;

  this.bundler = new CachingBundler(utile.mixin(
    options,
    this.config.get('couch')
  ));

  // Little known fact about flatiron: The logging plugin will log messages
  // emitted on the `log` namespace of the core app, which is an instance of
  // EventEmitter2.
  app.bundler.on('log::**', function (msg) {
    app.emit(this.event, msg);
  });

};

CachingBundler.init = Bundler.init;

The init method is exactly the same. The new behaviors are in our attach method. One of these behaviors is to pass our app's couchdb configuration to the object constructor. The other behavior is to re-emit logging events on the app object, whose log plugin will handle them.

I also created a ./lib/bundler/index.js which looks like so:

module.exports = require('./cache');

and allows us to simply require our new caching bundler in ./app.js like so:

var bundler = require('./lib/bundler');

Now, let's run our app and do another curl -X POST -d 'var traverse = require("traverse");' localhost:3600:

$ npm start
npm info it worked if it ends with ok
npm info using npm@1.1.14
npm info using node@v0.6.14
npm info prestart browserify-cdn@0.0.0-3
npm info start browserify-cdn@0.0.0-3

> browserify-cdn@0.0.0-3 start /home/josh/dev/nodejitsu/flatiron-tutorial
> node ./app.js

init: Waiting for npm.
init: npm is ready.
info: Browserify-CDN Listening on http://0.0.0.0:3600
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Trying to find bundle... event=log::info
warn: (md5 `RxLf0LntDlue3quYVoaVtw==`) Not found. event=log::warn
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Bundling. event=log::info
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse
npm info into /home/josh/dev/nodejitsu/flatiron-tutorial traverse@0.6.1
npm info installOne traverse@0.6.1
npm info unbuild /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info preuninstall traverse@0.6.1
npm info uninstall traverse@0.6.1
npm info postuninstall traverse@0.6.1
npm info preinstall traverse@0.6.1
npm info build /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info linkStuff traverse@0.6.1
npm info install traverse@0.6.1
npm info postinstall traverse@0.6.1
traverse@0.6.1 ./node_modules/traverse
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Caching bundle for next time... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info

Naturally, this request wasn't cached so our app had to bundle it. If you check your couch with futon, you'll see that our document has been stored by resourceful. So, if we curl our api again:

info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Caching bundle for next time... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Trying to find bundle... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Found bundle(s): ["f4328d0a958746dfa8b45856a900134e"] event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Match found. event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info

Since we've already bundled that particular source snippet, we may simply return it! Now we're cooking.

What Now?

In this quick tutorial, I showed you how I built the foundation for the new browserify-cdn project, which is hosted at http://browserify.nodejitsu.com, by starting with a "hello world" flatiron example and then iterating on it until we had a fully-working api server. Along the way, we learned a little about flatiron core's capabilities and many of its supporting libraries, as well as touching on concepts for organizing medium-sized apps.

The master branch has a number of tweaks and improvements, the most noticeable of which is serving an html landing page and static content using my static server middleware and adding more colors to the text output. Of course, now that you have your own browserify-cdn, you can change it however you want!

If you do, don't forget to send me a pull request.