Nodejitsu

Save time managing and deploying your node.js app. Code faster with jitsu and npm

npm, binary modules, and module foundry

About the author

Name
Location
Worldwide
nodejitsu nodejitsu

Last week we gave an update about #scalenpm. But scaling npm is about more than just increasing the amount of requests the npm registry can serve: it's about making npm better! And one of the places npm still needs improvement is making native modules easier to work with and build.

As of writing this post, there are 1,597 native modules published to npm.

$ curl http://isaacs.ic.ht/registry/_design/app/_view/needBuild -s | wc -l
    1597

Here's a quick look at the notable top native modules ranked by dependents.

#  npm git  dep pkg       
1  8   921  214 ws  
2  6   -    198 hiredis  
3  8   1455 131 canvas  
4  8   1199 124 pg  
5  13  785  94  bcrypt  
6  1   92   91  phantomjs  
7  2   637  80  websocket  
8  8   512  79  sqlite3  
9  2   872  70  serialport  
10 2   248  67  iconv  
11 5   340  65  libxmljs  
12 2   1109 64  fibers  
13 3   -    52  node-expat  
14 1   132  47  microtime  
15 3   718  46  zmq  
16 5   93   44  leveldown  

Building these native modules or applications using native modules is hard to get right on one platform let alone every platform. We know this from experience: at Nodejitsu we have run tens of thousands of Node.js application on our platform both on Linux and currently SmartOS. We build thousands of modules every day in production for our users (because applications are just big modules afterall!).

Over the last 18 months (thankfully) writing and using node.js modules with native code has gotten a lot easier due to two things: node-gyp and libuv. But it has to get better. Native modules are still far too difficult to use cross platform (especially on Windows) for a few reasons:

  • Native Dependencies: No, not other native Node.js modules, but native code that needs to be installed on the system for a certain module to build. Getting this right is even harder because on platform A you need to do X and on platform B you need to do Y, ad nausem.
  • Permissions: There is more to this process than just sudo, especially when trying to build concurrently.
  • Missing Tools: This is more of a problem for Windows users where the compiler toolchain and relevant SDKs aren't installed by default. It can be a problem on *nix systems as well where compiling native code can vary across platforms.
  • Fine tuning npm: This isn't particularly difficult when you're building against a single version of Node.js: just set unsafe-perm to true. When you need to support multiple versions on the same server, however, these settings become more complex.

Having seen these problems first hand we jumped at the chance to improve npm for the Node.js community. Over the last few months we've been working with Microsoft to create a better native module story that allows pre-built native binaries to be installed on multiple platforms. There are obviously several important steps to that goal: code signing, publishing and installing pre-built modules. These are all on the npm roadmap of course.

Before any of that can happen though: binary modules need to be built.


To solve that problem we are excited announce that we are releasing our production ready, battle-hardened build server, module-foundry, as Open Source under the MIT License. We've worked hard to make sure that it runs smoothly on Linux, SmartOS, and Windows so you can use it on Windows Azure, Joyent, or any other cloud provider or platform you want!

What is module-foundry?

Module foundry is a build server for Node.js modules and applications. That means it will accept a package.json file or tarball and return a fully built tarball. Simple right? Exactly.

Note that module-foundry is not a Continuous Integration server (but it could be). What's the difference you ask? Module foundry does not have any preconditions to run before building your module or application. In a Continuous Integration server you want to ensure that certain preconditions (such as tests) pass before continuing on in a build pipeline.

Installing module-foundry

For each platform of the main Node.js platforms we have created a set of scripts to install the best fit OS packages (e.g. apt, pkgsrc) that will satisfy the native dependencies of the most popular native modules.

Windows

Given the manual nature of many of the installers and MSIs for Windows we recommend that you read through the Windows installation instructions which will walk you through the process step-by-step.

SmartOS and Ubuntu

Given the more automated nature of using apt or pkgsrc these installations are relatively straight-forward:

  $ MY_PLATFORM='smartos' # MY_PLATFORM='ubuntu'
  $ ./module-foundry/scripts/$MY_PLATFORM/dependencies.sh
  $ ./module-foundry/scripts/setup-node-gyp.sh

Using module-foundry

When using module-foundry there are two distinct scenarios:

  • Running a module-foundry server: Starting up a module-foundry endpoint to build npm packages or node.js applications.
  • Requesting a build from module-foundry: Once your module-foundry server is started you can request a new build using foundry-build.

Running a module-foundry server

To start module-foundry simply install it globally with npm and then run module-foundry with the appropriate config file:

  npm install -g module-foundry
  [sudo] module-foundry -c /path/to/config/file.json

Note: sudo is required on *nix platforms to change uid and gid of npm processes.

Requesting a build from module-foundry

Once you have a module-foundry server running you can request a build by using foundry-build:

  npm install -g module-foundry
  foundry-build -p /path/to/package.json --url http://your-module-foundry-server:port

By default foundry-build will not respond with the tarball that was built. You must specify this explicitly by using --file|-f. For example, if we wanted to build the binary module bcrypt we would use this package.json:

example/packages/bcrypt.json

  {
    "engines": { "node": "0.8.x" },
    "repository": {
      "type": "npm",
      "package": "bcrypt",
      "version": "0.7.7"
    }
  }

Then all we need to do pass this to foundry-build:

  $ foundry-build -p bcrypt.json --url http://localhost:1337 --file bcrypt-0.7.7.tgz
  Requesting: http://localhost:1337/build
  Streaming output to bcrypt-0.7.7.tgz

For npm specific builds you can also pass this information directly into foundry-build. The below is equivalent to the above:

  $ foundry-build --url http://localhost:1337 --npm "bcrypt@0.7.7"
  Requesting: http://localhost:1337/build
  Streaming output to bcrypt-0.7.7.tgz

Full help for foundry-build can be found by using --help:

  $ foundry-build --help
  usage: foundry-build -p /path/to/package.json -u http://module-foundry:port

  Options:
    --package, -p  Path to the package.json to build.
    --npm, -n      npm package to build (e.g. "pg@2.7.0").
    --engine, -e   Version of node.js to request build against.  [default: "0.8.x"]
    --input, -i    Expects streaming input from stdin
    --url, -u      URL to the remote module-foundry server       [required]
    --remote, -r   Remote file location to upload to
    --file, -f     Path to local tarball to receive
    --command, -c  npm command to run [build, install]           [default: "build"]
    --help, -h     Display this message

What's does this mean for npm?

This is the first step towards installing pre-built native packages on platforms that do not ship with a full compiler tool-chain like Windows. It is also the first step towards trying to codify the base installation instructions for building and installing binary modules.

Want to help out? That's great. You can help us out today by simply running module-foundry! We like to think we've found a lot of the edge and corner cases for building Node.js modules, but we need your help running this on every app and every platform imaginable!

Got questions? Open an issue on Github or jump into #nodejitsu on Freenode and ask a jitsuka.