First- thank you! When npm started getting crashy and Nodejitsu needed your help to marshall the resources to improve it, the community stepped up and delivered. The outpouring of support has simply blown away our expectations. We never expected for #scalenpm to reach its goal in just over a week!
TL;DR? Thank you, thank you, thank you. Even npm is excited:
Since #scalenpm raised enough money to give the public npm registry the infrastructure and resources it needs to keep growing for the next 12 months we've decided to shut down donations a week early. This will give Isaac the time he needs to figure out a long-term plan for keeping it awesome. We love running the public npm registry and we like to think we've done a pretty good job up until now!
What this means is that donations will close on Wednesday December 18th, 2013. So besides that, what’s next? We're glad you asked!
Here’s a timeline of what’s coming up for the next few weeks. We still have a bunch of loose ends to tie: T-shirts to send, campaign summary posts to create, haikus to write. You know, the usual post-fundraiser stuff. If you made a $75 donation or greater, expect an email at the end of December asking you about your address.
- December 18th: Donations officially closed.
- December 26th: First blog post about how Nodejitsu is improving npm.
- December 28th: Gathering addresses for physical thank yous starts.
- January 2nd: npm.nodejitsu.com goes live with sponsor logo and information.
We know, we know: get to the point and tell us how npm is going to be better! Don’t worry: we’re excited about improving npm too! Want to keep an eye on the technical or financial details of how this is being done? Keep and eye on our blog for monthly reports on how the money is being spent and all of the nitty gritty technical details of what we’re doing to improve the public npm registry.
Thank you sponsors!
#scalenpm couldn’t have happened without all of the fantastic corporate sponsors who contributed over 70% of the donations thus far. We’ll be talking more about sponsors on our blog in the coming weeks, but for now why don’t you check some of them out? If you’re a node developer these are the places where you want to work- they get it.
What’s first on the agenda?
So npm is going to be awesome now. For serious. Jason, Charlie, and the rest of our operations team don’t like getting woken up at 2am to handle npm availability issues:
What features, you ask? We already went over this ... but what we’re tackling first all has to do with our high-availability via horizontally scaled multi-master clusters by default.
New consistent SSD-backed hardware: We had to rush into multi-master clustering because of the repeated downtimes in November. This meant that we were not able to make each of the master servers’ physical hardware consistent: one is a spinning disk and one is an SSD. We are provisioning three new SSD-backed servers over the next few days and we will be transitioning all npm traffic there before Monday. This will also allow us to regularly run CouchDB compaction in rotation across all three servers since one is a hot spare.
Better conflict resolution: We know that users have had issues with npm publish over the last few weeks since we switched to multi-master.
All of these problems stem from the same symptom: conflicts in CouchDB. If you’re new to CouchDB you can read up on conflicts here. Conflicts are caused in the npm registry because (depending on several factors) a given publish can involve multiple writes to the same document. When these writes do not hit the same CouchDB server conflicts are generated. There are other medium-term solutions to scaling writes (such as sticky HTTP sessions), but conflicts will inevitably arise so we must address the symptom as well as the cause.
- Monitor & audit replication between masters: We know that replicating the public npm registry needs to be easier first hand because replication between our two masters needs regular maintenance. If you want to learn more a good place to start would be to know the difference between the _replicator database and the _replicate API API in CouchDB. Long story short we’re working hard on some tools to make managing replication status easier which will help avoid conflicts in the first place.
Oh and good news! All of our conflict resolution and replication tools will be released as Open Source once we’ve had a few weeks to battle-harden them in production.
Thanks again, and expect more updates soon! Oh and if you were curious about the surprise module from substack for #scalenpm you can listen to the audio it generates here.