Thoughts on a plugin registry

miksago

Okay, so, this is continuing off the back of a bunch of other posts, and all that stuff I've been muttering on about in the irc channel.

Currently the plugin architecture in NodeBB is built on top of npm, this has massive benefits for us, but isn't without its drawbacks. A major benefit is that we don't need to setup and maintain some sort of registry and mechanism for publishing plugins (npm is that). However, a drawback is that the discoverability of plugins is fairly low, we don't have any awesome plugins.nodebb.org or anything like that.

I've been doing a little bit of research as to how far you can push (read: abuse) npm for an application-specific registry, without setting up an entirely separate registry. So, far, I've found out the following things:

It's super easy to monitor the official registry to discover newly published packages (see: https://gist.github.com/miksago/35eeafc0f031d2248269)
Using this monitoring, we can filter the changes in the registry by doing matching against the id value, such that we can find just changes to nodebb- prefixed packages.

Once we know when nodebb related packages are changed in npm, we can then do cool things like analyse their plugin metadata and find out all sorts of weird and wonderful things.

There is a caveat with how we'd currently have to implement this for nodebb plugins today: in order to get the plugin metadata, we need to read the plugin.json file. In order to read this file, we'd need to download the packages' tarball, extract it, and grab the contents of the plugin.json file. This means our servers would need to do a tonne more processing work, and we'd need to figure out some method to make this actually maintainable. (i.e., what happens if we fail to fetch the tarball, or we don't find a plugin.json file, or our servers run out of disk space?)

What we could do, as a massively breaking change, is to inline the plugin.json into the package.json file. I've done an example of what this might look like here: https://registry.npmjs.org/nodebb-plugin-test/0.0.2 — I'm using the nodebb key as the location in package.json for nodebb specific information. As for which versions of nodebb a plugin supports, I'm currently using this "versions" field.

I think it may be fair to say that we could put that information into the "engines" hash that package.json supports as a "nodebb": ">0.5.0" style value. You can do that for npm and not just node, which is typically what is in the engines hash. Is nodebb an engine?

Anyway, all of this considered, we'd be able to make a very lightweight front-end / index of nodebb plugins that's super up-to-date with npm by use of replication / the changes feed. Yay couchdb.

julian

Thanks for the insights @miksago

My thoughts

I've considered using the engines field, although I don't know if npm checks it when attempting to install plugins. If it does, that would be awesome, but my initial (albeit, light) testing provided no results
- If and when we publish NodeBB itself to npm, we can use peerDependencies instead
We are working on a "registry.nodebb.org" system so that NodeBB owners can vouch for the compatibility of plugins. We can work this system in as well.
- Right now, I believe it scraped npm nightly and stores it locally in Redis
I love the idea of moving plugin and theme metadata into package.json -- we're actually realising that packages and themes are quite similar (plugins are like themes, and vice versa), so having it all in package.json may not be a bad idea at all
- Of course, we'd have to support both systems for now, scheduling deprecation for v0.5.0 or v0.6.0

miksago

@julian I'm going to continue making a read-only replica which presents data in an interesting way, however, I don't think I'll do the system to extract plugin.json from the tarballs.

peerDependencies is apparently definitely what we should be using: https://twitter.com/indexzero/status/453379364517064704

Most likely I'll end up writing things out to mongodb, purely because storing the complete package.json's in it is stupidly easy.

miksago

@psychobunny apparently I should talk to you about: http://npm.aws.af.cm/discover in regards to this.

julian

@miksago Replied to indexzero. I do recall DocPad uses peerDeps in a similar system, so perhaps we can emulate that.