Could you post the full output from your terminal? That doesn't read as it should above. Everything from doing npm install downwards.

I'm pretty sure you can do require('npm') and then execute things like installs and uninstalls, which may be way nicer than shelling out to the npm process.

@psychobunny apparently I should talk to you about: http://npm.aws.af.cm/discover in regards to this.

@julian I'm going to continue making a read-only replica which presents data in an interesting way, however, I don't think I'll do the system to extract plugin.json from the tarballs.

peerDependencies is apparently definitely what we should be using: https://twitter.com/indexzero/status/453379364517064704

Most likely I'll end up writing things out to mongodb, purely because storing the complete package.json's in it is stupidly easy.

Okay, so, this is continuing off the back of a bunch of other posts, and all that stuff I've been muttering on about in the irc channel.

Currently the plugin architecture in NodeBB is built on top of npm, this has massive benefits for us, but isn't without its drawbacks. A major benefit is that we don't need to setup and maintain some sort of registry and mechanism for publishing plugins (npm is that). However, a drawback is that the discoverability of plugins is fairly low, we don't have any awesome plugins.nodebb.org or anything like that.

I've been doing a little bit of research as to how far you can push (read: abuse) npm for an application-specific registry, without setting up an entirely separate registry. So, far, I've found out the following things:

1. It's super easy to monitor the official registry to discover newly published packages (see: https://gist.github.com/miksago/35eeafc0f031d2248269)
2. Using this monitoring, we can filter the changes in the registry by doing matching against the id value, such that we can find just changes to nodebb- prefixed packages.

Once we know when nodebb related packages are changed in npm, we can then do cool things like analyse their plugin metadata and find out all sorts of weird and wonderful things.

There is a caveat with how we'd currently have to implement this for nodebb plugins today: in order to get the plugin metadata, we need to read the plugin.json file. In order to read this file, we'd need to download the packages' tarball, extract it, and grab the contents of the plugin.json file. This means our servers would need to do a tonne more processing work, and we'd need to figure out some method to make this actually maintainable. (i.e., what happens if we fail to fetch the tarball, or we don't find a plugin.json file, or our servers run out of disk space?)

What we could do, as a massively breaking change, is to inline the plugin.json into the package.json file. I've done an example of what this might look like here: https://registry.npmjs.org/nodebb-plugin-test/0.0.2 — I'm using the nodebb key as the location in package.json for nodebb specific information. As for which versions of nodebb a plugin supports, I'm currently using this "versions" field.

I think it may be fair to say that we could put that information into the "engines" hash that package.json supports as a "nodebb": ">0.5.0" style value. You can do that for npm and not just node, which is typically what is in the engines hash. Is nodebb an engine?

Anyway, all of this considered, we'd be able to make a very lightweight front-end / index of nodebb plugins that's super up-to-date with npm by use of replication / the changes feed. Yay couchdb.

Something that interested me when I first started hacking on NodeBB was how the platform uses realtime communication through socket.io (and websockets). This interest stems from work I did whilst working as a realtime engineer at Pusher several years ago. One of my research projects whilst there was to look into how websockets get used in production, and other use cases for websockets besides the standard "subscribe to a channel and receive events".

In 2011, I gave a talk at Keeping It Realtime conference in portland about some of my findings, you can watch it here: http://2011.krtconf.com/videos/micheil_smith (long hair and all) or check out the slides here: https://speakerdeck.com/miksago/krtconf-websockets-sub-protocols

How NodeBB currently seems to use realtime communication:

1. RPC: Request data from the server, get a response back, present that data to the user (or send data to a server like you would POST to a HTTP server)
2. Channels: Used for user to user chat messages
3. Events: Used for online/offline states of users.

I may have missed something here, so, let me know if I have.

Most of the websocket usage appears to be in RPC style messages: Get this piece of data do something with it. At present, all realtime communication for NodeBB is built on top of Socket.io, which uses a protocol more tailored for event type packets, rather than for RPC style packets. For this, something like JSON RPC may be more appropriate.

Refactoring this deep internal of NodeBB would make the platform more resource efficient on the wire, and also allow for better logging and handling of data. You'd know exactly what has failed on the client-side or server-side and be able to give an appropriate response. In the code, refactoring this level of things would make the code far cleaner, and give option to provide all HTTP Verbs over WebSocket or vica-verse (meaning if you wanted to, you could make 90% of NodeBB standard HTTP. You'd end up with an "api" module internally that is everything that can be exposed, which simple returns back standard javascript objects.

For user-to-user chat messaging you could still multiplex that onto the same actual websocket / socket.io connection (although it's arguable if this is actually beneficial).

Why do any of this writing or work? Interest more than anything else, but it may make the code to NodeBB cleaner, and improve maintainability.

@julian yeah, if I fork it, I'd have to republish as a different package on NPM, which isn't something that's ideal.

Small note on this: whilst I'm the original author, I've since left Kano, and as they have IP over it, and control access to the source code commits, I can no longer actually maintain it. It's up to them if they want to make me a maintainer of that repository.

@julian aye, for that, I could write a custom Winston transport to use, which just pumps JSON directly out to Bunyan. That way all logs would still be in JSON format, no matter where they originally came from.

Firstly, however, I'd be revamping the core logging.

@julian hahaha, alrighty then

@julian most likely a lot of it would get removed. This new logging would replace all existing stuff (including winston)

@aaron there's now a S3 plugin for NodeBB uploads, I developed it whilst working at Kano.

NodeBB doesn't use an MVC framework as such. For database access, majority of calls are just direct to Redis, or abstracted through lightweight objects. For websockets, it's using a custom "event" based RPC over Socket.io, and for serving the http parts, it's using Express. It's a weird conglomerate of systems, but they seem to work alright together.

There's also a relevant Github issue open for disabling local logins: https://github.com/designcreateplay/NodeBB/issues/1263

As for fronting your entire forum with a CDN, I'm not sure, but as for images and other assets, you could write a plugin to generate the correct image urls based on CDN configuration (such that the image urls fetched are CDN-enabled, but the urls stored are standard)

@Ezra-Sharp unless you've already done so, it may be worth while opening an issue on github for this: https://github.com/designcreateplay/nodebb/issues

@planner yup, node-bunyan exactly. Most likely I'll need to come up with a custom format for WebSocket logs, perhaps it'd make sense to split it further though:

• http request / response
• websocket open / close
• api calls
• platform events

As probably 40% of the information in a websocket data event is pointless to the end-user, and only marginally helpful to the developers of nodebb.

There was a motion by me to refactor all of the logging. If I were to do that, it'd be done with Bunyan most likely, such that all logs are In one format, even if they include more data than other sources (http and websocket combined).

This'd be a fair chunk of work. I might start experimenting on this soon.

@sunqi in that case, I'd be spawning a new box and moving, and then deleting the old one.

@aaron you'd probably use a terminator for doing HTTPS, purely because then you wouldn't have to modify NodeBB directly. You might find this guide from Digital Ocean to be useful: https://www.digitalocean.com/community/articles/how-to-create-a-ssl-certificate-on-nginx-for-ubuntu-12-04/

That said, I've never setup an SSL certificate before (have to learn soon though)