[nodebb-plugin-prometheus] Prometheus Monitoring for NodeBB

Dravere

This plugin introduces server monitoring for your NodeBB forum via Prometheus. Prometheus is a leading open-source monitoring solution that I personally use for all my servers. I wanted to have some of the information in NodeBB in my Prometheus monitoring environment. Thus being able to compare user activity with server load, memory usage, NodeJS event loop lag, 503 errors, etc. This plugin makes this possible.

You then can create nice looking graphs to get a nice overview of the data for example with Grafana. Here an example on how this might look. This is my server for the last 48 hours.

As you can see in the lower left corner, I also have some data about captchas. This is done via two hooks provided by this plugin and thus makes it possible for other plugins to provide monitoring metrics.

You can find the plugin in your plugin overview in the ACP. It is obviously also available on npmjs.com:
https://www.npmjs.com/package/nodebb-plugin-prometheus

There you can also find all the metrics that are currently collected. If you have any issues or questions please leave a post in this topic or open an issue on the official repository over at GitLab:
https://gitlab.com/cppnet/nodebb/nodebb-plugin-prometheus

An example dashboard for Grafana can be found here: https://grafana.com/dashboards/6578

exodo

good job

mates.dolda

Thanks a lot! Usefull!

Could you please share your nice grafana dashboard via https://grafana.com/dashboards ?

Dravere

@mates-dolda I'll look into it. Probably will first need to add some variables/templating to make it useful to be shared.

Dravere

@mates-dolda I shared the dashboard: https://grafana.com/dashboards/6578

Per0x

The plugin activation completely broke my forum, I can no longer launch it. Fortunately there are backups and anyway it will allow me to update to 1.10

I also tested to install it on a brand new NodeBB/Redis container on Docker and same result.

Per0x

Here are the logs:

info: Initializing NodeBB v1.10.0 https://forum.xxxxxx.com
info: [socket.io] Restricting access to origin: https://forum.xxxxxx.com:*
info: Routes added
error:  message=Value is not a valid number: 7, stack=TypeError: Value is not a valid number: 7
    at /usr/src/app/node_modules/prom-client/lib/gauge.js:219:10
    at Gauge.set (/usr/src/app/node_modules/prom-client/lib/gauge.js:85:31)
    at /usr/src/app/node_modules/nodebb-plugin-prometheus/src/library.js:33:28
    at /usr/src/app/src/database/redis/hash.js:44:4
    at /usr/src/app/src/database/redis/hash.js:50:4
    at Multi.callback (/usr/src/app/src/database/redis/hash.js:79:4)
    at multi_callback (/usr/src/app/node_modules/redis/lib/multi.js:89:14)
    at Command.callback (/usr/src/app/node_modules/redis/lib/multi.js:116:9)
    at normal_reply (/usr/src/app/node_modules/redis/index.js:726:21)
    at RedisClient.return_reply (/usr/src/app/node_modules/redis/index.js:824:9)
    at JavascriptRedisParser.returnReply (/usr/src/app/node_modules/redis/index.js:192:18)
    at JavascriptRedisParser.execute (/usr/src/app/node_modules/redis-parser/lib/parser.js:574:12)
    at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:274:27)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
error:  message=Not running, stack=Error: Not running
    at Server.close (net.js:1608:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at Server.emit (events.js:208:7)
    at emitCloseNT (net.js:1659:8)
    at _combinedTickCallback (internal/process/next_tick.js:135:11)
    at Immediate._tickCallback (internal/process/next_tick.js:180:9)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
[cluster] Child Process (43) has exited (code: 1, signal: null)
[cluster] Spinning up another process...
info: Initializing NodeBB v1.10.0 https://forum.xxxxxx.com

Dravere

@Per0x You could have just deactivated the plugin and everything would have worked perfectly again. You can do this from the command line via ./nodebb reset -p nodebb-plugin-prometheus. See also the NodeBB documentation.

Error is only related to forums with a Redis DB, since they store all values as text. Thanks for the log it is very useful to get that information. I'll go over the code to fix those issue and probably will release a new version in the next days.

Per0x

@dravere Thanks for the help. I can't wait to see the result with redis

Dravere

@Per0x Just published version 0.2.1 with the Redis fix. I've setup a Redis database for testing to hopefully avert such errors in the future. It is only a test database with limited data, but hopefully nothing as major should now arise in future versions. Otherwise report it immediately

Per0x

The activation of the plugin works well, but in grafana it seems not to want to make the connection.

0_1530553944966_screenshot--2018.07.02-19-51-06.png

And in logs of grafana: http: proxy error: dial tcp "MY_PUBLIC_IP":443: i/o timeout

When I go to the folder /metrics I see the metrics well in text format, like:

# HELP nodebb_maintenance_active If maintenance mode is on 1; otherwise 0
# TYPE nodebb_maintenance_active gauge
nodebb_maintenance_active 0

Is it just my network connection the problem, or something else ?

Anyway, well done for your responsiveness, you rocks !

Dravere

Grafana only displays some metrics from different data sources one of which is Prometheus. You need to setup Prometheus to collect the data first. My Plugin is only providing the metrics so that Prometheus can read and collect them. Have a look at the Prometheus documentation.

Per0x

Ok, I thought that it was built in the plugin. Never mind, I just have to launch a prometheus docker container, It shouldn't be too complicated.

Per0x

It's okay, now everything's working great with redis, thanks!
Just missing the number of online guests (not only users) and that would be perfect.

Dravere

@Per0x Will have a look at it. Will probably make it into the next version (0.3.0). Created an issue for it on the repository: https://gitlab.com/cppnet/nodebb/nodebb-plugin-prometheus/issues/3

Per0x

Great news! I had tried the plugin with google analytics but I was not really satisfied. Your plugin with grafana, there it really looks pro and especially, stats are really detailed.

Dravere

Great to hear. But I would like to just put out this notice:

Google Analytics and my plugin do very different stuff. My plugin should be considered as a part of network, system, or website monitoring. It is monitoring the software and its components. While Google Analytics is a web analytics which is about the users and how they use your website. An alternative for Google Analytics is Matomo.

Website monitoring and web analytics are usually not exclusive but are used both at the same time since they are used to monitor different things. Or actually one is monitoring a thing and the other is monitoring humans.

Imgbi

This post is deleted!

Dravere

I just release version 0.3.0 of this plugin. The following changes have been implemented:

Using action:analytics.increment for analytics counter (better performance and accuracy)
Added label viewer to page view that can have the values user, guest, or bot (inspired by an upcoming change in 1.11, you get it early here)
Added new metric nodebb_online_guests
Added new metric nodebb_eventloop_lag_seconds. This is measured by NodeBB which has a smoothing factored compared to what is measured by NodeJS and provided via nodejs_eventloop_lag_seconds
Added new metric nodebb_eventloop_maxlag_seconds. This is the maximum allowed lag for NodeBB. If the lag is getting closer to this value NodeBB will start to issue 503 error codes. The chance a user is getting a 503 is calculate by this formula: (lag - maxLage) / maxLag.

@Per0x Sorry it took a while but your metric for online guests is now finally available.