[nodebb-plugin-prometheus] Prometheus Monitoring for NodeBB

Per0x

The plugin activation completely broke my forum, I can no longer launch it. Fortunately there are backups and anyway it will allow me to update to 1.10

I also tested to install it on a brand new NodeBB/Redis container on Docker and same result.

Per0x

Here are the logs:

info: Initializing NodeBB v1.10.0 https://forum.xxxxxx.com
info: [socket.io] Restricting access to origin: https://forum.xxxxxx.com:*
info: Routes added
error:  message=Value is not a valid number: 7, stack=TypeError: Value is not a valid number: 7
    at /usr/src/app/node_modules/prom-client/lib/gauge.js:219:10
    at Gauge.set (/usr/src/app/node_modules/prom-client/lib/gauge.js:85:31)
    at /usr/src/app/node_modules/nodebb-plugin-prometheus/src/library.js:33:28
    at /usr/src/app/src/database/redis/hash.js:44:4
    at /usr/src/app/src/database/redis/hash.js:50:4
    at Multi.callback (/usr/src/app/src/database/redis/hash.js:79:4)
    at multi_callback (/usr/src/app/node_modules/redis/lib/multi.js:89:14)
    at Command.callback (/usr/src/app/node_modules/redis/lib/multi.js:116:9)
    at normal_reply (/usr/src/app/node_modules/redis/index.js:726:21)
    at RedisClient.return_reply (/usr/src/app/node_modules/redis/index.js:824:9)
    at JavascriptRedisParser.returnReply (/usr/src/app/node_modules/redis/index.js:192:18)
    at JavascriptRedisParser.execute (/usr/src/app/node_modules/redis-parser/lib/parser.js:574:12)
    at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:274:27)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
error:  message=Not running, stack=Error: Not running
    at Server.close (net.js:1608:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at Server.emit (events.js:208:7)
    at emitCloseNT (net.js:1659:8)
    at _combinedTickCallback (internal/process/next_tick.js:135:11)
    at Immediate._tickCallback (internal/process/next_tick.js:180:9)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
[cluster] Child Process (43) has exited (code: 1, signal: null)
[cluster] Spinning up another process...
info: Initializing NodeBB v1.10.0 https://forum.xxxxxx.com

Dravere

@Per0x You could have just deactivated the plugin and everything would have worked perfectly again. You can do this from the command line via ./nodebb reset -p nodebb-plugin-prometheus. See also the NodeBB documentation.

Error is only related to forums with a Redis DB, since they store all values as text. Thanks for the log it is very useful to get that information. I'll go over the code to fix those issue and probably will release a new version in the next days.

Per0x

@dravere Thanks for the help. I can't wait to see the result with redis

Dravere

@Per0x Just published version 0.2.1 with the Redis fix. I've setup a Redis database for testing to hopefully avert such errors in the future. It is only a test database with limited data, but hopefully nothing as major should now arise in future versions. Otherwise report it immediately

Per0x

The activation of the plugin works well, but in grafana it seems not to want to make the connection.

0_1530553944966_screenshot--2018.07.02-19-51-06.png

And in logs of grafana: http: proxy error: dial tcp "MY_PUBLIC_IP":443: i/o timeout

When I go to the folder /metrics I see the metrics well in text format, like:

# HELP nodebb_maintenance_active If maintenance mode is on 1; otherwise 0
# TYPE nodebb_maintenance_active gauge
nodebb_maintenance_active 0

Is it just my network connection the problem, or something else ?

Anyway, well done for your responsiveness, you rocks !

Dravere

Grafana only displays some metrics from different data sources one of which is Prometheus. You need to setup Prometheus to collect the data first. My Plugin is only providing the metrics so that Prometheus can read and collect them. Have a look at the Prometheus documentation.

Per0x

Ok, I thought that it was built in the plugin. Never mind, I just have to launch a prometheus docker container, It shouldn't be too complicated.

Per0x

It's okay, now everything's working great with redis, thanks!
Just missing the number of online guests (not only users) and that would be perfect.

Dravere

@Per0x Will have a look at it. Will probably make it into the next version (0.3.0). Created an issue for it on the repository: https://gitlab.com/cppnet/nodebb/nodebb-plugin-prometheus/issues/3

Per0x

Great news! I had tried the plugin with google analytics but I was not really satisfied. Your plugin with grafana, there it really looks pro and especially, stats are really detailed.

Dravere

Great to hear. But I would like to just put out this notice:

Google Analytics and my plugin do very different stuff. My plugin should be considered as a part of network, system, or website monitoring. It is monitoring the software and its components. While Google Analytics is a web analytics which is about the users and how they use your website. An alternative for Google Analytics is Matomo.

Website monitoring and web analytics are usually not exclusive but are used both at the same time since they are used to monitor different things. Or actually one is monitoring a thing and the other is monitoring humans.

Imgbi

This post is deleted!

Dravere

I just release version 0.3.0 of this plugin. The following changes have been implemented:

Using action:analytics.increment for analytics counter (better performance and accuracy)
Added label viewer to page view that can have the values user, guest, or bot (inspired by an upcoming change in 1.11, you get it early here)
Added new metric nodebb_online_guests
Added new metric nodebb_eventloop_lag_seconds. This is measured by NodeBB which has a smoothing factored compared to what is measured by NodeJS and provided via nodejs_eventloop_lag_seconds
Added new metric nodebb_eventloop_maxlag_seconds. This is the maximum allowed lag for NodeBB. If the lag is getting closer to this value NodeBB will start to issue 503 error codes. The chance a user is getting a 503 is calculate by this formula: (lag - maxLage) / maxLag.

@Per0x Sorry it took a while but your metric for online guests is now finally available.