Piwik and NodeBB

Introduction

So you have your shiny NodeBB forum up and running. Hopefully it is spilling over with content and you have many glorious visitors. But do you? How do you know? Analytics of course! Huh, analytics you say?

• Analytics
  the method of logical analysis ^1

So what do you do? You load a big giant googly companies analytics plugin. And then you realize this:

All your analytics are belong to us.

And then you are scared. What can you do about that? The answer is simple, you run your own analytics program! Piwik to the rescue. Go ahead, go there, get it all setup ^2. I'll wait.

Great! Welcome back. I trust you have your Piwik installation running and ready to analyze data from your NodeBB. The hard part is over now. All that is left is making a few customization tweaks to NodeBB.

NodeBB

Tracking Code

The tracking code. These are the bits that make the magic work. We need to tell Piwik about our visitor's actions. What are they looking at, what are they posting and even what are they searching for. NodeBB's custom header function is a wonderful fit for all of this. All that needs to be done is pasting in the following code with only a couple of slight adjustments.

Add the following tracking code to your NodeBB forum's custom header.

ACP > Appearance > Custom HTML & CSS > Custom Header

Change your.piwikurlhere.com to your actual Piwik URL. It could be a subdomain on your NodeBB domain or it could be an entirely different domain but it is the one you used when you installed and configured Piwik. If this is your first and only site that Piwik is tracking then the default site ID of 1 will be sufficient. If it isn't than update the site ID (setSiteID and idsite) to the appropriate value. You will know this from your Piwik installation and configuration.

<!-- Piwik -->
<script type="text/javascript">
var _paq = _paq || [];
(function () {
    var u = "//your.piwikurlhere.com/";

    function firePiwik (data) {
        if (app.user && app.user.uid > 0) {
            _paq.push(['setUserId', app.user.uid.toString()]);
            _paq.push(['setCustomVariable', 1, "appUserUsername", app.user.username, "visit"]);
        }
        _paq.push(['setDocumentTitle', document.title]);
        _paq.push(['setCustomUrl', location.href]);
        _paq.push(['enableHeartBeatTimer']);
        _paq.push(['appendToTrackingUrl', 'bots=1']);
		if (data.tpl === 'search') {
			_paq.push(['trackSiteSearch', ajaxify.data.search_query,, ajaxify.data.matchCount]);
		} else {
			_paq.push(['trackPageView']);
		}
        _paq.push(['enableLinkTracking']);
        _paq.push(['setTrackerUrl', u+'piwik.php']);
        _paq.push(['setSiteId', 1]);
    }

    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);

    $(window).on('action:ajaxify.contentLoaded', function(ev, data) {
        firePiwik(data);
    });
})();
</script>
<noscript><p><img src="//your.piwikurlhere.com/piwik.php?idsite=1&rec=1&bots=1" style="border:0;" alt="" /></p></noscript>
<!-- End Piwik Code -->

Set Enable Custom Header to on, and press that floppy disk^3 icon^4 to save all of your hard work.

Reload or restart your NodeBB forum to be sure that the system is now injecting the tracking code with the pages that NodeBB generates. Visit your NodeBB forum and check your Piwik dashboard for activity.

Conclusion

And there you have it. Analytics! Implementing Piwik like this also allows that other googly analytics plugin to work simultaneously if you so desire to have the benefits of that all seeing eye. It is also possible to add additional tracking services^5 along with Piwik in the custom header.

Known Working Compatibility

N'BB		Piwik
v1.1.0		2.16.1
v1.0.2		2.16.1, 2.16.0
v1.0.0		2.16.0, 2.15.0
v0.9.4		2.16.0, 2.15.0
v0.9.3		2.15.0

---

Contact me https://community.nodebb.org/user/rod

Revision 1.4, 2016-JUL-13 @667.beats

Let's Encrypt and NodeBB

Summary

NodeBB is a special beast. One that takes careful grooming and maintenance. Ah, gone are the golden days of LAMP. That glorious "P", so easy to use but oh so poor on performance. With Node.js and socket.io living behind your webserver you no longer have the easy liberty of throwing down files and opening directory access any where you would like. What's that? You have to shutdown your NodeBB just to renew your Let's Encrypt SSL certificates? You are using SSL right? Shame, shame if you are not.

Take notice! You no longer need to shutdown your webserver and NodeBB just to renew those Let's Encrypt certificates if you make a simple change to your webserver configuration.

Back story

Let's Encrypt is a free SSL certificate authority. You run their software on your server that authenticates with their master control. The key is proving that you have rights to the domain you are requesting a certificate for. Their software is a webserver so you can't already be running one on port 80. Theirs needs to bind to that port. To run their software, to renew my certificate, I have had to shutdown my webserver (which makes my NodeBB forum unavailable). It doesn't take long but I was annoyed that my NodeBB forum would have to be offline during that time. I knew there must be a solution to this problem and there is in their "webroot" option.

The webroot flag keeps their software from needing to run as a webserver and just places special authentication files in your already existing webserver's html path. That is all fine-and-dandy for a typical LAMP setup but doesn't behave as well with Node.js, socket.io and NodeBB. It can behave though if you tell your webserver to exclude the super secret Let's Encrypt webroot path from going to NodeBB and point off somewhere else.

Webserver

We must tell the webserver that the .well-known/acme-challenge directory is not to be proxied off to socket.io like it needs to be for NodeBB.

Nginx

These configurations need to be in the unencrypted (port 80) section of your webserver configuration.

Change yourdomainhere.com with your actual domain name.

server {
    listen       80;
    server_name  yourdomainhere.com;

    root   /var/www/yourdomainhere.com/html;
...

Add a location block for the Let's Encrypt directory

location /.well-known/acme-challenge/  {
    try_files $uri =404;
}

Bonus: If you want to only serve https traffic for your NodeBB forum make your root location block something like this

location / {
    rewrite ^(.*) https://yourdomainhere.com$1 permanent;
}

Reload Nginx for the changes to take affect.

sudo systemctl reload nginx

Apache (and others)

You will need to research how to setup the proper exclusion of the .well-known/acme-challenge directory from being proxied to socket.io.

Let's Encrypt

Linux

When it is time to renew your certificate (or create your certificate) you would run the following command.

Change yourdomainhere.com with your actual domain name.

sudo ./letsencrypt-auto certonly --webroot -w /var/www/yourdomainhere.com/html -d yourdomainhere.com -d www.yourdomainhere.com

• certonly means to only have the certificates generated and do not try to modify any webserver (Nginx) configuration files. That task is left to the student (you).
• --webroot tells Let's Encrypt to not run as a webserver and not to bind to port 80
• -w points to the actual path on disk that Nginx can serve files from. This is the place that the .well-known/acme-challenge directory will be created in and inside of that directory Let's Encrypt will place temporary authorization files
• -d the domain name that this certificate will authenticate for. Add additional -d flags for each of the names you may have.

Windows (and others)

You will need to research the syntax to execute the letsencrypt-auto command but the flags should be very similar to the Linux ones shown above.

Caveats

Symbolic Links

It has been reported that symbolic links do not work well with the Let's Encrypt software. Therefore please avoid attempting to sym link the .well-known/acme-challenge directory to another location on your file system.

Conclusion

What I have left out is the initial configuration of your webserver to accept SSL connections. There are numerous documents online to help with that portion. If it is done correctly for Let's Encrypt the certificate path will point to a live directory. Renewing the certificates only updates the files in the live directory and your webserver configuration is not touched.

With this configuration I am able to run the command to update my certificates without having to shutdown my webserver thus making NodeBB unreachable from my webserver proxy.

Contact me https://community.nodebb.org/user/rod#

Create & Manage a NodeBB Test/Dev Environment

Table of Contents

• Summary
• Caveats
• Webserver
  • Nginx
• NodeBB
  • PRD -> DEV
  • Adjustments to DEV
  • Upgrading DEV NodeBB
  • Delete DEV
  • Fresh NodeBB with PRD database

Summary

The old saying goes

Laws are like sausages, it is better not to see them being made.

Managing NodeBB isn't as bad as that but there could be a bit of a challenge the first few times that you attempt to upgrade your software or install a new feature. Especially on your production system! So what can you do?

Make a sausage factory of course! In our case the factory is a test/development environment to practice your upgrades or test a new plugin or CSS change. You do not necessarily need to spend additional money on a test/dev server, although you could. You can spin up a copy of your production system right on your production system. This is not very difficult due in part to how well NodeBB, and Node.js, were created.

Caveats

For disclosure I am not a developer of NodeBB or any of the supporting applications. These notes are just that, a collection of my notes as I was experimenting and learning. I do have twenty-four years of IT experience, mostly Unix (SunOS, Solaris, AIX, BSD, Linux) which includes twenty-four years of bad habits and ugly kludges. I do hope that in those years I have picked up a few gems and learned a few good traits. I will try to pass along only the best but please do accept the ugly as they come.

My NodeBB forum runs on a Linux (CentOS) server. I utilize Redis as my NodeBB database and use Nginx as my web server. This document, unless noted otherwise, will reference that environment.

I am making the assumption that you have a production NodeBB server that is fully operational and you wish to create a dev/test copy of it on the same server.

And finally, The risk is yours. I do not take responsibility for harm to your system. Please evaluate all of my suggestions as you deem necessary.

Webserver

Nginx

This addition should only need to be made once. After it is in place you should be able to create and delete dev NodeBB environments from your production without needing to adjust your Nginx configuration.

You will need to make a new server block in your Nginx domain configuration file. Most likely located in /etc/nginx/sites-available/yourforumdomain.com.conf. This new server block will have the specifics for your dev NodeBB. For example you may create an entry like:

server {
    listen       443 ssl;
    server_name  dev.yourforumdomain.com;
[...]

In your location / { location block you will need a proxy_pass entry that directs to the port that your dev NodeBB will listen on:

location / {
[...]
    proxy_pass http://127.0.0.1:4568;
[...]

Once you have your Nginx configuration adjusted appropriately do not forget to restart it. sudo systemctl restart nginx.

NodeBB

PRD -> DEV

For my examples the production NodeBB is running under the /opt/nodebb/ directory path and we will be creating a copy into /opt/devNodebb/

Let's make a copy of production into dev. This can be done "live" without needing to shutdown your production forum.

mkdir /opt/devNodebb
(cd /opt/nodebb; tar cf - .) | (cd /opt/devNodebb; tar xvf -)

If NodeBB was running when you made the copy you'll need to remove the pidfile. A pidfile typically contains the process id information of the running program. This is useful so that commands such as ./nodebb stop know which program to kill. The pidfile you copied over relates to the process id of your production NodeBB. You don't want to be killing that process when you try to startup your dev NodeBB.

rm /opt/devNodebb/pidfile

Since you made a copy of production you have all production information. If you attempted to start your dev NodeBB now you would have problems. You must edit your config.json file and tell it that you want a new instance of NodeBB.

vi /opt/devNodebb/config.json

Change your url, port and database entries. For the URL make it the dev domain that you configured your Nginx to listen for. For the port and database numbers I just incremented by one from production.

"url": "https://dev.yourforumdomain.com",
"port": "4568",
[...]
    "database": "1"

No you aren't ready to start up your dev NodeBB just yet. There isn't a database behind it.

Let's make a copy of the production Redis database for dev to use. I use redis-copy which is not part of the redis database distribution. I downloaded and built my copy from this Github repository redis-copy.

The following command makes a copy of the Redis database number 0 (production) to database number 1 (your dev/test). Database number 1 is the database we told the devNodebb config.json file above to use.

redis-copy localhost/0 localhost/1

Now we may startup the dev NodeBB instance.

cd /opt/devNodebb
./nodebb start

Adjustments to DEV

Once your dev forum is running you should make a few administrative adjustments to it. These are not necessarily critical but they will help you to keep track of which system you are working on and may reduce notifications to your users, etc. Keep in mind that you cloned your production system which includes the users. If you do not want them logging into your dev system you should limit their exposure to it.

You may want to adjust the site and browser title's to reflect a dev forum

ACP > Settings > General > Site Title > DEV YourForumName
ACP > Settings > General > Browser Title > DEV YourForumName

Disable email subscriptions

ACP > Settings > Email > Email Subscriptions > Disable subscriber notification emails

Disable plugins that may cause confusion, such as Google Analytics

ACP > Extend > Plugins > nodebb-plugin-google-analytics > Deactivate

There may be other plugins that you wish to disable. Experimenting will lead you to adjust this list appropriately.

There you have it. You now have a test/dev environment based off of your production system. Changes you make now will only be done in test/dev and your users will happily continue to use the production system until the time comes that you shutdown production to re-play changes or enhancements that you tested in dev.

Upgrading DEV NodeBB

So you went through all of the above steps and now you have, well, just a copy of production. Not really useful as it stands but now you have a platform to test upgrading without fear of damaging production.

Let's upgrade! If you'd like to bring your dev NodeBB up to the current general release of NodeBB the following steps should be adequate.

Stop your dev NodeBB

cd /opt/devNodebb
./nodebb stop

Tell Git to get the newest release information. This sets up NodeBB such that when you issue the upgrade command it will pull down all of the new or changed files as necessary.

cd /opt/devNodebb
git fetch
git checkout v1.x.x
git merge origin/v1.x.x

Issue the NodeBB upgrade command

cd /opt/devNodebb
./nodebb upgrade

Watch for upgrade errors. Note that any customizations or changes you have made to your theme's CSS may have been undone by a new feature or bug fix. That is why we have this test/dev environment, so you can test and look for problems.

If the upgrade appears to have worked without errors start up dev NodeBB and then switch into the logging mode so that you may watch the behavior of the system. Pressing ctrl-c will exit the logging mode.

cd /opt/devNodebb
./nodebb start
./nodebb log

Delete DEV

What if there were problems with your upgrade or you'd like to start this process over with a fresh copy of production? That is not a big problem. A few commands will delete your test/dev environment and then you can start the copy process over again.

Stop your dev NodeBB

cd /opt/devNodebb
./nodebb stop

Let's remove the test/dev software directory tree

rm -rf /opt/devNodebb

Delete the test/dev Redis database. This is probably the most dangerous command in this document. Please double check that the database number provided is the one for your test/dev environment and NOT the one for your production environment.

redis-cli -n 1 flushdb

And there you have it, test/dev is gone. You may now redo the PRD -> DEV or Fresh NodeBB with PRD database steps if you so desire.

Fresh NodeBB with PRD database

There may be times that you want to have a factory fresh NodeBB installation with your database content. You would end up with users, setttings, content, etc., but not files that you may have edited over time or plugins that you installed.

Run the Delete DEV section to be sure you have no remnants of a previous test/dev environment. Then proceed.

Tell Git to get the software and where to put it.

git clone -b v1.x.x https://github.com/NodeBB/NodeBB /opt/devNodebb

Copy your production config.json file and then edit it as you would have if you were doing the PRD -> DEV section

cp /opt/nodebb/config.json /opt/devNodebb/config.json

vi /opt/devNodebb/config.json

Change your url, port and database entries.

"url": "https://dev.yourforumdomain.com",
"port": "4568",
[...]
    "database": "1"

Copy over your Redis database

redis-copy localhost/0 localhost/1

Since the config.json has been created and contains the information that the NodeBB setup process would have asked for we can just run the NodeBB upgrade command (as opposed to the setup command). Bypassing the questions about which database to use, etc. Before you run the NodeBB upgrade you have to actually install the NodeBB software. This process was not necessary when we copied PRD to DEV earlier as it was done when you did your initial production NodeBB installation and was brought over in the tar copy process.

cd /opt/devNodebb
npm install
./nodebb upgrade

Now start it up!

cd /opt/devNodebb
./nodebb start

And there you have it. You now have a clean installation of NodeBB with your database content. As noted in the PRD -> DEV section you may wish to make adjustments to your dev forum so that you can easily identify which one you are working in. Ie., change the site and browser title names, etc.

---

Contact me https://community.nodebb.org/user/rod

Revision 1.2, 2016-APR-08 @645.beats

And finally upgraded to v1.4.2 for my production site.

@The-Penultimate-Defenestrator This exists through the nodebb-plugin-pushbullet plugin.

I am using it on this site (community.nodebb.org) as well as the forum I administer.

You get to it from your profile settings.

@AOKP I just looked at your high performance tutorial. Definitely a lot of information. Possibly too much all at once. I think people may find this succinct post just related to Let's Encrypt easier to digest.

@Jam said in Invalid CSRF Token:

SSL: no

Although you have SSL set to no, have you tried to resolve this problem by adding proxy_set_header X-Forwarded-Proto $scheme; to your Nginx configuration?

Sounds wonderful, off to crash my dev server. whistles

Is the nodebb-plugin-beep plugin installed?

NodeBB Forum Monitoring

Table of Contents

• Executive Summary
• Third Party Providers
  • Uptime Robot
  • Pushbullet
• Monitoring Your Forum
  • HTTP Check
  • Other Checks
• Setting Expectations
• Quagmire
• Known Working Compatibility
• Contact & Version Information

Executive Summary

While using a third party monitoring service your forum may be able to obtain the level of service you wish to commit your effort to providing. I identify one provider who's free service tier seems quite adequate for my level of need.

Third Party Providers

My cursory investigation into providers of monitoring systems turned up many to which mostly were paid services. I did find a few that offered a free-tier but they were limited to a bare minimum amount, and type, of service checks. I am not saying that paid services are bad and on the contrary paid services are wonderful. If your forum is of the caliber to require a paid monitoring service then please get one. If your forum is just not there yet to warrant paying then my work investigating and testing different solutions and services at the free level may benefit you. If so please enjoy my research.

Uptime Robot

I then found Uptime Robot. https://uptimerobot.com Like many of the other providers they offer a paid and a free tier. What separates them from many of the others is that their free service, although limited, does not seem limited to a detriment. Specifically their free tier sets the service interval check to five minutes, limits to fifty monitors, offers a limited method of notification and only provides 2 months of history.

Considering the price, free, I am happy to be constrained to those limitations. They are very generous actually. The forum I administer does not need service checks at a more granular level than five minutes and although SMS notification is not offered in the free tier they do offer email-to-sms (via your mobile carrier's email gateway -- if offered) but more to my liking they offer Pushbullet notification.

Pushbullet

If you aren't aware of Pushbullet take a look at their service. https://www.pushbullet.com Pushbullet provides iOS and Android apps and a plethora of web browser plugins for notification.

Pushbullet via their apps offer push notifications which to me are almost immediate from when the notification has been sent from the source system. In my testing a Pushbullet notification is as quick as a SMS message.

Lastly, a NodeBB plugin exists for Pushbullet. You could even add it to your forum for your users to receive your forum notifications.

Monitoring Your Forum

Go off and create your Uptime Robot account now. I'll wait. Oh, hi there. Welcome back. Be sure to create additional "Alert Contacts" if you want something other than email such as email-to-sms or Pushbullet or one of the other choices they offer.

Let's get to creating a service check for your forum.

HTTP Check

I consider this the most important check. It's nice to know if you do not have connectivity to your server via ping or know the latency of that connection but at the end of the day you are here to setup forum monitoring. The http check is our bread and butter. Just because the server responds to pings does not mean your forum is up and usable by your customers. The http check examines the response codes after visiting your site. A "200" response means that everything is working as planned. Deviations from that code, possibly a "502" or "503", would set off a notification to you.

• Click the "Add New Monitor" button
• Choose HTTP(S)
• Add a name for the check
• Enter the URL of your forum
• Choose to receive email and, or, another notification method by checking the appropriate check box
• Click to "Create Monitor"

Other Checks

Feel free to add additional checks as you deem necessary. You may wish to add a "ping" check to report if your entire server and not just your forum is unreachable. You may wish to have your database checked via the port check, etc.

Setting Expectations

Receiving notification regarding the health of your forum does not mean your forum is self-healing. If the forum goes down you or your team still have to diagnose and repair the problem. If you run a forum for fun and don't look forward to fixing things then don't setup a monitoring system to look at it! Deal with the problems on your time but if you want to attempt to offer a reliable service for your customers then by all means setup a monitoring system and have a plan on how you will respond.

Setting your own expectations ahead of time will go a long way to not being stressed when it is down. Here is a freebie for you. Take and have backups! Practice restoring and working with those backups before an emergency takes place.

Quagmire

You may ask, "why not run my own monitoring service?" That is a great question and a great idea. There are many suitable monitoring systems you could run which would be completely adequate for this problem. Nagios, for example, comes to mind. The catch-22 is how will you know if your NodeBB forum is having problems if your monitoring software is also having problems. This is most likely a truth if you attempt to run your monitoring system from the same server that your forum is running from. You could circumvent this predicament by operating another server on which your monitoring platform would reside but there is always the risk that your monitoring platform could run into problems or go off line undetected by you, thus leaving you in the same position as if you did not have monitoring at all. Let someone else manage your management service and benefit from their economy of scale.

Known Working Compatibility

This guide should be applicable to all versions of NodeBB.

---

Contact me https://community.nodebb.org/user/rod

Revision 1.2, 2016-MAY-26 @979.beats

I would like to trigger a specific action after a user submits a post. How would you recommend I do this?

I found an old post that references "fireHook", how could I use that to my advantage?

plugins.fireHook('action:post.purge', pid, function () { // all plugins are done });

In my case I don't want to purge a post but rather do something else.

Where would I even put the code chunk?

Thanks.

Well I am embarrassed, I figured out the problem! There is no problem with custom javascript or the custom header information. It turns out the nodebb-plugin-ns-twitch-monitor was the problem! The plugin does not seem to work after a certain version of NodeBB and where I had it placed in my widgets must have been causing the custom header code to not load or run properly.

I am still running v1.4.5 but am moving over to a new server, yes after all of these years. In doing so I have been incrementally stepping through upgrades, 1.4.5 -> 1.5.x -> 1.6.x -> 1.7.x... It was there at the 1.7.x that my custom Piwik (Matomo) code stopped working. I put NodeBB into "dev" mode and saw a number of errors related to the Twitch plugin. I removed the plugin and my analytics code now works fine.

It is bittersweet as I really like the Twitch plugin on my forum but the plugin stopped working quite some time ago. Twitch changed how their API works and the plugin was never updated to keep up.

Shouting out to a list of people for closure on this, in case you were all losing sleep on this since May of last year! @baris @julian @Nicolas

@baris Thanks for all of the pointers. The javascript does not seem to work in a Widget, as you deduced, either.

I'm at a loss as I'm not a JS programmer. I'll just stick with v1.4.5 for a while longer for my production forum. Possibly I will figure something else out.

@baris Still no success. I am watching the web server logs of the Piwik server and I see a hit for the loading of "/piwik.js" (as I should as you call the URL in your first script tag but I don't see any further web server entries after that.

I switch over to the my production NodeBB forum and while watching the Piwik web server log files I see all of the action I expect to see.

Is there any sort of special widget or plugin I could install in which I could put this javascript and have it run as it should. I wonder what in NodeBB is stomping on it?

I do see true in my browser console, from console.log(window.hasOwnProperty('_paq'))

@baris. I reverted my dev site back to my production '1.4.5' version and it logged properly to Piwik (Matomo.). I upgraded to v1.6.1 and that version worked fine also, reporting to Piwik.

I upgrade to v1.7.5 and voila reporting to Piwik no longer works. The 1.7.x branch is the one in which the Custom Javascript was broken out from Custom Header. I truly believe this is the root of the problem.

Has anyone else reported oddities with custom javascript after the introduction of the 1.7 branch?

Thank you.

@baris I cannot get this to work. My new plan is to destroy my 1.12.1 version and install 1.6.x and test. If that works do the same going to version 1.7.x and test, and keep jumping forward until it doesn’t work and then report back.

Thank you for all of your help thus far.

@baris You lost me on that last post. What section do I wrap in window.addEventListener('load', function ()?

@baris The odd thing is that it works in 1.4.5. Should it work if I move it to Custom Header with the script tags around it? Because when I try that I do get an error in the console and do NOT get any of that data form the console.log output.

The error I get when it is in Custom Header is ReferenceError: Can't find variable: ajaxify

@baris I added that to the function, I see data being displayed. Is there anything specific I should be looking for? I could give you access to the forum (it is a dev forum not prod) if you wanted to poke on it directly.

@baris said in In which version did Custom Javascript show up?:

firePiwik(ajaxify.data);

This has not solved the problem. Possibly closer, but not solved. I have my code back in Custom Javascript. I am not seeing errors for that in my browser console.

It now behaves like it just doesn't execute firePiwik(data); in the "$(window)" area. This is all so strange.

Thanks.