Piwik and NodeBB

Introduction

So you have your shiny NodeBB forum up and running. Hopefully it is spilling over with content and you have many glorious visitors. But do you? How do you know? Analytics of course! Huh, analytics you say?

• Analytics
  the method of logical analysis ^1

So what do you do? You load a big giant googly companies analytics plugin. And then you realize this:

All your analytics are belong to us.

And then you are scared. What can you do about that? The answer is simple, you run your own analytics program! Piwik to the rescue. Go ahead, go there, get it all setup ^2. I'll wait.

Great! Welcome back. I trust you have your Piwik installation running and ready to analyze data from your NodeBB. The hard part is over now. All that is left is making a few customization tweaks to NodeBB.

NodeBB

Tracking Code

The tracking code. These are the bits that make the magic work. We need to tell Piwik about our visitor's actions. What are they looking at, what are they posting and even what are they searching for. NodeBB's custom header function is a wonderful fit for all of this. All that needs to be done is pasting in the following code with only a couple of slight adjustments.

Add the following tracking code to your NodeBB forum's custom header.

ACP > Appearance > Custom HTML & CSS > Custom Header

Change your.piwikurlhere.com to your actual Piwik URL. It could be a subdomain on your NodeBB domain or it could be an entirely different domain but it is the one you used when you installed and configured Piwik. If this is your first and only site that Piwik is tracking then the default site ID of 1 will be sufficient. If it isn't than update the site ID (setSiteID and idsite) to the appropriate value. You will know this from your Piwik installation and configuration.

<!-- Piwik -->
<script type="text/javascript">
var _paq = _paq || [];
(function () {
    var u = "//your.piwikurlhere.com/";

    function firePiwik (data) {
        if (app.user && app.user.uid > 0) {
            _paq.push(['setUserId', app.user.uid.toString()]);
            _paq.push(['setCustomVariable', 1, "appUserUsername", app.user.username, "visit"]);
        }
        _paq.push(['setDocumentTitle', document.title]);
        _paq.push(['setCustomUrl', location.href]);
        _paq.push(['enableHeartBeatTimer']);
        _paq.push(['appendToTrackingUrl', 'bots=1']);
		if (data.tpl === 'search') {
			_paq.push(['trackSiteSearch', ajaxify.data.search_query,, ajaxify.data.matchCount]);
		} else {
			_paq.push(['trackPageView']);
		}
        _paq.push(['enableLinkTracking']);
        _paq.push(['setTrackerUrl', u+'piwik.php']);
        _paq.push(['setSiteId', 1]);
    }

    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);

    $(window).on('action:ajaxify.contentLoaded', function(ev, data) {
        firePiwik(data);
    });
})();
</script>
<noscript><p><img src="//your.piwikurlhere.com/piwik.php?idsite=1&rec=1&bots=1" style="border:0;" alt="" /></p></noscript>
<!-- End Piwik Code -->

Set Enable Custom Header to on, and press that floppy disk^3 icon^4 to save all of your hard work.

Reload or restart your NodeBB forum to be sure that the system is now injecting the tracking code with the pages that NodeBB generates. Visit your NodeBB forum and check your Piwik dashboard for activity.

Conclusion

And there you have it. Analytics! Implementing Piwik like this also allows that other googly analytics plugin to work simultaneously if you so desire to have the benefits of that all seeing eye. It is also possible to add additional tracking services^5 along with Piwik in the custom header.

Known Working Compatibility

N'BB		Piwik
v1.1.0		2.16.1
v1.0.2		2.16.1, 2.16.0
v1.0.0		2.16.0, 2.15.0
v0.9.4		2.16.0, 2.15.0
v0.9.3		2.15.0

---

Contact me https://community.nodebb.org/user/rod

Revision 1.4, 2016-JUL-13 @667.beats

Let's Encrypt and NodeBB

Summary

NodeBB is a special beast. One that takes careful grooming and maintenance. Ah, gone are the golden days of LAMP. That glorious "P", so easy to use but oh so poor on performance. With Node.js and socket.io living behind your webserver you no longer have the easy liberty of throwing down files and opening directory access any where you would like. What's that? You have to shutdown your NodeBB just to renew your Let's Encrypt SSL certificates? You are using SSL right? Shame, shame if you are not.

Take notice! You no longer need to shutdown your webserver and NodeBB just to renew those Let's Encrypt certificates if you make a simple change to your webserver configuration.

Back story

Let's Encrypt is a free SSL certificate authority. You run their software on your server that authenticates with their master control. The key is proving that you have rights to the domain you are requesting a certificate for. Their software is a webserver so you can't already be running one on port 80. Theirs needs to bind to that port. To run their software, to renew my certificate, I have had to shutdown my webserver (which makes my NodeBB forum unavailable). It doesn't take long but I was annoyed that my NodeBB forum would have to be offline during that time. I knew there must be a solution to this problem and there is in their "webroot" option.

The webroot flag keeps their software from needing to run as a webserver and just places special authentication files in your already existing webserver's html path. That is all fine-and-dandy for a typical LAMP setup but doesn't behave as well with Node.js, socket.io and NodeBB. It can behave though if you tell your webserver to exclude the super secret Let's Encrypt webroot path from going to NodeBB and point off somewhere else.

Webserver

We must tell the webserver that the .well-known/acme-challenge directory is not to be proxied off to socket.io like it needs to be for NodeBB.

Nginx

These configurations need to be in the unencrypted (port 80) section of your webserver configuration.

Change yourdomainhere.com with your actual domain name.

server {
    listen       80;
    server_name  yourdomainhere.com;

    root   /var/www/yourdomainhere.com/html;
...

Add a location block for the Let's Encrypt directory

location /.well-known/acme-challenge/  {
    try_files $uri =404;
}

Bonus: If you want to only serve https traffic for your NodeBB forum make your root location block something like this

location / {
    rewrite ^(.*) https://yourdomainhere.com$1 permanent;
}

Reload Nginx for the changes to take affect.

sudo systemctl reload nginx

Apache (and others)

You will need to research how to setup the proper exclusion of the .well-known/acme-challenge directory from being proxied to socket.io.

Let's Encrypt

Linux

When it is time to renew your certificate (or create your certificate) you would run the following command.

Change yourdomainhere.com with your actual domain name.

sudo ./letsencrypt-auto certonly --webroot -w /var/www/yourdomainhere.com/html -d yourdomainhere.com -d www.yourdomainhere.com

• certonly means to only have the certificates generated and do not try to modify any webserver (Nginx) configuration files. That task is left to the student (you).
• --webroot tells Let's Encrypt to not run as a webserver and not to bind to port 80
• -w points to the actual path on disk that Nginx can serve files from. This is the place that the .well-known/acme-challenge directory will be created in and inside of that directory Let's Encrypt will place temporary authorization files
• -d the domain name that this certificate will authenticate for. Add additional -d flags for each of the names you may have.

Windows (and others)

You will need to research the syntax to execute the letsencrypt-auto command but the flags should be very similar to the Linux ones shown above.

Caveats

Symbolic Links

It has been reported that symbolic links do not work well with the Let's Encrypt software. Therefore please avoid attempting to sym link the .well-known/acme-challenge directory to another location on your file system.

Conclusion

What I have left out is the initial configuration of your webserver to accept SSL connections. There are numerous documents online to help with that portion. If it is done correctly for Let's Encrypt the certificate path will point to a live directory. Renewing the certificates only updates the files in the live directory and your webserver configuration is not touched.

With this configuration I am able to run the command to update my certificates without having to shutdown my webserver thus making NodeBB unreachable from my webserver proxy.

Contact me https://community.nodebb.org/user/rod#

Create & Manage a NodeBB Test/Dev Environment

Table of Contents

• Summary
• Caveats
• Webserver
  • Nginx
• NodeBB
  • PRD -> DEV
  • Adjustments to DEV
  • Upgrading DEV NodeBB
  • Delete DEV
  • Fresh NodeBB with PRD database

Summary

The old saying goes

Laws are like sausages, it is better not to see them being made.

Managing NodeBB isn't as bad as that but there could be a bit of a challenge the first few times that you attempt to upgrade your software or install a new feature. Especially on your production system! So what can you do?

Make a sausage factory of course! In our case the factory is a test/development environment to practice your upgrades or test a new plugin or CSS change. You do not necessarily need to spend additional money on a test/dev server, although you could. You can spin up a copy of your production system right on your production system. This is not very difficult due in part to how well NodeBB, and Node.js, were created.

Caveats

For disclosure I am not a developer of NodeBB or any of the supporting applications. These notes are just that, a collection of my notes as I was experimenting and learning. I do have twenty-four years of IT experience, mostly Unix (SunOS, Solaris, AIX, BSD, Linux) which includes twenty-four years of bad habits and ugly kludges. I do hope that in those years I have picked up a few gems and learned a few good traits. I will try to pass along only the best but please do accept the ugly as they come.

My NodeBB forum runs on a Linux (CentOS) server. I utilize Redis as my NodeBB database and use Nginx as my web server. This document, unless noted otherwise, will reference that environment.

I am making the assumption that you have a production NodeBB server that is fully operational and you wish to create a dev/test copy of it on the same server.

And finally, The risk is yours. I do not take responsibility for harm to your system. Please evaluate all of my suggestions as you deem necessary.

Webserver

Nginx

This addition should only need to be made once. After it is in place you should be able to create and delete dev NodeBB environments from your production without needing to adjust your Nginx configuration.

You will need to make a new server block in your Nginx domain configuration file. Most likely located in /etc/nginx/sites-available/yourforumdomain.com.conf. This new server block will have the specifics for your dev NodeBB. For example you may create an entry like:

server {
    listen       443 ssl;
    server_name  dev.yourforumdomain.com;
[...]

In your location / { location block you will need a proxy_pass entry that directs to the port that your dev NodeBB will listen on:

location / {
[...]
    proxy_pass http://127.0.0.1:4568;
[...]

Once you have your Nginx configuration adjusted appropriately do not forget to restart it. sudo systemctl restart nginx.

NodeBB

PRD -> DEV

For my examples the production NodeBB is running under the /opt/nodebb/ directory path and we will be creating a copy into /opt/devNodebb/

Let's make a copy of production into dev. This can be done "live" without needing to shutdown your production forum.

mkdir /opt/devNodebb
(cd /opt/nodebb; tar cf - .) | (cd /opt/devNodebb; tar xvf -)

If NodeBB was running when you made the copy you'll need to remove the pidfile. A pidfile typically contains the process id information of the running program. This is useful so that commands such as ./nodebb stop know which program to kill. The pidfile you copied over relates to the process id of your production NodeBB. You don't want to be killing that process when you try to startup your dev NodeBB.

rm /opt/devNodebb/pidfile

Since you made a copy of production you have all production information. If you attempted to start your dev NodeBB now you would have problems. You must edit your config.json file and tell it that you want a new instance of NodeBB.

vi /opt/devNodebb/config.json

Change your url, port and database entries. For the URL make it the dev domain that you configured your Nginx to listen for. For the port and database numbers I just incremented by one from production.

"url": "https://dev.yourforumdomain.com",
"port": "4568",
[...]
    "database": "1"

No you aren't ready to start up your dev NodeBB just yet. There isn't a database behind it.

Let's make a copy of the production Redis database for dev to use. I use redis-copy which is not part of the redis database distribution. I downloaded and built my copy from this Github repository redis-copy.

The following command makes a copy of the Redis database number 0 (production) to database number 1 (your dev/test). Database number 1 is the database we told the devNodebb config.json file above to use.

redis-copy localhost/0 localhost/1

Now we may startup the dev NodeBB instance.

cd /opt/devNodebb
./nodebb start

Adjustments to DEV

Once your dev forum is running you should make a few administrative adjustments to it. These are not necessarily critical but they will help you to keep track of which system you are working on and may reduce notifications to your users, etc. Keep in mind that you cloned your production system which includes the users. If you do not want them logging into your dev system you should limit their exposure to it.

You may want to adjust the site and browser title's to reflect a dev forum

ACP > Settings > General > Site Title > DEV YourForumName
ACP > Settings > General > Browser Title > DEV YourForumName

Disable email subscriptions

ACP > Settings > Email > Email Subscriptions > Disable subscriber notification emails

Disable plugins that may cause confusion, such as Google Analytics

ACP > Extend > Plugins > nodebb-plugin-google-analytics > Deactivate

There may be other plugins that you wish to disable. Experimenting will lead you to adjust this list appropriately.

There you have it. You now have a test/dev environment based off of your production system. Changes you make now will only be done in test/dev and your users will happily continue to use the production system until the time comes that you shutdown production to re-play changes or enhancements that you tested in dev.

Upgrading DEV NodeBB

So you went through all of the above steps and now you have, well, just a copy of production. Not really useful as it stands but now you have a platform to test upgrading without fear of damaging production.

Let's upgrade! If you'd like to bring your dev NodeBB up to the current general release of NodeBB the following steps should be adequate.

Stop your dev NodeBB

cd /opt/devNodebb
./nodebb stop

Tell Git to get the newest release information. This sets up NodeBB such that when you issue the upgrade command it will pull down all of the new or changed files as necessary.

cd /opt/devNodebb
git fetch
git checkout v1.x.x
git merge origin/v1.x.x

Issue the NodeBB upgrade command

cd /opt/devNodebb
./nodebb upgrade

Watch for upgrade errors. Note that any customizations or changes you have made to your theme's CSS may have been undone by a new feature or bug fix. That is why we have this test/dev environment, so you can test and look for problems.

If the upgrade appears to have worked without errors start up dev NodeBB and then switch into the logging mode so that you may watch the behavior of the system. Pressing ctrl-c will exit the logging mode.

cd /opt/devNodebb
./nodebb start
./nodebb log

Delete DEV

What if there were problems with your upgrade or you'd like to start this process over with a fresh copy of production? That is not a big problem. A few commands will delete your test/dev environment and then you can start the copy process over again.

Stop your dev NodeBB

cd /opt/devNodebb
./nodebb stop

Let's remove the test/dev software directory tree

rm -rf /opt/devNodebb

Delete the test/dev Redis database. This is probably the most dangerous command in this document. Please double check that the database number provided is the one for your test/dev environment and NOT the one for your production environment.

redis-cli -n 1 flushdb

And there you have it, test/dev is gone. You may now redo the PRD -> DEV or Fresh NodeBB with PRD database steps if you so desire.

Fresh NodeBB with PRD database

There may be times that you want to have a factory fresh NodeBB installation with your database content. You would end up with users, setttings, content, etc., but not files that you may have edited over time or plugins that you installed.

Run the Delete DEV section to be sure you have no remnants of a previous test/dev environment. Then proceed.

Tell Git to get the software and where to put it.

git clone -b v1.x.x https://github.com/NodeBB/NodeBB /opt/devNodebb

Copy your production config.json file and then edit it as you would have if you were doing the PRD -> DEV section

cp /opt/nodebb/config.json /opt/devNodebb/config.json

vi /opt/devNodebb/config.json

Change your url, port and database entries.

"url": "https://dev.yourforumdomain.com",
"port": "4568",
[...]
    "database": "1"

Copy over your Redis database

redis-copy localhost/0 localhost/1

Since the config.json has been created and contains the information that the NodeBB setup process would have asked for we can just run the NodeBB upgrade command (as opposed to the setup command). Bypassing the questions about which database to use, etc. Before you run the NodeBB upgrade you have to actually install the NodeBB software. This process was not necessary when we copied PRD to DEV earlier as it was done when you did your initial production NodeBB installation and was brought over in the tar copy process.

cd /opt/devNodebb
npm install
./nodebb upgrade

Now start it up!

cd /opt/devNodebb
./nodebb start

And there you have it. You now have a clean installation of NodeBB with your database content. As noted in the PRD -> DEV section you may wish to make adjustments to your dev forum so that you can easily identify which one you are working in. Ie., change the site and browser title names, etc.

---

Contact me https://community.nodebb.org/user/rod

Revision 1.2, 2016-APR-08 @645.beats

And finally upgraded to v1.4.2 for my production site.

@Jam said in Invalid CSRF Token:

SSL: no

Although you have SSL set to no, have you tried to resolve this problem by adding proxy_set_header X-Forwarded-Proto $scheme; to your Nginx configuration?

@baris @bentael helped me out last night. Amongst other things he confirmed that too, that being on the index page results in an empty data.url. He did get me going using the location.href.

I have to say every one on this forum has been super helpful. I intend to make a donation to Nodebb in the near future.

Sounds wonderful, off to crash my dev server. whistles

@AOKP I just looked at your high performance tutorial. Definitely a lot of information. Possibly too much all at once. I think people may find this succinct post just related to Let's Encrypt easier to digest.

I feel that an edited post should be changed from read to un-read status therefore showing up again in the Unread and Recent thread displays. Why do I ask this? I posted a new topic in the tech support category which then had a handful of 'reads.' I have since added a lot more content to the post and now those people who have already read my post will not realize that I added more content. Maybe an elegant way to handle this would be if greater than x% changes (significant content change) change the post status to un-read but if less than x% changes (maybe just a few typographical corrections) leave the post as is.

What is your opinion on how to handle edited posts?

Thank you.

@The-Penultimate-Defenestrator This exists through the nodebb-plugin-pushbullet plugin.

I am using it on this site (community.nodebb.org) as well as the forum I administer.

You get to it from your profile settings.

Well I am embarrassed, I figured out the problem! There is no problem with custom javascript or the custom header information. It turns out the nodebb-plugin-ns-twitch-monitor was the problem! The plugin does not seem to work after a certain version of NodeBB and where I had it placed in my widgets must have been causing the custom header code to not load or run properly.

I am still running v1.4.5 but am moving over to a new server, yes after all of these years. In doing so I have been incrementally stepping through upgrades, 1.4.5 -> 1.5.x -> 1.6.x -> 1.7.x... It was there at the 1.7.x that my custom Piwik (Matomo) code stopped working. I put NodeBB into "dev" mode and saw a number of errors related to the Twitch plugin. I removed the plugin and my analytics code now works fine.

It is bittersweet as I really like the Twitch plugin on my forum but the plugin stopped working quite some time ago. Twitch changed how their API works and the plugin was never updated to keep up.

Shouting out to a list of people for closure on this, in case you were all losing sleep on this since May of last year! @baris @julian @Nicolas

@baris Thanks for all of the pointers. The javascript does not seem to work in a Widget, as you deduced, either.

I'm at a loss as I'm not a JS programmer. I'll just stick with v1.4.5 for a while longer for my production forum. Possibly I will figure something else out.

@baris Still no success. I am watching the web server logs of the Piwik server and I see a hit for the loading of "/piwik.js" (as I should as you call the URL in your first script tag but I don't see any further web server entries after that.

I switch over to the my production NodeBB forum and while watching the Piwik web server log files I see all of the action I expect to see.

Is there any sort of special widget or plugin I could install in which I could put this javascript and have it run as it should. I wonder what in NodeBB is stomping on it?

I do see true in my browser console, from console.log(window.hasOwnProperty('_paq'))

@baris. I reverted my dev site back to my production '1.4.5' version and it logged properly to Piwik (Matomo.). I upgraded to v1.6.1 and that version worked fine also, reporting to Piwik.

I upgrade to v1.7.5 and voila reporting to Piwik no longer works. The 1.7.x branch is the one in which the Custom Javascript was broken out from Custom Header. I truly believe this is the root of the problem.

Has anyone else reported oddities with custom javascript after the introduction of the 1.7 branch?

Thank you.

@baris I cannot get this to work. My new plan is to destroy my 1.12.1 version and install 1.6.x and test. If that works do the same going to version 1.7.x and test, and keep jumping forward until it doesn’t work and then report back.

Thank you for all of your help thus far.

@baris You lost me on that last post. What section do I wrap in window.addEventListener('load', function ()?

@baris The odd thing is that it works in 1.4.5. Should it work if I move it to Custom Header with the script tags around it? Because when I try that I do get an error in the console and do NOT get any of that data form the console.log output.

The error I get when it is in Custom Header is ReferenceError: Can't find variable: ajaxify

@baris I added that to the function, I see data being displayed. Is there anything specific I should be looking for? I could give you access to the forum (it is a dev forum not prod) if you wanted to poke on it directly.

@baris said in In which version did Custom Javascript show up?:

firePiwik(ajaxify.data);

This has not solved the problem. Possibly closer, but not solved. I have my code back in Custom Javascript. I am not seeing errors for that in my browser console.

It now behaves like it just doesn't execute firePiwik(data); in the "$(window)" area. This is all so strange.

Thanks.

@baris @julian That didn't help. My code snippet was just a piece of the javascript. Here is all of it, maybe that will help figuring out why the $(window)... bit no longer works since upgrading from 1.4.5

var _paq = _paq || [];
(function () {
    var u = "//your.piwikurlhere.com/";

    function firePiwik (data) {
        if (app.user && app.user.uid > 0) {
            _paq.push(['setUserId', app.user.uid.toString()]);
            _paq.push(['setCustomVariable', 1, "appUserUsername", app.user.username, "visit"]);
        }
        _paq.push(['setDocumentTitle', document.title]);
        _paq.push(['setCustomUrl', location.href]);
        _paq.push(['enableHeartBeatTimer']);
        _paq.push(['appendToTrackingUrl', 'bots=1']);
		if (data.tpl === 'search') {
			_paq.push(['trackSiteSearch', ajaxify.data.search_query,, ajaxify.data.matchCount]);
		} else {
			_paq.push(['trackPageView']);
		}
        _paq.push(['enableLinkTracking']);
        _paq.push(['setTrackerUrl', u+'piwik.php']);
        _paq.push(['setSiteId', 1]);
    }

    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);

    $(window).on('action:ajaxify.end', function(ev, data) {
        firePiwik(data);
    });
})();

The problem is that pesky $(window) area. I tried to replace it with the suggestion you made but that also didn't work and spewed number of other console errors.

I think you could just drop this whole code bit into your Custom Javascript area and see it in action. The URL won't be correct but that shouldn't matter.

I really want to get this working so I can stay current with my NodeBB versions. I really do.

Thanks!