How To: Piwik and NodeBB

Tutorials
  • Added v1.1.0 <-> 2.16.1 to known working versions.

  • @rod every Piwik version will work.

  • Hi there,

    Is there any other way to implement piwik other then adding it to the header. I've done it in my setup and it seems to be working but when I compare the internal dashboard with piwik the internal dashboard shows about approx three times more unique visits and page views.

    Is there parts of nodebb where the header isn't loaded or other oddity that could explain this.

  • @hmj That discrepancy can likely be explained by Piwik only tracking cold page loads, whereas NodeBB tracks page transitions as well (that is, going between pages).

    @rod I see there is code to fire Piwik when a page transition finishes:

        $(window).on('action:ajaxify.contentLoaded', function(ev, data) {
            firePiwik(data);
        });
    

    Any chance you could test this to ensure that it is actually working?

  • @julian said in How To: Piwik and NodeBB:

    @hmj That discrepancy can likely be explained by Piwik only tracking cold page loads, whereas NodeBB tracks page transitions as well (that is, going between pages).

    What do you mean by that? In my Piwik I see visitors moving between topics

    @rod I see there is code to fire Piwik when a page transition finishes:

        $(window).on('action:ajaxify.contentLoaded', function(ev, data) {
            firePiwik(data);
        });
    

    Any chance you could test this to ensure that it is actually working?

    Yes, I believe this is working. Suggestion on how to test this?

  • @rod @hmj does piwik count search bots as visitors or ignore them? That might explain the difference.

  • @pichalite I am quite sure that Piwik ignores them, installed a plugin to count bots now and I cannot see that it accounts for around 4k unique visits and at least 40k pageviews per day that are not counted in Piwik. I added GA tracking yesterday as well to compare to the two and GA seems to be in line with Piwik.

    So I'm a bit baffled by the amount of traffic Nodebb is showing in it's backend.

  • @hmj What other plugins do you have installed? Could be one of them are hammering some routes 😦

  • @julian Not sure here is a list of the plugins we are using.

    Activated plugins:
    nodebb-plugin-composer-default
    nodebb-plugin-dbsearch
    nodebb-plugin-emoji-extended
    nodebb-plugin-emoji-one
    nodebb-plugin-markdown
    nodebb-plugin-mentions
    nodebb-plugin-spam-be-gone
    nodebb-plugin-sso-oauth
    nodebb-widget-essentials

    So I've added GA to the mix. I did that with adding it in the same custom header as to validate that my Piwik installation is not on the fritz. Comparing the two I see a little fewer unique visits between the two in piwik but almost exactly the same amount of pageviews.

    Nodebb is still off by a factor of three in unique visits and around a factor of three to ten in pageviews.

  • Did this piwik code break for anyone else on the 1.6 upgrade?

    0_1508334234807_0f84cd27-16b4-442e-b972-e769f9d14e01.png

    Index 86: $(window).on('action:ajaxify.contentLoaded', function(ev, data) {
    Index 89: })();

  • @hmj v1.6.1? or v1.6.0... it may be that the piwik plugin needs updating.

    Edit: Actually, doesn't seem like there is a piwik plugin... are you just putting it in the custom HTML page?

  • @julian Yeah that code at top is put into Appearance -> Custom HTML & CSS -> Custom Header. That had worked fine until the last upgrade.

  • @hmj Are you on the develop branch? The code should still work fine in master.

    Also can you please paste the code here?

  • @julian We are on 1.6.0 and not Develop.

    Our code is almost a carbon copy of the one above in the first post.

     <!-- Piwik -->
    <script type="text/javascript">
    var _paq = _paq || [];
    (function () {
        var u = "https://our.piwik.com/";
    
        function firePiwik (data) {
            if (app && app.user && app.user.uid > 0) {
                _paq.push(['setUserId', app.user.uid.toString()]);
                _paq.push(['setCustomVariable', 1, "appUserUsername", app.user.username, "visit"]);
            }
            _paq.push(['setDocumentTitle', document.title]);
            _paq.push(['setCustomUrl', location.href]);
            _paq.push(['enableHeartBeatTimer']);
            _paq.push(['appendToTrackingUrl', 'bots=1']);
    		if (data.tpl === 'search') {
    			_paq.push(['trackSiteSearch', ajaxify.data.search_query,, ajaxify.data.matchCount]);
    		} else {
    			_paq.push(['trackPageView']);
    		}
            _paq.push(['enableLinkTracking']);
            _paq.push(['setTrackerUrl', u+'stats/piwik.php']);
            _paq.push(['setSiteId', 1]);
        }
        var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
        g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'stats/piwik.js'; s.parentNode.insertBefore(g,s);
    
        $(window).on('action:ajaxify.contentLoaded', function(ev, data) {
            firePiwik(data);
        });
    })();
    </script>
    <noscript><p><img src="https://our.piwik.com/stats/piwik.php?idsite=1&rec=1&bots=1" style="border:0;" alt="" /></p></noscript>
    <!-- End Piwik Code -->
    
  • @hmj can you provide the result of git rev-parse HEAD? I have a feeling you're on a commit with a feature we've temporarily rolled back

  • @julian said in How To: Piwik and NodeBB:

    git rev-parse HEAD

    02370b30d934006fe1db8575cccf6a42ac71431e

  • @julian Resurrecting an old thread, it is still relevant though.

    Since upgrading NodeBB from v1.4.5 to v1.12.1 my Piwik analytics code no longer works. It looks like @hmj had this same problem when he upgraded to v.1.6.0.

    Does anyone have suggestions on what may have changed such that this doesn't seem to work any longer?

    $(window).on('action:ajaxify.contentLoaded', function(ev, data) {
            firePiwik(data);
        });
    

    Thank you.


Suggested Topics


  • 3 Votes
    16 Posts
    512 Views

    @tankerkiller125 I too opt for /recent as I prefer to see all activity rather than categories

  • 0 Votes
    1 Posts
    594 Views
    Who Is This Document For?

    Well... like most things.... it depends....

    If one or more of following shoes fits: Just want to get NodeBB up and running with minimum of hassle please I refer you to the tl;dr commentary from my NodeBB Email Options article. Intermediate to advanced Linux user. You can likely get this on your own. May be of some value to those unfamiliar with network and server side aspects of SMTP? Seasoned email administrator? Right. Please do not be too harsh as I'm endeavoring to bridge a gap here... SME commentary welcome. 😉 If, on the other hand: You're attracted to the "Ops" side of DevOps. Like to know how things work. Medium/advanced "beginner" to the Unix/Linux command line and eager to learn. Have the time to invest in that learning (i.e. big boss man or some other urgency not twisting your arm to get this done yesterday) and not afraid to RTFM.

    Then.......

    For Those About to Rock...

    I confess to deploying a Null Client set up for my forums. I thought I'd expand upon this a bit for interested parties not already possessing subject matter expertise but comfortable with the command line and willing to roll up their sleeves. Follows is meant to be a thumbnail sketch to get the DIY inclined pointed in the proper directions rather than exhaustive step by step copy pasta.

    Firstly, let's take a look at some reasons why/when this use case may be a fit:

    Isolation and separation of duties. Some favor an all on one approach. I favor dedicated vm's for dedicated tasks. I'd rather not have my app servers initiating connections with the Big Bad Internet. Incoming http(s), smtp, etc. requests from non local VM's is proxied. Access to prexisting email infrastructure. Security and stability concerns. All software has bugs. Postfix has weathered extensive scrutiny and audit. As such, I expect it is less vulnerable to things like malformed responses. Flexibility moving forward. For e.g. you want a daemon to manage NodeBB's outgoing mail queue - and perhaps that daemon also comes in handy for other duties like emailing various reports or alerts to administrators.

    Presumptions:

    You have a Postfix Null Client configured as outlined in the Postfix Standard Configuration Readme. You have NodeBB configured to utilize such as depicted in my screenshot in my prior article linked above.

    I've mentioned this can get complex in a hurry. Let's see if we give it a little K.I.S.S.

    Incoming!! 💣 📬

    Let's get our biggest hurdle out of the way.

    Null clients do not receive incoming mail. Setting up SMTP relays, IMAP server, virus/malware scanner, etc. ups the ante considerably. We've already got enough on our plates, at least for now, embarking upon this Null Client journey so lets save ourselves some grief and off load those duties to a reputable third party service such as Proton Mail.

    Outgoing

    As always, there's more than one way to do it:

    Null client delivers email directly to one of the servers listed in recipient domains MX records.

    Null client forwards email to a dedicated SMTP relay host, e.g. the folks handling your incoming mail (you may need to coordinate with them to configure some knobs, e.g. which hosts they will relay email from). Advantages to using a relay host include: 1) Undeliverable mail does not get stuck on your NodeBB server, and 2) Our null client is now initiating outgong SMTP connections with only the relayhost(s) rather than the big bad Internet at large.

    Postfix needs to know about your relay host to get this working. Add the following line to your null client's main.cf

    relayhost = your-relay-host.domain.tld

    You may also choose to specify relayhost via alternate parameters, such as IP address. I recommend sticking with hostnames as IP addresses may change from time to time.

    DNS Record Cogs & Widgets ⚙

    DNS presents yet another challenge and can have a high price for mistakes. So you're best off offloading this to a third party. Good news is your email provider likely bundles this with their service. Your domain registrar surely does. Consider leveraging that. Submit a request to have forward and SPF records added for your NodeBB instance's hostname.

    Note: I've used traditional Bind zone file syntax in the examples below. Using this syntax in help desk requests helps avoid confusion. Your provider may well offer a web based UI.

    You want records that look similar to the following (mind the trailing periods on fully qualified host names):

    Forward Record

    null-host-name.your-domain.tld. IN A XXX.YYY.WWW.ZZZ

    SPF Record

    The purpose of an SPF record is to provide a list of servers authorized to send email from your domain and must therefore necessarily include your third party provder's servers as well as your null client(s).

    your-domain.tld. IN TXT "v=spf1 ip4:primary-ip, ip4:secondary-ip, ip4:null-host-ip -all"

    MX Records

    Your mailbox provider likely handles configuration of MX records seamlessly but for the sake of completeness....

    RFC's dictate primary and secondary mail exchangers. Get these from your mailbox host.

    @ IN MX 50 primary-mx.domain.tld. @ IN MX 80 secondary-mx.domain.tld.

    Reverse Record

    Your cloud provider is authoritative for reverse records for IP space assigned to them. Set up a PTR record for your NodeBB host using your providers management ui or submit a request ticket for them to create such for you.

    XXX.YYY.WWW.ZZZ.in-addr.arpa. IN PTR null-host-name.your-domain.tld.

    I manage my own DNS servers. DKIM is a royal pita for even seasoned adminstrators and imho an accident waiting to happen. Hence, unless you have a passion for learning about this stuff, I recommend letting your email/IMAP provider handle DNS for you, as it is lots easier to have them add an A or CNAME record as necessary for web servers and such than for you to configure and maintain DKIM. Else avoid DKIM entirely.

    Postmaster

    Other systems expect the existence of a valid postmaster address for communication of issues such as bounced mail, etc. Create "[email protected]", either as a dedicated mailbox or a forward to an existing mailbox, e.g. [email protected] if you prefer not having to check multiple mailboxes. Test this. Don't sweep postmaster mail under the rug lest it come back to haunt you at the most inopportune of times.

    Blacklists

    We need to ensure our newly minted Null Client is not blacklisted. MX Toolbox offers a meta blacklist check. Barracuda Networks will most likely do so before you've sent even a single email and requires jumping thru their hoops to become delisted. And jump ye' shall because their stuff is in wide spread use, particularly corporate side. It behooves you to check these blacklists from time to time, particularly with a newly minted server.

    Bippity Boppity Boo! 🎉

    Put it together and what have we got?

    Robust SMTP daemon relaying outgoing NodeBB email. Robust and reasonably secure IMAP mailboxes sans administration headaches. No worries about third party email plugin breakage between version upgrades.

    While this method adds complexity it's not too difficult to get set up once you understand some underlying concepts, moving pieces involved and what you're targeting. It also serves as a stepping stone to building out your own email infrastructure should you decide this stuff is all just too much fun.

    Enjoy! o/

  • 4 Votes
    9 Posts
    2k Views

    @ᴅᴀʀᴛʜ-ᴠᴀᴅᴇʀ It probably crashes because of the plugins you installed and nodebb can't find them (nodebb-theme-oxide in particular)

    I suggest you backup the redis database by copying the /data folder somewhere, then delete the contents of the original folder and also remove the config.json volume.

    Erase and restart redis and nodebb container (It should be like a fresh install) then installs the plugins and themes listed in the log file (nodebb-plugin-teamspeak-verify, nodebb-widget-search-bar, nodebb-theme-oxide, etc)

    When all plugins are installed and enabled, stop the nodebb container and restore the /data redis database with your backup, and remap config.json volume and I think it should be up.

    Next time you want to upgrade, disable all plugins before restarting the container with the new image.

  • NodeBB on Windows 10 / 2016

    Tutorials
    3 Votes
    7 Posts
    2k Views

    If anyone gets this error having the forum in a subfolder like blabla.com/forum:

    forumcategories Not Found

    Fix Here

  • NodeBB - Full Stack Setup

    Tutorials
    9 Votes
    13 Posts
    9k Views

    @omega I actually meant "according to the docs".
    I am sorry for my crappy English as I am not a native speaker.

    @Shard ok.