Frontend freeze on version later than v1.0.2

frissdiegurke

Looks like we got another case like this: https://community.nodebb.org/topic/8904/why-do-i-need-to-restart-my-nodebb-every-couple-of-hours

Sadly I still can't give any further suggestions... No idea what might be wrong; As @jarey mentioned it most likely is a problem of database inconsistency. I guess only the @administrators are aware of what changes happened lately to the database.

julian

@Jenkler Apologies if this was asked before, but have you tried running ./nodebb reset -p and seeing if the crashes still occur? If not, re-introduce plugins one-by-one until you find the culprit...

Bri

I've started seeing this behavior as well on my dev environment, maybe it's related?

Running 1.x.x branch latest as of May 20ish (1.0.3 @ a196c834d9aab4e1c27965dcca623ce12e61b4ae), mongo 3.0.7

I'll be browsing around, and can see about 10 connections in total opened up in mongo, and then after some variable number of minutes, the application just becomes unresponsive; no error logs from node bb or anything. However, I see logs from mongo like this (note the timestamps):

2016-05-29T21:38:52.811-0700 I ACCESS   [conn11] Successfully authenticated as principal nodebb on nodebb

// then, after about 19 minutes:
2016-05-29T21:57:19.778-0700 I NETWORK  [conn2] end connection 127.0.0.1:52223 (9 connections now open)
2016-05-29T21:57:19.778-0700 I NETWORK  [conn7] end connection 127.0.0.1:52234 (8 connections now open)
2016-05-29T21:57:19.779-0700 I NETWORK  [conn9] end connection 127.0.0.1:52236 (7 connections now open)
2016-05-29T21:57:19.779-0700 I NETWORK  [conn4] end connection 127.0.0.1:52231 (6 connections now open)
2016-05-29T21:57:19.779-0700 I NETWORK  [conn8] end connection 127.0.0.1:52235 (5 connections now open)
2016-05-29T21:57:19.782-0700 I NETWORK  [conn6] end connection 127.0.0.1:52233 (4 connections now open)
2016-05-29T21:57:23.474-0700 I NETWORK  [conn11] end connection 127.0.0.1:52251 (3 connections now open)
2016-05-29T21:57:23.474-0700 I NETWORK  [conn3] end connection 127.0.0.1:52224 (3 connections now open)
2016-05-29T21:57:23.474-0700 I NETWORK  [conn5] end connection 127.0.0.1:52232 (1 connection now open)
2016-05-29T21:57:28.231-0700 I NETWORK  [conn10] end connection 127.0.0.1:52237 (0 connections now open)

@Jenkler have you found mongo logs like this at all?

I can't think of any plugins that I have installed that would cause this, the only thing I've been changing is the theme and the composer plugin, and some fresh npm installs... I haven't been switching the NBB branch at all.

Bri

It seems pretty random, but here's another sample:

2016-05-29T22:58:13.486-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53521 #31 (1 connection now open)
2016-05-29T22:58:13.561-0700 I ACCESS   [conn31] Successfully authenticated as principal nodebb on nodebb
2016-05-29T23:02:00.357-0700 I NETWORK  [conn31] end connection 127.0.0.1:53521 (0 connections now open)
2016-05-29T23:02:01.362-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53603 #32 (1 connection now open)
2016-05-29T23:02:01.440-0700 I ACCESS   [conn32] Successfully authenticated as principal nodebb on nodebb
2016-05-29T23:04:03.492-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53632 #33 (2 connections now open)
2016-05-29T23:04:03.518-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53634 #34 (3 connections now open)
2016-05-29T23:04:23.842-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53638 #35 (4 connections now open)
2016-05-29T23:04:23.846-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53639 #36 (5 connections now open)
2016-05-29T23:04:23.846-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53640 #37 (6 connections now open)
2016-05-29T23:04:23.846-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53641 #38 (7 connections now open)
2016-05-29T23:04:23.848-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53642 #39 (8 connections now open)
2016-05-29T23:04:23.849-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53643 #40 (9 connections now open)
2016-05-29T23:04:23.850-0700 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:53644 #41 (10 connections now open)
2016-05-29T23:06:03.511-0700 I NETWORK  [conn33] end connection 127.0.0.1:53632 (9 connections now open)
2016-05-29T23:06:03.519-0700 I NETWORK  [conn34] end connection 127.0.0.1:53634 (8 connections now open)
2016-05-29T23:06:23.844-0700 I NETWORK  [conn35] end connection 127.0.0.1:53638 (7 connections now open)
2016-05-29T23:06:23.844-0700 I NETWORK  [conn36] end connection 127.0.0.1:53639 (7 connections now open)
2016-05-29T23:06:23.844-0700 I NETWORK  [conn37] end connection 127.0.0.1:53640 (5 connections now open)
2016-05-29T23:06:23.844-0700 I NETWORK  [conn38] end connection 127.0.0.1:53641 (4 connections now open)
2016-05-29T23:06:23.844-0700 I NETWORK  [conn39] end connection 127.0.0.1:53642 (3 connections now open)
2016-05-29T23:06:23.847-0700 I NETWORK  [conn40] end connection 127.0.0.1:53643 (2 connections now open)
2016-05-29T23:06:23.847-0700 I NETWORK  [conn41] end connection 127.0.0.1:53644 (1 connection now open)
2016-05-29T23:07:06.570-0700 I NETWORK  [conn32] end connection 127.0.0.1:53603 (0 connections now open)

Mikael Jenkler

@julian No, that one i can try. I have now reverted all my dbs but i can try to setup the same stuff aging with v1.0.3 and my db.

I will try the ./nodebb reset -p command
I will check my mongodb logs!

It would be nice to solve this

Edit1: Dev setup with my db and v1.x.x is now done. Will wait for first crash. Also looking in mongodb.log
Edit2: Don know why but i get 4 auth instead of 1 when the site auths to the db. Normal ? My v1.0.2 sites only get 1 every time
2016-05-30T11:53:10.080+0200 [conn75] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T11:53:10.085+0200 [conn76] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T11:53:10.087+0200 [conn77] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T11:55:13.907+0200 [conn78] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
Edit3: First crash, nothing in mongodblogs and this in nodebb dev
30/5 11:51 [68] - warn: [plugins] Expected method for hook 'filter:admin.header.build' in plugin 'nodebb-plugin-youtube-lite' not found, skipping.
30/5 11:52 [68] - verbose: [translator] No resource file found for sv/markdown, using provided fallback language file
30/5 11:52 [68] - warn: Route requested but not found: /vendor/jquery/timeago/locales/jquery.timeago.sv-short.js?_=1464602028233
info: Imgbed: cache miss
info: Imgbed: cache miss
info: Imgbed: cache miss
info: Imgbed: cache miss
info: Imgbed: cache miss
info: Imgbed: cache miss
...
Edit4 Running ./nodebb reset -p and start nodebb dev again
Edit5 Crash again. No errors no nothing Running ./nodebb reset -a and start nodebb dev again
Edit6 To many db connections if you ask me Something seams wrong here!
2016-05-30T12:20:34.571+0200 [conn115] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T12:20:34.575+0200 [conn116] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T12:20:34.579+0200 [conn117] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T12:20:34.589+0200 [conn118] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T12:20:34.592+0200 [conn119] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T12:20:34.634+0200 [conn120] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
2016-05-30T12:20:38.577+0200 [conn121] authenticate db: dev_nodebb_se { authenticate: 1, user: "dev_nodebb_se", nonce: "xxx", key: "xxx" }
Edit: Crash again and nothing more to test. Downgrading to v1.0.2 and see if it works again
Edit: If I downgrade to v1.0.2 and restore the plugins nodebb crashes. This mean that the upgrade over v1.0.2 fucks your DB in some way. Now i have a broken v1.0.2. Not that my original v1.0.2 works fine on the same system. This is a major BUG!

Something is wrong, but i cant find out what! More stuff i can try?

Diego Bao

I'm having the same problem as well.

I have checked all the things people has mentioned in this topic and several ones.

Disabling plug-in
Check memory (I have 1GB and 2GB of swap memory)
no errors in nodebb logs and mongodb logs.

Anything I tried is not working.

If the problem is with version 1.0.3, is there a way to downgrade to 1.0.2 without loosing data?

Thanks.

julian

Are any of you up for running git-bisect to determine which commit started causing these issues?

https://git-scm.com/docs/git-bisect

Because it only seems to occur on your instances, we cannot reproduce it and find a fix

We're running master branch here, with no issues.

That said, do these errors occur on master?

Mikael Jenkler

Yes, same issue at master!

Diego Bao

@julian I have installed nodebb less than a couple of weeks ago. I got master branch. Perhaps this only happens with new installations?

At this point the forums is useless sometimes it freezes after half an hour, sometimes it runs no problem for a few hours or a whole night.

If I somehow can help you diagnose the problem, tell me what I can do.

Mikael Jenkler

I am updating my system right now and wondering what nodejs version is recommended. I see now that I use an old version of nodejs.

0.12.6 <-- this is my old version
4.4.1 <-- this is the one I will update to

Release: Gentoo

Maybe thats the reason I will try it out in a few days!

@Diego-Bao Yes, indeed. > v1.0.2 is pretty useless hehe. Trashcan next in my case. Must find the issue And NO, this has nothing todo with new installation. I having trouble with all my old installations. Upgrade breaks the db! And it seams to be broken even if i downgrade so plz backup before upgrade !

julian

Can either of you send a database dump to me? (my username @ nodebb.org)

Perhaps it will reproduce on my local environment... but without traffic, I doubt it...

Mikael Jenkler

Yes, I will try to use nodejs 4x first and if that dont work i then i can send files and db dump to you Traffic is not the issue in my case, no traffic but it still hangs

Diego Bao

@julian Sorry I previously said I was running on master but I'm running on v1.x.x branch

Mikael Jenkler

I have it up and running again at v1.x.x branch

Mongo 2.6
Nodejs 4.4.1

Lets see how long it takes before hell breaks out again

Edit: Still running but even if this does not work its recommended to upgrade to Nodejs 4.4.1 Seams a lot faster

Diego Bao

@Jenkler it shouldn't take long lately it just takes me half an hour to break.

Mikael Jenkler

Wiiii, it seams that the problem is solved for me. I will test more today but no crash since yesterday.

The issue was with Nodejs 0.12.6, I upgraded to Nodejs 4.4.1 and now it works.

Thanks for all input I love NodeBB !

Diego Bao

I upgraded to Nodejs 4.4.5 and running Nodebb v1.0.3 (master). in less that 30 mins it froze again.

Diego Bao

I completely uninstall it and did a clean instalation of the code (left the dB alone). Installed version v1.0.2 and so far it's running fine.

Only thing I noticed is that the notifications number in my user shows 1, when I don't have any notification. I mark all as read, but if I refresh the page I get 1 again. I will keep updating here if I have more issues.

Mikael Jenkler

Try 4.4.1 it works for me with v1.x.x

Diego Bao

The problem with the user notification might be the issue I'm having. If I click to see the notification or unread messages the forum crashes and restarts on its own. But this only happens with my user.
Well @julian has my DB, so he probably can figure it out what it's happeneing.