Stuck at "Delete Sessions".

[Haitch]

I've a 1.4.5 instance of NodeBB that I tried previously tried to upgrade to 1.4.6 and again today tried to upgrade to 1.5.0. In both instances I ran into the same issue. During the schema update it gets to "Delete Sessions" and never moves beyond there. The CPU is running at 100%, but there appears to be no disk activity. I've left it running for over an hour and 0 progress past that step.

Any ideas?

1. Bringing base dependencies up to date... OK
2. Checking installed plugins for updates... OK
3. Updating NodeBB data store schema...
   28/5 15:22:11 [2428] - info: Beginning database schema update
   28/5 15:22:11 [2428] - info: [2016/09/22] Setting category recent tids - skipped!
   28/5 15:22:11 [2428] - info: [2016/10/8] favourite -> bookmark refactor - skipped!
   28/5 15:22:11 [2428] - info: [2016/10/14] Creating sorted sets for post replies - skipped!
   28/5 15:22:11 [2428] - info: [2016/11/22] Update global and user language keys - skipped!
   28/5 15:22:11 [2428] - info: [2016/11/25] Creating sorted sets for pinned topics - skipped!
   28/5 15:22:11 [2428] - info: [2017/2/25] Update global and user sound settings - skipped!
   28/5 15:22:11 [2428] - info: [2017/2/28] Update urls in config to /assets - skipped!
   28/5 15:22:11 [2428] - info: [2017/4/16] Delete sessions
   ^C

[teh_g]

@Haitch I would let it run, the sessions may take a while to clean up.

[PitaJ]

If you skip v1.4.6 and attempt to upgrade straight to v1.5.0 it should show the progress. This script only takes a significant amount of time if you're using redis as your sessions store. That is, redis as your only database, or redis in conjunction with mongodb. Since Redis is an in-memory db, that's why you see no disk activity.

If you really want to speed things up, I suggest first backing up your redis database, and, if you're only using it for session storage, then you can drop the entire db

[Haitch]

@PitaJ It is a redis database. It never progressed past the "Delete Sessions" stage though - what is a reasonable amount of time to expect that stage to take? Previous upgrades have never spent any significant time in the Schema Upgrade.

[PitaJ]

@Haitch it depends on how many sessions you have stored. Like I said, skip v1.4.6 if you want to see the progress.

[julian]

Depending on how many users you have and your general page load, you could have a LOT of sessions clogging up things... it can easily be longer than other upgrade scripts.

As @pitaj says you can go straight to latest and see progress bar.

[Haitch]

I'm running the upgrade (to 1.5.0) on a clone of the VM, it's been sitting at "Deleting accidently long-lived sessions" for over 2 hours. No sign of a progress bar. Just that message. Is there anything that can be done prior to running the upgrade? I really don't want to take my site down for this long.

[PitaJ]

@Haitch can you take a screenshot? Are you using mongo+redis, or just redis?

[PitaJ]

@Haitch if it literally says Deleting instead of Delete then you aren't on v1.5.0

[Haitch]

@PitaJ Just redis. Went to grab a screenshot, and it core dumped .....

Restarting the upgrade.

[PitaJ]

@Haitch try counting the number of session keys by doing this:

./redis-cli KEYS "sess:*" | wc -l

[Haitch]

@PitaJ said in Stuck at "Delete Sessions".:

./redis-cli KEYS "sess:*" | wc -l

The clone finally did completely process the upgrade - so I guess I'm just going to have to bite the bullet and take my site down for as long as it takes.

As for the "redis-cli KEYS "sess:*" | wc -l" command: 5,379,194

What the heck could have caused that many "long-lived" sessions?

[PitaJ]

@Haitch there was a bug which caused the sessions to be stored for 1000x the time they were supposed to be. That's why we're clearing the sessions in the first place.

[Haitch]

@PitaJ said in Stuck at "Delete Sessions".:

@Haitch there was a bug which caused the sessions to be stored for 1000x the time they were supposed to be. That's why we're clearing the sessions in the first place.

that's definitely an oops - but been, there done that in my own code.

Guess it's just bite the bullet and do it. I imagine not having to track over 5 million phantom sessions should make the site more responsive .....

[PitaJ]

@Haitch it is possible to run the upgrade script while your forum is running, I think

[julian]

It's... possible, given the type of upgrade scripts left... but generally, we don't recommend doing so.

[Haitch]

okay, upgrade completed - took almost 3 hours to delete the sessions, took the site down to do it, now running at 1.5.0.

New problem, when uploading images my users are getting: "something went wrong trying to parse the server response" - the upload URL appears in the post, but no pic. eg:

[Haitch]

@Haitch 1.5.1 update doesn't resolve it - still unable to upload images.

[PitaJ]

@Haitch open an issue here: https://github.com/nodebb/nodebb/issues

[baris]

Does it happen on all browsers? I just uploaded an image here and it worked fine, this forum is running 1.5.1 as well.