Power Off Droplet on DigitalOcean and save dump.rdb?

SimonQ

I want to create a snapshot of my droplet on digital ocean but, in order to do that I have to power off my droplet.

At first, I thought redis saved my dump.rdb file automatically but, I turned it off without doing anything in the redis command line.

This caused me to lose some of my settings that I had previously installed and had to re-work it all again.

Do I just run bgsave in redis-cli and then turn off the droplet to create my snapshot?

Don't want to risk losing all of my data again before powering it off.

I am using an Ubuntu server on the $5 tier offered by D.O.

julian

@SimonQ Hey there -- some things to note about Redis

By default:
- It persists the database to disk within 15 minutes if at least 1 key has changed
- 5 minutes if 10 keys have changed
- 1 minute if 10,000 keys have changed
During regular system shutdown, Redis will immediately persist the in-memory database to disk.
However -- If you hit "power off" in Digital Ocean, this is the same as hitting the power button on your computer. Redis will not have time to persist the database to disk.

So in the future, just shut down your droplet with sudo shutdown -h now

SimonQ

@julian Thanks for the reply

I ran sudo shutdown on the server to power it off and ended up losing the data. Does sudo shutdown -h now make any difference vs what I did?

Also, is there a way to change the default settings so that the database can save faster?

Edit: Before running sudo shutdown I was constantly receiving this error that prevented me from logging in:

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled.

I ran config set stop-writes-on-bgsave-error no and that seemed to fix the problem.

Is that my problem at the moment and why it's not saving? How would I go about fixing the MISCONF error?

julian

@SimonQ Yes that's a red flag...

Assuming the dump file is /var/lib/redis/dump.rdb, who owns that? (Run ls -l to find out)

SimonQ

@julian I redirected my dump so it is currently pointing to my root folder.

The owner is me simon and it is a sudo user. Do I have to change this file? If so, how do I go about doing that?

I checked the default dump.rdb in /var/lib/redis/dump.rdb and owner is redis. How would I change it to that?

julian

If the dump file redis is configured to use is not owned by redis, or if it's permissions are too restrictive, then redis will not be able to write to it.

SimonQ

@julian I changed the ownership/group and permissions to follow that of the default dump.rdb in /var/lib/redis/dump.rdb using sudo chown redis dump.rdb and it loaded fine after I restarted nodebb. However, it does not seem to be reading the font awesome icons and I can not click my profile to to see the menu dropdown from there. It works on my mobile though.

Is there a way to test if my redis database can persist to the disk now?

Edit: Just tried logging into my mobile and am still greeted with the same error:

Failed login attempt, please try again.
MISCONF Redis is configured to save RDB snapshots, but is currently not able to press on disk. Commands that may modify the data sets are disabled. Please check Redis logs for details about the error

I then did ./nodeebb log and everything seems to be working with the exception of some warnings from of a few plugins.

10/7 10:50 [19053] - warn: [plugins/nodebb-plugin-dbsearch] This plugin may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing.
10/7 10:50 [19053] - warn: [plugins/nodebb-plugin-dbsearch] In the event of an unresponsive NodeBB caused by this plugin, run ./nodebb reset plugin="nodebb-plugin-dbsearch".

10/7 10:50 [19053] - warn: [plugins/nodebb-theme-lavender] This plugin may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing.
10/7 10:50 [19053] - warn: [plugins/nodebb-theme-lavender] In the event of an unresponsive NodeBB caused by this plugin, run ./nodebb reset plugin="nodebb-theme-lavender".

10/7 10:50 [19053] - info: [plugins/spam-be-gone] Settings loaded
10/7 10:50 [19053] - info: NodeBB Ready
10/7 10:50 [19053] - info: NodeBB is now listening on: 0.0.0.0:4567

SimonQ

@julian

I just checked /var/log/redis/redis-server.log and although I have changed the ownership and group to redis and can confirm that it's not saving the .rdb with permission denied still:

933:M 10 Jul 11:42:00.054 * 1 changes in 900 seconds. Saving...
933:M 10 Jul 11:42:00.054 * Background saving started by pid 19861
19861:C 10 Jul 11:42:00.055 # Failed opening .rdb for saving: Permission denied
933:M 10 Jul 11:42:00.155 # Background saving error

The dump.rdb that redis is configured to use has the following that I recently changed to:

from:
-rw-rw---- 1 simon simon 36296 Jul 8 16:50 dump.rdb

to:
-rw-rw---- 1 redis redis 36296 Jul 8 16:50 dump.rdb

Is there another reason that the dump.rdb file is not saving?

SimonQ

@julian I think I managed to solve this problem, not 100% though

I changed the permission to 755 for my working root directory and the dump.rdb permissions to 644. I also changed the ownership of my working root directory to redis as well.

Upon running redis-cli bgsave and then checking the redis-server.log I am greeted with this:

933:M 10 Jul 12:52:17.908 * Background saving started by pid 20754
20754:C 10 Jul 12:52:17.917 * DB saved on disk
20754:C 10 Jul 12:52:17.918 * RDB: 0 MB of memory used by copy-on-write
933:M 10 Jul 12:52:17.946 * Background saving terminated with success

It says 0MB of memory used by copy-on-write? Does this mean it saved?

julian

Looks like you're good to go Hooray for problem solving!

SimonQ

@julian I got it to work however when I restarted the server I was greeted with an error upon starting nodebb up again. I found that the error was the pidfile was rewriting as root and I keep getting ./pidfile rewrite: operation denied or something similar

more importantly when I try to do ./nodebb stop I get

Stopping NodeBB. Goodbye!
./nodebb: line 43: kill: (1843) - Operation not permitted

but works when I do sudo ./nodebb stop same with sudo ./nodebb start

Is there a way to change that? This only happened since I changed the directory of the root folder to redis in order to persist to the DB.

julian

There's no need to keep the redis dump file in with your NodeBB install. Keeping it where it belongs (/var/lib/redis) is best.

Then you can reset your NodeBB directory's permissions

SimonQ

@julian This may sound silly but, if I were to place the dump.rdb file back into /var/lib/redis (from my NodeBB install) and reset my server from there would anything horrible go wrong?