Redis database lost somehow 17 days worth of forum data

Pilvinen

I just lost 17 days worth of forum data when I rebooted the server machine. I did have backups from yesterday because of the bug from yesterday so I only lost 1 days worth of data in the end ... but how is this even possible?

nodebb stop
cd /var/lib/redis
redis-cli -> shutdown (as root)
redis-cli -> quit (as root)
/sbin/reboot
cd /var/lib/redis
redis-server & (as root)
./nodebb start

And... all the sudden the forum database reverted back 17 days. WHAT?! Where the F* did it even GET that old data?

I'm getting royally pissed off how unreliable this whole setup with Nodebb is. I can't even do a simple reboot without everything getting f'd up.

WHAT?! Help!

Pilvinen

Actually my guestion is: how do I prevent this from happening again?

frissdiegurke

I'm not an expert on redis, but this topic seems like it may be helpful (I guess the main points ain't outdated): https://community.nodebb.org/topic/932/redis-useful-info

But since he quotes

Otherwise make sure to shutdown the database using the SHUTDOWN command:
$ redis-cli shutdown
This way Redis will make sure to save the data on disk before quitting.

I guess you did the shutdown correctly.

For further help the others (those who know more about redis than me) you may need to show your redis-configuration.

EDIT: Otherwise you may want to consider using mongodb instead.

lenovouser

@Pilvinen

You need to make the storage persistent. You do that by putting that in your /etc/redis/redis.conf:

save 60 1000

This saves your DB every 60 seconds if you change more than 1000 keys. If you want to be really careful you can change that to lower values like save 30 500 etc.

You should also enable the append-only file

appendonly yes

From now on, every time Redis receives a command that changes the data it will append it to the AOF. When you restart Redis it will re-play the AOF to rebuild the state.

If you want to know more you can go to:

redis.io/documentation and
redis.io/topics/persistence

julian

@Pilvinen Redis uses fairly sane defaults for persistence, so if your on-disk database is stale by 17 days, then Redis isn't doing it's job (or the config was altered and persistence was turned off).

Do you get errors when Redis starts up? If it can't persist to disk, it will complain loudly (yet silently) to the log file at /var/log/redis

Charly86

@julian
good to know this useful information, I checked on mine (not changed any redis db config when installed nodebb) so in doubt I checked, here the log file, sounds to works fine and default are

save 900 1
save 300 10
save 60 10000

and logs

[574] 17 Sep 20:02:36.228 * Background saving terminated with success
[574] 17 Sep 20:07:37.008 * 10 changes in 300 seconds. Saving...
[574] 17 Sep 20:07:37.009 * Background saving started by pid 16535
[16535] 17 Sep 20:07:37.196 * DB saved on disk
[16535] 17 Sep 20:07:37.197 * RDB: 0 MB of memory used by copy-on-write
[574] 17 Sep 20:07:37.210 * Background saving terminated with success
[574] 17 Sep 20:12:38.037 * 10 changes in 300 seconds. Saving...
[574] 17 Sep 20:12:38.039 * Background saving started by pid 16538
[16538] 17 Sep 20:12:38.227 * DB saved on disk
[16538] 17 Sep 20:12:38.228 * RDB: 1 MB of memory used by copy-on-write
[574] 17 Sep 20:12:38.239 * Background saving terminated with success
[574] 17 Sep 20:17:39.093 * 10 changes in 300 seconds. Saving...
[574] 17 Sep 20:17:39.094 * Background saving started by pid 16543
[16543] 17 Sep 20:17:39.308 * DB saved on disk
[16543] 17 Sep 20:17:39.310 * RDB: 0 MB of memory used by copy-on-write
[574] 17 Sep 20:17:39.395 * Background saving terminated with success
[574] 17 Sep 20:22:40.096 * 10 changes in 300 seconds. Saving...
[574] 17 Sep 20:22:40.097 * Background saving started by pid 16550
[16550] 17 Sep 20:22:40.276 * DB saved on disk
[16550] 17 Sep 20:22:40.277 * RDB: 0 MB of memory used by copy-on-write
[574] 17 Sep 20:22:40.298 * Background saving terminated with success