Lost all my data ? Help please


  • Same issue here on a DigitalOcean hosting. The database has first been wiped out while running v0.8.2, then again last night while running v0.9.0.

  • Gamers

    Though it is likely unrelated, lots of Digital Ocean people involved 😕


  • What's amazing is that happening to several people in the same days interval on different redis version and nodeBB version without touching anything from days !!!

    I've backup up also every hour now !!!

    crontab - e then add

    # 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
    #
    # For more information see the manual pages of crontab(5) and cron(8)
    #
    # m h  dom mon dow   command
    MAILTO=""
    13 * * * *  /root/redis-backup.sh
    

    And /root/redis-backup.sh (grabbed from Internet, don't remember where and modified to fit my needs).

    dont forget to chmod ug+x /root/redis-backup.sh

    #!/bin/bash
    #
    ## redis backup script
    ## usage
    ## redis-backup.sh port backup.dir
    
    port=${1:-6379}
    backup_dir=${2:-"/var/lib/redis/backup"}
    
    cli="/usr/bin/redis-cli -p $port"
    rdb="/var/lib/redis/dump.rdb"
    
    test -f $rdb || {
      echo "[$port] No RDB Found" ; exit 1
    }
    test -d $backup_dir || {
      echo "[$port] Create backup directory $backup_dir" && mkdir -p $backup_dir
    }
    
    # perform a bgsave before copy
    echo bgsave | $cli
    echo "[$port] waiting for 5 seconds..."
    sleep 5
    try=10
    while [ $try -gt 0 ] ; do
      ## redis-cli output dos format line feed '\r\n', remove '\r'
      bg=$(echo 'info Persistence' | $cli | awk -F: '/rdb_bgsave_in_progress/{sub(/\r/, "", $0); print $2}')
      ok=$(echo 'info Persistence' | $cli | awk -F: '/rdb_last_bgsave_status/{sub(/\r/, "", $0); print $2}')
      if [ "$bg" = "0" ] && [ "$ok" = "ok" ] ; then
        dst="$backup_dir/$port-dump.$(date +%Y%m%d%H%M).rdb"
        cp $rdb $dst
        if [ $? = 0 ] ; then
          echo "[$port] redis rdb $rdb copied to $dst."
    
          # delete rdb created 30 days ago
          cd $backup_dir
          find . \( -name "$port-dump*" \) -mtime +30 -exec rm -f {} \;
          exit 0
        else
          echo "[$port] >> Failed to copy $rdb to $dst!"
        fi
      fi
      try=$((try - 1))
      echo "[$port] redis maybe busy, waiting and retry in 5s..."
      sleep 5
    done
    
    

    If it can help, it can be better and not in root folder, done in a hurry, you know what I mean 🙂


  • Is someone here can explain me where to find if I have a backup.. How can I just lose all my data without a reason ?? Please someone explain me what's could have happened to my website ? I lose a nine months of hard work !!!

    What's command I have to execute to try restoring my datas ?


  • @bitspook I don't want to throw out blind judgement here... Since I never used Digital Ocean. I've always used my own full scale dedicated. But does someone experience issue's with MongoDB and Digital Ocean? And like many people over NodeBB said, keep making backups.
    Edit: I'm also wondering on what scale database wipes occur. They seem to be pretty rare.

  • Gamers

    @Kowlin not that it couldnt be Digital Ocean; just that the correlation is anecdotal as of yet. 😉


  • @bitspook Who knows, maby its a blind curse 😉. Anyways is someone running more Node.JS apps or Redis databases on there servers? Did those get wiped at one point or another?

  • GNU/Linux Admin

    If you had Redis persisting to disk, it would be at /var/lib/redis.

    It sounds like someone (or some script, rather) is going around trying to connect to port 6379, and if successful, flushing the database.

    Better use AUTH and have good backups!!

  • GNU/Linux Admin

    @kacemlight This seems to be the case. I am able to connect to the redis server at nodejsworld.com:

    $ redis-cli -h nodejsworld.com
    nodejsworld.com:6379> keys *
    1) "sess:0d5qvchIPxvQ-lsGNTJkWL7s1SQBrd8v"
    2) "ip:recent"
    3) "schemaDate"
    

    Please take a look at this topic: http://redis.io/topics/security.

    I realise this is a very rough time for you, @kacemlight, as you've lost your site data. I have every confidence that you will be more careful in the future and utilise backups in addition to setting up authentication for your database.


  • @julian Always a posibility too be honest. I'll have some fun opening my Redis port tonight to my server. Not like I have critical data on there anyways.

  • GNU/Linux Admin

    @Kowlin I've found the easiest way is to just use ufw. If you only connect from one PC, then disallow requests from all IPs except your own, and set the default behaviour to DENY.


  • Well am I the only one a bit troubled about the fact that the default Redis security is so easy to avoid. I opened my Redis port from my router and connected to it without any single problem...

  • Community Rep

    @kacemlight said:

    Is someone here can explain me where to find if I have a backup.. How can I just lose all my data without a reason ?? Please someone explain me what's could have happened to my website ? I lose a nine months of hard work !!!

    What's command I have to execute to try restoring my datas ?

    Did you take a backup of any sort?

  • Gamers

    @julian @Kowlin Redis is that bad? Did not know that. Planned on migrating to Mongodb before production anyway 😄


  • @bitspook Like @julian said, security is retentively easy. But I'm surprised and scarred that its so weak by default. If it wasn't for my router I think my databases also could have been dropped.

  • Community Rep

    @kacemlight said:

    Is someone now why this happen ?!! Is it can be related to the hosting provider Digitalocean ?

    That would not make any logical sense. DO is only your IaaS platform and does not touch the system itself. You would not suspect a hardware vendor in a physical deployment of deleting your database.

  • GNU/Linux Admin

    @Kowlin said:

    Well am I the only one a bit troubled about the fact that the default Redis security is so easy to avoid.

    IIRC the default Redis config is "bind to 0.0.0.0", which means open to everyone. Might be the same with Mongo.

    However, Ubuntu locks these configs down by binding to localhost only, so installing via apt-get is usually better than installing by downloading and compiling on your own.

  • Community Rep

    @Kowlin said:

    Well am I the only one a bit troubled about the fact that the default Redis security is so easy to avoid. I opened my Redis port from my router and connected to it without any single problem...

    That is expected. That means that the firewall is open on your OS (is the firewall running?) and Redis does this because it is designed to be in a cluster and would need the port open to talk to itself or to a Sentinel.

  • Gamers

    @julian said:

    It sounds like someone (or some script, rather) is going around trying to connect to port 6379, and if successful, flushing the database.

    maliciously?

  • GNU/Linux Admin

    @bitspook Probably.

    Looks like antirez added a large warning block about it here, and it may be uncommented by default now...
    https://github.com/antirez/redis/blob/unstable/redis.conf#L40-L61

Suggested Topics

| |