Very weird behaviour when hosting multiple instances


  • Hey, guys!

    I'm currently facing very weird behaviour when hosting multiple instances of NodeBB. My setup is as follows, I'm running Arch Linux:

    • 1 account to host NodeBB under
    • 3 folders for 3 different NodeBB installations
    • 3 different Redis databases
    • 3 different ports used
    • 3 different "secrets" used

    Now, when I send a chat message to a friend, it will also appear on all other hosted forums. Same with notifications. What the heck happened here? 😮

    ~Netto


  • The first thing that comes in my mind (I am sure that I'm wrong), is maybe due to sockets?

  • NodeBB

    Can you post your config.json files for each instance for reference.


  • @baris Sure thing!

    Instance 1

    {
    "base_url": "https://hanasu.me",
    "port": "2339",
    "secret": "such_secret (I changed this)",
    "bind_address": "0.0.0.0",
    "database": "redis",
    "secondary_database": "none",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "password": "very password",
        "database": "0"
    },
    "bcrypt_rounds": 12,
    "upload_path": "/public/uploads",
    "use_port": false,
    "relative_path": ""
    }
    

    Instance 2

    {
    "base_url": "http://unser-schleswig.de",
    "port": "2340",
    "secret": "very_secret (changed again)",
    "bind_address": "0.0.0.0",
    "database": "redis",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "password": "wow",
        "database": "1"
    },
    "bcrypt_rounds": 12,
    "upload_path": "/public/uploads",
    "use_port": false,
    "relative_path": ""
    }
    

    Instance 3

    {
    "base_url": "http://37.59.31.162",
    "port": "4567",
    "secret": "do I have to do that again?",
    "bind_address": "0.0.0.0",
    "database": "redis",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "password": "-",
        "database": "2"
    },
    "bcrypt_rounds": 12,
    "upload_path": "/public/uploads",
    "use_port": true,
    "relative_path": ""
    }
    

    Never touched the config after installation. Just followed the docs and thats it.

  • NodeBB

    Your configs look good, when you get a chat notification on the other instances do you actually get the chat message? If you reload the page and open up your chats do you see chat messages from users on other forums? If not then this might be isolated to socket io and the redis store we use to pub/sub socket messages across worker processes.


  • Yes, the chat messages are actually delivered on other instances. After reload, the chat messages are not displayed any more.

    This is @Master-Antonio 's forum with a chat between him and me. He has UID #1 and I got #3.
    chat1.png

    This is my forum where I got UID #1 and it seems as I'm chatting with myself.
    chat2.png

  • Gamers

    @Netto-Hikari
    Yes, i confirm.
    This is big Problem.
    And i confirm if reload the page message not appears.

  • NodeBB

    So the messages are not actually saved into the wrong database, its just socket io seems to deliver it to all the instances running on the same redis instance even though they use different database ids.


  • @baris That's correct.

  • NodeBB

    From redis docs

    Database & Scoping
    Pub/Sub has no relation to the key space. It was made to not interfere with it on any level, including database numbers.
    Publishing on db 10, will be heard on by a subscriber on db 1.
    If you need scoping of some kind, prefix the channels with the name of the environment (test, staging, production, ...).
    

    So at this point nodebb wont work with 3 instances using the same redis instance on different databases.

    You can however have 3 redis instances on different ports ie (6379, 6380, 6381 so on) and use db 0 for all of them.


  • @baris Okay, then I'll do that. 🙂 Is there a way to easily dump and restore a single database from a redis instance?

  • NodeBB

    I think you will have to get a copy of your *.rdb file and copy it to all redis instances. I don't know a quick way of moving db 1 to db 0 etc.

  • NodeBB

    Also looks like this might be a non issue when we upgrade to socket io 1.0+ and use the redis-adapter seems like it supports prefixes https://github.com/Automattic/socket.io-redis/blob/master/index.js#L48


  • @baris Thank you so much for your help. I'm running 3 instances of redis now and it seems to be working.

Suggested Topics

  • 11
  • 2
  • 10
  • 4
  • 6
| |