Again: «Looks like your connection to NodeBB was lost, please wait while we try to reconnect.» - in Docker Swarm

mwaeckerlin

I have exactly the same problem as many other and this is really annoying. Everything is setup correctly, but still does not work at all. I read all posts and still did not find a solution.

You can try at: forum.mrw.sh — feel free to register, I'll reset the database as soon as the problem is fixed.

I have the following setup:

docker swarm
an nginx reverse proxy in a docker container, running mwaeckerlin/reverse-proxy
a docker container running nodebb/docker

So user enters https://forum.mrw.sh, which is directed to container mwaeckerlin/reverse-proxy running on swarm master host named jupiter, listening on external port 443, which is port 8443 in the container. Then nginx redirects to http://jupiter:8036 which is redirected by the docker swarm lead master to the docker container nodebb/docker which listens on external port 8036, which is port 4567 in the container.

So, what exactly is the url, that must be entered in config.json?!? I suppose the url as it is visible from outside, which is https://forum.mrw.sh?

Documentation of nodebb/docker is extremely bad and incomplete! Especially the volumes that must be persistent are not specified! As far as I have seen, these are the config.json file and the upload directory.

First, I have a problem: The docker image has a chickem-egg-problem with the config file: The config file should be mounted into the container, but it is created in the container and creating an empty config file at startup fails. Better solution: specify a configuration directory and mount the whole directory. But this means to specify an alternate directory, but that does not work with the ./nodebb script! Also, calling node src/cli --config /etc/nodebb/config.json start does not work, first it must be built.

So I had to change the docker command to: /bin/bash -c "node src/cli --config /etc/nodebb/config.json bui
ld && node src/cli --config /etc/nodebb/config.json start"

Next question is: What will happen on nodebb updates?

Anyway, this is the configuration:

Docker:

version: '3.3'
services:

  mongodb:
    image: mongo
    volumes:
      - type: bind
        source: /srv/volumes/forum-mrw-sh/mongodb
        target: /data/db
    deploy:
      resources:
        limits:
          memory: 1G

  nodebb:
    image: nodebb/docker
    ports:
      - 8036:4567
    labels:
      - 'url=https://forum.mrw.sh'
    volumes:
      - type: bind
        source: /srv/volumes/forum-mrw-sh/nodebb
        target: /etc/nodebb
    environment:
      - CONFIG=/etc/nodebb/config.json
    command: /bin/bash -c "node src/cli --config /etc/nodebb/config.json build && node src/cli --config /etc/nodebb/config.json start"
    deploy:
      resources:
        limits:
          memory: 1G

config.json:

{
    "url": "https://forum.mrw.sh",
    "secret": "****",
     "database": "mongo",
     "port": 4567,
     "mongo": {
         "host": "mongodb",
         "port": "27017",
         "database": "nodebb"
     }
}

Nginx:

map $http_accept_language $lang {
  default en;
  ~*^de de;
}
server { # redirect http to https
  listen 80;
  server_name forum.mrw.sh;
  server_name www.forum.mrw.sh;
  location /.well-known {
      alias /acme/.well-known;
  }
  location / {
    return 302 https://forum.mrw.sh:443$request_uri;
  }
}
server { # redirect www to non-www
  listen 443 ssl http2;
  server_name www.forum.mrw.sh;
  add_header Strict-Transport-Security max-age=15552000 always;
  return 302 $scheme://forum.mrw.sh:443$request_uri;
  ssl on;
  ssl_certificate /etc/letsencrypt/live/forum.mrw.sh/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/forum.mrw.sh/privkey.pem;
}
server {
  listen 443 ssl http2;
  server_name forum.mrw.sh;
  add_header Strict-Transport-Security max-age=15552000 always;
  ssl on;
  ssl_certificate /etc/letsencrypt/live/forum.mrw.sh/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/forum.mrw.sh/privkey.pem;
  error_page 502 /502.html;
  error_page 504 /504.html;
  error_page 404 /404.html;
  location ~ ^/(502|504|404)\.html$ {
    root /etc/nginx/error/$lang;
  }
  location ~ ^/(502|504|404)\.jpg$ {
    root /etc/nginx/error;
  }
  location / {
    include proxy.conf;
    if ($request_method ~ ^COPY$) {
      rewrite /(.*) /$1 break;
    }
    proxy_cookie_domain jupiter forum.mrw.sh;
    proxy_pass http://jupiter:8036/;
    proxy_redirect off;
  }
  location /.well-known {
      alias /acme/.well-known;
  }
}

proxy.conf:

proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
set $ssl off;
if ($scheme = https) {
    set $ssl on;
}
proxy_set_header X-Forwarded-Ssl $ssl;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Accept-Encoding "";
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Original-Request $request_uri;
proxy_pass_request_headers on;
#proxy_cache off;
#proxy_buffering off;
client_max_body_size 4096m;
client_body_buffer_size 128k;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 86400;
send_timeout 600;
proxy_buffers 32 4k;
#subs_filter_types text/css text/javascript text/xml;
set $fixed_destination $http_destination;
if ( $http_destination ~* ^https(.*)$ ) {
  set $fixed_destination http$1;
}
proxy_set_header Destination $fixed_destination;

proxy_ssl_verify off;

# WebSocket proxying
# http://nginx.org/en/docs/http/websocket.html
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

# referrer-policy
add_header "Referrer-Policy" "no-referrer";

And in the Log:

NodeBB v1.10.2 Copyright (C) 2013-2014 NodeBB Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions.
For the full license, please visit: http://www.gnu.org/copyleft/gpl.html

Clustering enabled: Spinning up 1 process(es).

2018-11-23T10:58:24.570Z [65] - info: Initializing NodeBB v1.10.2 https://forum.mrw.sh
2018-11-23T10:58:24.735Z [65] - warn: You have no mongo username/password setup!
2018-11-23T10:58:30.466Z [65] - warn: You have no mongo username/password setup!
2018-11-23T10:58:30.490Z [65] - info: [socket.io] Restricting access to origin: https://forum.mrw.sh:*
2018-11-23T10:58:31.929Z [65] - info: Routes added
2018-11-23T10:58:31.933Z [65] - info: NodeBB Ready
2018-11-23T10:58:31.943Z [65] - info: Enabling 'trust proxy'
2018-11-23T10:58:31.949Z [65] - info: NodeBB is now listening on: 0.0.0.0:4567

So: What's the problem?

Principially, socks.js and node.js works in my projects in the same environment, but I never restricted access to origin for socks.js.

PitaJ

@mwaeckerlin have you tried a very cut down nginx config which follows the ones at docs.nodebb.org

mwaeckerlin

@PitaJ, I can't arbitrarily change the ngnix configuration, because it is part of a docker container that serves a huge reverse proxy which handles ~50 different services. All changes apply to all services. All other services work perfectly — including some running on node.js with sockets. But everything mentioned in your cod should have been applied, if I checked it right.

Tom Rade

The outside URL https://forum.mrw.sh is the one that should be inside config.json.

And the directory /etc/nodebb doesn't exist normally with the Docker image. The default, official docker image uses /usr/src/app.

PitaJ

@mwaeckerlin there are certain options which are dependent on the order in which they're applied in the config. I'd recommend trying to cut down the config, at least that pertinent to the NodeBB forum, to the SSL config at the following link

NginX - NodeBB Documentation

Documentation portal for NodeBB Forum Software

(docs.nodebb.org)

mwaeckerlin

@Tom_Rade said in Again: «Looks like your connection to NodeBB was lost, please wait while we try to reconnect.» - in Docker Swarm:

The outside URL https://forum.mrw.sh is the one that should be inside config.json.

And the directory /etc/nodebb doesn't exist normally with the Docker image. The default, official docker image uses /usr/src/app.

As you see in my YAML, I create /etc/nodebb, as I explain above, this is because it is not possible to mount single file as volume when this file has to be created at first run. And as you also see, I had to replace the command.

mwaeckerlin

@PitaJ, what options depend on the order? I checked all lines in NginX - NodeBB Documentation and they all should be in my configuration too. I even enhanced my docker image so, that now line proxy_redirect off is possible (before that was the only difference). But with no result.

Is there any way to debug? It seems that your NodeBB generates a 403, so is there any possibility to debug why a 403 is generated, what options/parameters does NodeBB get?

Has anyone ever tried the combination to run NodeBB in a docker swarm environment behind a reverse-proxy?

mwaeckerlin

In the code I see nconf. This would help to override some configurations. But I didn't find any documentation on how to use nconf. How can I set configuration options using nconf?

I found this code in src/socket.io/index.js:

        if (process.env.NODE_ENV !== 'development') {
                var domain = nconf.get('cookieDomain');
                var parsedUrl = url.parse(nconf.get('url'));
                var override = nconf.get('socket.io:origins');
                if (!domain) {
                        domain = parsedUrl.hostname;    // cookies don't prov
ide isolation by port: http://stackoverflow.com/a/16328399/122353
                }

                if (!override) {
                        io.origins(parsedUrl.protocol + '//' + domain + ':*')
;
                        winston.info('[socket.io] Restricting access to origin: ' + parsedUrl.protocol + '//' + domain + ':*');
                 } else {
                        io.origins(override);
                        winston.info('[socket.io] Restricting access to origin: ' + override);
               }
   }

By now, I configured NODE_ENV=development, this solves my problem, but is only a workaround.

mwaeckerlin

See: https://socket.io/docs/server-api/

It is possible to pass a function instead of only an url to io.origins. This way, you can trace before you reject the access. That helps all of your users to debug the problem. I suggest that you change your code accordingly.

I'll fork and provide a patch…

mwaeckerlin

Ok, I traced down the problem:

I changed the socket origin configuration to:

		var originUrl = override ? override : parsedUrl.protocol + '//' + domain;
		winston.info('[socket.io] Restricting access to origin: ' + originUrl);
		io.origins((origin, callback) => {
			if (origin.startsWith(originUrl)) {
				return callback(null, true);
			} else {
				winston.error('[socket.io] rejecting origin: ' + origin);
				winston.error('[socket.io] expected origin: ' + originUrl);
				return callback('origin not allowed', false);
			}
		})

This is the error I get:

2018-11-24T10:40:41.056Z [66] - error: [socket.io] rejecting origin: *
2018-11-24T10:40:41.056Z [66] - error: [socket.io] expected origin: http://forum.mrw.sh

That means: The problem is not the configuration, the problem is detected origin!

I'll prepare a patch to enable NodeBB in a docker environment, including a Dockerfile that works.

mwaeckerlin

Ok, I fixed the problem and created a pull request. Please accept it as soon as possible.

This is then a working Docker Swarm compose file:

version: '3.3'
services:

  mongodb:
    image: mongo
    volumes:
      - type: bind
        source: /srv/volumes/forum-mrw-sh/mongodb
        target: /data/db

  nodebb:
    image: mwaeckerlin/nodebb
    ports:
      - 8036:4567
    volumes:
      - type: bind
        source: /srv/volumes/forum-mrw-sh/nodebb/config
        target: /usr/src/app/config
      - type: bind
        source: /srv/volumes/forum-mrw-sh/nodebb/uploads
        target: /usr/src/app/public/uploads

The image here is mwaeckerlin/nodebb, as soon as my fix has been pulled into your repository, that can be changed to nodebb/docker.

In the online-setup, chose MongoDB and thet the db-url to mongodb.

Tom Rade

@mwaeckerlin The way I solved it was to mount /usr/src/app/temp, then start the server, move all files to the temporary "temp" directory, then stop the server and change the mount point to /usr/src/app. That worked quite nicely.

I think all your plugins and some settings will get deleted once you restart the server if you only mount config and uploads.

mwaeckerlin

@Tom_Rade, mounting, stopping, moving, moving mountpoint is not the way how a proper docker installation should work.

Could you give me a list of all files that should remain persistent?

Yes, in my nextcloud docker image I had to persist the apps (=plugins) path too

PitaJ

@mwaeckerlin socket origin is restricted to prevent what is called "cross site websocket hijacking". This is when another site connects to the socket server and acts as if it's the NodeBB client.

The reason your detected origin is wrong is almost certainly a configuration issue, most likely in your reverse proxy setup. This is why I suggest trying a cut down nginx config, as nginx is responsible for passing down the origin header. proxy_redirect off is one of the order-dependent options.

But I didn't find any documentation on how to use nconf

nconf options are set within config.json, documented here:

config.json - NodeBB Documentation

Documentation portal for NodeBB Forum Software

(docs.nodebb.org)

Dev mode sets this to "*" which enables connections from anywhere.

Tom Rade

@mwaeckerlin I just did the entire /usr/src/app directory, not sure if that's correct - let me know if you think otherwise.

mwaeckerlin

@Tom_Rade, no, that's not correct. Docker (and clean programming) require a strict separation between code and data.

mwaeckerlin

@PitaJ, the problem is, that not the url setting is *, but the origin that arrives in socket.io.

With a reverse proxy, I can handle same origin in nginx, centralized before it passes through to NodeBB, even more: NodeBB always gets all requests from only one source, from the reverse proxy server. So I suggest that the same origin feature should be optional: On by default, but it should be possible to disable it in reverse proxy environments.

I see that the origin reported from socket.io is *, but I don't understand yet, why and how.

mwaeckerlin

@PitaJ, exactly which option in the suggested nginx configuration passes the origin to the backend NodeBB?

I want to understand. Up to now, everything sounds like voodoo, such as «certain options which are dependent on the order» or «almost certainly a configuration issue». So what exactly is the problem? Which options are responsible for setting the origin? Which options depend in the order? What is why the correct order? AFAIK, order is not relevant for all options. I would like to have an educated understanding of what happens and why, not something that works by coincidence.

PitaJ

@mwaeckerlin said in Again: «Looks like your connection to NodeBB was lost, please wait while we try to reconnect.» - in Docker Swarm:

@PitaJ, the problem is, that not the url setting is *, but the origin that arrives in socket.io.

Yes I understood your problem.

With a reverse proxy, I can handle same origin in nginx, centralized before it passes through to NodeBB, even more: NodeBB always gets all requests from only one source, from the reverse proxy server. So I suggest that the same origin feature should be optional: On by default, but it should be possible to disable it in reverse proxy environments.

Did you go to the link? You can configure the socket.io origins by adding the following to config.json:

"socket.io": {
  "origins": "*"
}

Them you can add the same origin check to nginx.

I see that the origin reported from socket.io is *, but I don't understand yet, why and how.

I'm no nginx wizard either. It's pretty much voodoo to me as well. That's why I suggested trying a cut down config. If a config based closely on the one in our docs worked, then it's likely possible to figure out what the issue is based on differences between them.

I don't have enough experience to diagnose exactly what is wrong.

exactly which option in the suggested nginx configuration passes the origin to the backend NodeBB?

I want to understand. Up to now, everything sounds like voodoo, such as «certain options which are dependent on the order» or «almost certainly a configuration issue». So what exactly is the problem? Which options are responsible for setting the origin? Which options depend in the order? What is why the correct order? AFAIK, order is not relevant for all options. I would like to have an educated understanding of what happens and why, not something that works by coincidence.

I don't know enough about nginx to answer these questions. I just know that our config does work, and yours doesn't.

One more thing: none of us core devs know much about docker, which is why the documentation and defaults aren't very good for it.

mwaeckerlin

Ok, I suggest two thing:

I'll help you to elaborate a Dockerfile that works fine and stable, also in swarm environments (next step will be OpenShift and upgrades).
It would be helpful to add logging information when socket.io rejects a connection, i.e. log the detected vs the configured origin.

Agree, @PitaJ, @Tom_Rade?