400 and 522 errors since enabling SSL via Nginx proxy

KingCat

So everything was working great but due to the nature of my site and it's users, SSL is required, especially since I'll be using SSO exclusively for login. Once I enabled SSL I started receiving both 400 and 522 errors on a certain amount of the socket.io polls. I'm not certain what the percentage is of failed polling attempts but it's not 100% because the site still works and some data still makes it through.

Here's a screenshot of the errors in my safari console (yes, it happens in chrome and firefox too)

Image 2015-04-12 at 10.27.45 AM.png

And here's my nginx vhost config

server {
    listen 80;

    server_name forum.nzb.cat;

    return 301    https://forum.nzb.cat$request_uri;

}
server {
    listen 443 ssl;

    server_name forum.nzb.cat;

    ssl_certificate /etc/nginx/ssl/forum.nzb.cat.crt;
    ssl_certificate_key /etc/nginx/ssl/nzbcat.key;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;

        proxy_pass http://127.0.0.1:4567/;
        proxy_redirect off;

        # Socket.IO Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

And here's the error in the nginx error.log. I googled and googled and did everything the stack overflow people suggested to no avail:

2015/04/12 10:39:03 [error] 17037#0: *146836 upstream prematurely closed connection while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: forum.nzb.cat, request: "GET /socket.io/?EIO=3&transport=polling&t=1428849532722-383&sid=ECw2nTDnOQgEjGcdAAKp HTTP/1.1", upstream: "http://127.0.0.1:4567/socket.io/?EIO=3&transport=polling&t=1428849532722-383&sid=ECw2nTDnOQgEjGcdAAKp", host: "forum.nzb.cat", referrer: "https://forum.nzb.cat/"

I'm at my wit's end here so any help would be appreciated. I didn't bother commenting out any of the URL's as it's a public site.

Moritz Friedrich

I had a hard time too while configuring. Try this one:

### redirects http requests to https
server {
    listen 80;
    server_name forum.example.org;
    
    return 301 https://forum.example.org$request_uri;
}

### the https server
server {
    # listen on ssl, deliver with speedy if possible
    listen 443 ssl spdy;

    server_name forum.example.org;

    ssl on;
    
    # change this path!
    ssl_certificate /path/to/cert/bundle.crt;

    # change this path!
    ssl_certificate_key /path/to/cert/forum.example.org.key;
    
    # enables all versions of TLS, but not SSLv2 or 3 which are weak and now deprecated.
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    
    # disables all weak ciphers
    ssl_ciphers 'AES128+EECDH:AES128+EDH';
    
    ssl_prefer_server_ciphers on;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;

        proxy_pass http://127.0.0.1:4567/;
        proxy_redirect off;

        # Socket.IO Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Moritz Friedrich

Alternatively, my working configuration for clustering enabled + SSL:

# redirects http requests to https
server {
  listen 80;
  
	# change this!
  server_name example.com;

	# change this!
  return 301 https://example.com$request_uri;
}

# create multiple instances for scaling
upstream io_nodes {
    ip_hash;
    server 127.0.0.1:4567;
    server 127.0.0.1:4568;
    server 127.0.0.1:4569;
}

# the actual encrypted server
server {
  listen 443 ssl spdy;

	# change this!
	 server_name example.com;

  ssl on;
  
  # change this!
  ssl_certificate /etc/nginx/ssl/bundle.crt;
  
  #change this!
  ssl_certificate_key /etc/nginx/ssl/site.de.key;

  # enables all versions of TLS, but not SSLv2 or 3 which are weak and now deprecated.
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

  # disables all weak ciphers
	ssl_ciphers 'AES128+EECDH:AES128+EDH';
  ssl_prefer_server_ciphers on;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;
    proxy_redirect off;

    # Socket.io Support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    
    # watch out - no quotes!
    proxy_set_header Connection upgrade;

    gzip            on;
    gzip_min_length 1000;
    gzip_proxied    off;
    gzip_types      text/plain application/xml application/x-javascript text/css application/json;

    location @nodebb {
        proxy_pass http://io_nodes;
    }

    location ~ ^/(images|language|sounds|templates|uploads|vendor|src\/modules|nodebb\.min\.js|stylesheet\.css|admin\.css) {
       
        # change this! 
        root /var/www/nodebb/public/;
        try_files $uri $uri/ @nodebb;
    }

    location / {
			proxy_pass http://io_nodes;
    }

}

nhl.pl

An issue with terminating SSL connections?
http://blog.mixu.net/2011/08/13/nginx-websockets-ssl-and-socket-io-deployment/

KingCat

@Moritz-Friedrich Unfortunately none of those config changes made a difference I'm behind cloud flare ssl so I'm not sure if that's what's causing it (even though I disabled it completely for awhile and it still happened)

Moritz Friedrich

@KingCat , if you run sudo nginx -t, what happens exactly?

KingCat

@Moritz-Friedrich

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

julian

It's cloudflare. Keep using it, but disable acceleration. Websocket issues will be fixed.

KingCat

@julian said:

It's cloudflare. Keep using it, but disable acceleration. Websocket issues will be fixed.

I did what you suggested and it seems to have fixed the issue in chrome. However in safari it still happens. I'm gonna wait a while and see if maybe safari just has something weird cached (even though i cleared browser and dns cache).

KingCat

So it's been a few days and i'm still getting the errors in safari. The errors don't show up in any other browser though.

If I had to guess, I'd say theres some kind of technology or protocol that safari doesn't support (maybe the wss:// protocol?) so the socket.io has to fallback to using http instead and that seems to be why the error happens. Totally a guess though.

If this is the best I can do I'm OK with that as 0.1% of my users are on safari and it's not a site-breaking bug either. I'll consider this closed unless anyone has anything else to try.

Also, not sure if it matters but I'm using a free StartSSL certificate. It's not a self-signed one and it gives the green lock in safari but just in case it matters for some reason.

KingCat

By the way, I visited the link in the error messages in safari (https://forum.nzb.cat/socket.io/?EIO=3&transport=polling&t=1429031828694-10&sid=E7eEDbKPU53mKENvAAW3)

and got this:

{"code":1,"message":"Session ID unknown"}

Not sure what this means for the cause of the 400/502 errors.

Schamper

@KingCat I don't really have anything to contribute, just wanted to say that I saw your username and went "... NZBCat?" and lo and behold... Two things I use combined together

KingCat

@Schamper Haha yep. I may be the KingCat but even I need support sometimes

KingCat

Welp, I solved it.

Turns out I had the wrong directory specified in nginx for static assets. It was pointing to a different install of nodebb. For some reason that caused issues with socket.io (and only with safari/opera for some reason).

Now, here's the weird thing. I only recently started using nginx to serve the assets. Recently as in like this morning. Before that I simply had a proxy pass setup and let nodebb serve the assets. I'm not sure what was causing the 400/502 errors then but I'm just glad their solved now.