What do typical production server requirements look like?
-
Don't know
But it took me a while to figure this stuff out too... 
-
Another important thing I would consider is a replacement for Redis. I am currently working on a new backend and therefore am also looking for major improvements in my backend.
I replaced Redis with SSDB and actually have to say I am quite satisfied with it. It even offers an option to migrate Redis to SSDB.
Not only that it is faster, but more efficient as it uses your HDD and your RAM as cache only.
But basically to get back to your main question. I would suggest a decentralized database system straight ahead (but I guess that this should be clear for you anyway).
An example taken from my new system:
Database server (MariaDB + SSDB):- 4GB RAM (SSDB is very efficient!!!, before I needed 8GB)
- NVMe SSD (CephFS)
- Intel Xeon E5-2687W v3 (2vCores)
Website itself:
- NGINX 1.9.6 with HTTP/2 and PageSpeed
- HHVM 3.10.1 (contains Redis PHP extension, which is required to migrate to SSDB)
- NodeJS 0.10.38 (will be replaced)
- RAID10 SATA SSD
- Intel Xeon E5-4650 (2vCores - will be replaced)
- 2GB RAM
While you cannot tweak too much at NodeJS you can at NGINX. For example by settings worker processes and maximum connections, GZIP, Caching, etc.
-
Is SSDB a drop-in replacement for Redis?
-
@julian it says on the GitHub page that redis clients are supported, so yes, I guess.
-
Neat, now NodeBB supports SSDB
-
@Kowlin said:
@lenovouser CloudFlare can websockets with one of there enterprise plans if I remember correctly. Searching the forums with just the keyword CloudFlare should give enough results
Yes, you need the enterprise plan.
-
10K concurrent users is absolutely enormous. That would put you pretty high on the list of biggest websites in the world. I've worked for someone who was doing 30K concurrent and they are by far one of the biggest websites anywhere. 10K concurrent means millions of views per hour, likely. Are you sure that concurrent is what you mean? What will be driving that kind of traffic? That would suggest that you will hit hundreds of thousands, maybe millions of users every hour. That would be the total populations of your target countries each day. Those are localized Google scale numbers.
-
You will surely want dedicated Redis and MongoDB clusters and many front end app nodes. Look at something like Chef to handle auto-scaling for you.
-
10K concurrent means millions of views per hour, likely. Are you sure that concurrent is what you mean? What will be driving that kind of traffic? That would suggest that you will hit hundreds of thousands, maybe millions of users every hour.
That's a lot. The NodeBB team hasn't seen that kind of traffic since our previous lives working in the videogame industry... FB games that had real-time interactions on live national television and such.
For our forum software, as @julian mentioned earlier we haven't had a chance to manage a forum with that much load yet. We'd love the opportunity to do so - I think we have a ton of experience here... if you'd like to offload the server management to us give us a shout
-
Lets say you serve 10.000 clients a second, this means:
10.000x60=60.000 per minute
60.000x60=3.600.000 per hour
3.600.000x24=864.000.000 per day
864.000.000x30=25.920.000.000 per monthSo just to have a relation. This means you would serve 3,7 times a month the world population.
If you want I can calculate a system for this size. Could be funny I guess
-
@AOKP that number is not quite that unreasonable because 10K would be his peak, not his sustained, and real world websites do much larger numbers than that because single users stay on for longer than a second and, we assume, return. But the number is still enormous.
-
@scottalanmiller yeah I am aware of this fact.
But I just was curious like you, if he does not means 10.000 visitors a day or if this is a simple general question how NodeBB deals with it.
-
10K a day is easy. We do that constantly. 10K concurrent is easily more than 10K every second, though!!
-
The question is about spikes in traffic. I need generalized data for scaling projections, cost estimates and hardware decisions. The correctly sized server for each process will act as a base for scaling to the required demand. It is not uncommon to reach ten's of thousands of uniques when facebook is involved.
If 10k concurrent were constant, i'd hire someone from outside the team.
-
Concurrent users will be more of a traffic issue than anything on its own. If that is 10K concurrent readers you get one thing, if you are looking at 10K concurrent posters then MongoDB is going to be struggling on any platform.
-
Do you have an existing site to compare this data from? How did you come up with 10K concurrent? To give perspective, http://mangolassi.it/ is an extremely busy forum, we are told we are the busiest in our industry (IT) and concurrent rarely tops 80. Not 80K... 80. That's with thousands of daily users, nearly a thousand daily posts, etc. You are looking at more than 100 times our peak, let alone our load. Doable, but huge.
-
@bitspook said:
The question is about spikes in traffic. I need generalized data for scaling projections, cost estimates and hardware decisions. The correctly sized server for each process will act as a base for scaling to the required demand. It is not uncommon to reach ten's of thousands of uniques when facebook is involved.
If 10k concurrent were constant, i'd hire someone from outside the team.
Well in my eyes there are several factors I would consider.
Do you ONLY want to run NodeBB or are there other applications running on the server as for example WordPress, which requires PHP or is the server dedicated for NodeJS?Do you use SSL encryption?
Whats your webserver (NGINX would be perfect for this)?
How fast is your connection (mainly important for an end-user)?
So basically it is important what you actually want to do. If its about simply running and serving NodeBB to 10K connections it is not so hard, but if you want to run several tasks at once on as few servers as possible things are slightly different.
Edit:
This is what I would do, when I would be have to serve the same amount of traffic as you do. I assume you are going to use SSL and will also run a dynamic PHP application with NodeJS.Additionally I will not consider the fact, that you maybe want your site to be served world wide.
I would go with 3 servers.
1x Redis (SSDB)
1x MariaDB (for dynamic PHP application)
1x Webserver (where your files are on)So basically 2 database servers and 1 for your static files.
Why shall you use SSDB?
According to my experience it is worth the try and many big sites see that similar, if we can trust the SSDB site.
Why shall we use MariaDB?
Well, I guess this is almost self explaining.
I would recommend to cache as much as possible of the database and also use a disk-controller with a write cache (see more at the setup).What webserver is the best?
For high traffic purposes NGINX is definitely the best, you can get for free.
If you want to use SSL I would even go for NGINX 1.9.6 with HTTP/2 support, but thats everything else than mandatory and is just an option.
Otherwise we can make use of NGINX 1.8.0 (current stable release) , PageSpeed, IPv6 and HHVM.The setups
As a matter of fact, high I/O and a big RAM are definitely the most important things for databases.SSDB:
- 32GB DDR4 RAM
- Intel Xeon E5-2630 v3 (slightly slower than a 1650, but less TDP)
- 400GB NVMe SSD (Intel 750)*
- Sockel 2011-3 Mainboard
- 500-700W PSU
- Linux
Cost: ca. 1500€-2000€ (ca. 1750-2200$)
MariaDB:
- 64GB DDR4 RAM
- Intel Xeon E5-2630 v3 (slightly slower than a 1650, but less TDP)
- 128GB RAID 10 SSD*
- Sockel 2011-3 Mainboard
- 500-700W PSU
- Linux
Cost: ca. 1500€-2000€ (ca. 1750-2200$)
Notice:
You can again use a NVMe SSD, but if there are no bigger R/W processes, it would be a waste. An example for that would be a simple cooperation website built with a CMS.
With an additional plugin you can even generate static sites, which no longer require SQL to be served.Webserver:
- 64GB DDR4 RAM
- Intel Xeon E5-2630 v3 (slightly slower than a 1650, but less TDP)
- 128GB RAID 10 SSD*
- Sockel 2011-3 Mainboard
- 500-700W PSU
- Linux
Cost: ca. 1500€-2000€ (ca. 1750-2200$)
*Size depends on actual need
Software:
- NGINX with PageSpeed, SSL and IPv6 support
- HHVM 3.10.1 - Click to compare with PHP
- NodeJS 0.10.x or 4.x (depends on NodeBB branch)
General requirements to all 3 servers:
Dpending on the speed you want to deliver the site I would at very least recommend 1Gbits connections. For optimal experience 10Gbits are more than enough.Side note:
-
@AOKP said:
Well in my eyes there are several factors I would consider.
Do you ONLY want to run NodeBB or are there other applications running on the server as for example WordPress, which requires PHP or is the server dedicated for NodeJS?Do you use SSL encryption?
Whats your webserver (NGINX would be perfect for this)?
How fast is your connection (mainly important for an end-user)?
So basically it is important what you actually want to do. If its about simply running and serving NodeBB to 10K connections it is not so hard, but if you want to run several tasks at once on as few servers as possible things are slightly different.
-
PHP only for the cart software. All NodeBB + webRTC signal server
-
Only Partial SSL; for cart
-
Yes; NGINX
-
Connection would be determined by AWS/GCE
I am looking to architect the hardware footprint as a scaling collection of minimally sized servers so that during lulls in traffic the costs drop to the core Reserved Instances.
Although some content will be created and stored locally, the vast majority of the content will be embedded links to other sites. It is almost all just http traffic; NodeBB, chat, etc. Even the local content is being largely offloaded with Peer5 and the chat with webRTC. Minimalism.
-
-
@bitspook said:
@AOKP said:
Well in my eyes there are several factors I would consider.
Do you ONLY want to run NodeBB or are there other applications running on the server as for example WordPress, which requires PHP or is the server dedicated for NodeJS?Do you use SSL encryption?
Whats your webserver (NGINX would be perfect for this)?
How fast is your connection (mainly important for an end-user)?
So basically it is important what you actually want to do. If its about simply running and serving NodeBB to 10K connections it is not so hard, but if you want to run several tasks at once on as few servers as possible things are slightly different.
-
PHP only for the cart software. All NodeBB + webRTC signal server
-
Only Partial SSL; for cart
-
Yes; NGINX
-
Connection would be determined by AWS/GCE
I am looking to architect the hardware footprint as a scaling collection of minimally sized servers so that during lulls in traffic the costs drop to the core Reserved Instances.
Although some content will be created and stored locally, the vast majority of the content will be embedded links to other sites. It is almost all just http traffic; NodeBB, chat, etc. Even the local content is being largely offloaded with Peer5 and the chat with webRTC. Minimalism.
Before proceding with AWS or GCE, I would like to know where you are located.
You could use OVH. Their prices are pretty low, they have DC's in Canada and France and even the right servers for this project, except NVMe SSD's, if I am not mistaken.So maybe also tell us your budget. There are definitely cheaper providers than AWS and GCE, offering the same reliability and performance.
Edit:
Be careful about SSDB. It definitely is better than Redis, but NodeBB does not fully supports it, yet. As an appropriate connector is missing.
Therefore you still have to stick to Redis for now - an import to SSDB should be possible at any time though.
-
-
I live in North america and the target audience is similarly located. Only one AZ is truly needed.
I would prefer mostly upfront capex, rather than ongoing opex. A combination of Reserved Instances for the core and Spot/Preemptable instances for the autoscaling would be ideal. The initial core of the servers would be relatively cheap reserved instances on three years term. The scaling costs are intended to be amortized on-the-fly as demand increases with ad revenue and internal sources.
