I run a forum with currently 4 nodebb processes which are proxied to from nginx, with a mongodb database backend and redis enabled (which from my understanding is used for sessions only in this configuration).

In this configuration every nodebb process eats a fair amount of memory (to note it was reduced significantly with one of the recent versions. 1.15.x I think - I did see some cache related changes).

Please correct me if I'm wrong, but is each nodebb process in this configuration handling its own post cache etc?

It would seem like a good idea to be able to use redis as a cache backend on top of mongodb - but perhaps I am way off how it works. My knowledge of the codebase is very limited.

[Edit] I forgot to say, thanks very much for your continued work on this excellent forum software. Is it possible to make a donation as I've been running it for 4 years for our project and am not on a paid plan etc.

1.12.0 to 1.12.1

Realised no email was getting sent from my NodeBB after an update to 1.12.1

Checking the logs it was trying to connect to localhost on port 587. However by default exim4 only listens on port 25, so NodeBB must have changed this (or the default plugin nodebb uses for sending email).

I couldn't see this documented anywhere but it may affect other users so mentioning it here.

On Debian I changed /etc/default/exim4 to include

SMTPLISTENEROPTIONS='-oX 25:587'

Might be worth mentioning this somewhere if it's not already been done.

the hashing finally completed. I am still interested to know how much data may be stored here, as it took 2 hours to complete. Not sure that much "recent" ip data is needed.

It's still processing - I'm inclined to stop it and restore the db backup until I have verification on this. Certainly if it is something that should take this long, it would be useful if it had some sort of progress bar.

Am currently updating to 1.10 (from 1.9.x) and it's taking a very long time with

4. Updating NodeBB data store schema...
Parsing upgrade scripts... 
OK | 4 script(s) found, 53 skipped
  → [2017/6/22] Hash all IP addresses stored in Recent IPs zset...

(Mongodb 3.4.15)

I assume it's going through a very large number of ips - as cpu usage is high and it's still processing. It's taken over 30 mins so far. Is there any way of checking on progress manually via mongodb cli ? Also how can I see the size of the data set ? I wonder if the list of ips stored is larger than it should be due to the time taken.

@julian Using the registration queue would add a lot more work for admins/mods though - but certainly it would help with this also.

Looks like the above has helped, but we are still getting some registrations - the spam accounts are now just putting the URL in the "Full name" field.

I suspect a setting for X posts before they can change this info might be more suitable than reputation as we want to distinguish spammers from users, rather than the quality of users posts etc.

I saw the "minimum reputation" settings for the user profile. Apologies but I had missed this. I will use that to restrict the spammers user of the user profile info.

I use the anti spam plugin which helps with posts, but I have noticed recently a large number of spammers registering and using the user info / profile page to create an "Advert" for their services. They are not posting, just creating profiles which link to their website etc.

I don't believe this feature exists, but it would be useful if you could specify a user to not be able to set up a user profile info page until they have X posts or similar. This should stop them being able to do this.

I may open a ticket for the spam plugin to also send the profile info to akismet as I suspect it doesn't do that currently.

Any ideas are welcome. Going to go and delete the 20+ new spam accounts now that have appeared in the last 2 days.

Thanks for the workaround.

@torn2 because they referenced an external CSS which was updated to v4.0 (bootstrap) which broke existing skins that were built on v3.3.x

I decided to clear the redis cache completely as I don't think it's going to complete anytime soon otherwise (I assume this is ok, since using mongodb backend).

@baris Doing the update now. It's taking a long time to delete all the sessions. Can I just clear the redis cache completely in my set-up without breaking anything or use redis commandline to just clear the sessions ?

Thanks.

@baris Thanks for the fix!

@Justin-Kamerman-0 MongoDB is not running or is running on another port ? default port is 6379 for mongodb I believe. Check mongodb and nodebb configs.

you need to use v 0.3.1 for nodebb v1.4.5

npm i [email protected] --save

@baris Sorry, didn't reply/tag you correctly - but can't edit post due to the akismet spam filter flagging it on edit (I have noticed this issue with the spam plugin on retropie also when editing posts - even as an admin editing a users post - but that's another issue).

@barisThanks for the reply.

Nothing else should be using redis on the server. Apart from nodebb we just have a wordpress site on it, and service up static content from nginx.

Here's the full redis info

{
    "aof_current_rewrite_time_sec": "-1",
    "aof_enabled": "0",
    "aof_last_bgrewrite_status": "ok",
    "aof_last_rewrite_time_sec": "-1",
    "aof_last_write_status": "ok",
    "aof_rewrite_in_progress": "0",
    "aof_rewrite_scheduled": "0",
    "arch_bits": "64",
    "blocked_clients": "0",
    "client_biggest_input_buf": "0",
    "client_longest_output_list": "0",
    "cluster_enabled": "0",
    "config_file": "/etc/redis/redis.conf",
    "connected_clients": "20",
    "connected_slaves": "0",
    "db0": "keys=5814342,expires=5814342,avg_ttl=1206519971687",
    "evicted_keys": "0",
    "executable": "/usr/bin/redis-server",
    "expired_keys": "0",
    "gcc_version": "4.9.2",
    "hz": "10",
    "instantaneous_input_kbps": "1.64",
    "instantaneous_ops_per_sec": "15",
    "instantaneous_output_kbps": "2.22",
    "keyspace_hits": "438506",
    "keyspace_misses": "3627",
    "latest_fork_usec": "40722",
    "loading": "0",
    "lru_clock": "15957072",
    "master_repl_offset": "0",
    "maxmemory": "0",
    "maxmemory_human": "0B",
    "maxmemory_policy": "noeviction",
    "mem_allocator": "jemalloc-4.0.3",
    "mem_fragmentation_ratio": "0.26",
    "migrate_cached_sockets": "0",
    "multiplexing_api": "epoll",
    "os": "Linux 3.16.0-4-amd64 x86_64",
    "process_id": "682",
    "pubsub_channels": "2077",
    "pubsub_patterns": "0",
    "rdb_bgsave_in_progress": "0",
    "rdb_changes_since_last_save": "964",
    "rdb_current_bgsave_time_sec": "-1",
    "rdb_last_bgsave_status": "ok",
    "rdb_last_bgsave_time_sec": "65",
    "rdb_last_save_time": "1492351903",
    "redis_build_id": "56206ba45ffc95ca",
    "redis_git_dirty": "0",
    "redis_git_sha1": "00000000",
    "redis_mode": "standalone",
    "redis_version": "3.2.8",
    "rejected_connections": "0",
    "repl_backlog_active": "0",
    "repl_backlog_first_byte_offset": "0",
    "repl_backlog_histlen": "0",
    "repl_backlog_size": "1048576",
    "role": "master",
    "run_id": "774d8aa38bad1c0ed3bab3af7be6a0a23f8b7180",
    "sync_full": "0",
    "sync_partial_err": "0",
    "sync_partial_ok": "0",
    "tcp_port": "6379",
    "total_commands_processed": "965441",
    "total_connections_received": "31",
    "total_net_input_bytes": "382839654",
    "total_net_output_bytes": "1172279275",
    "total_system_memory": "4156342272",
    "total_system_memory_human": "3.87G",
    "uptime_in_days": "0",
    "uptime_in_seconds": "84451",
    "used_cpu_sys": "144.46",
    "used_cpu_sys_children": "1518.90",
    "used_cpu_user": "134.70",
    "used_cpu_user_children": "6927.99",
    "used_memory": "1943342064",
    "used_memory_human": "1.81G",
    "used_memory_lua": "37888",
    "used_memory_lua_human": "37.00K",
    "used_memory_peak": "1943542624",
    "used_memory_peak_human": "1.81G",
    "used_memory_rss": "513413120",
    "used_memory_rss_human": "489.63M"
}

Have a machine with 4gb ram. Running on mongodb + redis with 4 nodebb backends (proxying to them with nginx). Running nodebb 1.4.5.

I have noticed recently redis is using a lot of memory (40-50%). This seems a lot for session storage etc. Is there any way to reduce memory used?. I can for example clear the redis cache, but I'm wondering why it seems to be growing - I suspect there is old data in there such as old sessions etc ? We have 8.5k users, and about 70k posts.

Just for reference - my config.

{
    "url": "https://retropie.org.uk/forum",
    "bind_address": "127.0.0.1",
    "secret": "REMOVED",
    "database": "mongo",
    "port": [4567, 4568, 4569, 4570],
    "mongo": {
        "host": "127.0.0.1",
        "port": "27017",
        "username": "REMOVED",
        "database": "REMOVED",
        "password": "REMOVED"
    },
    "redis": {
        "host":"127.0.0.1",
        "port":"6379",
        "database": 0
    }
}

admin stats

mongodb
MongoDB Version 3.2.12
Uptime in Seconds 7,716
Storage Engine wiredTiger
Collections 3
Objects 3,151,142
Avg. Object Size 100.16 b
Data Size 301.00 mb
Storage Size 255.95 mb
Index Size 270.75 mb
Resident Memory 651 mb
Virtual Memory 965 mb
Mapped Memory 0 mb

Redis
Redis Version 3.2.8
Uptime in Seconds 7,717
Uptime in Days 0
Connected Clients 20
Connected Slaves 0
Blocked Clients 0
Used Memory 1.79G
Memory Fragmentation Ratio 0.75
Total Connections Received 31
Total Commands Processed 108,891
Instantaneous Ops. Per Second 16
Keyspace Hits 48,982
Keyspace Misses 422