awesome!#%!@%O)@!

I stressed tested it a little today in combination with all of my data, lightning fast & seems very stable.

Please refer to Github issues #320 and #321 for discussion regarding the issues raised here.

These optimizations will be implemented and acid-tested on a new branch before being merged into master.

Cool! Yea I was unsure of why there were so many queries on category load.

Regarding the external page click 'interception': Is it possible to make that customizable as well? On my forum, people post tons of external links, they always expect a "new tab" if it's outside of the adarq.org domain. I do some other things such as lightbox etc, but that's easy to customize on my end.

If you guys ever want to experiment live with the 93k posts data, I can easily give you access for testing.

cya!

The UX looks slick, and I think a more realtime real-time forum is a pretty novel idea.

The only problem is that Redis is an in-memory database, and for a forum like mine, where I want to be able to keep all of the posts, that simply will not work.

If all you're doing is a key value store then check out ueberDB which will give you a ton of database options out of the box. https://github.com/Pita/ueberDB

Hi varx,

You could always persist from the slave, it's a pretty efficient way to do it. Assuming you will have a master/slave setup for your preferred DB choice, you're going to replicate, so with redis you would do the same (slaveof) and then persist there.

redis persistence:

http://dev.paperlesspost.com/blog/2011/12/21/redis-tips-persistence-only-slaves-and-the-slowlog/

http://redis.io/topics/persistence

Would be nice for these backends to be plugins and not "forced on you", i'd continue to use the redis backend

I'm also interested in seeing is cheerio is really the bottleneck (which it very well may be).

Just add the following line after the filter:post.parse hook is fired (line 195 of postTools.js):

plugins.fireHook('filter:post.parse', raw, function(parsed) {
    return callback(null, parsed);   //   <-- new line

Fingers crossed, if this takes care of the majority of the slowdown, then we can try to recreate the skipped code without using cheerio.

That speeds it up significantly. Fresh clone/test:

1. Without the 'cheerio bypass' (lol): 18.x seconds

2. With the 'cheerio bypass': 10-12s

btw all of these numbers I do multiple tests, it's very consistent.

So, bypassing cheerio is shaving 6-8 seconds off of the load time. It's also doing a massive amount of redis queries though too so, if that was tuned, maybe all of this would even become a non-issue.

i'll msg you the log.

Hey @adarqui,

Thanks for all of the hard work you're putting into this! We haven't had a large dataset to test with, so it's really great to see how NodeBB handles under a huge load.

17 seconds is definitely not scalable, and if the majority of it is taken up by postTools.toHTML, then that's a good place to start.

As for whether it is feasible to switch it over to client side, that's a difficult question to answer immediately, as it hooks into our plugin system. NodeBB saves the raw markup (or markdown rather, heh) as typed into the posts composer, and parses it using markdown (or any other plugins). That particular subsystem might be difficult (if not impossible) to move to the client side. If the bottleneck is cheerio, however, then I think we may be able to do something about that...

As a point of reference... how long did it take to load the "Progress Journals" subforum in SMF?

hey!

http://www.adarq.org/progress-journals-experimental-routines/

It takes ~1.5 sec.

It's only doing a few queries to achieve that, since everything is packed into a few mysql tables.

I cloned the latest NodeBB repo, no modifications, here's what I get when I load progress journals subforum:

Two separate tests:

redis-cli MONITOR > /tmp/journals.monitor

^C
root@serv:~/admin/git/NodeBB/src# wc -l /tmp/journals.monitor
58137 /tmp/journals.monitor
root@serv:~/admin/git/NodeBB/src# grep hgetall /tmp/journals.monitor |wc -l
42954

redis-cli MONITOR > /tmp/journals.monitor

^C
root@serv:~/admin/git/NodeBB/src# wc -l /tmp/journals.monitor
58137 /tmp/journals.monitor
root@serv:~/admin/git/NodeBB/src# grep hgetall /tmp/journals.monitor |wc -l
42954

Someone possibly added more queries, last time I checked that (a few days ago), it was 25k. This is probably just a small tuning issue. I'll send you the redis-cli MONITOR log.

I'm going to re-migrate data back into nodebb tonight, so, if you want I can even give your team access to my testing server if they want to play around.

cya!

Hey,

I'll post some stuff here regarding my migration from SMF 2.x forum to NodeBB. I've had a forum for ~4 years and it's about time I revamped it. I'm a huge node.js and redis fan, so having a forum written in php is blasphemous.

So far, i've been working on migrating the data from my SMF forum to NodeBB. Everything is going good here, i've been able to get everything into NodeBB with the correct timestamps etc etc. So that's cool.

NodeBB is crazy fast, and can be even faster.

One issue i've noticed so far, is that once I loaded some huge amounts of data into NodeBB, I started seeing a serious drop in performance. The good news is that we can fix it.

Here's a small screencast of what my 'load times' look like when visiting the "progress journals" subforum. This sub forum isn't "that big", 71 posts. However, some of the threads are enormous, so nodebb is doing say, 25k-30k redis operations just to load it, then it's doing a ton of server side html/dom processing which seems to be the biggest issue.

http://screencast.com/t/d1ooHZtEvf

Ok so, in order as is seen in the video, here are the "benchmarks".

1. NodeBB unmodified code: 17.x seconds
2. NodeBB modified code, using a redis.multi pipeline to send all of the hgetall's to redis in one request: 15.x seconds
3. NodeBB modified code, using the same pipeline above AND getting rid of posts.toHTML(): 3.x seconds

**CORRECTION:
Also, all of that would actually be slower, but I've changed the redis code to use a UNIX DOMAIN SOCKET (/tmp/redis.sock) etc, which speeds it up a bit. I've also installed the hiredis native C redis package, that actually seems pretty significant as well.
**

I'm not so great with the DOM/"front end stuff", so i'm not sure how hard/easy/feasible it would be to move that posts.toHTML()/cheerio stuff to the browser, client side? If data were sanitized on the way in (during a post from a user), then you could simply return it to the browser and have it do the DOM crunching client side, which would free up nodebb server-side significantly?

So far, that seems to be the only issue i've come across.

Ok so i'll look for a few more spots to possibly speed it up initially. Right now i'm thinking the initial dashboard, it uses a bunch of hmget's maybe 'multi' will have an impact.

peace!

Hey, after forking the 0.0.6 repo and cloning it etc, the latest-post column (in any subcategory) seems to bleed out into the other topics etc. Here's a screenshot:

http://i.imgur.com/fuqw6gq.png

Here's the hash for the effected one:

3eb594d|1379724494| (HEAD, origin/master, origin/HEAD, master)

For my 0.0.6 that's working fine, here's the hash for that:

0831649|1379353991| (HEAD, origin/master, origin/HEAD, master)

i'm not sure if it's only me or if it is in fact an issue, my git skills need sharpening to figure it out based on the commits.

peace

test

reply

test

reply