Problem of solr plugin

sharonyue

Another issue is that I found solr ate memory a lot! I have only 2G memory, when nodebb is running, it takes around 50%, if solr is running, solr and nodebb would take 90% of the memory..

See @julian said in this post:

However, the big downside is that Solr is a beast... it requires more resources than NodeBB itself does, and runs on Java+Tomcat, which I have next to no experience debugging

https://community.nodebb.org/topic/11103/default-search-solr-importance-of-full-text-search/8

More info:
https://stackoverflow.com/questions/22149296/solr-always-use-more-than-90-of-physical-memory

gotwf

@sharonyue Yes, Solr is a beast! Necessary evil if you want/need top quality, full index search function. I explored it briefly. Balanced costs and benefits and decided builtin search was "good enough".

If you do want to run Solr, I would recommend running it on a second VM, as it is a pretty fat stack. That will give you more granular control over your bottlenecks: Scale up the Solr and Nodebb servers independently, as necessary. And yeah, you are likely going to want a couple gigs of memory on that Solr server.

P.S.; You can fairly easily config Solr to run on your port of choice. Or tune NodeBB accordingly.

sharonyue

@gotwf I am not familiar with front-end so I have no idea how to achieve that. But I will do it several years later I think... when my site is large enough

You can fairly easily config Solr to run on your port of choice. Or tune NodeBB accordingly.

My solr is running good at port 8983, I did not install tomcat. do you think tomcat is necessary? solr must be deployed as webapps in tomcat?

gotwf

@sharonyue Been years since I set up a Solr server so I am a bit hazy. You should not need Tomcat, as Solr bundles Jetty. Hence, a dedicated Tomcat instance is not necessary.

Solr Requirements.

As for port, there should be a file at some path, server/solr.xml. In there will be a reference to jetty's port. Tweak that as you desire.

Are you sure Solr is "listening" on 8983? What is output of e.g. :

netstat -na | grep -i listen

or, more specifically:

netstat -na | grep -i 8983

and... if so, can you connect to that port, e.g.:

telnet localhost 8983

Yeah, I know there are fancier ways of testing that nowadays but they escape my mind at present and telnet will get the job done.

Edit: Okay, I just brought up Solr 8.5.1 on my workstation, which sports an abundance of memory. Solr wants a good 2 gigs just to launch, sans any collections to index. Solr is listening to 8983 on all interfaces (wh/you may want to restrict to localhost).

Also, confirm that you can connect to port 8983 as your nodebb user.

One last thing, I am unsure as to the path you specify above. So double check that NodeBB actually knows about that and has permissions to access??

Hope his helps. Or if not, at least rules a couple things out. Have fun!

P.S.; Maybe you want to grab a binary and follow docs here to confirm it works before braving Tomcat?

Installing Solr

sharonyue

Thanks for your suggestion.

@iZ8vb5zw2bx60ytuhcwlwiZ:~/solr-8.5.1$ netstat -na | grep -i 8983
tcp6       0      0 :::8983                 :::*                    LISTEN     
tcp6       0      0 IPaddress:8983      IPaddress:36399    ESTABLISHED
tcp6       0      0 IPaddress:8983      IPaddress:36403    ESTABLISHED
tcp6       0      0 IPaddress:8983      IPaddress:36400    ESTABLISHED

In there will be a reference to jetty's port.

If I got you right, solr should be used with Jetty? I heard from the internet that solr can be combined with tomcat or jetty before: https://community.nodebb.org/topic/12157/search-unreliable/18?_=1589503004418 , but I am not sure.

Edit: https://github.com/ckan/ckan/issues/4762

gotwf

@sharonyue Yeah, should be able to use Solr w/Tomcat, JBoss, whatever.

The binary distributions linked from the install page contain a built in jetty though, so go for the low hanging fruit until you get Solr and NodeBB working and then try transitioning to some other java app/servlet server. Jetty is okay for smaller gigs and testing. Maybe not the best for enterprise scale.

So, is everything on that box using IPV6? NodeBB is not running only IPv4, by any chance?

Were you able to telnet localhost 8983? I want to confirm some local firewall rule is not blocking it.

I do not use Ubuntu, so cannot speak to whatever they have lurking in apt. Debian and Ubuntu like to tweak stuff. Sometimes they break things. Suggest testing with the binary bits from Solr direct.

Extract that to a dir somewhere, then cd to that dir and bin/solr start. See if that works. If so, create a dedicated unprivileged user for solr and then config systemd to manage it.

Also, something is connecting to Solr on those 36xxx ports.

P.S.; The boytoy issue you linked above references building Solr from sources. Grab the binary.

sharonyue

@gotwf said in Problem of solr plugin:

I do not touch anything of IPV6 IPV4, so it should be as default (I think IPV4).

telnet localhost 8983

@iZ8vb5zw2bx60ytuhcwlwiZ:~/nodebb$ telnet localhost 8983
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Have a look of my solr, its running,

捕获.JPG

捕获2.JPG

Another thing is that now I am running nodebb on https.
So I can access solr by: http://example.com:8983/solr, but not https://example.com:8983/solr.

I also tried to modify Host from localhost to http://example.com, https://example.com they do not work. I am not sure if this malfunction is related to https.

gotwf

@sharonyue Humor me and test 127.0.0.1 in your nodebb config above rather than "localhost".

As for http(s): one thing at a time ;D

Oh, snap!! See my latest post.

gotwf

This post is deleted!

gotwf

@sharonyue said in Problem of solr plugin:

Background: recently I moved my nodebb forum to another server, till now it runs surprisingly good. I only found one problem: the solr searching plugin. It does not work, and the problem comes from the basic settings. After checking the internet a whole day. I made some progress:

As I understand, solr plugin needs to be deployed first and nodebb needs to connect it. At first I tried the easiest way: sudo apt install solr-tomcat, but it does not work: tomcat is fine, I can access it by example:8080, but solr does not work. I suspect the reason is that the new version of solr is a separated app (I learnt it from the internet).

Then I tried to download the solr-8.5.1.tgz, and put it somewhere, then run the solr service. It runs surprisingly, by such an easy way... The problem is that solr runs at example:8983. Then, I set as the following in the solr plugin, it is still not connected.

So not I dont know what am I supposed to do next. Any ideas?

^^^^

Ha! Uncheck "Secure" in your config since solr is not working via https at present.

sharonyue

sad,

does not work either...

捕获.JPG

gotwf

@sharonyue Boo Hiss!! Well, nothing obvious so looks like somebody smarter than me is requisite for this one. ;(

Hmm... one last thing. Here's a netstat from Centos:

tcp        0      0 127.0.0.1:4569          0.0.0.0:*               LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:25                  :::*                    LISTEN

The first one is ipv4 while the last two ipv6. From your netstat I am unsure if your solr is listening to ipv4, as those multiple colons tend point to ipv6.

That you can telnet localhost would indicate things should work but I've no clue what NodeBB may be doing internally w.r.t. that stuff, so maybe a possibility?

sharonyue

@gotwf This problem is way out or my ability. I have no knowledge of ipv4 something... Thank you anyway. I will wait a few more days to see if someone can come up a solution. I hope

sharonyue

I checked my old server. It runs tomcat and a quite old solr version. Everything is fine.

I guess the solr plugin can be only used by combining tomcat+solr, or it is not compatible with latest solr? I am not sure.

@julian Do you have any comments on it?

julian

The solr plugin works on any solr instance, as it just interfaces via port.

If you can't get the connection icon to go green, check the hostname, port, or subfolder.

sharonyue

After a very heavy work. Finally I made it work:

Ubuntu 18 enables ipv6 as default, please follow this instruction as written to disable it. link text
Download an old version of solr (I am using solr-5.3.0), I verified solr-8 does not work. Run it. Here is the settings:

It should be noted that I am not sure if the first step is necessary. Since I did it, but it does not work. Then I downgrade solr, it works.

Hope it helps the others, especially those who are using the latest Ubuntu and Solr. Have fun!!

捕获.JPG

gotwf

@sharonyue Ha! Was suspicious of the ipv6 stuff. But seems likely more to do with plugin's Solr version compatibility. Would be nice if someone more knowledgeable than I could help tease this out.

In any event, glad you got it sorted! And now that you have, I would be interested in follow up reports as to Solr's effectiveness, especially compared and contrasted with NodeBB's builtin search. I am guessing you are preferring Solr due to better multilingual support?

Enjoy!

sharonyue

@gotwf Yes. My forum is in Chinese so I must use solr. dbsearch does not support Chinese as you know so I use solr. And I found the searching results is satisfactory. Why do you use solr? The drawback of solr is just it costs too much memory. But I can accept that.

julian

@sharonyue An alternative is to use something like elasticsearch. There is a nodebb plugin for it and it is a fork of solr, so it should work similarly

sharonyue

Yes I found that https://github.com/joe1chen/nodebb-plugin-search-elasticsearch It looks it is not updated lately