Search unreliable

[TLINDEN]

The search function doesn't work reliably.

I've got two Threads with the (same) subject Semmelknödel a la Tom. If I search for knödel both will be found, however, if I search for Semmel only one of them will be shown.

NodeBB version is 1.8.1.

[TLINDEN]

Ah, there's a pattern, it doesn't find older posts!

Nope, it finds the old, but not the new one. But there's an older posting "jobangebot" and if I search for "job", it wont be found.

Also, subjects are not being searched at all. It seems, only post bodies are included - but not all, see above.

So, here are the two threads:

Search for Semmel in subject and body returns 1 result:

When searching only the subject, nothing will be found:

[julian]

Hm... have you tried re-indexing your forum? That might fix the missing search key...

[TLINDEN]

I did but it didn't help.

[julian]

Funny, searching knödel here returns 1 post, but Semmel returns two!

@baris

[TLINDEN]

wrong thread, sorry.

[TLINDEN]

Is there any solution to this? Because if there's no solution, I must migrate the forum to some other software, as a reliable search function is key to my users application: they log recipes to the forum and use search to check for duplicates. But since the search doesn't work, they actually produce duplicates instead of avoiding them!

[PitaJ]

You can redirect the search function to Google, but it won't work for non-public posts. There is also an elasticsearch plugin you might try.

[baris]

Are you on mongodb or redis?

[TLINDEN]

@PitaJ thanks for your reply, but it's not what I'm asking for I can't use google, of course, and I won't add another plugin which would just add to the already annoying load problem I'm experiencing.

So, I think this is clearly a bug. Whatever the dbsearch plugin is trying to do- it is doing it completely wrong. In fact, I think the index it creates on MongoDB is wrong, since I can reproduce the invalid search results from mongo shell as well:

> db.searchtopic.find( { $text: { $search: "Semmel" }})
[no output]

> > db.searchpost.find( { $text: { $search: "Semmel" }})
> { "_id" : ObjectId("5a9d818cd19a243c1a362c28"), "id" : 1201, [..]

So, only searching topics returns no result and posts returns 1 result. However, in reality there are 3 topics matching the string Semmel:

> db.objects.find( { _key: "topic:178" });
{ "_id" : ObjectId("5a9d818cd19a243c1a362c15"), "_key" : "topic:178", "tid" : 178, "uid" : 3, "cid" : 24, "mainPid" : 1201, "title" : "Semmelknödel ala Tom", "slug" : "178/semmelknödel-ala-tom", "timestamp" : 1520271756137, "lastposttime" : 1520277378816, "postcount" : 7, "viewcount" : 12, "locked" : 0, "deleted" : 0, "pinned" : 0, "upvotes" : 2, "downvotes" : 0, "teaserPid" : "1212" }

> db.objects.find( { _key: "topic:362" });
{ "_id" : ObjectId("5ac49e3cd19a243c1a36e9f3"), "_key" : "topic:362", "tid" : 362, "uid" : 6, "cid" : 29, "mainPid" : 2826, "title" : "Semmelknödelteig", "slug" : "362/semmelknödelteig", "timestamp" : 1522835004299, "lastposttime" : 1522847285369, "postcount" : 4, "viewcount" : 10, "locked" : 0, "deleted" : 0, "pinned" : 0, "teaserPid" : "2840" }

> db.objects.find( { _key: "topic:367" });
{ "_id" : ObjectId("5ac4f7ced19a243c1a36ed50"), "_key" : "topic:367", "tid" : 367, "uid" : 6, "cid" : 23, "mainPid" : "2852", "title" : "Semmelknödel a la Tom...", "slug" : "367/semmelknödel-a-la-tom", "timestamp" : 1522857934277, "lastposttime" : 1522866479077, "postcount" : 20, "viewcount" : 12, "locked" : 0, "deleted" : 0, "pinned" : 0, "teaserPid" : "2907", "thumb" : "", "upvotes" : 2, "downvotes" : 0 }

(and many more posts)

What is weird as well is the index language setting: "default_language" : "english", but I'm using german as forum language.

[baris]

@tlinden what does the following query return? db.searchtopic.find({id: 178});

[randoms]

I'm using chinese as forum language and I'm experiencing the same problem.

[TLINDEN]

@baris said in Search unreliable:

@tlinden what does the following query return? db.searchtopic.find({id: 178});

@baris this::

> db.searchtopic.find({id: 178})
{ "_id" : ObjectId("5a9d818cd19a243c1a362c1c"), "id" : 178, "content" : "Semmelknödel ala Tom", "cid" : "24", "uid" : "3" }

[baris]

@tlinden ok so it is in the searchtopic collection, does searching for it like this find it?

db.searchtopic.find( { $text: { $search: "Semmelknödel" }}

I don't think full text search supports partial word matches right now. https://stackoverflow.com/questions/44833817/mongodb-full-and-partial-text-search?rq=1

I will hopefully add language support to dbsearch plugin if mongodb is used https://github.com/barisusakli/nodebb-plugin-dbsearch/issues/33

barisusakli created this issue in barisusakli/nodebb-plugin-dbsearch

closed add support for languages if using mongodb #33

[TLINDEN]

Ah, that explains it! Searching for Semmelknödel reveals 1 topic and 2 posts (which is correct). So, we have to seach for whole words until partial matches are supported sometime.

Ok, still a little annoying for the girls, but I'm going to blame you

However, thanks for clarifying @baris !

best,
Tom

[baris]

@tlinden Yeah I agree it's a bit annoying, there is an issue for it here https://jira.mongodb.org/browse/SERVER-15090 so it might be supported in the future.

[baris]

nodebb-plugin-dbsearch@2.0.14 adds support for different languages. You can switch to German and see if it improves your results.

[Juan G.]

You can also try Solr search for NodeBB.

First install solr-jetty (jetty is lighter and has better performance than tomcat):

sudo apt-get install solr-jetty

(This also automatically installs the many needed dependencies, basically Java packages).

Optional: Temporarily open port 8080 if you want to see Solr admin (not needed):

http://example.com:8080/solr/

(Port 8080 worked for me; if not, try 8983).

Then install NodeBB's Solr plugin:

NodeBB Admin > Extend > Plugins > Find Plugins > nodebb-plugin-solr > Install

Then activate the plugin, restart NodeBB, reload admin page, and go to:

Plugins > Apache Solr

Configuration:

Host: localhost
Port: 8080
Path: /solr

Save Settings

Reload NodeBB and admin page.

Connected

Indexing Enabled

Rebuild Search Index

Disable port 8080 (or 8983) on firewall for security.

It works well.

[TLINDEN]

Thanks for the tutorial, but I won't do that, it's like we say in Germany: "mit Kanonen auf Spatzen schiessen" (shooting with canons on sparrows). Also, I can't install software, because I'm on a shared system w/o root.

best,
Tom

[sharonyue]

@Juan-G I found it does not work with https, do you have any suggestions?