Is Google complaining about nodebb or something else?

NodeHam

@phenomlab said in Google constantly complains about nodebb forums:

@NodeHam said in Google constantly complains about nodebb forums:

We used to spend a great deal of time on SEO but found it was a never ending game with Google and could not justify spending so much time at it.

And that one statement speaks a multitude. That's exactly what it is, and it's very easy to focus all of your effort on that rather than creating compelling content. I turned away from SEO years ago, but do post regularly on my own forum (https://sudonix.com) and it's ranking has improved over time - all organic, and no "tricks".

Yes but compelling content that cannot be found is exactly where we are now :). We decided to do just that, spend more time on relevant and good content but if we're not being indexed, no one is finding that great content anyhow.

phenomlab

@NodeHam I know the feeling. It's hard to find a niche that hasn't already been "done to death".

NodeHam

The niche is not the problem, it's being found on the Internet that is. I suggested nodebb because it seemed to have a better shot at being indexed than the previous forums that were being used but now I find this.

Here are the plugins on the site. Pretty basic.

nodebb-plugin-2factor - Installed 5.0.2 | Latest 5.0.2. Not active
nodebb-plugin-composer-default - Installed 9.1.1 | Latest 9.1.1. Active
nodebb-plugin-cppnet-markdown - Installed 0.7.0 | Latest 0.7.0. Not active
nodebb-plugin-dbsearch - Installed 5.1.5 | Latest 5.1.5. Active
nodebb-plugin-emoji - Installed 4.0.4 | Latest 4.0.4. Active
nodebb-plugin-emoji-android - Installed 3.0.0 | Latest 3.0.0. Active
nodebb-plugin-markdown - Installed 10.1.0 | Latest 10.1.0. Active
nodebb-plugin-mentions - Installed 3.0.11 | Latest 3.0.11. Active
nodebb-plugin-soundpack-default - Installed 2.0.1 | Latest 2.0.1. Not active
nodebb-plugin-spam-be-gone - Installed 1.0.0 | Latest 1.0.0. Not active
nodebb-rewards-essentials - Installed 0.2.1 | Latest 0.2.1. Active

nodebb-theme-lavender - Installed 6.0.0 | Latest 6.0.0.
nodebb-theme-persona - Installed 12.1.1 | Latest 12.1.1
nodebb-theme-slick - Installed 2.0.2 | Latest 2.0.2
nodebb-theme-vanilla - Installed 12.1.18 | Latest 12.1.18
nodebb-widget-essentials - Installed 6.0.0 | Latest 6.0.0

phenomlab

@NodeHam Yes, nothing of any concern there. I also use the Q&A plugin, and I know that creates better SEO as mine increased after I installed it - I have a use case for this because my site is a consultancy where members can ask questions and get assistance around technology.

NodeHam

Robots in admin/settings/web-crawler is set as follows. At least this is the most current testing since finding this problem.

User-Agent: *
Sitemap: https://xxx.com/sitemap/topics.xml
Sitemap: https://xxx.com/sitemap/topics.1.xml
Sitemap: https://xxx.com/sitemap/categories.xml
Sitemap: https://xxx.com/sitemap/pages.xml

NodeHam

@phenomlab, thanks very much. I'll install that plugin for sure if it helps even a little bit

NodeHam

I tried that plugin but it just breaks the editing tools. The site is fully updated, up to date, no other issues that I know of.
Guess I can't use it.

phenomlab

@NodeHam Odd - seems fine for me

NodeHam

One problem at a time I guess :). I'll try it again at some point.

Need to focus on how to get these forums indexed.
There's a ton of searchable, relevant content that Google is not seeing and that's a bad thing since I only see two pages found for the last year of indexing.

julian

@NodeHam looking at this list...

viewtopic.php?f=13&t=133
user/seo-crawler/followers
viewtopic.php?p=193
user/sergrid/posts
viewtopic.php?p=70
user/ibm-research-bot/groups
viewtopic.php?p=80
viewtopic.php?p=208
user/toliklus/topics?sort=lastpost
groups/registered_coppa

A lot of it seems to be pages that have query strings that do nothing in NodeBB. e.g. ?p=, ?sort=lastpost, even viewtopic.php is not a route we maintain, etc.

They signify that Google is hitting the 404s because it remembers indexing them from before, likely due to older software you were using on the same site, does that sound right?

As for indexing, you'll also need to make sure that the "spiders" group has access to all of your categories, otherwise Google won't be able to index them

Juuuuust for fun here's ours:

julian

@NodeHam Let's focus on the two main culprits:

Alternate page with proper canonical tag

This suggests that pages were found that have a <link rel="canonical" ..> pointing to another page. In an of itself not an issue, although I think you can expand that and see a sample of URLs that fall under this category. They should all make sense (e.g. /topic/123/5 should have a canonical url /topic/123, since it's just pointing to the 5th post in topic 123).

Page with redirect

Digging into ours, I am seeing that most of the entries here fall under two categories:

/post/12312
/topic/{tid}/{slug}?lang=en-GB

The former is expected, it's a permalink that sends you directly to that pid. The latter is also valid (in that it loads the same content with or without the query string), but is concerning insomuch that I don't know where Google finds the ?lang=en-GB parameter from.

I will mark it for review, but that said, it shouldn't matter that it redirects, the original page should still be indexed properly.

NodeHam

@julian

likely due to older software you were using on the same site, >does that sound right?

Yes, you're right on that, those would be the old phpbb forums links so it answers that. Still doesn't answer why the rest of our forums aren't being indexed however so reading on.

As for indexing, you'll also need to make sure that the "spiders" >group has access to all of your categories, otherwise Google >won't be able to index them

Ok, you might have hit on something again. Looking under Manage, Groups, this is what I'm seeing.

As you can see, some of those options cannot be changed however.

gotwf

Very interesting. Indeed. Indubitably, even.

julian

@NodeHam That's fine, those privileges are specific to using the search functionality.

You'll want to select a category from the dropdown and check the viewing privileges, e.g.

julian

Googlebot picking up `?lang=en-GB` from somewhere · Issue #10993 · NodeBB/NodeBB

I am not certain whether this is a bug, but it is causing noisy entries in webmaster tools. We've received two separate reports (one privately, one here) that the query string ?lang=en-GB is added to URLs reported in the webmaster tools....

GitHub (github.com)

julian

FWIW it still does seem new topics get indexed, and fairly quickly, at that. This topic already shows up on both Google and DDG

So it's really a matter of understanding that, yeah, Google marks a whole whackload of pages as duplicates, but that's ok, because they really are duplicates (e.g. redirects to actual topics, topic indices, etc.)

As long as the actual content does get indexed, we're ok, and so far that still seems to be the case.

NodeHam

I do recall now that I posted about setting this up when first starting to use these forums.

Maybe I need to share the site so that it's easier to see what I'm seeing or at least thinking I'm seeing :).

When searching using site: sitename.com, next to nothing shows up.

NodeHam

This post is deleted!

NodeHam

This post is deleted!

NodeHam

I can search an exact specific topic AND include the name of the site and nothing shows up. You would think it would be the very first result considering how specific it is.

I'm having this discussion on another site and really could use help to confirm what the problem is.
Is it a problem with NodeBB? Are my forums set correctly or is Google simply not indexing us for some reason?