SOLVED: Robot getting 403, why?

[NodeHam]

I'm using semrush SEO to analyse how things are looking on our new nodebb. However, the robot keeps getting a 403 for this url.

https://forums..domain.com/search

the only robots.txt was the default empty so I added the following but it still happens.

User-agent: *
Disallow:
https://forums.domain.com/sitemap.xml

I've checked and cannot see any reason for this happening. It isn't happening when going to that link but it is for semrush.

Solved in that it's ok for bots to get 403 on some pages which in this case, is the search which redirects to login.

[baris]

Check if guests/spiders have privileges to search the forum. If they don't then loading that url as a guest/spider will result in 403.

[NodeHam]

@baris

Hi, thanks for your reply.

You're on to something.
The search function requires logging in to use but it redirects to the login page. I'm not sure why the bot is getting a 403.

Spiders and Guests have view users, view tags and view groups only.

[julian]

@nodeham we automatically redirect users to the login page, but spiders we detect and serve them a 403 for pages they're not allowed into, since it doesn't make sense to send them to /login

[NodeHam]

@julian

Yes, that's what I'm seeing but I'm not seeing how I can fix that or are you saying there is no point in fixing it?

In terms of permissions, I've tried to find a list of default installed groups compared to mine since mine was converted from phpbb.

Some groups seem to he hidden, for lack of better word. when I go to groups, I see the following groups but never see one which from what I've read is a default called registered-users. Instead, I see two other groups both with part of the users.

I wanted to move all registered users from the old board to the registered-users group to consolidate those and to get rid of any non nodebb groups that were created during the conversion.

When I search, then I can see more groups... (notice the number of users in the registered-group)

The original phpbb groups were;

Administrators
Bots
Global moderators
Guests
Newly registered users
Registered users
Registered COPPA users

And after the conversion, the nodebb groups are;
administrators (1)
bots (45)
Global Moderators (2)
newly_registered (73)
registered (52)
global_moderators (3)
registered_coppa (0)

And finally, permissions are as follow;

From what I have tested, guest/non logged in visitors can see everything and so search engines should be indexing the site but since semrush gets a 403 it makes me nervous that search engines are not properly indexing the site.

So, I've been unsure about migrating some groups, deleting un-needed, etc. I wish I would have started from scratch at this point.

[julian]

@nodeham "registered-users" should contain the same number of members as users in your forum. If that's not the case, you'll probably want to manually add them back. The other groups were migrated, but are mostly informational, you can probably remove them (but not knowing your dataset, I can't recommend you do so for certain).

If semrush reports seeing a 403 in /search, that's okay. It should still be able to access the posts as long as the "spiders" group has the "access topics" privilege checked for all categories.

You're looking at the "Global" privilege list. The dropdown lets you see privileges for individual categories.

[NodeHam]

@julian

Thanks for the input. yes, I checked those as well and everything looks fine other than the groups thing I mentioned. If you don't think the 403 to search is an issue, I will disregard it then.

[julian]

Glad you got it resolved, best of luck!