SOLVED: Robot getting 403, why?
-
I'm using semrush SEO to analyse how things are looking on our new nodebb. However, the robot keeps getting a 403 for this url.
https://forums..domain.com/search
the only robots.txt was the default empty so I added the following but it still happens.
User-agent: *
Disallow:
https://forums.domain.com/sitemap.xmlI've checked and cannot see any reason for this happening. It isn't happening when going to that link but it is for semrush.
Solved in that it's ok for bots to get 403 on some pages which in this case, is the search which redirects to login.
-
Hi, thanks for your reply.
You're on to something.
The search function requires logging in to use but it redirects to the login page. I'm not sure why the bot is getting a 403.Spiders and Guests have view users, view tags and view groups only.
-
Yes, that's what I'm seeing but I'm not seeing how I can fix that or are you saying there is no point in fixing it?
In terms of permissions, I've tried to find a list of default installed groups compared to mine since mine was converted from phpbb.
Some groups seem to he hidden, for lack of better word. when I go to groups, I see the following groups but never see one which from what I've read is a default called registered-users. Instead, I see two other groups both with part of the users.
I wanted to move all registered users from the old board to the registered-users group to consolidate those and to get rid of any non nodebb groups that were created during the conversion.
When I search, then I can see more groups... (notice the number of users in the registered-group)
The original phpbb groups were;
Administrators
Bots
Global moderators
Guests
Newly registered users
Registered users
Registered COPPA usersAnd after the conversion, the nodebb groups are;
administrators (1)
bots (45)
Global Moderators (2)
newly_registered (73)
registered (52)
global_moderators (3)
registered_coppa (0)And finally, permissions are as follow;
From what I have tested, guest/non logged in visitors can see everything and so search engines should be indexing the site but since semrush gets a 403 it makes me nervous that search engines are not properly indexing the site.
So, I've been unsure about migrating some groups, deleting un-needed, etc. I wish I would have started from scratch at this point.
-
@nodeham "registered-users" should contain the same number of members as users in your forum. If that's not the case, you'll probably want to manually add them back. The other groups were migrated, but are mostly informational, you can probably remove them (but not knowing your dataset, I can't recommend you do so for certain).
If semrush reports seeing a 403 in
/search, that's okay. It should still be able to access the posts as long as the "spiders" group has the "access topics" privilege checked for all categories.You're looking at the "Global" privilege list. The dropdown lets you see privileges for individual categories.
