SOLVED: Robot getting 403, why?

General Discussion
  • I'm using semrush SEO to analyse how things are looking on our new nodebb. However, the robot keeps getting a 403 for this url.

    the only robots.txt was the default empty so I added the following but it still happens.

    User-agent: *

    I've checked and cannot see any reason for this happening. It isn't happening when going to that link but it is for semrush.

    Solved in that it's ok for bots to get 403 on some pages which in this case, is the search which redirects to login.

  • Check if guests/spiders have privileges to search the forum. If they don't then loading that url as a guest/spider will result in 403.

  • @baris

    Hi, thanks for your reply.

    You're on to something.
    The search function requires logging in to use but it redirects to the login page. I'm not sure why the bot is getting a 403.

    Spiders and Guests have view users, view tags and view groups only.

  • @nodeham we automatically redirect users to the login page, but spiders we detect and serve them a 403 for pages they're not allowed into, since it doesn't make sense to send them to /login

  • @julian

    Yes, that's what I'm seeing but I'm not seeing how I can fix that or are you saying there is no point in fixing it?

    In terms of permissions, I've tried to find a list of default installed groups compared to mine since mine was converted from phpbb.

    Some groups seem to he hidden, for lack of better word. when I go to groups, I see the following groups but never see one which from what I've read is a default called registered-users. Instead, I see two other groups both with part of the users.

    I wanted to move all registered users from the old board to the registered-users group to consolidate those and to get rid of any non nodebb groups that were created during the conversion.


    When I search, then I can see more groups... (notice the number of users in the registered-group)

    The original phpbb groups were;

    Global moderators
    Newly registered users
    Registered users
    Registered COPPA users

    And after the conversion, the nodebb groups are;
    administrators (1)
    bots (45)
    Global Moderators (2)
    newly_registered (73)
    registered (52)
    global_moderators (3)
    registered_coppa (0)

    And finally, permissions are as follow;

    From what I have tested, guest/non logged in visitors can see everything and so search engines should be indexing the site but since semrush gets a 403 it makes me nervous that search engines are not properly indexing the site.

    So, I've been unsure about migrating some groups, deleting un-needed, etc. I wish I would have started from scratch at this point.

  • @nodeham "registered-users" should contain the same number of members as users in your forum. If that's not the case, you'll probably want to manually add them back. The other groups were migrated, but are mostly informational, you can probably remove them (but not knowing your dataset, I can't recommend you do so for certain).

    If semrush reports seeing a 403 in /search, that's okay. It should still be able to access the posts as long as the "spiders" group has the "access topics" privilege checked for all categories.

    You're looking at the "Global" privilege list. The dropdown lets you see privileges for individual categories.


  • @julian

    Thanks for the input. yes, I checked those as well and everything looks fine other than the groups thing I mentioned. If you don't think the 403 to search is an issue, I will disregard it then.

  • Glad you got it resolved, best of luck!

Suggested Topics

  • 0 Votes
    4 Posts

    @eveh Make sure the spiders pseudo-group is grant permission to view topics... otherwise NodeBB hides them. spiders are not treated the same as registered-users or guests

    By default they should have access, though.

  • 0 Votes
    7 Posts

    @yariplus So I tried the "npm install nodebb-plugin-markdown@latest" and I got permanent 502 gateway error. Then, I tried deactivate > install markdown from the ACP. In the second attempt, markdown table works. In both cases I saw this.

    Thanks for the help. Now I can finally have nice looking tables 👍

  • 0 Votes
    1 Posts

    Had a few slight problems getting this working. It crashed my forum and my blog, which reside on separate servers. I'm running NodeBB 0.7.0-dev with a new Wordpress install.

    I installed the plugin and followed the guide on how to set it up. So all was working. I posted a blog article to test the comments engine (using forum plugin), and when I clicked the button under the article to send the article to the forum, it produced an error, which said I either had to be a member of publishers group or admin to post. I was logged into NodeBB as admin, and Wordpress too.

    My forum started grinding to a halt, throwing up 502 errors in the process.
    My blog collapsed and is currently sitting with a "Error establishing a database connection" notice. It's still like this, and i'll repair that when I get the chance.

    The strangest aspect however is that the blog DID post to the forum. 32,800 times, no less! You can see the topic here - I set up a category for the blog articles to be posted to.

    I realise to get a further insight i'll have to look into the error logs which I intend to do, but can anyone help? I've since removed the blog's server from DNS so it's no longer visitable through links and the forum is back to being the homepage.

  • 0 Votes
    1 Posts

    I have nodebb setup with mongodb. Whenever I login as admin and go to the plugins page. I see this message in the log. I am not able to install any plugins. Clicking on "Install" doesn't do anything. Please help.

  • 0 Votes
    3 Posts

    it disappear after restart.