Question for all folk that self host their websites. Do you have a robots.txt and if so what's advisable to put in it ???#HomeLab #SelfHosted
-
Question for all folk that self host their websites. Do you have a robots.txt and if so what's advisable to put in it ???
#HomeLab #SelfHosted -
Drew DeVaultreplied to Justine Smithies last edited by
@justine sourcehut's robots.txt might be a good place to start:
It has comments on why each entry is there
-
Justine Smithiesreplied to Drew DeVault last edited by
@drewdevault Hey Drew thank you.
-
Justine Smithiesreplied to Justine Smithies last edited by
@drewdevault Daft question Drew cgit robots.tx does not have very much in so should I append so of the stuff you shared to stop bots ? Or best leave as is ? TIA
-
Drew DeVaultreplied to Justine Smithies last edited by
@justine I think you don't need much for cgit in terms of blocking expensive routes but you can keep an eye on it and adapt over time
Copyright © 2024 NodeBB | Contributors