Robots.txt: Difference between revisions
From DWIKI
Tag: wikieditor |
Tag: wikieditor |
||
| Line 34: | Line 34: | ||
User-agent: Applebot | User-agent: Applebot | ||
Disallow: / | |||
User-Agent: Thinkbot | |||
Disallow: / | Disallow: / | ||
</nowiki> | </nowiki> | ||
Latest revision as of 13:35, 10 November 2025
Links
bad bots
Documentation
Examples
- https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html
- Good and bad bots
Examples
User-agent: PetalBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Dotbot Disallow: / User-agent: Amazonbot Disallow: / User-agent: GPTBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Applebot Disallow: / User-Agent: Thinkbot Disallow: /
