Robots.txt

Links

bad bots

• List of tool/bots you might want to block
• zipbomb protection

Documentation

• The ultimate guide to robots.txt
• Google doc about robots.txt

Examples

• https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html
• Good and bad bots

Examples

User-agent: PetalBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: Dotbot
Disallow: /


User-agent: Amazonbot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: Applebot
Disallow: /

User-Agent: Thinkbot
Disallow: /