Robots.txt: Difference between revisions
From DWIKI
Created page with "=Links= * https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html =Examples=" Tag: wikieditor |
Tag: wikieditor |
||
| (7 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=Links= | =Links= | ||
* https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html | ==bad bots== | ||
*[https://www.humansecurity.com/learn/blog/crawlers-list-known-bots-guide/ List of tool/bots you might want to block] | |||
*[https://idiallo.com/blog/zipbomb-protection zipbomb protection] | |||
==Documentation== | |||
*[https://yoast.com/ultimate-guide-robots-txt/ The ultimate guide to robots.txt] | |||
*[https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt Google doc about robots.txt] | |||
==Examples== | |||
*https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html | |||
*[https://simtechdev.com/blog/good-and-bad-bots-to-control-to-save-server-resources-and-improve-performance/ Good and bad bots] | |||
=Examples= | =Examples= | ||
<nowiki> | |||
User-agent: PetalBot | |||
Disallow: / | |||
User-agent: AhrefsBot | |||
Disallow: / | |||
User-agent: Dotbot | |||
Disallow: / | |||
User-agent: Amazonbot | |||
Disallow: / | |||
User-agent: GPTBot | |||
Disallow: / | |||
User-agent: SemrushBot | |||
Disallow: / | |||
User-agent: Applebot | |||
Disallow: / | |||
User-Agent: Thinkbot | |||
Disallow: / | |||
</nowiki> | |||
Latest revision as of 13:35, 10 November 2025
Links
bad bots
Documentation
Examples
- https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html
- Good and bad bots
Examples
User-agent: PetalBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Dotbot Disallow: / User-agent: Amazonbot Disallow: / User-agent: GPTBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Applebot Disallow: / User-Agent: Thinkbot Disallow: /
