Robots.txt: Difference between revisions
From DWIKI
m →Links Tag: wikieditor |
Tag: wikieditor |
||
| Line 2: | Line 2: | ||
==Documentation== | ==Documentation== | ||
*[https://yoast.com/ultimate-guide-robots-txt/ The ultimate guide to robots.txt] | *[https://yoast.com/ultimate-guide-robots-txt/ The ultimate guide to robots.txt] | ||
*[https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt Google doc about robots.txt] | |||
==Examples== | |||
*https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html | *https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html | ||
*[https://simtechdev.com/blog/good-and-bad-bots-to-control-to-save-server-resources-and-improve-performance/ Good and bad bots] | *[https://simtechdev.com/blog/good-and-bad-bots-to-control-to-save-server-resources-and-improve-performance/ Good and bad bots] | ||
=Examples= | =Examples= | ||
Revision as of 14:28, 28 May 2025
Links
Documentation
Examples
- https://www.link-assistant.com/news/10-robots-txt-files-worth-to-have-a-look-at.html
- Good and bad bots
