Robots.txt: Difference between revisions

From DWIKI
← Older editNewer edit →
Tony (talk | contribs)
7,300 edits
Tony (talk | contribs)
7,300 edits
Tag: wikieditor
Tony (talk | contribs)
7,300 edits
Tag: wikieditor
Line 1: Line 1:
=Links=
=Links=
*[https://stopbadbots.com/http-tools/ List of tool/bots you might want to block]
==bad bots==
*[https://www.humansecurity.com/learn/blog/crawlers-list-known-bots-guide/ List of tool/bots you might want to block]


==Documentation==
==Documentation==

Revision as of 08:34, 24 June 2025

Links

bad bots

Documentation

Examples

Examples

Retrieved from "https://wiki.dhits.nl/index.php?title=Robots.txt&oldid=9779"