- This topic has 1 reply, 2 voices, and was last updated 1 year, 6 months ago by
.
-
Topic
-
Viewing 1 replies (of 1 total)
-
Replies
-
Friendly BOTS are fine provided I have allowed them to crawl the site.
Most BOTS you see will actually be ‘web crawlers’ such as Google and Bing
Web Crawler bots use data provided by the webmaster (me) of the web site which is provided by a combination of a site map, robot.txt and the webmaster accounts on sites like Google.
Currently, I have set all crawlers not to index the forum until I have removed all the test posts over the next week.
Viewing 1 replies (of 1 total)
- You must be logged in to reply to this topic.