i just looked at the list of currently active members and i find these unusal things or peopleYahoo! Slurp Spider ,Google spider etcso what is this are they trackes from google or yahoo?did they come here to promote our site in their search lists?
i think using robots.txt file u can restrict these spiders to some level. u may not need this now but may be in future. now that google indexing is much better so u can expect more and more hits as time passes
most spiders are smart enough to slow down indexing if they find out that the server is not able to cope up with the traffic. google provides an option to slow down the speed in google webmaster panel.yahoo follows robots.txt. currently i have nothing to worry about the search engine spider load.
This website uses affiliate links. This means that if you click on a link and make a purchase, we may receive a commission. This does not affect the price you pay for the product.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.