who or what is this?

coolbal

Regulars
[OP]
Regulars
Jul 15, 2006
509
1
i just looked at the list of currently active members and i find these unusal things or peopleYahoo! Slurp Spider ,Google spider etcso what is this are they trackes from google or yahoo?did they come here to promote our site in their search lists?
 

amish

Star gazer
Regulars
Jul 3, 2006
24,699
1,068
Mumbai
so many spiders? wudnt they chew up bandwidth?
 


Sushubh

Administrator
Oct 29, 2004
416,850
12,926
Gurugram
pretty true. i have not marketed this forum personally. all the traffic it has is due to search engines.



a good example of how good google is with indexing of sites. incidentally techwhack is banned in msn live search engine.
 


amish

Star gazer
Regulars
Jul 3, 2006
24,699
1,068
Mumbai
no no i know spiders are good for promotion. but yesterday i saw way too many yahoo spiders.
 

Sushubh

Administrator
Oct 29, 2004
416,850
12,926
Gurugram
there is nothing like too many search engine spiders... :) as long as you dont run out of SQL connections... and we are no longer. for now atleast.
 

amish

Star gazer
Regulars
Jul 3, 2006
24,699
1,068
Mumbai
i think using robots.txt file u can restrict these spiders to some level. u may not need this now but may be in future. now that google indexing is much better so u can expect more and more hits as time passes
 

Sushubh

Administrator
Oct 29, 2004
416,850
12,926
Gurugram
most spiders are smart enough to slow down indexing if they find out that the server is not able to cope up with the traffic. google provides an option to slow down the speed in google webmaster panel.yahoo follows robots.txt. currently i have nothing to worry about the search engine spider load.