First, a web site cannot get your browser's cookies that were put onto your computer from
other web sites.
The robots/spiders simply retrieve the public content from every website, index it, and then let you search it. Google is obviously the most famous of these search engines that uses spiders.
So, there is some confusion being introduced in the posts above. Yes, there are ways to track your behavior, but none of these are done by spiders. The site you are on (including this one) does tracking via cookies. Once the site has established a "connection" with your browser, unless you turn off cookies, that site can send a small, unique code to your computer (a "cookie") and your browser will store that code, along with thousands of others, in the cookie cache. When you re-visit the same site, that site can ask for any cookies that it put into your browser, retrieve them, and then know it is you.
These cookies not only let the site recognize you so you don't have to log on each time you visit, but can also be used to track what you click on. This lets the site offer suggestions that might be useful to you.
Amazon does this better than almost any other site, where they offer product suggestions based on past behavior.
Another type of tracking is when you click on a link to go to some external site. The site to which you link can be told which site you were on, and the referring site can get a "kick-back" fee for the referral.
Finally (well, finally for me, because there is much more to this topic), if you click on an ad, you are not actually taken directly to the site, even though it may feel like that. Instead, most ad clicks are directed through an intermediate site which logs the referral, so the referring site can get paid; takes care of actually generating invoices and payments; but most important to we users, it plants a cookie in your computer, and when you are on a completely unrelated site, that cookie can be retrieved by another site, but only by going back to this intermediate company who is the only company that can retrieve that cookie, because they put it on your browser. It is that intermediate ad company that provides the "magic" of correlating your behavior across multiple web sites. Without that intermediary, no web site can know what you have done at other sites.
I don't normally provide links to other forums because, unlike this forum which has a very high percentage of knowledgeable people, most sites contain more junk than not. However, the first few posts in this conversation provides more meat to what I just described:
Can a webpage read another page's cookies?
