This is not a question about indexers, most of them obey robots.txt file.
But maybe you noticed publicized cases where Facebook or Skype followed URL's posted by users or which were sent via private instant message using these platforms. I believe these two are not (or soon won't be) alone doing this.
The problem is that their visit to specific URL doesn't necessarily need to obey robots.txt and they might perform 'backup' of text what they see. Although I manage quite standard sites (non-profit orgs, hobby, blog), I don't like this 'ultimate marketing tool' (this was the best assumption what it is). Therefore I'm thinking of elegant way of avoiding such a sniffing/crawling coming from these sites after they get the link. Something like
the entire page is not displayed until a human test is passed (then a cookie is set so this is not needed any more) – OR –
the site is accessible (page with all modules is present etc., no problem with article titles published in them) but main article content (com_content) is initially empty and auto-replaced using Ajax shortly
Do you have any ideas how to implement prevention against these crawlers so they won't get any/some content of your site's URL they visited?