Stop robots from getting site/article content

Question

This is not a question about indexers, most of them obey robots.txt file.

But maybe you noticed publicized cases where Facebook or Skype followed URL's posted by users or which were sent via private instant message using these platforms. I believe these two are not (or soon won't be) alone doing this.

The problem is that their visit to specific URL doesn't necessarily need to obey robots.txt and they might perform 'backup' of text what they see. Although I manage quite standard sites (non-profit orgs, hobby, blog), I don't like this 'ultimate marketing tool' (this was the best assumption what it is). Therefore I'm thinking of elegant way of avoiding such a sniffing/crawling coming from these sites after they get the link. Something like

• the entire page is not displayed until a human test is passed (then a cookie is set so this is not needed any more) – OR –

• the site is accessible (page with all modules is present etc., no problem with article titles published in them) but main article content (com_content) is initially empty and auto-replaced using Ajax shortly

Do you have any ideas how to implement prevention against these crawlers so they won't get any/some content of your site's URL they visited?

Answer 1

There is no easy solution, because bots have different behaviour. I would split them into 4 categories and solution for each:

1. Legitimate crawlers (i.e. Google) - these usually respect robots.txt and as you said, you are not interested in that.

2. Legitimate scrappers (i.e. Facebook) - these usually have the proper user agent, so you can block these based on that (the user agent).

3. Crawlers that don't respect robots.txt - easiest way is to create an invisible link (for humans) on your page, put it rel=nofollow (to not have issues with legitimate crawlers) to that link. The link should be going to a page, where you will log IP, UA and similar stuff of the crawler. Then you can implement a check against that log and if there is match, you will simply not should the content.

4. One of scrappers - similar to Facebook, but with fake user agent. The only viable solution is to check for cookies support, yet it is not 100% guarantee it will work. It will also block users who have disabled cookies.

Answer 2

One method would be to block accesses from crawlers, though this depends on being able to identify them.

The following article has some good examples of how to enhance the Joomla security.

http://docs.joomla.org/Htaccess_examples_%28security%29

One method is to use the HTTP_REFERER attribute to block access for example to images unless they are loaded from a page on your site.

Another is to use the HTTP_USER_AGENT attribute to block access to particular engines.

To determine additional the HTTP_USER_AGENT strings you would need to inspect your access logs, or, what I've done on one occasion is to modify the Redirect extension to add the HTTP_USER_AGENT in the comment.

Answer 3

If you don't want content to be publicly accessible then use Joomla's ACL to restrict access, removing access for the default (public) usergroup. If you don't do that, or something similar, then your content is public, people can see it, bots can see it.

If you don't want your content viewed by anyone, don't publish it to the world. If you do publish it to the world you don't get to have much say in who views it.