5

This is not a question about indexers, most of them obey robots.txt file.

But maybe you noticed publicized cases where Facebook or Skype followed URL's posted by users or which were sent via private instant message using these platforms. I believe these two are not (or soon won't be) alone doing this.

The problem is that their visit to specific URL doesn't necessarily need to obey robots.txt and they might perform 'backup' of text what they see. Although I manage quite standard sites (non-profit orgs, hobby, blog), I don't like this 'ultimate marketing tool' (this was the best assumption what it is). Therefore I'm thinking of elegant way of avoiding such a sniffing/crawling coming from these sites after they get the link. Something like

  • the entire page is not displayed until a human test is passed (then a cookie is set so this is not needed any more) – OR –

  • the site is accessible (page with all modules is present etc., no problem with article titles published in them) but main article content (com_content) is initially empty and auto-replaced using Ajax shortly

Do you have any ideas how to implement prevention against these crawlers so they won't get any/some content of your site's URL they visited?

  • Thank you for all your insights. I'll try implementing some and then will return to mark an answer. – miroxlav May 21 '14 at 6:17
3

There is no easy solution, because bots have different behaviour. I would split them into 4 categories and solution for each:

  1. Legitimate crawlers (i.e. Google) - these usually respect robots.txt and as you said, you are not interested in that.

  2. Legitimate scrappers (i.e. Facebook) - these usually have the proper user agent, so you can block these based on that (the user agent).

  3. Crawlers that don't respect robots.txt - easiest way is to create an invisible link (for humans) on your page, put it rel=nofollow (to not have issues with legitimate crawlers) to that link. The link should be going to a page, where you will log IP, UA and similar stuff of the crawler. Then you can implement a check against that log and if there is match, you will simply not should the content.

  4. One of scrappers - similar to Facebook, but with fake user agent. The only viable solution is to check for cookies support, yet it is not 100% guarantee it will work. It will also block users who have disabled cookies.

  • Thank you for splitting the problem into categories and insight into them. What do you think about using Ajax article loader? Would it prevent most bots from scraping? – miroxlav May 14 '14 at 9:03
  • Depends on the bot, bots can follow AJAX request, yet not all of them. – Ivo May 19 '14 at 12:26
3

One method would be to block accesses from crawlers, though this depends on being able to identify them.

The following article has some good examples of how to enhance the Joomla security.

http://docs.joomla.org/Htaccess_examples_%28security%29

One method is to use the HTTP_REFERER attribute to block access for example to images unless they are loaded from a page on your site.

Another is to use the HTTP_USER_AGENT attribute to block access to particular engines.

To determine additional the HTTP_USER_AGENT strings you would need to inspect your access logs, or, what I've done on one occasion is to modify the Redirect extension to add the HTTP_USER_AGENT in the comment.

  • HTTP_USER_AGENT can be spoofed, only few programs leave something like Java/1.6.0_30. I would use this recognition only as last resort. – miroxlav May 8 '14 at 22:30
  • 1
    Yes, though you had made specific reference to crawling by well known sites, so this should block the majority of cases. But yes, not all of them. And it requires diligence to maintain the user agent strings. – Peter Wiseman May 8 '14 at 22:36
3

If you don't want content to be publicly accessible then use Joomla's ACL to restrict access, removing access for the default (public) usergroup. If you don't do that, or something similar, then your content is public, people can see it, bots can see it.

If you don't want your content viewed by anyone, don't publish it to the world. If you do publish it to the world you don't get to have much say in who views it.

  • you wrote: ...people can see it, bots can see it.. And I'm wondering how a difference can be made here :) 'Public' is being redefined with technology progress. I think many people wouldn't object if passer-by wants to take picture of their distinctive house from public place outside their property. But if they find that guy installing time-lapse camera (although taking the same scene as before), I'm sure many will object. Remember affair of Google sniffing WiFi routers. And why there are places where no photo-taking is allowed? I understand that someone doesn't care. Someone does. – miroxlav May 9 '14 at 0:04
  • If you don't want your content to be public, you don't have to make it public. The choice is yours. – Seth Warburton May 9 '14 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.