Block a directory using robots.txt

How to block a directory and its subdirectories using the robots.txt file.


Edited: 2017-12-19 17:14

Sometimes you may have a directory containing decorative images or temp files, and you do not necessarily want those files to be indexed by search engines.

Keep in mind that the Robots.txt file is not intended for access control. If you want to keep some data safe, then you should not have made it publicly accessible in the first place. See also: Robots.txt and Security

The best way to make sure certain files will not be indexed, is to add password protection to the directory. However, such extreme measures are not always necessary, and besides, the content of the directory could still be used on publicly available pages. In such cases, we can try to noindex the directory to keep it out of the major search engines.

Blocking a directory with robits.txt

If you just have a single directory, such as a directory called images, containing files and sub-directories you do not want indexed, you can use the below piece of code in your robots.txt file:

User-agent: *
Disallow: /images/

If you want to prevent access to multiple directories, simply list them below each other, like done in this example:

User-agent: *
Disallow: /images/
Disallow: /temp/
Disallow: /cgi-bin/

Tell us what you think:

  1. Drop in traffic doing recent helpful content updates; time will tell if I will recover.
  2. Let us investigate what is up with those mass spammed *-k.html backlinks that many of us are seeing in our link reports.
  3. An in-dept look at the use of headings (h1-h6) and sections in HTML pages.
  4. Pagination can be a confusing thing to get right both practically and programmatically. I have put a lot of thought into this subject, and here I am giving you a few of the ideas I have been working with.
  5. The Video outside the viewport is properly not worth spending time on solving; it is probably intended to solve a specific issue, and every single little video probably does not need to get indexed anyway.

More in: SEO