Block a directory using robots.txt
How to block a directory and its subdirectories using the robots.txt file.
Edited: 2017-12-19 17:14
Sometimes you may have a directory containing decorative images or temp files, and you do not necessarily want those files to be indexed by search engines.
Keep in mind that the Robots.txt file is not intended for access control. If you want to keep some data safe, then you should not have made it publicly accessible in the first place. See also: Robots.txt and Security
The best way to make sure certain files will not be indexed, is to add password protection to the directory. However, such extreme measures are not always necessary, and besides, the content of the directory could still be used on publicly available pages. In such cases, we can try to noindex the directory to keep it out of the major search engines.
Blocking a directory with robits.txt
If you just have a single directory, such as a directory called images, containing files and sub-directories you do not want indexed, you can use the below piece of code in your robots.txt file:
User-agent: * Disallow: /images/
If you want to prevent access to multiple directories, simply list them below each other, like done in this example:
User-agent: * Disallow: /images/ Disallow: /temp/ Disallow: /cgi-bin/
Tell us what you think: