Not many internet master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be terribly useful in keeping the spiders indexing your actual pages and not different info, like wanting through your stats!

The robots.txt file is useful keep your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web web site content. You can select to own the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the internet stats portion of your site.

Several search engines cannot view dynamically generated content properly, mainly created by programming languages, like PHP or ASP. If you have got an on-line store programmed in your hosting account, and it’s in a seperate directory, you would be wise to block out the spiders from this directory so it solely finds relevant information.

The robots.txt file should be placed within the directory where your main files for your hosting are located. Thus you would be suggested to create a blank text file, and put it aside as robots.txt, and then upload it to your net hosting to the identical directory your index.htm file is located.

Here is samples of the use of the robots.txt file:

To dam out a directory in a very robots.txt file, such as a subdirectory for your on-line store known as /store/ you would do the subsequent: Disallow: /store/

Another example to dam out your stats directory: Disallow: /stats/

You may also wish to disallow individual files that you are doing not need searched by the search engines. For example you dont wish search.php to be parsed by the Search Engines. To try and do this you type in the subsequent on its own line:

Disallow: /search.php

Following the rules made public and making the robots.txt file, you’ll keep search engine spiders out of unwanted files and directories, and letting them bear the vital files to determine what your net site is all about!

To dominate 1st spot on every search engine, visit this site: seo cost. seo cost generates traffic and high ranking for your websites. You can use seo cost now to increase website’s rank on search engines and boost your business as well.