Create a Robots.txt File
A robots.txt file lets you specify the robots (also known as "bots" or "spiders") you want to block from crawling files or directories on your site.
There are two important considerations when using a robots.txt:
- Robots can ignore your robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- The robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
Here is a sample robots.txt file that has many restrictions for commonly known scrapers and malware robots. Sample robots.txt
Contact | Tools | SEO Information | Sitemap
-
Quick Links
-
SEO Community Links
-
SEO Information
-
