Robots.txt Generator

Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Robots.txt is a file that contains instructions on how to crawl a website. It is also known as bot exclusion protocol, and sites use this standard to tell bots what to do. part of your website needs indexing. In addition, you can specify which areas you do not want these trackers to process; such areas contain duplicate content or are under development. Bots like malware scanners, email harvesters do not follow this standard and will look for weaknesses in your settings, and there is a considerable chance that they will start scanning your site from the areas you don't want. index.

A complete Robots.txt file contains "User-agent", and below it, you can write other directives like "Allow", "Disallow", "Crawl-Delay", etc., if is typed manually, can be time-consuming, and can enter multiple lines of commands in one file. If you want to exclude a page, you must write "Don't allow: the link you don't want bots to visit", the same goes for the permission attribute. If you think that's all there is to the robots.txt file, then it's not easy, one wrong line can exclude your page from the index queue. Therefore, it is better to leave the task to the professionals, let our Robots.txt generator take care of the file for you. More about  Robots.txt...

The Robots txt file is easy to make, but people who don't know how to do it should follow the instructions below to save time.

• When you have reached the page of the generator of new robots txt , you will see the new robots txt generator. a couple of options, not all are required, but you should choose carefully. The first row contains default values ​​for all bots and whether to keep a crawl delay. Leave them as they are if you don't want to change them as shown in the image below:
• The second row is about the sitemap, make sure you have one and don't forget to mention it in the robot's txt file.
• After this you can choose from a couple of options for the search engines if you want the search engine bots to crawl or not, the second block is for images if you are going to If you want to allow it to be indexed, the third column is for the mobile version of the website.
• The last option is to disallow, where it will restrict the option. It allows crawlers to index the areas of the page. Be sure to add the forward slash before completing the field with the address of the directory or page.