How to update robots.txt file

As site owner you can simply update the robots.txt file from Site Settings area.

You can use the following content for the robots.txt file:

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty “/robots.txt” file, or don’t use one at all)

 

To exclude all robots from the entire server

User-agent: *
Disallow: /

 

To allow a single robot

User-agent: Google
Disallow:

 

Alternatively you can explicitly disallow all disallowed pages:

User-agent: *
Disallow: /~folder/junk.html
Disallow: /~folder/foo.html
Disallow: /~folder/bar.html

 

For advanced usage of robots.txt file, please check the following page: http://www.robotstxt.org/robotstxt.html

As an alternative you can check the robots.txt file using the following tool: http://tools.seobook.com/robots-txt/analyzer/