How to update robots.txt file

As site owner you can simply update the robots.txt file from Site Settings area.

You can use the following content for the robots.txt file:

To allow all robots complete access

User-agent: *

(or just create an empty “/robots.txt” file, or don’t use one at all)


To exclude all robots from the entire server

User-agent: *
Disallow: /


To allow a single robot

User-agent: Google


Alternatively you can explicitly disallow all disallowed pages:

User-agent: *
Disallow: /~folder/junk.html
Disallow: /~folder/foo.html
Disallow: /~folder/bar.html


For advanced usage of robots.txt file, please check the following page:

As an alternative you can check the robots.txt file using the following tool: