Ultineo Web

Robots and search engines

Search engines send robots to read your site. The robots.txt file tells them what they may crawl. On your published site, the base file lets them crawl every public page and gives them the address of your sitemap. Your preview is always closed to robots: that protection is automatic and cannot be changed.

Create the base file

  1. Open Website in the menu, then the Robots tab.
  2. Under Environment, choose Live site.
  3. When No custom crawler rules is shown, click Generate base file.
  4. Check the Website file address, then click Generate the file.

The base file contains User-agent: *, Allow: / and the address of your main sitemap. Click Open robots.txt to read it.

Add a rule for a robot

  1. On the Robots tab, with Live site chosen, click Add a rule.
  2. In Bot, write the name of a robot, such as Googlebot, or * for every robot that follows the rules.
  3. Under Rule, choose Allow or Disallow.
  4. In Path, write the part of your site concerned, starting with /, for example /drafts/.
  5. Click Save.

To change or delete a rule, use Edit the rule or Delete the rule on its line.

Note — If the tab says This site is on a trial hosting, where search engines always stay blocked., your site is on a trial hosting: search engines crawl none of its pages, whatever your rules. Your rules stay saved, but they do not change the robots.txt file of that hosting.

Important — Robots rules guide the robots that respect them; they do not protect anything. Never rely on them to hide a confidential page.