Yoast SEO: robots.txt - functional specification

This documentation explains how Yoast SEO modifies robots.txt files.

Default content

When a site does not have a user-defined robots.txt file (at the point of installation/activation), Yoast SEO generates a default output.

# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK

If a robots.txt file already exists and contains non-empty rules, this block should be prefixed before any existing rules.

Controls

Users must be able to edit the contents of their robots.txt file.