Live
    Free tool

    Robots.txt Generator

    Create a valid robots.txt file to control how search engines crawl your site. Free and instant.

    User-agent: *
Allow: /
Disallow: /admin
Disallow: /login
Disallow: /dashboard

Sitemap: https://example.com/sitemap.xml

    What is robots.txt?

    The robots.txt file lives at the root of your domain and tells search-engine crawlers which paths they may or may not fetch. It's a polite request, not a security control — anything sensitive should be behind auth, not just disallowed.

    How to use this generator

    1. Pick which crawlers the rules apply to (all bots, or a specific user-agent like Googlebot).
    2. Add Allow and Disallow paths (e.g. /admin, /cart).
    3. Reference your XML sitemap with a Sitemap: line.
    4. Copy the output and upload it to https://yourdomain.com/robots.txt.

    Common patterns

    • Disallow: / under User-agent: * — blocks the whole site (careful — a common accidental production issue).
    • Disallow: /admin/ — hide admin routes from crawlers.
    • Allow: / with a specific Disallow: for a bot — carve out per-crawler behaviour.

    Related tools & guides

    FAQs

    Will disallowing a URL remove it from Google?

    No. Disallowing blocks crawling but not indexing. Use a noindex meta tag or HTTP header to remove pages from the index.

    Can I have more than one robots.txt?

    No — only one per host, at the root path.

    Automate everything

    GenPressa automates content production, optimisation and internal linking across your whole site.