Free browser tool

robots.txt generator

Build a robots.txt file from user-agent, allow, disallow, and sitemap settings.

Input and settings

How to use it

Generate a simple robots.txt group with one User-agent, optional Allow/Disallow lines and an optional Sitemap URL.

Enter the User-agent, Allow paths and Disallow paths.
Add a sitemap URL if needed, or use one of the presets in the helper area.
Select Generate, then copy or download robots.txt.

User-agent: One group
Allow / Disallow: One path per line
Sitemap: Optional absolute URL
Output: Plain-text robots.txt

Block /admin/ while allowing the rest

User-agent: * with Allow: / and Disallow: /admin/ creates a basic crawler rule group plus the sitemap line if provided.

robots.txt controls crawler access guidance; it is not an authentication or privacy mechanism. Do not use it to protect confidential URLs.

Allow and Disallow lines are written in the order provided

The tool does not simulate crawler-specific precedence rules. It simply builds the text from your entries.

One generator run creates one User-agent group

Complex robots.txt files with several crawler-specific groups need manual editing or multiple generated sections.

Use a complete Sitemap URL

Enter the full Sitemap address, including https://, so the generated line can be used directly in robots.txt.

Blocking crawling is different from removing a page from search

A URL already known to a search engine can sometimes remain indexed even when crawling is blocked. Indexing control may require noindex or removal workflows.