What to know before using it
robots.txt mainly controls crawler access. Blocking a page can prevent a crawler from seeing a noindex directive on that page.
Free browser tool
Check common directives and basic syntax in a pasted robots.txt file.
robots.txt syntax checker processes your input in the browser, then lets you inspect or download the result.
Keep originals when editing important files and verify the downloaded result before replacing the source.
robots.txt mainly controls crawler access. Blocking a page can prevent a crawler from seeing a noindex directive on that page.
For an internal page, choose noindex or authentication based on the goal rather than relying on robots.txt alone.