Does robots.txt hide pages from the internet?

No. robots.txt is a set of crawl instructions for compliant bots. It does not enforce access control—use authentication for private content.

Should I include a Sitemap line in robots.txt?

Usually yes. Adding a Sitemap directive helps crawlers discover your sitemap faster, especially on new sites.

What does Disallow: / do?

It tells crawlers not to crawl any paths on your site for the specified user-agent. It's commonly used for staging or private sites.

Is robots.txt the same as noindex?

No. robots.txt controls crawling. noindex is a directive (usually via meta robots) that controls indexing. If you block a page in robots.txt, Google may not crawl it and therefore may not see a noindex tag on that page.

Should I block /api or /_next?

Generally you don't need to block Next.js internal assets if they're not indexable pages. Focus on blocking sensitive/admin areas and low-value or duplicate content routes.

Generate robots.txt rules fast – Free Robots.txt Generator

Learn more

Need to generate a robots.txt file? Create allow and disallow rules plus a sitemap line you can copy to the root in seconds with this free tool.

robots.txt

Need robots.txt? Generate Allow/Disallow rules fast

Experience

What should bots do?

Note: robots.txt is guidance for crawlers, not access control. Don’t rely on it to protect private content.

Include sitemap

Adds a Sitemap: line to help crawlers discover URLs.

Sitemap URL