What is robots.txt?
seoDefinition
robots.txt is a text file placed at the root of a website (e.g., toolsdock.in/robots.txt) that instructs web crawlers which pages or directories they should or should not access. It uses Robots Exclusion Protocol directives like User-agent (which bot), Allow (permit access), and Disallow (block access). It is a request, not enforcement — crawlers can ignore it.
Detailed Explanation
Search engines like Google send automated programs called 'crawlers' or 'spiders' (e.g., Googlebot) to discover and index web pages. The robots.txt file tells these crawlers which parts of your site are off-limits for indexing.
A basic robots.txt that allows all crawlers: User-agent: * \n Allow: /. To block a specific directory: User-agent: * \n Disallow: /admin/. To block a specific bot: User-agent: AhrefsBot \n Disallow: /.
Critically: robots.txt does NOT prevent pages from being indexed if they have external links pointing to them. To prevent indexing of a specific page, use a <meta name="robots" content="noindex"> tag instead. Generate a compliant robots.txt for your site using ToolsDock's Robots.txt Generator.
A simple robots.txt, line by line
A minimal file might read: User-agent: * (applies to every crawler), Disallow: /admin/ (do not crawl that folder), Allow: /admin/help/ (an exception) and Sitemap: https://example.com/sitemap.xml. Rules are matched by path prefix, and Google also supports * as a wildcard and $ to mean end of URL, for example Disallow: /*.pdf$.
The file must sit at the root of the host, for example https://toolsdock.in/robots.txt. A robots.txt in a subfolder is ignored, and each subdomain needs its own. The Robots.txt Generator builds a valid file from a few choices.
What robots.txt cannot do
It does not hide a page. A blocked URL can still appear in search results, without a snippet, if other sites link to it. To keep a page out of the index, leave it crawlable and add a noindex robots meta tag or an X-Robots-Tag header; Google must be able to fetch the page to see that instruction. Google also does not support a noindex rule inside robots.txt.
It is not a security measure either. The file is public and well-behaved crawlers follow it by choice, so never rely on it to protect private data. Use authentication for that.
Mistakes that quietly cost traffic
The classic disaster is leaving 'Disallow: /' from a staging site on the live domain, which blocks the whole site. Another is blocking CSS and JavaScript folders, which stops Google rendering your pages properly. Crawl-delay is ignored by Google, though some other engines respect it.
After any change, test the file in Search Console and watch the Page indexing report for 'Blocked by robots.txt' entries. Pair it with an accurate XML sitemap so crawlers know which URLs matter.
🤖 Robots.txt Generator
Create a robots.txt file