What is robots.txt?
seoDefinition
robots.txt is a text file placed at the root of a website (e.g., toolsdock.in/robots.txt) that instructs web crawlers which pages or directories they should or should not access. It uses Robots Exclusion Protocol directives like User-agent (which bot), Allow (permit access), and Disallow (block access). It is a request, not enforcement — crawlers can ignore it.
Detailed Explanation
Search engines like Google send automated programs called 'crawlers' or 'spiders' (e.g., Googlebot) to discover and index web pages. The robots.txt file tells these crawlers which parts of your site are off-limits for indexing.
A basic robots.txt that allows all crawlers: User-agent: * \n Allow: /. To block a specific directory: User-agent: * \n Disallow: /admin/. To block a specific bot: User-agent: AhrefsBot \n Disallow: /.
Critically: robots.txt does NOT prevent pages from being indexed if they have external links pointing to them. To prevent indexing of a specific page, use a tag instead. Generate a compliant robots.txt for your site using ToolsDock's Robots.txt Generator.
🤖 Robots.txt Generator
Create a robots.txt file