Skip to content
BEAD

Robots.txt Generator

Build a robots.txt for your site with a guided UI.

SEOWeb
Preset
Block 1
robots.txt
User-agent: *
Disallow:

🔒 Generated entirely in your browser.

</>Use this tool programmaticallycurl · JavaScript · MCP

Same tool, callable from any HTTP client or from Claude (via MCP). Anonymous: 100 req/day per IP. Sign up for 1,000 req/day.

curl
curl https://api.b-e-a-d.com/tools/robots-txt-generator/run \
  -H "Content-Type: application/json" \
  -d '{
  "blocks": [
    {
      "agent": "a",
      "disallow": [],
      "allow": []
    }
  ]
}'
JavaScript (fetch)
const res = await fetch("https://api.b-e-a-d.com/tools/robots-txt-generator/run", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
  "blocks": [
    {
      "agent": "a",
      "disallow": [],
      "allow": []
    }
  ]
}),
});
const data = await res.json();
console.log(data.result);
MCP (Claude Desktop / Claude Code)
# In Claude Desktop / Claude Code, add to your MCP config:
{
  "mcpServers": {
    "bead": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://mcp.b-e-a-d.com"]
    }
  }
}

# Then ask Claude to call:
mcp__bead__bead_robots_txt_generator

Full reference: developer docs · OpenAPI spec

About robots.txt

A robots.txt file tells well-behaved crawlers which parts of your site to skip. It is advisory — malicious bots ignore it. Use it for crawl management, not security.

  • Put the file at the site root: https://example.com/robots.txt
  • User-agent: * applies to all crawlers
  • Disallow: alone (empty value) means allow everything
  • Include a Sitemap: line so crawlers can find your sitemap

You might also like