SEO
robots.txtrobots.txt
robots.txt is a plain-text file at /robots.txt that tells crawlers which paths they are allowed to fetch. It can explicitly allow named crawlers, such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot, and disallow private paths like /manage. Allowing AI crawlers is a deliberate choice so pages can be cited in AI answers.
What is robots.txt?
robots.txt is a plain-text file at /robots.txt that tells crawlers which paths they are allowed to fetch. It can explicitly allow named crawlers, such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot, and disallow private paths like /manage. Allowing AI crawlers is a deliberate choice so pages can be cited in AI answers.
In practice
Public Pulse's robots.txt explicitly allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot for the public site, and explicitly disallows /manage and /api/auth. The AI-crawler allow-list is intentional — we want our pages cited in AI answers.
FAQ
Frequently asked.
- What does robots.txt mean?
- robots.txt is a plain-text file at /robots.txt that tells crawlers which paths they are allowed to fetch. It can explicitly allow named crawlers, such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot, and disallow private paths like /manage. Allowing AI crawlers is a deliberate choice so pages can be cited in AI answers.
- Why does robots.txt matter for Bangladesh brands?
- Public Pulse's robots.
- Does Public Pulse use robots.txt in client engagements?
- Yes. robots.txt is part of how we deliver seo for clients. Walk through it on a free consultation call.