SEO
robots.txtrobots.txt
A plain-text file at /robots.txt that tells crawlers which paths they're allowed to fetch.
What is robots.txt?
A plain-text file at /robots.txt that tells crawlers which paths they're allowed to fetch.
In practice
Public Pulse's robots.txt explicitly allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot for the public site, and explicitly disallows /manage and /api/auth. The AI-crawler allow-list is intentional — we want our pages cited in AI answers.
FAQ
Frequently asked.
- What does robots.txt mean?
- A plain-text file at /robots.txt that tells crawlers which paths they're allowed to fetch.
- Why does robots.txt matter for Bangladesh brands?
- Public Pulse's robots.
- Does Public Pulse use robots.txt in client engagements?
- Yes. robots.txt is part of how we deliver seo for clients. Walk through it on a free consultation call.