SuperstrateBot

1. What SuperstrateBot is

SuperstrateBot is the website reader used by Superstrate. It reads the public pages of websites whose owners use Superstrate, so their on-site assistant answers from up-to-date content.

It also reads a site's public pages when a website check is requested for that site.

2. How to recognise it

User agent:

Mozilla/5.0 (compatible; SuperstrateBot/<version>; +https://superstrate.com.au/bot)

The version number changes with our releases; match on the word SuperstrateBot.

We do not publish fixed IP addresses (the reader runs on Google Cloud and its addresses change), so please identify it by user agent, not by IP.

3. How it behaves

• Reads public pages only. It never logs in, never submits forms, never buys or books anything.
• Follows robots.txt: rules for SuperstrateBot first, otherwise the * rules.
• Honours Crawl-delay (up to 10 seconds).
• Is gentle: at most 2 requests at a time to one site, at least half a second apart.
• May load a page in a headless browser when the plain page has too little text.

4. How to block it

Add to robots.txt:

User-agent: SuperstrateBot
Disallow: /

5. How to allow it

If your firewall or bot protection blocks it:

robots.txt

User-agent: SuperstrateBot
Allow: /

Cloudflare

Create a WAF custom rule with the expression

(http.user_agent contains "SuperstrateBot")

and the action Skip (skip the bot protection features your plan allows).

Note that on some plans Bot Fight Mode cannot be skipped by a rule and may need to be turned off for the reader to get through.

6. Contact