CrowBot
The crawler behind Crow. It visits public online storefronts to capture how they present themselves to shoppers.
What CrowBot does
Crow studies how online stores are built. For each storefront on a list that people choose by hand, CrowBot requests a few public pages, such as the homepage and product pages, and records what a visitor would see.
- The page HTML, both as the server sent it and as it looks after the page's JavaScript has run.
- Screenshots at desktop and mobile sizes, including marketing popups and cookie banners as a visitor first sees them.
- Public product data where the store publishes it, and the store's own sitemaps to find pages.
CrowBot does not sign in, add items to a cart, check out, or touch anything behind a login. It only reads pages that any visitor can open, and it may close a popup or cookie banner so the page underneath can be captured.
How it behaves
- Robots first. It reads
robots.txtbefore requesting anything else and skips every URL the file disallows. - Slow by design. It waits at least one second between requests to the same site.
- Slows down when asked. On an HTTP 429 it waits as long as
Retry-Aftersays, retries a few times, then stops visiting that site for the run. - Bounded. Every page has a fixed time budget, and large sitemaps are capped.
- Honest about who it is. It always sends the user agent below and signs its requests, so sites can tell it apart from impersonators.
Identification
- User-Agent
CrowBot/2.0 (+https://botcrow.com)- Verification
- CrowBot signs its requests with HTTP Message Signatures (Cloudflare Web Bot Auth, Ed25519).
- Public keys
- /.well-known/http-message-signatures-directory
- Operator
- Unit 203 Technologies Inc.
Opt out or get in touch
To keep CrowBot off your site, or to slow it down, add a rule for CrowBot to your robots.txt. CrowBot reads it on every visit, so a change takes effect on the next one.
For anything else, such as unexpected traffic, a removal request, or a question, write to support@trycrow.ai.