CormorantDocs
Menu

Limits & politeness

Cormorant is a good citizen of the web on your behalf. That shapes how fast you can go — mostly per domain, not per project.

Hard limits#

Limit Value What happens
Requests per project 200/s by default (operator can set per project) 429 with Retry-After: 1. A batch call counts as one request.
Per-key limits set by the operator per key (see below) 429 key_quota_exhausted, key_budget_exhausted, key_rate_limited; 403 key_category_not_allowed
Synchronous wait 150 s (or timeout_s + 15 s) 504 {"detail": {"error": "timeout", …}}
timeout_s 5–150 s validation error outside the range
Items per batch job 50,000 413
Request body 64 MB 413 — split big batches
Fetch body size (max_bytes) default 15 MB, at most 50 MB error: "too_large"
PDFs in batch fetches up to 200 MB when the item leaves max_bytes unset (an explicit max_bytes applies to PDFs too) error: "too_large"
Large PDF download time 5 minutes in total (or timeout_s, when shorter) error: "timeout"
PDF text first 5,000 pages / 5,000,000 characters text_truncated: true; the full PDF is at body_url
Redirects at most 10 followed per fetch the fetch fails
Search max_results 1–50 validation error
Job page size (limit) 1–1,000 validation error
Retention jobs and results: 30 days after the job finishes; bodies: at least 30 days after your last retrieval 404 afterwards

Per-key limits#

The operator can give each API key its own limits, enforced on every request (ask for them, e.g. to give a client or a notebook a bounded key):

Limit What happens
Monthly spend cap (your invoice currency) 429 key_budget_exhausted until the month resets
Monthly requests per category: search, fetch (SEC EDGAR requests count here), fetch_premium, company search/fetch/company: 429 key_quota_exhausted; fetch_premium used up: fetches still run, without the premium routes
Requests per second 429 key_rate_limited with Retry-After: 1
Allowed categories (e.g. search only) 403 key_category_not_allowed; without fetch_premium, fetches run without the premium routes

Batch jobs are checked when created and again for each item when it runs: items beyond a quota fail with the same error code and are not billed. Request counts are exact; the spend cap compares the charges of the aggregates (refreshed every 5 minutes) plus an estimate of the requests since then, so a key can overshoot its cap by a few minutes of traffic.

Every authenticated response tells you where you stand:

RateLimit-Policy: "project";q=200;w=1, "key";q=10;w=1
RateLimit: "project";r=187;t=1, "key";r=9;t=1
X-Cormorant-Key-Budget-Remaining: 12.40 USD
X-Cormorant-Quota-Remaining: fetch=18766, search=900

RateLimit-Policy/RateLimit follow the IETF draft RateLimit header fields for HTTP (q limit, w window in seconds, r remaining, t seconds to reset). The two X-Cormorant-… headers appear only when the key has a spend cap or quotas. GET /v1/quota has the full picture, and POST /v1/alerts registers a signed webhook for 50 / 80 / 100 % of a limit.

robots.txt#

Cormorant checks robots.txt for the user-agent token CormorantBot (then *) before fetching and answers error: "robots_disallowed" when a path is off limits. robots.txt files are cached for a day; a missing robots.txt (4xx) allows everything.

Per-domain politeness#

To avoid hammering any single site, Cormorant limits concurrency per domain and egress IP, across all of its hosts:

  • at most 2 requests in flight to the same domain from the same egress,
  • at least 250 ms between request starts,
  • after a 429 or a bot challenge the interval doubles (up to 30 s) and decays again after about 10 minutes.

What this means for you:

  • Many URLs on many domains go fast. A batch of 10,000 URLs spread over 2,000 sites runs in parallel across all workers.
  • Many URLs on one domain go at the domain's polite rate — a few requests per second at best, slower if the site pushes back. 5,000 pages from one site take tens of minutes however you send them.
  • A domain that blocks every tier is skipped for a while. After three consecutive fetches where every tier failed, fetches for that domain answer domain_unavailable with retry_after_s (10 minutes at first, up to 6 hours) instead of spending time and money on it.
  • Batches handle this for you. Admission is domain-aware: a domain that dominates your job drains at its polite rate while the other domains proceed, and items waiting for a busy domain do not block workers.
  • Synchronous calls wait up to 15 s for a slot on a busy domain. When the expected wait is longer (the requests ahead of yours, divided over the domain's slots, times how long a request to that site takes), the call answers at once with HTTP 503, error: "domain_busy", retry_after_s and a Retry-After header instead of holding your connection for a minute. Hitting one domain with many parallel sync calls is the slowest way to use Cormorant.
  • A polite burst for a site you need several pages of. Send domain_concurrency (up to 4) with a fetch to let up to that many requests to the domain run at once — e.g. the 10–15 pages of a company's own site. It is ignored while the site pushes back (429s, challenges) and for hosts with their own site-wide limits (e.g. Wikimedia); the 250 ms between request starts still applies.

For an enrichment run (say 10–15 pages of each company's own site), create one batch job per slice of companies with an idempotency_key you can re-post safely, and set domain_concurrency: 4 on the items:

JSON
{
  "idempotency_key": "chipiron-run-2026-10-20-sites-0001",
  "items": [
    {"url": "https://www.nestle.com/investors", "domain_concurrency": 4,
     "project_ref": "company:nestle"},
    {"url": "https://www.nestle.com/sustainability", "domain_concurrency": 4,
     "project_ref": "company:nestle"}
  ]
}

Batch items never fail with domain_busy: a busy domain's items go back to the queue for a few seconds while other companies proceed, so a job of 10,000 companies × 12 pages finishes at the polite rate of every site without holding a connection or a worker. If the job creation request itself fails (timeout, 5xx), re-post it with the same key — you get the existing job, not a duplicate.

Search throughput#

Each SearXNG instance takes about 15 queries per second (a query takes ~4 s because engines are queried upstream); bursts queue instead of failing. The Brave fallback is capped cluster-wide. For hundreds of queries, use POST /v1/batch/search.

Practical advice#

  • Prefer one batch over many parallel sync calls.
  • Do not run your own retry loop on politeness_timeout or all_tiers_failed in tight succession — see Errors & retries.
  • Re-fetching the same URL within its cache lifetime is instant and billed as a cache hit; cache_ttl_s: 0 forces a real request — use it only when you need it.
  • If a site needs a contact User-Agent (SEC EDGAR, Wikimedia), ask the operator to add yours to the project allowlist and send it with user_agent.

View as Markdown