Fetching in depth

POST /v1/fetch turns a URL into text, metadata and provenance. This guide explains what happens in between and how to steer it.

The order of operations#

1. Normalise the URL (lower-case host, default port and fragment dropped, tracking parameters such as utm_*, gclid, fbclid removed, remaining parameters sorted). This is the cache key.
2. Cache. A fresh entry that satisfies your request answers at once, with cached: true and cost_usd: 0. The cache is shared by all projects: HTML pages stay fresh for 7 days, PDFs for 30. A stale entry is revalidated with a conditional request: if the origin answers 304 Not Modified, you get the cached copy with cached: true and revalidated: true, billed as a cache hit.
3. Safety. Private, loopback, link-local and metadata addresses are refused (blocked_destination), also after redirects. A host whose name does not exist (NXDOMAIN twice within 6 h) answers error: "domain_dead" at once for 6 h — or its archived copy, see below.
4. robots.txt is honoured for CormorantBot and * (robots_disallowed). A few hosts are fetched only through the premium routes, never from Cormorant's own addresses. For them, allow_premium: false answers premium_route_required, robots.txt is read through the proxy, and a site-wide throttle applies.
5. Domain memory. A domain where every tier failed in 3 consecutive fetches is skipped for a while (10 min, then 30 min, 2 h, 6 h) and answers error: "domain_unavailable" at once, with retry_after_s. Any success ends the skip. Cormorant also remembers which tier (and rendering difficulty) worked for a domain and starts there next time, and which domains refuse even the premium routes: those skip proxy/proxy_render for 6 h (then 24 h, 3 days) and go to the archive and alternative sources instead, so a blocked domain stops costing premium bytes (premium_skipped_domain_refuses attempts).
6. Tiers run until one produces a good page.

Tiers and escalation#

The ladder starts at direct (or at a tier learned for the domain, see architecture). It escalates on network errors, HTTP 403/429/451/5xx and bot-challenge pages (Cloudflare, Akamai, DataDome, PerimeterX, Imperva, Kasada, generic interstitials). Thin main text escalates direct → browser only. archive and the premium routes are used only when the other standard routes were blocked — never for a page that is simply short. Definitive answers (404, 410, 401, …) are returned without escalation — a 404/410 on direct never reaches a premium route (at most the archived copy, see below). No CAPTCHA is solved unless the operator enabled it for the project.

Domains that refuse the premium routes too. When a fetch fails on every tier and every premium route it tried was refused by the site (403/429/451, a challenge, a rendering-tier captcha or "cannot render", the provider's content policy — not provider errors or timeouts) in 2 fetches in a row, the domain is premium-blocked for 6 hours (then 24 h, then 3 days on repeats): its fetches still try direct, browser and the archive, but skip proxy and proxy_render (attempt reason premium_skipped_domain_refuses, no premium bytes) and go straight to the alternative sources, or fail with all_tiers_failed. 5 % of these fetches re-probe the premium routes anyway; one premium success ends the block.

Timing. Each attempt's ms splits into wait_ms (waiting for a polite per-domain slot) and upstream_ms (the site, browser, archive or premium route). timing sums them for the request and adds queue_ms, the time in Cormorant's queues before a worker ran it (synchronous: from the API to a worker; batch: since admission, politeness requeues included; plus the render queue after a browser handoff): {"queue_ms": 35, "politeness_wait_ms": 0, "upstream_ms": 412}. The same three components are stored per request in the request log and summarised in GET /v1/usage.

Every attempt carries premium_bytes (what the residential network billed for it; 0 for standard routes and proxy_render, which bills per call). The proxy route can show more than one attempt: when the site refuses a residential exit, or the network fails, the request is retried at once through another residential network, each attempt with its own status, reason and premium bytes. Prices are per route and do not depend on which network served the page. A failed fetch says which premium routes were tried in premium_routes_tried and summarises every attempt in error_detail, e.g. "all_tiers_failed: direct (http_403), browser (challenge_akamai), proxy (http_403, 16211 premium bytes), proxy_render (http_403); premium routes tried: proxy, proxy_render (16211 premium bytes)".

Dead domains and missing pages. A host whose name does not exist (NXDOMAIN — not a temporary DNS failure such as SERVFAIL or a timeout, which stay blocked_destination with reason dns_error) twice within 6 hours is remembered as dead for 6 hours: fetches answer error: "domain_dead" at once, with a single {"tier": "dns", "reason": "domain_dead"} attempt and no tier work. When a fetch ends in domain_dead or a 404/410 and allow_archive is true (and tiers, if given, includes archive), Cormorant looks for a Wayback snapshot (≤ 365 days) and, if there is one, returns it instead: tier: "archive", archived_at, and served_from_archive_because: "domain_dead" or "not_found"; attempts still shows the 404 or the dead host. These stand-in copies are not cached. The archive's paywall rules apply (no copies for known paywalled domains or of articles marked subscriber-only, archive_blocked_paywall). Without a snapshot the answer stays domain_dead or http_404/http_410 (with an archive_not_found attempt), and only then are alternative sources looked for. Other definitive statuses (401, …) never use the archive.

render=never uses direct → archive → proxy; render=always uses browser → proxy_render.

Known hard domains. For sites on Cormorant's list of hard domains (nestle.com, nestleusa.com, beyondmeat.com, apnews.com, nytimes.com, researchgate.net, … — see Fetching → Known hard domains) tiers that do not work for the site are skipped (hard_domain_skip attempts), the residential exit country is pinned (unless the request sets country) and proxy_render starts at the site's difficulty. Errors — and archive or alternative copies standing in — carry hint (the recommended route) and, on errors, alternative_urls. An explicit tiers list overrides the route. The proxy tier sends GET requests with Chrome's TLS/HTTP2 fingerprint, like direct.

attempts[].reason values include ok, thin_content, not_modified, challenge_<vendor>, http_<status>, timeout, tls_error, dns_error, dns_nxdomain, domain_dead, network_*, not_configured, budget_exhausted, hard_domain_skip, archive_not_found, premium-route reasons (…_captcha_rejected, …_no_credits, …_render_503), paywall (teaser of a hard paywall, returned as is), paywall_skip_premium, premium_skipped_domain_refuses, archive_blocked_paywall, and for the alternative-source steps (tier: source) alternate_<source> (with the copy's URL in detail), syndication_candidates_<n>, open_access_none, open_access_miss, guardian_api_miss, publisher_feed_miss, publisher_feed_teaser, common_crawl_none, common_crawl_not_free, common_crawl_paywalled_domain, common_crawl_index_error, common_crawl_record_error, source_timeout. proxy_render attempts carry detail with the difficulty used (low, medium, high); other attempts omit detail. Busy rendering-tier answers (…_render_503) are retried once and appear as two attempts.

What you get back#

Non-HTML content#

content_kind tells you how to read text:

• json — text is the decoded document (still valid JSON);
• xml — feeds, sitemaps, XBRL…: one text node per line, title from the first <title>; external entities are never resolved;
• text — plain text, CSV, Markdown, decoded;
• binary — spreadsheets, archives, images…: text is empty, download the bytes from body_url.

All of these are successful fetches (error: null).

PDFs#

PDFs are accepted by default (accept_pdf: false turns them into error: "pdf_not_accepted"). Text is extracted per page; use pdf_text_pages to cite page numbers. Large reports may need a higher max_bytes (default 15 MB, at most 50 MB); a bigger body is error: "too_large".

Exact bytes and provenance#

Every fetch stores the exact bytes it retrieved, content-addressed by sha256, and returns body_url (null when object storage is unavailable). body_kind says what the hash covers:

• origin: the HTTP entity body the origin (or the archive, for tier = archive) sent, after content-encoding (gzip/br) is removed — the bytes a plain HTTP client would hand you;
• rendered_dom: the document serialized by a browser (browser, proxy_render) after scripts ran — use it for text, not as proof of the origin's bytes. Non-HTML documents a browser downloads (PDFs) are origin.

GET /v1/bodies/{sha256} (same Bearer key) streams the bytes with the origin Content-Type, Content-Length, ETag: "<sha256>", X-Body-Kind and Cache-Control: immutable; If-None-Match gives 304; HEAD returns the headers only. A project can download only bodies it has retrieved itself (fresh or from the shared cache); anything else is 404. Bodies are kept while a cache entry points to them or a project retrieved them within the last 30 days. Downloading a body (Python and TypeScript continue the Quickstart client):

curl -sS "$BODY_URL" -H "Authorization: Bearer $CORMORANT_API_KEY" -o report.pdf

page = client.post(
    "/v1/fetch", json={"url": "https://www.example.com/annual-report-2025.pdf"}
).json()
if page["body_url"]:
    with client.stream("GET", page["body_url"]) as response:
        response.raise_for_status()
        with open("report.pdf", "wb") as out:
            for chunk in response.iter_bytes():
                out.write(chunk)

const bytes = await fetch(page.body_url, { headers }).then((r) => r.arrayBuffer());

For citations, keep final_url, retrieved_at, sha256 and body_kind with the quote. For tier = archive, final_url is the Wayback Machine copy and archived_at the snapshot time.

Cache semantics#

• cache_ttl_s is the maximum age you accept. 0 bypasses the cache and forces a fresh fetch.
• A cache hit is used only if it satisfies your request: render: "always" needs a rendered copy and render: "never" an origin copy; HTML text must reach your min_text_chars; a cached body larger than your max_bytes answers too_large, exactly as a fresh fetch would; with tiers set, hits from other tiers are not used.
• Pages that are simply short (direct and the browser both gave less text than min_text_chars, nothing blocked) are cached for a day as such and answer later requests; thin pages of blocked fetches are not cached.
• A redirected page is cached under the URL the redirects ended at too.
• Hits are billed as cache hits (fetch_cached), even if the page originally came from a premium route.
• Cache classes, TTLs and how to get hits across runs: Caching and reuse across runs.

Per-request controls#

A cost-capped, latency-capped fetch for an agent:

{
  "url": "https://www.holcim.com/investors",
  "allow_premium": false,
  "timeout_s": 45,
  "include_html": false
}

A filing from a service that requires a contact User-Agent:

{
  "url": "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193",
  "user_agent": "Example Research contact@example.org",
  "tiers": ["direct"],
  "render": "never"
}

Wikipedia, Wikidata and other Wikimedia sites#

Fetches of *.wikipedia.org, *.wikidata.org, *.wikimedia.org and the sister projects follow Wikimedia's User-Agent policy for you: Cormorant identifies itself with its own descriptive User-Agent and the operator's contact (or your allowlisted user_agent, when you send one), uses the direct tier only (no browser or premium routes, so these fetches are never charged as premium) and keeps the number of concurrent requests to all Wikimedia hosts low across all clients. Under load, expect requests to wait briefly for a slot rather than fail; if Wikimedia answers 429 the fetch fails (all_tiers_failed, attempts shows http_429) and every worker slows down for a few minutes, so retry later rather than at once. Prefer the APIs (https://en.wikipedia.org/api/rest_v1/page/summary/Zurich, https://www.wikidata.org/wiki/Special:EntityData/Q72.json) to HTML pages: they are smaller and cached the same way.

Known hard domains#

Some sites block automated access from datacenter addresses on every route Cormorant has, or answer only behind a paywall. Cormorant ships a list of them with the route that works best per site (operators can extend or override it with HARD_DOMAINS_FILE):

Skipped tiers show as hard_domain_skip attempts; a request's own tiers list overrides the route, and its country wins over the route's exit country. When such a fetch fails — or an archived or alternative copy stands in — the result carries hint (the route in one sentence) and, on errors, alternative_urls (e.g. the page's Wayback Machine URL, or an OpenAlex search built from a ResearchGate URL). The residential proxy tier always presents Chrome's TLS/HTTP2 fingerprint (like direct), and the browser tier presents a headful Chrome identity (no HeadlessChrome in the User-Agent or client hints).

For a company's own site (10–15 pages per company), use a batch fetch with an idempotency_key: batch items wait out busy and blocked domains instead of failing, and may use Common Crawl.

Errors#

The fetch-level error codes, what they mean and whether to retry them are in Errors & retries.