# Fetching in depth

`POST /v1/fetch` turns a URL into text, metadata and provenance. This guide
explains what happens in between and how to steer it.

## The order of operations

1. **Normalise** the URL (lower-case host, default port and fragment dropped,
   tracking parameters such as `utm_*`, `gclid`, `fbclid` removed, remaining
   parameters sorted). This is the cache key.
2. **Cache.** A fresh entry that satisfies your request answers at once,
   with `cached: true` and `cost_usd: 0`. The cache is shared by all
   projects: HTML pages stay fresh for 7 days, PDFs for 30. A stale entry is
   revalidated with a conditional request: if the origin answers `304 Not
   Modified`, you get the cached copy with `cached: true` and
   `revalidated: true`, billed as a cache hit.
3. **Safety.** Private, loopback, link-local and metadata addresses are
   refused (`blocked_destination`), also after redirects. A host whose name
   does not exist (NXDOMAIN twice within 6 h) answers `error: "domain_dead"`
   at once for 6 h — or its archived copy, see below.
4. **robots.txt** is honoured for `CormorantBot` and `*`
   (`robots_disallowed`).
   A few hosts are fetched **only through the premium routes**, never from
   Cormorant's own addresses. For them, `allow_premium: false` answers
   `premium_route_required`, robots.txt is read through the proxy, and a
   site-wide throttle applies.
5. **Domain memory.** A domain where every tier failed in 3 consecutive
   fetches is skipped for a while (10 min, then 30 min, 2 h, 6 h) and answers
   `error: "domain_unavailable"` at once, with `retry_after_s`. Any success
   ends the skip. Cormorant also remembers which tier (and rendering
   difficulty) worked for a domain and starts there next time, and which
   domains refuse even the premium routes: those skip `proxy`/`proxy_render` for
   6 h (then 24 h, 3 days) and go to the archive and alternative sources
   instead, so a blocked domain stops costing premium bytes
   (`premium_skipped_domain_refuses` attempts).
6. **Tiers** run until one produces a good page.

## Tiers and escalation

| Tier | What | Route |
|---|---|---|
| `direct` | HTTP with Chrome's TLS/HTTP2 fingerprint and headers, redirects, compression | standard |
| `browser` | Chromium (new headless) with per-site cookies; waits out JS interstitials | standard |
| `archive` | recent Wayback Machine snapshot (≤ 365 days), only when blocked | standard |
| `proxy` | residential network (several networks, failover between them) | premium (also per GB) |
| `proxy_render` | rendering service behind residential addresses | premium |

The ladder starts at `direct` (or at a tier learned for the domain, see
architecture). It escalates on network errors, HTTP 403/429/451/5xx and
bot-challenge pages (Cloudflare, Akamai, DataDome, PerimeterX, Imperva,
Kasada, generic interstitials). Thin main text escalates `direct → browser`
only. `archive` and the premium routes are used only when the other standard routes were
*blocked* — never for a page that is simply short. Definitive answers (404,
410, 401, …) are returned without escalation — a 404/410 on `direct` never
reaches a premium route (at most the archived copy, see below). No CAPTCHA is
solved unless the operator enabled it for the project.

**Domains that refuse the premium routes too.** When a fetch fails on every tier
and every premium route it tried was refused by the site (403/429/451, a
challenge, a rendering-tier captcha or "cannot render", the provider's
content policy —
not provider errors or timeouts) in 2 fetches in a row, the domain is
*premium-blocked* for 6 hours (then 24 h, then 3 days on repeats): its fetches
still try `direct`, `browser` and the archive, but skip `proxy` and
`proxy_render` (attempt reason `premium_skipped_domain_refuses`, no premium bytes)
and go straight to the alternative sources, or fail with `all_tiers_failed`.
5 % of these fetches re-probe the premium routes anyway; one premium success ends
the block.

**Timing.** Each attempt's `ms` splits into `wait_ms` (waiting for a polite
per-domain slot) and `upstream_ms` (the site, browser, archive or
premium route). `timing` sums them for the request and adds `queue_ms`, the time
in Cormorant's queues before a worker ran it (synchronous: from the API to a
worker; batch: since admission, politeness requeues included; plus the render
queue after a browser handoff): `{"queue_ms": 35, "politeness_wait_ms": 0,
"upstream_ms": 412}`. The same three components are stored per request in the
request log and summarised in `GET /v1/usage`.

Every attempt carries `premium_bytes` (what the residential network billed for it;
`0` for standard routes and `proxy_render`, which bills per call). The `proxy`
route can show more than one attempt: when the site refuses a residential
exit, or the network fails, the request is retried at once through another
residential network, each attempt with its own status, reason and premium
bytes. Prices are per route and do not depend on which network served the
page. A failed fetch says
which premium routes were tried in `premium_routes_tried` and summarises every attempt
in `error_detail`, e.g. `"all_tiers_failed: direct (http_403, block: hard_block),
browser (challenge_akamai, block: challenge), proxy (http_403, 16211 premium bytes,
block: hard_block), proxy_render (http_403, block: hard_block); premium routes tried:
proxy, proxy_render (16211 premium bytes)"`.

**Block types.** Attempts that reached a site (or failed talking to it) carry a
neutral `block_type`: `ok`, `rate_limit`, `challenge` (a JavaScript or CAPTCHA
check), `hard_block`, `geo_block`, `login_wall`, `paywall`, `soft_404` (a
"not found" page answered with 200), `empty_shell` (a JavaScript application
without content), `fingerprint_refusal` (the site refused the client's TLS or
browser fingerprint), `upstream_error` or `not_found`. Skipped tiers have none.
It is derived from the status, headers and the page and never changes how a
fetch escalates.

**Dead domains and missing pages.** A host whose name does not exist
(NXDOMAIN — not a temporary DNS failure such as SERVFAIL or a timeout, which
stay `blocked_destination` with reason `dns_error`) twice within 6 hours is
remembered as dead for 6 hours: fetches answer `error: "domain_dead"` at once,
with a single `{"tier": "dns", "reason": "domain_dead"}` attempt and no tier
work. When a fetch ends in `domain_dead` or a 404/410 and `allow_archive` is
true (and `tiers`, if given, includes `archive`), Cormorant looks for a Wayback
snapshot (≤ 365 days) and, if there is one, returns it instead: `tier:
"archive"`, `archived_at`, and `served_from_archive_because: "domain_dead"` or
`"not_found"`; `attempts` still shows the 404 or the dead host. These stand-in
copies are not cached. The archive's paywall rules apply (no copies for known
paywalled domains or of articles marked subscriber-only, `archive_blocked_paywall`).
Without a snapshot the answer stays `domain_dead` or `http_404`/`http_410`
(with an `archive_not_found` attempt), and only then are alternative sources
looked for. Other definitive statuses (401, …) never use the archive.

`render=never` uses `direct → archive → proxy`; `render=always` uses
`browser → proxy_render`.

**Known hard domains.** For sites on Cormorant's list of hard domains
(`nestle.com`, `nestleusa.com`, `beyondmeat.com`, `apnews.com`, `nytimes.com`,
`researchgate.net`, … — see [Fetching → Known hard
domains](/docs/fetching.md#known-hard-domains)) tiers that do not work for the
site are skipped (`hard_domain_skip` attempts), the residential exit country is
pinned (unless the request sets `country`) and `proxy_render` starts at the
site's difficulty. Errors — and archive or alternative copies standing in —
carry `hint` (the recommended route) and, on errors, `alternative_urls`. An
explicit `tiers` list overrides the route. The `proxy` tier sends GET requests
with Chrome's TLS/HTTP2 fingerprint, like `direct`.

`attempts[].reason` values include `ok`, `thin_content`, `not_modified`,
`challenge_<vendor>`, `http_<status>`, `timeout`, `tls_error`, `dns_error`,
`dns_nxdomain`, `domain_dead`, `network_*`, `not_configured`, `budget_exhausted`,
`hard_domain_skip`,
`archive_not_found`, premium-route reasons (`…_captcha_rejected`,
`…_no_credits`, `…_render_503`), `paywall` (teaser
of a hard paywall, returned as is), `paywall_skip_premium`, `premium_skipped_domain_refuses`,
`archive_blocked_paywall`, and
for the alternative-source steps (`tier: source`) `alternate_<source>` (with the copy's
URL in `detail`), `syndication_candidates_<n>`, `open_access_none`,
`open_access_miss`, `guardian_api_miss`, `publisher_feed_miss`,
`publisher_feed_teaser`, `common_crawl_none`, `common_crawl_not_free`,
`common_crawl_paywalled_domain`, `common_crawl_index_error`,
`common_crawl_record_error`, `source_timeout`. `proxy_render` attempts carry
`detail` with the difficulty used (`low`, `medium`, `high`); other attempts
omit `detail`. Busy rendering-tier answers (`…_render_503`) are retried
once and appear as two attempts.
## What you get back

| Field | What it is |
|---|---|
| `text` | The retained text: main content for HTML (trafilatura, readability fallback, then visible body text), pages joined by blank lines for PDFs, the decoded document for JSON/XML/plain text, empty for binary. Deterministic — the same bytes always give the same text, and cache hits return identical text, so you can anchor quotes against it. |
| `pdf_text_pages`, `pdf_page_count`, `text_truncated` | For PDFs: `[{"page": 1, "text": "…"}, …]`, the number of pages, and whether the text stops early (very long documents). |
| `meta` | For HTML pages: author, published/modified dates, language, canonical URL, publisher, site name, description, section and word count ([details](/docs/api.md#article-metadata-meta)). |
| `html` | The decoded HTML of HTML pages (sync only by default; see `include_html`). |
| `title` | Document title, if found. |
| `content_kind` | `html`, `pdf`, `text`, `json`, `xml` or `binary` — from the content type, sniffed when it is missing or generic. |
| `status`, `final_url`, `redirects` | The HTTP status, the URL after redirects (`null` when unknown, for `proxy_render`) and each redirect hop. |
| `content_type`, `origin_content_type`, `headers` | What the body is, what the origin said it was, and an allowlist of the origin's response headers (never cookies). |
| `tier`, `attempts` | Which tier produced the page, and every attempt with its reason, duration (`ms` = `wait_ms` for a polite slot + `upstream_ms`) and `premium_bytes`. |
| `timing` | Where the time went: `queue_ms` (Cormorant's queues), `politeness_wait_ms` (busy domain), `upstream_ms` (the site or provider). |
| `error_detail`, `premium_routes_tried` | On failures: a one-line account of every attempt, and the premium routes that were actually tried. |
| `sha256`, `bytes`, `body_url`, `body_kind` | The exact retrieved bytes and where to download them. |
| `cached`, `retrieved_at`, `etag`, `last_modified`, `archived_at` | Freshness and provenance. |
| `cost_usd`, `error`, `request_id` | Spend, outcome and the id to quote. |

### Non-HTML content

`content_kind` tells you how to read `text`:

- `json` — `text` is the decoded document (still valid JSON);
- `xml` — feeds, sitemaps, XBRL…: one text node per line, `title` from the
  first `<title>`; external entities are never resolved;
- `text` — plain text, CSV, Markdown, decoded;
- `binary` — spreadsheets, archives, images…: `text` is empty, download the
  bytes from `body_url`.

All of these are successful fetches (`error: null`).

### PDFs

PDFs are accepted by default (`accept_pdf: false` turns them into
`error: "pdf_not_accepted"`). Text is extracted per page; use
`pdf_text_pages` to cite page numbers. Large reports may need a higher
`max_bytes` (default 15 MB, at most 50 MB); a bigger body is
`error: "too_large"`.

## Exact bytes and provenance

Every fetch stores the exact bytes it retrieved, content-addressed by
`sha256`, and returns `body_url` (null when object storage is unavailable).
`body_kind` says what the hash covers:

- `origin`: the HTTP entity body the origin (or the archive, for
  `tier = archive`) sent, after content-encoding (gzip/br) is removed — the
  bytes a plain HTTP client would hand you;
- `rendered_dom`: the document serialized by a browser (`browser`,
  `proxy_render`) after scripts ran — use it for text, not as proof of the
  origin's bytes. Non-HTML documents a browser downloads (PDFs) are `origin`.

`GET /v1/bodies/{sha256}` (same Bearer key) streams the bytes with the origin
`Content-Type`, `Content-Length`, `ETag: "<sha256>"`, `X-Body-Kind` and
`Cache-Control: immutable`; `If-None-Match` gives `304`; `HEAD` returns the
headers only. A project can download only bodies it has retrieved itself
(fresh or from the shared cache); anything else is `404`. Bodies are kept
while a cache entry points to them or a project retrieved them within the
last 30 days.
Downloading a body (Python and TypeScript continue the [Quickstart](/docs/quickstart.md) client):

```sh
curl -sS "$BODY_URL" -H "Authorization: Bearer $CORMORANT_API_KEY" -o report.pdf
```

```python
page = client.post(
    "/v1/fetch", json={"url": "https://www.example.com/annual-report-2025.pdf"}
).json()
if page["body_url"]:
    with client.stream("GET", page["body_url"]) as response:
        response.raise_for_status()
        with open("report.pdf", "wb") as out:
            for chunk in response.iter_bytes():
                out.write(chunk)
```

```ts
const bytes = await fetch(page.body_url, { headers }).then((r) => r.arrayBuffer());
```

For citations, keep `final_url`, `retrieved_at`, `sha256` and `body_kind`
with the quote. For `tier = archive`, `final_url` is the Wayback Machine copy
and `archived_at` the snapshot time.

## Cache semantics

- `cache_ttl_s` is the **maximum age you accept**. `0` bypasses the cache and
  forces a fresh fetch.
- A cache hit is used only if it satisfies your request: `render: "always"`
  needs a rendered copy and `render: "never"` an origin copy; HTML text must
  reach your `min_text_chars`; a cached body larger than your `max_bytes`
  answers `too_large`, exactly as a fresh fetch would; with `tiers` set,
  hits from other tiers are not used.
- Pages that are simply short (direct and the browser both gave less text
  than `min_text_chars`, nothing blocked) are cached for a day as such and
  answer later requests; thin pages of blocked fetches are not cached.
- A redirected page is cached under the URL the redirects ended at too.
- Hits are billed as cache hits (`fetch_cached`), even if the page originally came from a premium route.
- Cache classes, TTLs and how to get hits across runs:
  [Caching and reuse across runs](/docs/caching.md).

## Per-request controls

| Field | Use it to |
|---|---|
| `render` | `auto` (default) climbs the ladder; `never` avoids browsers (`direct → archive → proxy`); `always` goes straight to a browser (`browser → proxy_render`) for JavaScript-only pages. |
| `allow_premium` | `false` never uses `proxy` or `proxy_render` — the fetch stays on the standard routes. |
| `tiers` | Restrict the ladder, e.g. `["direct", "browser"]`. If nothing remains: `error: "no_tier_allowed"`. |
| `allow_archive` | `false` never answers from the Wayback Machine — neither when the live site blocks us nor in place of a dead domain or a 404/410 (`served_from_archive_because`). |
| `timeout_s` | A deadline (5–150 s) for the whole fetch across tiers. When it runs out you get `error: "timeout"` — or the best page found so far. |
| `min_text_chars` | Main text shorter than this (default 500) counts as thin and escalates `direct → browser` only. Lower it for short pages (product pages, abstracts). |
| `max_bytes`, `accept_pdf` | Size cap and PDF acceptance. |
| `country` | Geotargeting for the premium routes (ISO 3166-1 alpha-2). |
| `accept_language` | The `Accept-Language` header / browser locale, e.g. `de-CH,de;q=0.9`. |
| `user_agent` | Your own User-Agent for `direct`, `browser` and `proxy` — some services (SEC EDGAR, Wikimedia) require a contact address. It must be on your project's allowlist, which the operator maintains; otherwise `403 user_agent_not_allowed`. You do not need one for Wikimedia: see below. |
| `include_html` | Return `html` (default `true` for `/v1/fetch`, `false` in batches). Turn it off when you only need text — responses get much smaller. |
| `project_ref` | Your own text stored with your usage, e.g. the entity the fetch was for. |

A cost-capped, latency-capped fetch for an agent:

```json
{
  "url": "https://www.holcim.com/investors",
  "allow_premium": false,
  "timeout_s": 45,
  "include_html": false
}
```

A filing from a service that requires a contact User-Agent:

```json
{
  "url": "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193",
  "user_agent": "Example Research contact@example.org",
  "tiers": ["direct"],
  "render": "never"
}
```

### Wikipedia, Wikidata and other Wikimedia sites

Fetches of `*.wikipedia.org`, `*.wikidata.org`, `*.wikimedia.org` and the
sister projects follow Wikimedia's
[User-Agent policy](https://meta.wikimedia.org/wiki/User-Agent_policy)
for you: Cormorant identifies itself with its own descriptive User-Agent and
the operator's contact (or your allowlisted `user_agent`, when you send one),
uses the `direct` tier only (no browser or premium routes, so these fetches are
never charged as premium) and keeps the number of concurrent requests to all
Wikimedia hosts low across all clients. Under load, expect requests to wait
briefly for a slot rather than fail; if Wikimedia answers 429 the fetch fails
(`all_tiers_failed`, `attempts` shows `http_429`) and every worker slows down
for a few minutes, so retry later rather than at once. Prefer the APIs
(`https://en.wikipedia.org/api/rest_v1/page/summary/Zurich`,
`https://www.wikidata.org/wiki/Special:EntityData/Q72.json`) to HTML pages:
they are smaller and cached the same way.

### Known hard domains

Some sites block automated access from datacenter addresses on every route
Cormorant has, or answer only behind a paywall. Cormorant ships a list of them
with the route that works best per site (operators can extend or override it
with `HARD_DOMAINS_FILE`):

| Domain | Protection | Route |
|---|---|---|
| `nestle.com` | Cloudflare managed challenge | skip the browser; residential proxy with a Chrome fingerprint from a Swiss exit, then the rendering tier (`proxy_render`, `medium`) |
| `nestleusa.com` | Cloudflare | as above, US exit |
| `beyondmeat.com` | Cloudflare | skip the browser; residential proxy (US exit), then the rendering tier |
| `apnews.com` | Cloudflare | skip the browser; archive, residential proxy, then the wire copy elsewhere (`syndication`) or a Common Crawl capture — also on synchronous calls |
| `nytimes.com` | DataDome + hard paywall | no browser or premium routes: only open copies (archive, publisher feed, syndication) |
| `researchgate.net` | Cloudflare on every route | no browser or premium routes; Common Crawl; look for the DOI / open-access copy (`alternative_urls`: OpenAlex, Semantic Scholar) |
| `medium.com`, `indeed.com` (all country sites) | Cloudflare | skip the browser (Indeed: and the rendering tier) |
| `reuters.com`, `barrons.com` | robots.txt + DataDome | no browser or premium routes; wire copies via syndication |

Skipped tiers show as `hard_domain_skip` attempts; a request's own `tiers`
list overrides the route, and its `country` wins over the route's exit
country. When such a fetch fails — or an archived or alternative copy stands
in — the result carries `hint` (the route in one sentence) and, on errors,
`alternative_urls` (e.g. the page's Wayback Machine URL, or an OpenAlex
search built from a ResearchGate URL). The residential proxy tier always
presents Chrome's TLS/HTTP2 fingerprint (like `direct`), and the browser tier
presents a headful Chrome identity (no `HeadlessChrome` in the User-Agent or
client hints).

For a company's own site (10–15 pages per company), use a batch fetch with an
`idempotency_key`: batch items wait out busy and blocked domains instead of
failing, and may use Common Crawl.

## Errors

The fetch-level `error` codes, what they mean and whether to retry them are in
[Errors & retries](/docs/errors.md).