# Cormorant API v1

Base URL: `https://api.cormorant.ch`. All endpoints below are
under `/v1` except health and metrics. The contract is versioned: fields are
only ever **added** within v1; nothing is renamed or removed. Guides, the
interactive reference and `llms.txt` are served at `/docs` (see the
[changelog](/docs/changelog.md) for what changed).

## Authentication

Every `/v1` request needs a project API key (except the public
[service status](#service-status), `/v1/status*`):

```
Authorization: Bearer cmr_...
```

Keys belong to exactly one project, are stored only as HMAC-SHA256 hashes, and
are created/revoked by the operator (Mission Control, or
`python -m app.cli create-key <project> [--role account_admin]`,
`revoke-key <prefix>`). Missing or invalid keys get `401`.

There are two kinds of keys, and the prefix tells them apart:

| Prefix | Role (`X-Cormorant-Key-Role`) | Sees |
|---|---|---|
| `cma_…` | `account_admin` — the **account admin key** | everything of the customer account where an endpoint offers it: all its keys, projects, usage and invoices |
| `cmr_…` | `standard` — an **additional key** | its own usage only |

Both authenticate every `/v1` endpoint the same way. A key's role is fixed when
it is created; to change it, have a new key issued and the old one revoked.
Keys issued before roles existed are standard keys. Every authenticated `/v1`
response carries the header `X-Cormorant-Key-Role: account_admin` or
`X-Cormorant-Key-Role: standard`. Each project has a
request-rate limit (default 200 requests/s, fixed one-second window); excess
requests get `429` with `Retry-After: 1`. A batch call counts as one request.
Every authenticated response carries `RateLimit-Policy` and `RateLimit`
(IETF draft; `"project";q=200;w=1` and, for a key with its own limit,
`"key";q=10;w=1`).

**Debug requests (available on request).** When you report a problem with a
specific request, Cormorant support can enable debugging for one of your keys.
Send that key's synchronous `POST /v1/search` or `POST /v1/fetch` request with
the header `X-Cormorant-Debug: 1`: the request is recorded in full detail for
support, and the answer carries `debug_url`, a link to that record (it opens in
the operator console, not with an API key). Quote it in your report. Without
debugging enabled for the key, the header is ignored and the answer is the
usual one. Debug requests are billed like any other request.

**Per-key limits.** The operator can set, per key, a monthly spend cap (in the
customer's invoice currency; provider cost in USD for projects without a
customer), monthly request quotas per category (`search`, `fetch`,
`fetch_premium`, `company`; SEC EDGAR requests count as `fetch`), requests per
second and the
allowed categories. They are
enforced on every request and batch item: `403 key_category_not_allowed`,
`429 key_quota_exhausted` / `key_budget_exhausted` (`Retry-After` until the
month resets, UTC) and `429 key_rate_limited`. A key without `fetch_premium`
(not allowed, or its quota used up) still fetches, without the premium routes.
Keys with limits get `X-Cormorant-Key-Budget-Remaining: 12.40 USD` and
`X-Cormorant-Quota-Remaining: fetch=18766, search=900` on every response.
Request counts are exact; spend is the aggregates (5 minutes) plus an
estimate since then.
Keys of a suspended customer get `403` with
`{"detail": {"error": "customer_suspended", "message": "…"}}`.

Every response carries a `request_id` (`req_…`; batch items use
`<job_id>:<index>`). Quote it when debugging — Mission Control can show the
full attempt history for any request id.

`cost_usd` is Cormorant's provider spend caused by *this* request in USD
(search fallback and premium routes); cache hits and standard routes cause no provider cost. Spend is attributed only to
the project whose request made the provider call; the caches are shared. **Keys of
an external customer get `cost_usd: null`**: customers see what they are
charged — per request in `GET /v1/requests` (`charge`), aggregated in
`GET /v1/usage` — never Cormorant's provider cost. Every search, fetch and
SEC result (also batch items) carries `billing_category`
(`search`, `search_fallback`, `search_cached`, `fetch_own`, `fetch_premium`,
`fetch_cached`, `sec`, `sec_cached`; company data requests count as
`company` / `company_cached`; new categories may be added): the category,
and so the plan meters, the request counts toward. SEC EDGAR requests (`/v1/sec/…` and fetches of
SEC hosts) are `sec` / `sec_cached`: no provider cost, billed at your plan's
price for them, and counted against a key's `fetch` quota.

## POST /v1/search

```json
{
  "query": "Nestlé sustainability report 2025",
  "language": "en",        // optional, ISO 639-1 (SearXNG language, Brave search_lang)
  "country": "CH",         // optional, ISO 3166-1 alpha-2 (combined with language for SearXNG)
  "max_results": 20,       // 1..50, default 20
  "freshness": "month",    // optional: day | week | month | year (pd/pw/pm/py accepted)
  "cache_ttl_s": 604800,   // optional: max acceptable cache age; 0 = bypass the cache
  "project_ref": "company:4711"  // optional your own text, stored with usage
}
```

Response `200`:

```json
{
  "query": "Nestlé sustainability report 2025",
  "results": [
    {
      "url": "https://www.nestle.com/sustainability/...",
      "title": "Creating Shared Value and Sustainability Report 2025",
      "snippets": ["Our 2025 report covers ...", "Download the full report ..."],
      "engines": ["google", "bing", "brave_api"],
      "published_at": "2026-03-02T00:00:00",
      "rank": 1
    }
  ],
  "provider": "searxng",          // "searxng" | "brave"
  "fallback_reason": null,        // why Brave was used (see below), else null
  "unresponsive_engines": [],     // SearXNG engines that failed for this query
  "cached": false,
  "cost_usd": 0.0,                // null for keys of an external customer
  "billing_category": "search",   // search | search_fallback | search_cached
  "error": null,
  "request_id": "req_6b1f..."
}
```

Order of operations: shared search cache (key = normalised query + language,
country, max_results, freshness; 7 days) → our SearXNG instances (least-loaded
across all hosts, a second instance on transport failure) → Brave Search API.

Brave is used when SearXNG errors or times out (`searxng_error`,
`searxng_timeout`, `no_searxng_instance`), reports unresponsive engines
(`unresponsive_engines`, or `captcha_engines` when any reason mentions a
CAPTCHA), or returns fewer than `MIN_RESULTS` (3) usable results
(`too_few_results`). By default unresponsive engines trigger the fallback only
when fewer than two engines returned results (`SEARCH_FALLBACK_ON_UNRESPONSIVE=
quorum`, `SEARCH_QUORUM_ENGINES=2`); `majority` (as many engines failed as
answered) and `any` are stricter. A Brave rate limit (429) is retried after the
indicated wait; batch items that still fail with it are retried later.

Results are deduplicated by normalised URL (scheme, `www.`, trailing slash and
tracking parameters ignored). **No snippet is lost**: every distinct snippet
and engine seen for a URL is kept, in order of first appearance. When Brave is
used, Brave's results come first, followed by any SearXNG-only results.

If Brave is unavailable or the project budget is exhausted, whatever SearXNG
returned is served (not cached) with `fallback_reason` set. A **genuinely
empty** search — the engines (and Brave, when consulted) answered and found
nothing — returns `results: []` with `error: null`. Only when the providers
could not answer is `error` set (e.g. `budget_exhausted`, `brave_http_429`,
`searxng_timeout`).

## POST /v1/fetch

```json
{
  "url": "https://www.example.com/annual-report-2025.pdf",
  "render": "auto",          // auto | never | always
  "max_bytes": 15728640,     // default 15 MB, max 50 MB (batch items left at the default: PDFs up to 200 MB)
  "accept_pdf": true,
  "cache_ttl_s": 604800,     // optional max acceptable cache age; 0 = bypass
  "min_text_chars": 500,     // below this, HTML main text counts as "thin"
  "allow_archive": true,     // allow an archived copy when the live site blocks us, is gone or 404s
  "include_html": true,      // default true here, false for batch items
  "allow_premium": true,        // false: never use proxy / proxy_render
  "tiers": ["direct", "browser", "archive"],   // optional allowlist of tiers
  "timeout_s": 60,           // optional overall deadline, 5–150 s
  "country": "CH",           // optional geotargeting (premium routes)
  "user_agent": "Example Research contact@example.org",  // must be on the project allowlist
  "accept_language": "de-CH,de;q=0.9",
  "project_ref": "company:4711"
}
```

Response `200` (also for unsuccessful fetches — check `error`):

```json
{
  "url": "https://www.example.com/annual-report-2025.pdf",
  "final_url": "https://www.example.com/annual-report-2025.pdf",  // null when unknown (proxy_render)
  "redirects": [                    // one entry per redirect before final_url
    {"url": "http://example.com/ar25", "status": 301, "location": "https://www.example.com/annual-report-2025.pdf"}
  ],
  "status": 200,
  "content_type": "application/pdf", // what the body is (rendered pages: text/html)
  "origin_content_type": "application/pdf",  // what the origin sent; null when unknown
  "headers": {"content-type": "application/pdf", "last-modified": "Tue, 03 Mar 2026 09:00:00 GMT"},
  "tier": "direct",                 // direct | browser | archive | proxy | proxy_render
  "attempts": [
    {"tier": "direct", "status": 200, "reason": "ok", "ms": 412}
  ],
  "html": null,                     // decoded HTML for HTML pages
  "text": "Annual Report 2025\n\n...",
  "pdf_text_pages": [{"page": 1, "text": "Annual Report 2025"}, {"page": 2, "text": "..."}],
  "pdf_page_count": 148,            // PDFs: number of pages; null otherwise
  "text_truncated": false,          // PDF text stopped at the extraction limits
  "title": "Annual Report 2025",
  "meta": null,                     // article metadata for HTML pages (see below)
  "sha256": "a3f1...",              // of the retrieved bytes
  "bytes": 2483112,
  "cached": false,
  "revalidated": false,             // true: expired cache entry confirmed by the origin (304)
  "retrieved_at": "2026-10-19T08:12:44.120Z",
  "etag": "\"5f2-1a\"",
  "last_modified": "Tue, 03 Mar 2026 09:00:00 GMT",
  "archived_at": null,              // snapshot time (tier = archive) or common_crawl capture time
  "served_from_archive_because": null,  // domain_dead | not_found when the archive stands in
  "body_url": "https://api.cormorant.ch/v1/bodies/a3f1...",  // exact retrieved bytes
  "body_kind": "origin",            // origin | rendered_dom (what sha256 covers)
  "content_kind": "pdf",            // html | pdf | text | json | xml | binary
  "paywall": null,                  // HTML only: none | metered | hard | unknown
  "paywall_evidence": [],           // e.g. ["jsonld_not_free", "vendor_piano"]
  "text_complete": null,            // false when text is only a paywall teaser
  "alternate_of": null,             // set when text comes from a verified copy elsewhere
  "alternate_source": null,         // publisher_feed | guardian_api | open_access | syndication | common_crawl
  "match_score": null,
  "primary_sources": [],            // [{url, kind, text}] for paywalled/unavailable pages
  "cost_usd": 0.0,                  // null for keys of an external customer
  "billing_category": "fetch_own",  // fetch_own | fetch_premium | fetch_cached | sec | sec_cached
  "error": null,
  "request_id": "req_91c2..."
}
```

`text` is exactly the retained text Cormorant stores and serves again on cache
hits (main text via trafilatura, readability fallback, then visible body text;
whitespace normalised). Anchor quotes against it. For PDFs, `text` is the pages
joined with blank lines and `pdf_text_pages` has the per-page text.

**Large PDFs.** `max_bytes` applies to every response of `/v1/fetch`. In batch
fetches whose items leave `max_bytes` unset, a PDF (by
`Content-Type: application/pdf` or the `%PDF-` signature) may be up to
**200 MB**; other content keeps the default 15 MB. An item that sets
`max_bytes` gets exactly that limit, for PDFs too. Such a PDF is fetched by
the `direct` or `proxy` tier (the browser, archive and rendering tiers keep
`max_bytes`); its exact bytes are available from `body_url`. Downloading it
may take at most 5 minutes in total (or the item's `timeout_s`, when
shorter); a slower download ends with `error: timeout` and an attempt with
reason `pdf_download_timeout`. Text extraction stops after 5,000 pages or
5,000,000 characters of page text: then `text_truncated` is `true`,
`pdf_text_pages` covers the pages read so far (the last one possibly cut)
and `pdf_page_count` still counts every page. To skip PDFs entirely, set
`accept_pdf: false`.

**Non-HTML content.** `content_kind` classifies the response (content type,
sniffed when missing or generic). `json`: `text` is the decoded document
(still valid JSON). `xml` (feeds, sitemaps, XBRL…): `text` is the text content,
one node per line, `title` from the first `<title>`; external entities are
never resolved. `text` (plain, CSV, Markdown): decoded text. `binary`
(spreadsheets, archives, images…): `text` is empty — the bytes are in
`body_url`. All of these are successful fetches (`error: null`).

**Paywalls.** HTML pages carry a `paywall` verdict. Cormorant detects
paywalls and never circumvents them (no crawler User-Agents, cookie clearing,
script blocking or referer tricks).

| `paywall` | Meaning | `text_complete` |
|---|---|---|
| `hard` | `text` is only the teaser: the publisher declares the article subscriber-only (`isAccessibleForFree: false`) and the text is cut short, or a subscription call to action follows a teaser together with paywall markup, vendor scripts or truncation | `false` |
| `metered` | a meter ("2 articles left this month") or a declared paywall, but the full text came through | `true` |
| `unknown` | weak signals only (a paywall vendor's script, paywall-like markup) | `true` |
| `none` | no paywall signal | `true` |

`paywall_evidence` lists the signals: `jsonld_not_free`, `vendor_<name>`
(piano, poool, zephr, pelcro, laterpay, steady, blendle, …),
`paywall_markup`, `truncation_markup`, `phrase_<lang>`, `metered_<lang>` (en,
de, fr, it) and `length_mismatch` (declared word count far above the
extracted text). A `hard` teaser is a successful fetch (`error: null`)
returned at once — other tiers see the same paywall — and is not cached.
Domains whose articles are repeatedly hard-paywalled skip the premium routes
(`paywall_skip_premium` attempts). Wayback copies of articles the publisher marks
as subscriber-only are not served (`archive_blocked_paywall`); the operator can allow
them per project (`archive_paywalled`).

**Alternative sources.** When the page is a hard-paywall teaser,
disallowed by robots.txt, gone (404/410, `domain_dead`) or blocked on every tier
(`all_tiers_failed`, `domain_unavailable`), Cormorant looks for a verified
open copy elsewhere (`allow_alternates`, default `true`):

| `alternate_source` | How it is found and verified |
|---|---|
| `publisher_feed` | tried first, for paywall teasers and blocked pages only: the article's item in the publisher's own full-text RSS/Atom feed (feeds Cormorant found linked from the publisher's pages or the operator added, and detected to carry whole articles). The item must link to the requested URL (tracking parameters ignored; `match_score` 1.0) or have the same title and contain ≥ 50 % of the teaser's opening word 5-grams; items that are themselves teasers (short, or ending in a subscribe call-to-action) are never used. `final_url` is the feed |
| `guardian_api` | theguardian.com URLs via the Guardian Content API (when the operator configured a key) |
| `open_access` | a DOI in the URL or the page's citation metadata → the best open-access location OpenAlex knows (repository, arXiv, Europe PMC, publisher OA), fetched with the standard `direct` route; `match_score` 1.0 |
| `syndication` | the headline is searched; copies on other sites (wire services first) are fetched with `direct` and accepted only if ≥ 50 % of the teaser's opening word 5-grams appear in them and they are not paywalled themselves; `match_score` is that share |
| `common_crawl` | **batch fetches only** (too slow for synchronous calls), for pages that are gone (404/410, `domain_dead`) or blocked on every tier — never for paywall teasers or robots-disallowed pages: the newest HTTP 200 capture of the exact URL in the last three Common Crawl crawls (CDX index, then the single WARC record by HTTP Range request). Domains known to be paywalled are skipped, and a capture whose markup declares the article subscriber-only (`isAccessibleForFree: false`) or that is itself a paywall teaser is not served. `archived_at` is the capture time, `final_url` the captured URL, `html` the captured page; `match_score` 1.0 |

The result then has `tier: "source"`, `alternate_of` (the URL asked for),
`final_url` (the copy), `text_complete: true` and `paywall` still describing
the original page. Without a verified copy the original answer stands (the
flagged teaser or the error) and `attempts` show what was tried.
`primary_sources` lists links from the page to court decisions,
government/regulator pages, press releases, DOIs and PDFs. Searching for
copies may use the Brave fallback (billed like a search). The operator can
turn alternates off per project (`set-option <project> alternates off`).

**Provenance.** `redirects`, `headers` (an allowlist of the origin's final
response headers: content type/length/language/disposition, ETag,
Last-Modified, Cache-Control, Expires, Date, Link, X-Robots-Tag — never
cookies) and `origin_content_type` are kept with cache entries and returned on
hits. For `tier = archive`, `final_url` is the archived copy's URL
(`https://web.archive.org/web/<timestamp>id_/<url>`) and `archived_at` the
snapshot time. The rendering tier (`proxy_render`) does not report the post-redirect
URL or origin headers: `final_url`, `origin_content_type` are `null` and
`headers` is empty rather than guessed.

**Per-request controls.** `allow_premium=false` removes the premium routes;
`tiers` restricts the ladder to the listed tiers (an empty intersection is
`error: no_tier_allowed`; cache hits from other tiers are not used).
`timeout_s` is a deadline for the whole fetch across tiers — when it runs out
the result is `error: timeout` with a `request_timeout` attempt (or the best
page so far). `country` geotargets the premium routes (`proxy`, `proxy_render`).
`user_agent` replaces our browser User-Agent for `direct`, `browser` and
`proxy` (not `proxy_render`); it must be on the project's allowlist, which the
operator sets (`set-option <project> allowed_user_agents '["…"]'`) — services
such as SEC EDGAR and Wikimedia require a contact User-Agent. Otherwise
`403 user_agent_not_allowed`. `accept_language` sets the Accept-Language
header / browser locale header.

**Cache hits** are used only when they satisfy the request: `render=always`
needs a rendered copy, `render=never` an origin copy; HTML text must reach
`min_text_chars`; a cached body larger than `max_bytes` answers `too_large`
(as a fresh fetch would). Otherwise the page is fetched again. Pages returned
as *best effort* (every tier gave thin text) are never cached.

**Revalidation.** An expired entry (or one older than `cache_ttl_s`) that
would otherwise satisfy the request and has an `ETag` or `Last-Modified` is
revalidated: the `direct` tier sends `If-None-Match` / `If-Modified-Since`.
`304 Not Modified` returns the cached copy with `cached: true`,
`revalidated: true`, a single `direct` attempt with reason `not_modified`,
`cost_usd: 0`, a new `retrieved_at`, and renews the entry's freshness (new
validators from the 304 are kept). Any other answer takes the normal path (a
`200` replaces the entry). `cache_ttl_s: 0` never revalidates. Other tiers
never send conditional requests.

### Article metadata: meta

For HTML pages `meta` describes the article; it is `null` for other content
kinds and on errors. It is kept with cache entries (identical on hits) and
included in batch results.

```json
"meta": {
  "author": "Anna Muster; Beat Beispiel",
  "published_at": "2026-03-19T08:30:00Z",
  "modified_at": "2026-03-19T10:05:12Z",
  "language": "de-CH",
  "canonical_url": "https://www.example.com/news/central-bank-cuts-rates",
  "publisher": "Example Media AG",
  "site_name": "Example News",
  "description": "The central bank cut its policy rate by 0.25 percentage points.",
  "section": "Economy",
  "word_count": 1240
}
```

| Field | Meaning |
|---|---|
| `author` | Author name(s); several are separated by `; ` |
| `published_at` | Publication date, ISO 8601 (format below) |
| `modified_at` | Last modification date, ISO 8601 |
| `language` | Language tag (BCP 47), e.g. `de-CH` |
| `canonical_url` | Canonical URL the page declares, absolute |
| `publisher` | Publishing organisation |
| `site_name` | Name of the site or periodical |
| `description` | The page's own summary (at most 2,000 characters) |
| `section` | Section or category (the first one) |
| `word_count` | Words in `text` — what was retrieved, not what the page declares |

Every field is `null` when the page does not say. Sources, in order of
preference: JSON-LD (the `NewsArticle`, `Article`, `BlogPosting`,
`ScholarlyArticle` or `Report` node, also inside `@graph` or arrays;
malformed JSON-LD is skipped or repaired, never an error), OpenGraph and
`article:*` meta tags, Highwire `citation_*` meta tags (scholarly pages),
`<html lang>` and `<link rel=canonical>` (which wins for `canonical_url`),
then trafilatura's metadata detection. Dates are
`YYYY-MM-DDTHH:MM:SSZ` (converted to UTC) when the page gives a time zone,
`YYYY-MM-DDTHH:MM:SS` when it gives a time without zone, and `YYYY-MM-DD`
(or `YYYY-MM`, `YYYY`) when it gives only a date — no time or zone is
invented.

### Exact bytes: GET /v1/bodies/{sha256}

Every fetch stores the exact bytes it retrieved, content-addressed by
`sha256`, and returns `body_url` (null when object storage is unavailable).
`body_kind` says what the hash covers:

- `origin`: the HTTP entity body the origin (or the archive, for
  `tier = archive`) sent, after content-encoding (gzip/br) is removed — the
  bytes a plain HTTP client would hand you;
- `rendered_dom`: the document serialized by a browser (`browser`,
  `proxy_render`) after scripts ran — use it for text, not as proof of the
  origin's bytes. Non-HTML documents a browser downloads (PDFs) are `origin`.

`GET /v1/bodies/{sha256}` (same Bearer key) streams the bytes with the origin
`Content-Type`, `Content-Length`, `ETag: "<sha256>"`, `X-Body-Kind` and
`Cache-Control: immutable`; `If-None-Match` gives `304`; `HEAD` returns the
headers only. A project can download only bodies it has retrieved itself
(fresh or from the shared cache); anything else is `404`. Bodies are kept
while a cache entry points to them or a project retrieved them within the
last 30 days.

### Tiers and escalation

| Tier | What | Route |
|---|---|---|
| `direct` | HTTP with Chrome's TLS/HTTP2 fingerprint and headers, redirects, compression | standard |
| `browser` | Chromium (new headless) with per-site cookies; waits out JS interstitials | standard |
| `archive` | recent Wayback Machine snapshot (≤ 365 days), only when blocked | standard |
| `proxy` | residential network (several networks, failover between them) | premium (also per GB) |
| `proxy_render` | rendering service behind residential addresses | premium |

The ladder starts at `direct` (or at a tier learned for the domain, see
architecture). It escalates on network errors, HTTP 403/429/451/5xx and
bot-challenge pages (Cloudflare, Akamai, DataDome, PerimeterX, Imperva,
Kasada, generic interstitials). Thin main text escalates `direct → browser`
only. `archive` and the premium routes are used only when the other standard routes were
*blocked* — never for a page that is simply short. Definitive answers (404,
410, 401, …) are returned without escalation — a 404/410 on `direct` never
reaches a premium route (at most the archived copy, see below). No CAPTCHA is
solved unless the operator enabled it for the project.

**Domains that refuse the premium routes too.** When a fetch fails on every tier
and every premium route it tried was refused by the site (403/429/451, a
challenge, a rendering-tier captcha or "cannot render", the provider's
content policy —
not provider errors or timeouts) in 2 fetches in a row, the domain is
*premium-blocked* for 6 hours (then 24 h, then 3 days on repeats): its fetches
still try `direct`, `browser` and the archive, but skip `proxy` and
`proxy_render` (attempt reason `premium_skipped_domain_refuses`, no premium bytes)
and go straight to the alternative sources, or fail with `all_tiers_failed`.
5 % of these fetches re-probe the premium routes anyway; one premium success ends
the block.

**Timing.** Each attempt's `ms` splits into `wait_ms` (waiting for a polite
per-domain slot) and `upstream_ms` (the site, browser, archive or
premium route). `timing` sums them for the request and adds `queue_ms`, the time
in Cormorant's queues before a worker ran it (synchronous: from the API to a
worker; batch: since admission, politeness requeues included; plus the render
queue after a browser handoff): `{"queue_ms": 35, "politeness_wait_ms": 0,
"upstream_ms": 412}`. The same three components are stored per request in the
request log and summarised in `GET /v1/usage`.

Every attempt carries `premium_bytes` (what the residential network billed for it;
`0` for standard routes and `proxy_render`, which bills per call). The `proxy`
route can show more than one attempt: when the site refuses a residential
exit, or the network fails, the request is retried at once through another
residential network, each attempt with its own status, reason and premium
bytes. Prices are per route and do not depend on which network served the
page. A failed fetch says
which premium routes were tried in `premium_routes_tried` and summarises every attempt
in `error_detail`, e.g. `"all_tiers_failed: direct (http_403, block: hard_block),
browser (challenge_akamai, block: challenge), proxy (http_403, 16211 premium bytes,
block: hard_block), proxy_render (http_403, block: hard_block); premium routes tried:
proxy, proxy_render (16211 premium bytes)"`.

**Block types.** Attempts that reached a site (or failed talking to it) carry a
neutral `block_type`: `ok`, `rate_limit`, `challenge` (a JavaScript or CAPTCHA
check), `hard_block`, `geo_block`, `login_wall`, `paywall`, `soft_404` (a
"not found" page answered with 200), `empty_shell` (a JavaScript application
without content), `fingerprint_refusal` (the site refused the client's TLS or
browser fingerprint), `upstream_error` or `not_found`. Skipped tiers have none.
It is derived from the status, headers and the page and never changes how a
fetch escalates.

**Dead domains and missing pages.** A host whose name does not exist
(NXDOMAIN — not a temporary DNS failure such as SERVFAIL or a timeout, which
stay `blocked_destination` with reason `dns_error`) twice within 6 hours is
remembered as dead for 6 hours: fetches answer `error: "domain_dead"` at once,
with a single `{"tier": "dns", "reason": "domain_dead"}` attempt and no tier
work. When a fetch ends in `domain_dead` or a 404/410 and `allow_archive` is
true (and `tiers`, if given, includes `archive`), Cormorant looks for a Wayback
snapshot (≤ 365 days) and, if there is one, returns it instead: `tier:
"archive"`, `archived_at`, and `served_from_archive_because: "domain_dead"` or
`"not_found"`; `attempts` still shows the 404 or the dead host. These stand-in
copies are not cached. The archive's paywall rules apply (no copies for known
paywalled domains or of articles marked subscriber-only, `archive_blocked_paywall`).
Without a snapshot the answer stays `domain_dead` or `http_404`/`http_410`
(with an `archive_not_found` attempt), and only then are alternative sources
looked for. Other definitive statuses (401, …) never use the archive.

`render=never` uses `direct → archive → proxy`; `render=always` uses
`browser → proxy_render`.

**Known hard domains.** For sites on Cormorant's list of hard domains
(`nestle.com`, `nestleusa.com`, `beyondmeat.com`, `apnews.com`, `nytimes.com`,
`researchgate.net`, … — see [Fetching → Known hard
domains](/docs/fetching.md#known-hard-domains)) tiers that do not work for the
site are skipped (`hard_domain_skip` attempts), the residential exit country is
pinned (unless the request sets `country`) and `proxy_render` starts at the
site's difficulty. Errors — and archive or alternative copies standing in —
carry `hint` (the recommended route) and, on errors, `alternative_urls`. An
explicit `tiers` list overrides the route. The `proxy` tier sends GET requests
with Chrome's TLS/HTTP2 fingerprint, like `direct`.

`attempts[].reason` values include `ok`, `thin_content`, `not_modified`,
`challenge_<vendor>`, `http_<status>`, `timeout`, `tls_error`, `dns_error`,
`dns_nxdomain`, `domain_dead`, `network_*`, `not_configured`, `budget_exhausted`,
`hard_domain_skip`,
`archive_not_found`, premium-route reasons (`…_captcha_rejected`,
`…_no_credits`, `…_render_503`), `paywall` (teaser
of a hard paywall, returned as is), `paywall_skip_premium`, `premium_skipped_domain_refuses`,
`archive_blocked_paywall`, and
for the alternative-source steps (`tier: source`) `alternate_<source>` (with the copy's
URL in `detail`), `syndication_candidates_<n>`, `open_access_none`,
`open_access_miss`, `guardian_api_miss`, `publisher_feed_miss`,
`publisher_feed_teaser`, `common_crawl_none`, `common_crawl_not_free`,
`common_crawl_paywalled_domain`, `common_crawl_index_error`,
`common_crawl_record_error`, `source_timeout`. `proxy_render` attempts carry
`detail` with the difficulty used (`low`, `medium`, `high`); other attempts
omit `detail`. Busy rendering-tier answers (`…_render_503`) are retried
once and appear as two attempts.

### Errors

| `error` | Meaning |
|---|---|
| `invalid_url` | not an http(s) URL |
| `blocked_destination` | private, loopback, link-local or metadata address (also after redirects) |
| `robots_disallowed` | robots.txt disallows the path for `CormorantBot`/`*` |
| `too_large` | body exceeds `max_bytes` (batch PDFs with `max_bytes` unset: 200 MB) |
| `pdf_not_accepted` | PDF with `accept_pdf=false` |
| `http_<status>` | definitive upstream status (e.g. `http_404`) |
| `all_tiers_failed` | every allowed tier failed or was blocked (`status` = last HTTP status; `error_detail` and `premium_routes_tried` say what was tried) |
| `domain_unavailable` | every tier failed for this domain in 3 consecutive fetches: it is skipped for a while (10 min, then 30 min, 2 h, 6 h); `retry_after_s` says when to try again. Any success ends the skip |
| `domain_dead` | the host name does not exist (NXDOMAIN twice within 6 h); answered at once for 6 h, without trying. With `allow_archive`, a snapshot is returned instead when one exists (`served_from_archive_because: "domain_dead"`) |
| `timeout` | `timeout_s` ran out (see per-request controls), or a large PDF took longer than 5 minutes to download (attempt reason `pdf_download_timeout`) |
| `no_tier_allowed` | `tiers`/`allow_premium` left no tier to try |
| `politeness_timeout` | could not get a per-domain slot in time (transient; batch items are requeued) |
| `domain_busy` | synchronous calls only, with **HTTP 503** and `Retry-After`: the expected wait for one of the domain's polite slots exceeds the 15 s a synchronous call waits; `retry_after_s` is that expected wait. Lower the parallelism per domain or use a batch (whose items wait instead) |
| `sec_not_configured` | SEC EDGAR URL, but the operator has not configured Cormorant's SEC identity: SEC is disabled |
| `rate_limited` | SEC EDGAR URL: the cluster-wide SEC rate limit (at most 10 requests/s for all clients) had no slot within 10 s; retry after `retry_after_s` (transient) |
| `sec_unavailable` | SEC EDGAR throttled (403/429/503: all SEC requests back off), failed (5xx) or could not be reached; retry after `retry_after_s` (transient) |
| `redirect_not_allowed` | SEC EDGAR URL that redirected to a host outside SEC (not followed) |
| `premium_route_required` | a host fetched only through premium routes with `allow_premium: false` or a `tiers` list without `proxy`/`proxy_render` |
| `budget_exhausted` | such a host, and the project's budget leaves nothing for the premium routes |

When no tier reaches `min_text_chars`, the attempt with the most text is
returned (no error).

## Batch: POST /v1/batch/search and POST /v1/batch/fetch

```json
{
  "items": [{"query": "..."}, {"query": "..."}],       // same objects as the single endpoints
  "idempotency_key": "chipiron-run-2026-10-20-search-0001",
  "callback_url": "https://chipiron.example/hooks/cormorant"   // optional
}
```

Response `202` `{"job_id": "…", "created": true}`. Re-posting the same
`idempotency_key` returns the existing job (`200`, `created: false`); the same
key with a different kind or item count is `409`. Up to 50,000 items per job.
Fetch items may set `"include_html": true` to keep the HTML in stored results
(default: text only). Batch fetch items that leave `max_bytes` unset accept
PDFs up to 200 MB (see *Large PDFs* above).

Items are admitted to the queues gradually (fair share across running jobs),
executed with the same logic as the single endpoints, and retried with backoff
only for transient problems. A batch fetch item blocked on our own tiers is
retried once after a cool-down before a premium route is used.

When the job completes, `callback_url` receives a `POST` with
`{"job_id","kind","status","total","succeeded","failed","finished_at"}` (3
attempts; private addresses are refused). The host is resolved once, checked,
and the connection goes to that checked address (Host header and TLS name
stay the callback's host name), so a DNS answer that changes in between cannot
redirect it; redirects are not followed.

### GET /v1/jobs/{job_id}

```json
{
  "job_id": "1d0c...",
  "kind": "fetch",
  "status": "running",          // queued | running | completed | cancelled
  "counts": {"total": 3000, "pending": 1200, "queued": 380, "done": 1390, "failed": 30, "cancelled": 0},
  "created_at": "...", "updated_at": "...", "finished_at": null,
  "callback_url": null, "callback_status": null,
  "items_url": "/v1/jobs/1d0c...?include=items&offset=0&limit=100"
}
```

Add `?include=items&offset=0&limit=1000` (max 1000) to page through item
results, optionally `&status=failed`:

```json
"items": [
  {"index": 0, "status": "done", "input": {"url": "..."}, "result": { ...fetch response... }, "error": null}
]
```

**Retention:** jobs and their item results are kept for 30 days after the job
finishes, then deleted (`JOB_RETENTION_DAYS`). Download results you need to
keep before that. If a stored result cannot be read (object storage
unavailable), the item has `"result": null` and `"error": "result_unavailable"`;
retry the request later.

### DELETE /v1/jobs/{job_id}

Cancels the job: no new items are admitted (the admission pump runs every
3 s, well within 60 s), pending items become `cancelled`, queued items skip
themselves. Returns the job status. Jobs are only visible to their project.

## SEC EDGAR: /v1/sec/...

Company and filing data from SEC EDGAR. Cormorant sends **its own** declared
identity to SEC (`"<Org name> <contact email>"`, set by the operator) and
keeps all clients together under SEC's fair-access limit: one cluster-wide
rate limit per egress IP (default 8 requests/s, never more than 10), queued
for up to 10 s, and a shared backoff when SEC throttles (403/429/503). Only
the direct tier is used — never the browser, archived copies, the residential network or
alternative sources. **Do not send your own User-Agent** for SEC: a
`user_agent`/`accept_language` in `POST /v1/fetch` is ignored for SEC hosts.
The same policy applies to `POST /v1/fetch` and batch items for URLs on
`www.sec.gov`, `data.sec.gov` and `efts.sec.gov`.

All endpoints need an API key, count towards the project's rate limit and
are metered as kind `sec` (no provider cost). Answers are HTTP 200 also when
they failed; check `error`:

| `error` | Meaning |
|---|---|
| `unknown_identifier` | neither SEC nor Cormorant knows this ticker, CIK, name or ISIN (a home-market ticker of a non-US company: use its ADR ticker or the ISIN) |
| `not_sec_filer` | the company is known (a non-US listing such as `NESN.SW` / `XSWX:NESN`, the company register, GLEIF for an ISIN) but not registered with the SEC under this identifier; `hint`: `company`, `listing`, `sec_candidates` (SEC filers with its name, e.g. the ADR) |
| `not_found` | SEC has no such filing or document (also: no filing of the requested forms) |
| `invalid_identifier` | a CIK, accession number, document name, form type, date or query is malformed (`message` says which); nothing was requested from SEC |
| `rate_limited` | no slot in the shared SEC rate limit within 10 s; retry after `retry_after_s` |
| `sec_unavailable` | SEC throttles (all SEC requests back off) or failed; retry after `retry_after_s` |
| `sec_not_configured` | SEC is not enabled on this deployment |
| `timeout`, `too_large`, `robots_disallowed`, … | the fetch errors |

Identifiers: CIKs as a number or 10 digits (`100493`, `0000100493`);
accession numbers with or without dashes (`0000100493-25-000070`,
`000010049325000070`); form types comma-separated (`10-K,20-F,40-F,10-K/A`).
Only the SEC URL patterns behind each endpoint can be requested.

Every answer has `error`, `message`, `retry_after_s`, `cached` (`true` when
all SEC documents came from Cormorant's cache), `retrieved_at` (oldest
document), `source_url` (the main SEC document), `body_url` and `sha256`
(its exact bytes, e.g. the raw submissions JSON) and `request_id`.

**Caching.** Filing documents and indexes (`/Archives/edgar/data/{cik}/{accession}/…`)
are immutable and cached for 365 days (revalidated only when you ask with
`cache_ttl_s`); `company_tickers.json`, submissions and XBRL facts for 12 h,
then revalidated with conditional requests; full-text search for 1 h.

### GET /v1/sec/companies/resolve?ticker=TSN

Also `?cik=100493` or `?name=tyson foods` (exactly one). From
`company_tickers.json` and `company_tickers_exchange.json`:

```json
{"cik": 100493, "cik_padded": "0000100493", "title": "TYSON FOODS, INC.",
 "tickers": ["TSN"], "exchanges": ["NYSE"], "ambiguous": false,
 "candidates": [{"cik": 100493, "cik_padded": "0000100493", "title": "TYSON FOODS, INC.", "tickers": ["TSN"], "exchanges": ["NYSE"]}],
 "error": null, "cached": true, "source_url": "https://www.sec.gov/files/company_tickers.json", "…": "…"}
```

`BRK.B` and `BRK-B` are the same ticker. A name matches exact titles first,
then titles starting with or containing it; when several companies match,
`ambiguous` is `true`, the company fields are `null` and `candidates` lists
up to 10. A CIK without a listed ticker is resolved from its submissions.

### GET /v1/sec/companies/{cik}/submissions

Query: `forms`, `since`, `until` (filing dates, `YYYY-MM-DD`), `limit`
(default 100, max 10,000), `history` (default `true`: also read the pages of
older filings SEC lists under `files`, only those that can hold filings
after `since`). Company fields `cik`, `cik_padded`, `name`, `title`,
`tickers`, `exchanges`, `entity_type`, `sic`, `sic_description`, `category`,
`fiscal_year_end`, `state_of_incorporation`, `ein`, `lei`, `website`,
`phone`, `business_address`, `former_names`; `filings` (newest first, at most
`limit`), `filings_total`, `truncated`, and `files` (`name`, `filing_count`,
`filing_from`, `filing_to`, `url`, `fetched`, `body_url`). Each filing:

| Field | From SEC's column |
|---|---|
| `accession` | `accessionNumber` |
| `form`, `filing_date`, `report_date`, `acceptance_datetime` | `form`, `filingDate`, `reportDate`, `acceptanceDateTime` |
| `act`, `file_number`, `film_number`, `items`, `core_type`, `size` | `act`, `fileNumber`, `filmNumber`, `items`, `core_type`, `size` |
| `is_xbrl`, `is_inline_xbrl` | `isXBRL`, `isInlineXBRL` (as booleans) |
| `primary_document`, `primary_document_description` | `primaryDocument`, `primaryDocDescription` |
| `primary_document_url`, `index_url` | built: the document and the `-index.htm` page (`null` URL when the primary document is in a sub-folder, e.g. ownership forms) |
| `is_amendment` | built: the form ends in `/A` |

### GET /v1/sec/companies/{cik}/filings/latest?forms=10-K,20-F,40-F

The newest filing (by filing date) of `forms` — default `10-K,20-F,40-F`
(annual reports; amendments only when you list them, e.g. `10-K,10-K/A`):
the filing fields above at the top level (`accession`, `form`,
`filing_date`, `report_date`, `primary_document`, `primary_document_url`,
`index_url`, `is_amendment`, …) plus `cik`, `cik_padded`, `name`, `tickers`,
`exchanges`. Older filing pages are searched when the recent list has none;
none at all: `not_found`.

### GET /v1/sec/filings/{cik}/{accession}

The filing index: `form`, `filing_date`, `report_date`, `accepted`,
`company`, `is_amendment`, `index_url`, `folder_url`, `accession`,
`accession_nodash`, `index_source` (`index_htm`, or `index_json` when only the
folder listing has the documents) and `documents`: `sequence`,
`description`, `document` (file name), `type` (`10-K`, `EX-21`, `EX-101.SCH`,
…), `size`, `url`. Sub-folders are not listed.

### POST /v1/sec/documents/fetch

```json
{"cik": 100493, "accession": "0000100493-25-000070", "document": "tsn-20250927.htm",
 "max_bytes": 52428800}
```

Fetches one document of a filing (HTML/iXBRL, PDF, exhibit, XML) and answers
**exactly like `POST /v1/fetch`** (`text`, `pdf_text_pages`, `content_kind`,
`body_url`, `sha256`, `body_kind: origin`, …). Optional: `include_html`
(default `true`), `accept_pdf`, `cache_ttl_s`, `timeout_s`, `project_ref`.
Invalid identifiers answer `error: invalid_identifier` in that shape.

### GET /v1/sec/companies/{cik}/facts

A summary of `data.sec.gov/api/xbrl/companyfacts/CIK##########.json`:
`cik`, `cik_padded`, `name`, `taxonomies` (concepts per taxonomy),
`concepts`, `bytes`; the complete JSON is at `body_url`.

### GET /v1/sec/search?q=…&forms=…&dateRange=…

EDGAR full-text search (filings since 2001). Query: `q` (EDGAR syntax),
`forms`, `dateRange` (`all`, `10y`, `5y`, `1y`, `30d`, or `custom` with
`startdt`/`enddt`), `from` (offset, pages of 100). Answer: `query`, `total`,
`total_relation`, `hits` (`accession`, `ciks`, `display_names`, `form`,
`root_forms`, `file_type`, `file_description`, `file_date`, `period_ending`,
`document`, `url`, `index_url`, `score`).

### Batch: POST /v1/batch/sec

Up to 50,000 SEC operations as one job of kind `sec`:

```json
{
  "items": [
    {"op": "resolve", "ticker": "TSN"},
    {"op": "latest", "ticker": "TSN", "forms": "10-K"},
    {"op": "latest", "cik": 100493},
    {"op": "document", "cik": 100493, "accession": "0000100493-25-000070", "document": "tsn-20250927.htm"}
  ],
  "idempotency_key": "chipiron-run-2026-10-20-sec-0001",
  "callback_url": "https://chipiron.example/hooks/cormorant"
}
```

`op` is `resolve` (`ticker`, `cik` or `name`), `submissions` (`cik` or
`ticker`, `forms`, `since`, `until`, `limit`, `history`), `latest` (`cik` or
`ticker`, `forms`), `facts` (`cik` or `ticker`), `filing` (`cik`,
`accession`) or `document` (`cik`, `accession`, `document`, `max_bytes`,
`accept_pdf`, `include_html` — default `false` in a batch); `project_ref`
on any item. Identifiers are checked when the job is created (`422
invalid_item` naming the item). Item results have the shape of the single
endpoint (plus `op`); `document` results that of `POST
/v1/sec/documents/fetch`. Items share the cluster-wide SEC rate limit with
every other SEC request (at most 10 requests/s per egress IP) and are
admitted a few at a time: a full limiter or a SEC backoff puts an item back
in the queue instead of failing it. Each item is metered once as kind `sec`
(a ticker resolved inside an item is not extra) and counts as `fetch` for
key limits.

## GET /v1/usage?from=YYYY-MM-DD&to=YYYY-MM-DD

Usage of the caller's scope and what it costs. Inclusive dates (UTC), default:
current month to date, at most 400 days. Read from aggregates refreshed every
5 minutes, never from the request log, so it answers in the same time at any
volume.

**Who sees what.** A standard key (`cmr_…`) sees only its own requests. An
account admin key (`cma_…`) sees every key and project of its customer —
revoked keys included, and usage recorded before per-key attribution as
`"key": null`. A project without a customer is its own account. Nothing ever
crosses a customer boundary; a key or project you cannot see is `404`, like
one that does not exist.

**Parameters**

| Parameter | Meaning |
|---|---|
| `from`, `to` | Range of UTC days (inclusive). |
| `group_by` | Comma-separated, combinable: `key`, `project`, `category`, `day`, `hour` (hourly data: last 35 days). |
| `key` | Only these keys (comma-separated prefixes, e.g. `cmr_AbCdEf`). |
| `project` | Only these projects (comma-separated names). |
| `category` | Only these billing categories: `search`, `search_fallback`, `search_cached`, `fetch_own`, `fetch_premium`, `fetch_cached`, `company`, `company_cached`, `sec`, `sec_cached`, `company_profile`, `company_profile_cached`. |
| `format` | `csv` (same as `Accept: text/csv`). |

```json
{
  "project": "acme-crawler", "key": "cma_XyZaBc", "role": "account_admin",
  "scope": {"customer": "Acme", "projects": ["acme-crawler", "acme-search"], "keys": "all"},
  "from": "2026-10-01", "to": "2026-10-19",
  "as_of": "2026-10-19T11:05:02+00:00", "final": false, "currency": "USD",
  "totals": {"requests": 9120, "succeeded": 8800, "failed": 320, "success_rate": 0.9649,
             "cached": 1200, "cache_hit_rate": 0.1316,
             "errors": {"http_404": 210, "all_tiers_failed": 110},
             "latency_ms_p50": 900, "latency_ms_p95": 7400, "bytes": 812345678,
             "premium_requests": 90, "premium_share": 0.0113, "charge": "15.96",
             "billable": {"fetch_own": {"quantity": "8710", "included": "1000",
                                        "billable": "7710", "amount": "15.42"},
                          "fetch_premium": {"quantity": "90", "included": "0",
                                         "billable": "90", "amount": "0.36"},
                          "premium_gb": {"quantity": null, "included": null,
                                      "billable": null, "amount": "0.18"}}},
  "search": {"requests": 1200, "cached": 300, "cache_hit_rate": 0.25, "succeeded": 1198,
             "success_rate": 0.998, "bytes": 0, "premium_bytes": 0,
             "latency_ms_p50": 1100, "latency_ms_p95": 3900, "fallbacks": 60,
             "fallback_rate": 0.0667, "providers": {"searxng": {"requests": 840, ...}, "brave": {...}}},
  "fetch": {"requests": 7920, "cached": 900, "tiers": {"direct": {...}, "proxy": {...}},
            "errors": {"http_404": 210},
            "by_outcome": {"succeeded": {"requests": 7600, "bytes": 800000000,
                                         "premium_bytes": 8800000, "premium_requests": 80},
                           "failed": {"requests": 320, "bytes": 0,
                                      "premium_bytes": 323456, "premium_requests": 10}}, ...},
  "company_profile": {},  // extended company profiles (kind company_profile), same fields; {} when none
  "group_by": ["key"],
  "groups": [{"key": "cmr_AbCdEf", "key_label": "crawler", "key_role": "standard",
              "key_status": "active", "project": "acme-crawler", "requests": 9000, ...,
              "charge": "15.70", "billable": {...}}],
  "keys": [{"prefix": "cmr_AbCdEf", "label": "crawler", "role": "standard",
            "project": "acme-crawler", "active": true, "requests": 9000, "cached": 1150,
            "succeeded": 8690, "search_requests": 1100, "fetch_requests": 7900, "company_requests": 0,
            "company_profile_requests": 0,
            "premium_bytes": 9123456, "charge": "15.70"}],
  "account": [{"period": "2026-10", "from": "2026-10-01", "to": "2026-10-19", "currency": "USD",
               "complimentary": true, "usage_charges": "15.96", "account_level_amount": "-15.96",
               "account_lines": [{"kind": "complimentary", "description": "Complimentary plan: 100 % discount",
                                  "amount": "-15.96"}],
               "list_total": "15.96", "net_total": "0.00"}]
}
```

- **Metrics** (in `totals` and every group): `requests` (cache hits
  included), `succeeded`, `failed`, `cached`, `errors` (failures by error
  code, top 50), `latency_ms_p50`/`latency_ms_p95` of uncached requests
  (estimated from a latency histogram with bounds 100, 250, 500 ms, 1, 2, 5,
  10, 20, 30, 60 and 120 s — exact to the bucket), `bytes`, `premium_requests`
  and `premium_share` (fetches that needed the premium routes, of all
  uncached fetches).
- **Charges** (projects with a customer): `charge` is what the group costs at
  your plan's prices in your invoice currency, as a decimal string; `billable`
  lists each price-plan meter with the measured `quantity`, the `included`
  units it used (its pro-rata share of the plan's included units of the
  month), `billable` and `amount`. Meters billed on provider cost show only the
  amount. The period is rated exactly like the invoice and each usage line is
  split to the cent, first per key, then per day and category: **any grouping
  adds up to the same total**, and the keys add up to the usage lines of the
  estimate (`GET /v1/account`) or invoice. Base fee, minimum, contract
  discount and the complimentary credit are account-level lines: account admin
  keys see them per period in `account`. Complimentary accounts see list-price
  charges and the 100 % `complimentary` line.
- **Provider cost** — `cost_usd` and the project `budget` (Cormorant's spend
  with its search and proxy providers) appear only for projects without an external
  customer (internal use). Customers see what they pay, never what Cormorant
  pays.
- **Freshness**: `as_of` is the time up to which every request is included;
  `final: true` (top level, and per group with `group_by=day`) means the days
  are closed and will not change (a day becomes final two hours after it
  ends).
- `search` and `fetch` keep the per-kind blocks of earlier versions (now for
  the caller's scope), with `providers`/`tiers` breakdowns. With `key` in
  `group_by`, `keys` keeps the earlier per-key list as well.
- **Latency components**: every per-kind block has `latency_components`
  (`queue_ms`, `politeness_wait_ms`, `upstream_ms`, each `{p50, p95}` of
  uncached requests), and each entry of `tiers`/`providers` adds
  `latency_ms_p50/p95` and `<component>_p50/_p95`, so a slow tier can be told
  apart from a busy domain or a queue backlog. Estimated from histograms like
  the other percentiles; requests from before these were recorded are not
  included.
- `by_outcome` (in every per-kind block) splits the same requests into
  `succeeded` and `failed`, each with `requests`, `bytes`, `premium_bytes`,
  `premium_requests` (fetch) and `cost_usd` (provider cost, internal projects
  only): `fetch.by_outcome.failed.premium_bytes` is what blocked pages cost
  before the fetch gave up. Whether failed requests are billed depends on
  your plan (`billable`).

**CSV.** `Accept: text/csv` (or `format=csv`) returns one row per group — or
one row of totals without `group_by` — with the dimensions, the metrics,
`charge`, the meters as `<meter>_quantity`, `<meter>_included`,
`<meter>_billable`, `<meter>_amount` and `errors` as `code=n;code=n`.

```sh
curl -sS "$CORMORANT_URL/v1/usage?from=2026-10-01&to=2026-10-31&group_by=key,day" \
  -H "Authorization: Bearer $CORMORANT_API_KEY" -H "Accept: text/csv" -o usage.csv
```

## GET /v1/usage/timeseries

The same metrics per `granularity=hour` (last 35 days; default range: today)
or `granularity=day` (default: month to date), one point per bucket up to now
— buckets without usage are zero, so the points can be charted directly.
`group_by` (`key`, `project`, `category`, combinable) gives one series per
value; `key`, `project` and `category` filter as for `GET /v1/usage`. Same
scope rules, same charges (an hour's charge is its share of the day's), CSV
with `Accept: text/csv`.

```json
{"granularity": "hour", "from": "2026-10-19", "to": "2026-10-19", "as_of": "2026-10-19T11:05:02+00:00",
 "final": false, "currency": "USD", "group_by": ["key"],
 "series": [{"key": "cmr_AbCdEf", "key_label": "crawler", "key_role": "standard",
             "key_status": "active", "project": "acme-crawler",
             "points": [{"t": "2026-10-19T00:00:00+00:00", "final": false, "requests": 410,
                         "succeeded": 402, "failed": 8, "latency_ms_p95": 6100,
                         "premium_share": 0.01, "charge": "0.82", ...}, ...]}]}
```

## GET /v1/usage/domains

The busiest fetched domains of the caller's keys over the range (kept 90
days): `requests`, `succeeded`, `success_rate`, `premium_requests` and
`premium_share` per domain, busiest first (`limit`, default 50, at most 500).
`group_by=key` returns a top list per key. Built from each key's 100 busiest
domains per day; usage recorded before per-key attribution is not included.

```json
{"from": "2026-10-01", "to": "2026-10-19",
 "domains": [{"domain": "www.example.com", "requests": 1200, "succeeded": 1150,
              "success_rate": 0.9583, "premium_requests": 40, "premium_share": 0.0333}]}
```

## GET /v1/requests

The request ledger, newest first and in real time: one row per search, fetch
or batch item of the caller's scope (a standard key: its own requests; an
account admin key: the whole customer). Kept as long as the request log (90
days; default range: the last 30 days).

| Parameter | Meaning |
|---|---|
| `from`, `to` | Range of UTC days. |
| `key`, `project`, `category` | As for `GET /v1/usage`. |
| `kind` | `search`, `fetch`, `company` (`/v1/companies/…`), `sec` (SEC EDGAR: `/v1/sec/…` and fetches of SEC hosts) or `company_profile` (extended company profiles). |
| `outcome` | `succeeded` or `failed`. |
| `error`, `domain`, `project_ref`, `job_id`, `cached` | Exact matches. |
| `limit` | Rows per page, 1–1000 (default 100). |
| `cursor` | `next_cursor` of the previous page. |

```json
{"from": "2026-09-20", "to": "2026-10-19", "as_of": "2026-10-19T11:05:02+00:00",
 "currency": "USD", "limit": 100, "next_cursor": "MjAyNi0xMC0xOVQxMTowMjoxMyswMDowMHw5ODc2",
 "requests": [{"request_id": "req_6b1f2c0e9a7d4b5c8e3f1a2b3c4d5e6f",
               "time": "2026-10-19T11:02:13+00:00", "key": "cmr_AbCdEf", "key_role": "standard",
               "project": "acme-crawler", "project_ref": "company:4711", "job_id": null,
               "kind": "fetch", "domain": "www.example.com", "tier": "direct", "status": 200,
               "ok": true, "error": null, "latency_ms": 840, "cached": false, "bytes": 48211,
               "premium": false, "category": "fetch_own", "charge": "0.001842"}]}
```

Page with `cursor` until `next_cursor` is `null`; the next page is also in
the `Link: <…>; rel="next"` header (and `X-Next-Cursor`). Pages are keyset
pages, so they stay fast and consistent however deep you go, while new
requests arrive on top. `charge` is the request's share of its key's
charges for that day and category (their average price, 6 decimals); it is
`null` until the request is aggregated (within 5 minutes) and settles when
the day is final. `GET /v1/requests/{request_id}` returns one request with
`reason` and the tiers tried (`attempts`); another key's (standard keys) or
another customer's request is `404`. CSV with `Accept: text/csv` (one page;
the next cursor is in `X-Next-Cursor`).

## GET /v1/quota

Every limit that applies to the caller this month, each with `limit`,
`used`, `reserved`, `remaining`, `resets_at`, `projected_end_of_period`
(linear extrapolation to the end of the month) and `status` (`ok`,
`warning` from 80 %, `exhausted`): the key's spend cap
(`monthly_spend`, money as decimal strings), request quotas
(`monthly_requests.<category>`) and rate limit; the project's rate limit
(and provider budget, for projects without an external customer). An account
admin key gets every active key of the account and, for a customer, the plan's
included units (`included.<meter>`).

```json
{"key": "cmr_AbCdEf", "role": "standard", "period": "2026-10",
 "resets_at": "2026-11-01T00:00:00+00:00", "as_of": "2026-10-19T11:05:02+00:00",
 "usage_refreshed_at": "2026-10-19T11:05:00+00:00",
 "keys": [{"key": "cmr_AbCdEf", "key_label": "crawler", "key_role": "standard",
           "project": "acme-crawler", "is_self": true, "allowed_categories": null,
           "usage": {"requests": {"search": 120, "fetch": 81234, "fetch_premium": 412},
                     "spend": "38.12", "currency": "USD"},
           "limits": [{"name": "monthly_requests.fetch", "unit": "requests", "limit": 100000,
                       "used": 81234, "reserved": 0, "remaining": 18766,
                       "resets_at": "2026-11-01T00:00:00+00:00",
                       "projected_end_of_period": 132550, "status": "warning"},
                      {"name": "monthly_spend", "unit": "USD", "limit": "50.00", "used": "37.90",
                       "reserved": "0.22", "remaining": "11.88", "status": "ok", ...}]}],
 "project": {"project": "acme-crawler",
             "limits": [{"name": "rate_limit", "unit": "requests/s", "limit": 200, "used": 3, ...}]}}
```

`reserved` is the spend estimated since the last aggregation (or, for the
provider budget, spend held for calls in flight). `events` lists the
thresholds crossed this month in your scope (see *Alerts* below).
`alert_email` says whether alert e-mails reach your account's alert
addresses (there may be several): `status` is `deliverable`, `not_set`,
`partially_undeliverable` (some addresses) or `undeliverable` (all), with
`addresses`, `undeliverable` (counts), and `reason` (`bounced` or
`complained`) and `since` of the first undeliverable one. After a hard
bounce or a spam complaint no further alert mail goes to that address until
it is changed (contact us); webhooks are not affected.

## Alerts: POST, GET, DELETE /v1/alerts

A webhook receives an event whenever something in its scope crosses **50, 80
or 100 %** of a limit or allowance in a month (each threshold fires once per
month): a key's spend cap or request quota, a project's provider budget
(projects without an external customer) or the plan's included units of the
account. The same events are listed in `GET /v1/quota` (`events`).

```json
POST /v1/alerts
{"url": "https://hooks.example.com/cormorant", "scope": "customer", "thresholds": [80, 100]}

201 {"id": 7, "url": "https://hooks.example.com/cormorant", "scope": "customer", "key": null,
     "project": null, "thresholds": [80, 100], "created_at": "2026-10-19T11:00:00+00:00",
     "last_delivery_at": null, "last_status": null, "secret": "whsec_9f2c…"}
```

- `scope`: `key` (one key; `key` = its prefix, default the calling key),
  `project` (`project` = name; the project and its keys) or `customer` (the
  whole account). A standard key can only watch itself (`403
  account_admin_required` otherwise); an account admin key anything in its
  account. Default: `key` for standard keys, `customer` for admin keys.
- `thresholds`: any of `50`, `80`, `100` (default all).
- `url` must be `https` and resolve to a public address; at most 20 webhooks
  per account. `secret` is returned **only once**.
- `GET /v1/alerts` lists the webhooks you can see (never secrets) with
  `last_delivery_at` and `last_status`; `DELETE /v1/alerts/{id}` removes one.

**Deliveries** are `POST <url>` with the event as JSON:

```json
{"id": "evt_42", "created_at": "2026-10-19T11:05:03+00:00", "type": "usage.threshold",
 "scope": "key", "customer": "Acme", "project": "acme-crawler", "key": "cmr_AbCdEf",
 "key_role": "standard",
 "limit": {"name": "monthly_spend", "unit": "USD", "limit": "50.00", "used": "40.12",
           "threshold": 80, "period": "2026-10", "resets_at": "2026-11-01T00:00:00+00:00"}}
```

Headers: `X-Cormorant-Event: usage.threshold`, `X-Cormorant-Delivery: dlv_…`
and `X-Cormorant-Signature: t=1760871903,v1=5257a8…`, where `v1` is the hex
HMAC-SHA256 of `"<t>.<raw body>"` with your `secret`. Verify it in constant
time and reject a `t` older than a few minutes:

```python
import hashlib, hmac, time


def verify(secret: str, body: bytes, header: str) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    mac = hmac.new(secret.encode(), f"{parts['t']}.".encode() + body, hashlib.sha256)
    fresh = abs(time.time() - int(parts["t"])) < 300
    return fresh and hmac.compare_digest(mac.hexdigest(), parts["v1"])
```

Any `2xx` answer counts as delivered; other answers and network errors are
retried after 1, 5, 15, 60, 180, 360 and 720 minutes (8 attempts). Redirects
are not followed, and each delivery connects to the address that passed the
check (no DNS rebinding). Events are evaluated with the usage aggregates (every 5
minutes). Use `id` to ignore duplicates.

## GET /v1/keys and GET /v1/keys/self

The keys the caller may see — by prefix, label and role, never the key itself:
every key of the customer's projects for an account admin key (revoked keys
included), only itself for a standard key. The prefix always shows the role:
`cma_…` is an account admin key, `cmr_…` a standard key.

```json
{"role": "account_admin",
 "keys": [{"prefix": "cma_XyZaBc", "label": "ops", "role": "account_admin",
           "project": "acme-crawler", "status": "active",
           "created_at": "2026-09-01T08:00:00+00:00", "revoked_at": null,
           "last_used_at": "2026-10-19T11:02:00+00:00", "is_self": true},
          {"prefix": "cmr_AbCdEf", "label": "crawler", "role": "standard", "status": "revoked", ...}]}
```

Each key also has `limits` (`monthly_spend_cap`, `currency`, `monthly_requests`,
`rate_limit_rps`, `allowed_categories`; `null` without limits).
`GET /v1/keys/self` returns the calling key in the same shape plus its use
this month (`usage`) and quota state (`quota`, the entries of `GET /v1/quota`).

## Account and invoices

Keys belong to a project; a project may belong to a **customer** (the billed
account). These endpoints show the key's own project usage plus the plan,
current-period estimate and issued invoices of that customer — never another
customer's data, never drafts. Money is a decimal **string** in the invoice
currency (USD, CHF or EUR). Usage is aggregated every 5 minutes (`usage_refreshed_at`).
See [billing.md](/docs/billing.md) for pricing rules.

### GET /v1/account

```json
{
  "project": {"id": 7, "name": "acme-crawler"},
  "period": {"period": "2026-10", "from": "2026-10-01", "to": "2026-10-31"},
  "usage_refreshed_at": "2026-10-19T11:00:04+00:00",
  "project_usage": {"fetch_own": {"requests": 9120, "succeeded": 8800, "bytes": 812345678,
                    "bytes_ok": 800000000, "premium_bytes": 0, "premium_bytes_ok": 0,
                    "cost_usd": "0", "cost_ok_usd": "0"}, "search": {...}},
  // cost_usd / cost_ok_usd: null for projects of an external customer
  "customer": {"name": "Acme", "company_name": "Acme Research AG", "currency": "USD",
               "status": "active", "complimentary": true},
  "customer_usage": {...},
  "plan": {"id": 1, "code": "standard-usd", "version": 1, "name": "Standard (USD)",
           "currency": "USD", "complimentary": false},
  "estimate": {"period": "2026-10", "currency": "USD",
               "lines": [{"kind": "usage", "component": "fetch_own",
                          "description": "Page fetches (1,000 included)", "quantity": "8800",
                          "unit": "fetch", "unit_price": null, "amount": "15.60",
                          "detail": {"included": "1000", "billable": "7800",
                                     "tiers": [{"from": "0", "to": "100000", "quantity": "7800",
                                                "unit_price": "0.002", "amount": "15.60"}]}},
                         {"kind": "complimentary", "description": "Complimentary plan: 100 % discount", "amount": "-15.60", ...}],
               "subtotal": "15.60", "discount_total": "15.60", "net_total": "0.00",
               "vat_rate": "8.1", "vat_amount": "0.00", "total": "0.00", "complimentary": true,
               "list_total": "15.60"}
}
```

`keys` breaks the period down per API key. An account admin key (`cma_…`)
sees every key of the customer's projects (or of this project without a
customer); an additional key (`cmr_…`) sees only itself. Keys appear by
`prefix`, `label` and `role`, never ids or secrets, with its usage by category and `amount`, its share of
the usage charges at list price. Each usage line of the estimate is split
between keys in proportion to their share of that meter (to the cent), so the
amounts add up to the usage lines; base fee, minimum, discounts and the
complimentary credit stay on the account. A row with `"prefix": null` is usage recorded
before per-key attribution.

```json
"keys": [{"prefix": "cma_XyZaBc", "label": "billing", "role": "account_admin", "project": "acme-crawler",
          "active": true, "requests": 9120, "succeeded": 8800,
          "usage": {"fetch_own": {"requests": 9120, ...}}, "amount": "15.60"}]
```

Projects without a customer get `"customer": null, "plan": null, "estimate": null`
(and `amount: null` per key).
Categories: `search`, `search_fallback`, `search_cached`, `fetch_own`,
`fetch_premium`, `fetch_cached`, `company`, `company_cached`, `sec`, `sec_cached`,
`company_profile`, `company_profile_cached`.

### GET /v1/account/invoices

`{"invoices": [{"number": "CORM-2026-0001", "status": "issued|paid|void", "period": "2026-09",
"period_start", "period_end", "currency", "plan_label", "complimentary", "subtotal", "discount_total",
"net_total", "vat_rate", "vat_amount", "total", "vat_note", "issued_at", "due_at", "paid_at",
"voided_at", "pdf_url": "/v1/account/invoices/CORM-2026-0001/pdf"}]}`

### GET /v1/account/invoices/{number} and /pdf

The invoice with its lines (and tier breakdown), or the PDF. Another
customer's number (or a draft) is `404`.

## Company data

The company register: one source-attributed record (`company/v1`) per
company Cormorant has been asked about, kept and versioned. Sources are
official registers, GLEIF and other open, legitimate sources
(`GET /v1/companies/sources`). Every data call is metered as **`company`**
(billing category and meter `company`; one request per call or batch item).
`GET /v1/companies/sources`, `GET /v1/schemas/company` and watch management
are not metered.

| Endpoint | Purpose |
|---|---|
| `GET /v1/companies/search?q=&country=&identifier=&status=&legal_form=&limit=&cursor=&live=` | Register matches (trigram/full-text over current, former and trading names; exact on `identifier`) plus live hits from the sources covering `country` on the first page (unless the register has a near-exact hit, score ≥ 0.95; `live=true` always, `live=false` never). `match_score` 0–1, `in_register`, `ref`; `live_sources[]` per source `{source, ok, status, error, ms}` |
| `GET /v1/companies/{id}` | The record. `id` = `cmp_…` or `scheme:value`. `max_age`, `refresh`, `include=people,ownership,filings,financials,events,claims`, `sources=` |
| `GET /v1/companies/{id}/history?field=` | Claims timeline: every value each source asserted, with `observed_at`, `valid_from`, `valid_to`, `current` |
| `GET /v1/companies/{id}/{events,people,ownership,relationships,filings,financials,sources}` | Sub-resources with paging (`limit`, `cursor`); `people` accepts `include_former` |
| `POST /v1/companies/resolve` | `{name, country, city, identifiers, isin, persist}` → `match` (confident only; added to the register unless `persist: false`, so it has its `cmp_` `id` and `in_register: true`) and ranked `candidates` |
| `POST /v1/companies/batch`, `POST /v1/companies/batch/csv` | Job of kind `company`: `get`, `refresh` or `resolve` per item; JSON or CSV in; results via `GET /v1/jobs/{job_id}` or `GET /v1/companies/batch/{job_id}.csv` |
| `GET /v1/companies/changes?since=&cursor=&country=` | Change feed of the register (company and changed top-level fields), oldest first |
| `POST/GET /v1/companies/watches`, `DELETE /v1/companies/watches/{id}` | Signed `company.changed` webhooks for a list of companies |
| `GET /v1/companies/sources` | Sources: countries, schemes, capabilities, precedence, licence URL, freshness classes, live health |
| `GET /v1/schemas/company` | JSON Schema (2020-12) of `company/v1` |

**Identifiers** are written `scheme:value` and normalised (spaces, dots and
case do not matter; check digits are verified): `ch_uid`
(`CHE-101.329.561`), `ch_chid`, `ch_ehraid`, `lei`, `us_cik`, `uk_crn`,
`de_register` (`München:HRB:6684`), `fr_siren`, `fr_siret`, `no_orgnr`,
`dk_cvr`, `fi_ytunnus`, `ee_regcode`, `cz_ico`, `pl_krs`, `be_kbo`, `au_abn`,
`au_acn`, `nz_nzbn`, `jp_corporate_number`, `br_cnpj`, `sg_uen`, `ca_corp`,
`vat` (`DE129273398`, `CHE101329561`), `isin` (`CH0038863350`: the issuer,
found through GLEIF's ISIN-to-LEI mapping; the ISIN is then kept on the
record), `ticker` (`XNAS:AAPL`), `wikidata`. An invalid identifier (e.g. a
wrong ISIN check digit) is `422 invalid_identifier`.

**Refresh on request.** A request answers from the register. Each source's
data about the company is re-fetched first when it is older than the
source's freshness class for that field group (e.g. registry core 24 h,
people 7 days) or than `max_age` (seconds); `refresh=true` re-fetches
everything. The answer's `refresh` lists per source and group what was
`refreshed`, `fresh`, `failed` (error class; the register still answers),
`not_found`, `in_progress`, `pending` (a refresh slower than 2 s, `error:
"deadline"`, or of a source whose canaries fail, `error: "canary_failed"`:
it finishes in the background and the next request sees its data) or
`skipped` (the source backs off after repeated failures, `error:
"backoff"`). Stale sources are refreshed concurrently and the answer waits
at most about 2 s for them; batch items wait for every refresh. A company not yet in the register is looked
up in the sources that know the identifier's scheme and added; unknown
everywhere is `404 company_not_found`, and remembered (ISIN 24 h, other
schemes 1 h; `refresh=true` asks again): within that time the sources are
not asked and `refresh` shows `not_found` with `error: "cached"`. Companies nobody asks for are never
re-scraped; official change feeds (e.g. GLEIF's daily updates) keep the ones
in the register current.

**Provenance.** Every value is a claim of a source; the record shows the
resolved value of each field, `provenance` (source, `observed_at`,
`confirmed_at`, `sha256` of the raw document, `conflict`), `quality`
(`completeness`, `conflicts` where sources disagree) and `sources[]` (raw
documents with `body_url` — downloadable with `GET /v1/bodies/{sha256}` once
you have received the record). `include=claims` adds every current claim.
Precedence when sources disagree: official register > official gazette >
GLEIF > regulatory filings > crosswalks > commercial sources, then each
field's declared rule. `names.legal` is the national register's primary
entry and does not depend on refresh order (CH: `zefix_rest` >
`lindas_zefix` > `uid_register` > `ch_cantonal_excerpt` > `sogc` > `gleif`);
other sources' forms are in `names.alternates` (`{name, source}`), language
versions in `names.translations` (with `source`). A merged
record's `cmp_` id keeps working and answers with the surviving record and
`redirected_from`.

```sh
curl -s "https://api.cormorant.ch/v1/companies/ch_uid:CHE-101.329.561?include=ownership" \
  -H "Authorization: Bearer $CORMORANT_API_KEY"
```

```json
{
  "schema_version": "company/v1",
  "id": "cmp_01JA2B3C4D5E6F7G8H9J0KMNPQ",
  "identifiers": [{"scheme": "ch_uid", "value": "CHE-101.329.561", "issuer": "Federal Statistical Office (BFS)", "status": "active", "parts": null, "sources": ["gleif"]},
                  {"scheme": "lei", "value": "BFM8T61CT2L1QCEMIK50", "issuer": "GLEIF", "status": "active", "parts": null, "sources": ["gleif"]}],
  "names": {"legal": "UBS AG", "previous": [{"name": "SBC AG", "lang": "en"}], "trading": [], "translations": [], "alternates": []},
  "legal_form": {"local_code": "AG", "local_name": "Company limited by shares", "elf_code": "MVII", "category": "corporation"},
  "status": {"value": "active", "local_value": "ACTIVE"},
  "jurisdiction": {"country": "CH", "subdivision": null, "register": {"authority": "RA000549", "court": null, "entity_id": "CHE-101.329.561"}},
  "addresses": [{"type": "registered", "lines": ["Bahnhofstrasse 45"], "street": "Bahnhofstrasse 45", "house_number": null, "postal_code": "8001", "city": "Zurich", "region": "CH-ZH", "country": "CH"}],
  "dates": {"incorporated": "1978-02-28", "registered": null, "deregistered": null, "fiscal_year_end": null},
  "ownership": [{"type": "direct_parent", "holder": {"name": "UBS Group AG", "identifiers": [{"scheme": "lei", "value": "549300SZJ9VS8SGXAN81"}], "country": "CH"}, "source": "gleif"}],
  "provenance": {"names.legal": {"source": "gleif", "observed_at": "2026-10-19T08:00:00+00:00", "confirmed_at": "2026-10-19T08:00:00+00:00", "sha256": "9f86…", "conflict": false}},
  "sources": [{"source": "gleif", "url": "https://api.gleif.org/…/BFM8T61CT2L1QCEMIK50", "retrieved_at": "2026-10-19T08:00:00+00:00", "sha256": "9f86…", "body_url": "/v1/bodies/9f86…", "content_type": "application/vnd.api+json", "licence": "https://www.gleif.org/en/meta/lei-data-terms-of-use", "parser_version": "gleif/1"}],
  "quality": {"completeness": 0.778, "conflicts": []},
  "refresh": [{"source": "gleif", "group": "core", "status": "fresh"}, {"source": "gleif", "group": "ownership", "status": "refreshed", "at": "2026-10-19T08:00:01+00:00"}],
  "redirected_from": null,
  "updated_at": "2026-10-19T08:00:01+00:00"
}
```

**Latency.** A company in the register is answered from it: `resolve`
with a confident register match and `search` with a near-exact register hit
ask no source. Otherwise the live sources are asked concurrently and waited
for at most 2.5 s (batch `resolve` items: 12 s); slower sources finish in
the background, sources backing off after repeated failures or with failing
canaries are not asked. `live_sources` (search and resolve) reports each
source as `ok`, `failed` (`error`: error class), `timeout` or `skipped`
(`error`: `backoff` or `canary_failed`) with `ms`. A newly stored `resolve`
match gets the other sources' data in the background.

**Batch.** `POST /v1/companies/batch` takes `{action, items: [{ref} | {name,
country, city, identifiers, isin}], include, max_age, persist, idempotency_key,
callback_url}`; `POST /v1/companies/batch/csv` takes a CSV body (header
`ref,name,country,city,isin,identifiers` or any subset, identifiers
separated by `;`) with the same options as query parameters. An item with an
invalid ISIN fails alone (`invalid_identifier`). Item results: `{company_id, record}` for
`get`/`refresh` (the record as above), `{match, candidates, company_id}` for
`resolve`, or `{error}` (`company_not_found`, `invalid_identifier`).

**Change feed and watches.** `GET /v1/companies/changes` returns `{items:
[{seq, company_id, kind: created|updated|merged|split, fields, country,
changed_at}], next_cursor}`; start with `since` (ISO 8601) and continue with
`cursor`. `POST /v1/companies/watches` with `{companies: [ids or
scheme:value], url}` (HTTPS, public address) returns the watch with its
`secret` once (at most 20 watches per project, 1000 companies each). Changes
of watched companies are POSTed every minute as `{event: "company.changed",
id, watch_id, changes}` with `X-Cormorant-Event`, `X-Cormorant-Delivery` and
`X-Cormorant-Signature: t=<unix>,v1=<hex>` (HMAC-SHA256 of `"<t>.<raw
body>"`, as for usage alerts). Any 2xx is delivered; otherwise retried after
1, 5, 15, 60 and 180 minutes, then hourly, without skipping changes. A
watched record merged into another is followed.

**Errors**: `404 company_not_found`, `422 invalid_identifier`,
`422 invalid_parameter` (e.g. unknown `include`), `422 invalid_item` (batch);
otherwise as for every endpoint (`401`, `403`, `429`).

## Service status

Public status of the service for your monitoring, and the page at
[`/status`](/status). **No authentication** (an `Authorization` header is
accepted and ignored), CORS-enabled, refreshed at most every 15 s and shared
by all callers (`Cache-Control: public, max-age=15`, `ETag` /
`If-None-Match` → `304`). Limited to 120 requests per minute per client IP
(`429` with `Retry-After`). The status never contains host names, addresses,
instance URLs, project names, keys or internal error strings — only component
ids, statuses, reason codes and aggregate numbers.

The status endpoints answer **`200` whatever the service status**: the
status is in the body. Alert on the body (`status` is not `operational`), and
treat a timeout, a connection error or any other HTTP code as "Cormorant is
unreachable" — a status endpoint that answered `503` during an outage could
not be told apart from a broken proxy or network in between.

| Endpoint | Returns |
|---|---|
| `GET /v1/status` | overall `status`, every component with `reasons` and `metrics`, incidents and maintenance |
| `GET /v1/status/summary` | only `status`, `description`, `updated_at` (for uptime monitors) |
| `GET /v1/status/components/{id}` | one component plus the incidents and maintenance naming it (`404` for an id not in this deployment) |
| `GET /v1/status/history?hours=24` | worst status per 5 minutes (per hour beyond 24 h, up to 168 h) and uptime per component; overall as `_overall` |

```json
{
  "status": "degraded",
  "description": "Degraded performance",
  "updated_at": "2026-10-19T09:41:15+00:00",
  "window_s": 600,
  "components": [
    {"id": "api", "name": "API", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": null},
    {"id": "fetch", "name": "Fetch", "description": "…", "status": "degraded", "reasons": ["low_success_rate"], "metrics": {}, "group": null},
    {"id": "company_gleif", "name": "Company data: GLEIF", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": "company_data"}
  ],
  "incidents": [
    {"id": "inc_3f9a1c0b7e", "title": "Elevated fetch failures", "status": "identified",
     "impact": "degraded", "components": ["fetch"], "created_at": "…", "updated_at": "…",
     "resolved_at": null, "updates": [{"at": "…", "status": "identified", "message": "…"}]}
  ],
  "maintenance": []
}
```

**Statuses**, best to worst: `operational`, `maintenance`, `degraded`,
`partial_outage`, `major_outage`. The overall `status` is the worst component
status, except that a major outage of a component other than `api`,
`search`, `fetch` or `storage` counts as a partial outage, and it is
`maintenance` when the only components not operational are under announced
maintenance. An unresolved incident shows its components at least at its
`impact`. Components not part of the deployment are not listed (`archive`
when archived copies are disabled, `premium_routes` when no premium route is
configured, `sec` while no SEC identity is configured, `company_profiles` while no
premium route is configured) — do not alert on a missing component.

Counts and rates cover the last `window_s` = 600 s. A component with fewer
than its minimum sample there (20 uncached requests; 10 attempts for a tier)
reports `requests`/`attempts`, `failures` and `success_rate` over the **last
hour** instead; `metrics.window_s` (`<tier>_window_s` in `premium_routes`) says
which window was used, and the status is judged on that window. `null` rates
mean even the hour has too few samples.

| Component | Metrics (rates 0–1; `failures` and `window_s` with every rate) | Not operational when |
|---|---|---|
| `api` | — | database or cache unreachable (major) |
| `search` | `success_rate`, `requests`, `engines_healthy_share`, `engines`, `instances_healthy_share`, `fallback_available`, `fallback_share`, `fallback_success_rate`, `workers_alive`, `queue_lag_s` | success rate of uncached searches < 0.95 / 0.80 / 0.50 (degraded / partial / major); healthy search instances < 50 % (degraded) or none (partial with the fallback search service, major without); healthy engines < 34 % (degraded); no worker; queue lag. `engines_healthy_share`: share of search engines (per instance) whose latest answer in the last 15 minutes returned results (the instance share until searches were observed); `engines`: `[{engine, status}]` with `ok` (healthy everywhere), `degraded` (on some instances) or `down` (rate limited, CAPTCHA or suspended everywhere) |
| `fetch` | `success_rate`, `requests`, `latency_p95_ms`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | success rate of uncached fetches < 0.85 / 0.60 / 0.30 (4xx answers of the site, robots.txt, invalid or blocked URLs, too large, PDF not accepted and deferred are not failures); p95 > 60 s (degraded); no worker (major); queue lag |
| `browser_rendering` | `success_rate`, `attempts`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | browser attempts failing for infrastructure reasons: success < 0.80 / 0.50 / 0.10; no browser worker (major) |
| `archive` | `success_rate`, `attempts` | archive unreachable or erroring: success < 0.80 / 0.50 / 0.10 |
| `premium_routes` | `proxy_success_rate`, `proxy_attempts`, `rendering_success_rate`, `rendering_attempts` (configured tiers only) | provider errors: success < 0.80 / 0.50 / 0.10 |
| `sec` | `success_rate`, `requests`, `throttle_wait_p95_ms` (p95 wait for a slot in the cluster-wide SEC rate limiter), `backoff_active`, `rate_limit_rps` | SEC backing off after 403/429/503 (degraded, `sec_backoff`); success rate of uncached SEC requests < 0.85 / 0.60 / 0.30 (4xx, invalid identifiers and not found are not failures); p95 wait > 5 s (degraded, `throttled`) |
| `providers` | — | an outside service Cormorant relies on (search fallback, premium routes, alternative sources, notifications) currently refuses our access (degraded, `provider_access`); which one is not shown — the operator is alerted |
| `company_profiles` | `success_rate`, `requests`, `latency_p95_ms` (uncached extended company profile operations), `cache_hit_rate`, `refused_share` (operations the source refused: `blocked`, `rate_limited`), `rendering_share` (share served by the rendering service rather than the residential proxy), `throttle_wait_p95_ms` (p95 wait for a slot in the shared rate limit), `backoff_active`, `proxy_skipped` (the residential proxy is being skipped after a recent refusal); always over the last 600 s | success rate of uncached operations < 0.85 / 0.60 / 0.30 (not found, invalid identifiers, robots.txt, premium routes not allowed and exhausted budgets are not failures; `null` below 10); refused share ≥ 0.20 (degraded, `upstream_refusing`); the site throttle backing off after 429 (degraded, `site_backoff`); p95 wait > 10 s (degraded, `throttled`) |
| `batch_jobs` | `scheduler_last_run_s`, `workers_alive`, `running_jobs`, `oldest_waiting_job_s`, `queue_backlog` | scheduler last ran > 30 s (degraded) / > 120 s ago (partial); no batch worker (major); a job waiting to start > 15 min (degraded) / > 60 min (partial) |
| `storage` | `database_ok`, `cache_ok`, `object_storage_ok` (`null`: not configured) | database or cache unreachable (major); object storage unreachable (degraded) |
| `company_<source>` (group `company_data`, one per active company data source, e.g. `company_gleif`) | `success_rate`, `requests` (last hour of calls to the source; `null` below 10), `latency_p95_ms`, `parse_failure_rate`, `canary_ok`, `backoff_active`, `schema_drift` (fields that used to be filled and now come back empty), `change_feed_lag_s` | success < 0.85 / 0.50 / 0.20; canaries (known companies checked every 15 minutes) failing (partial); backoff after repeated failures, schema drift, parse failures > 10 % (degraded). Counts at most as `degraded` for the overall status |

Queue lag (`queue_lag_s`, `sync_queue_lag_s`: how long the oldest synchronous
request has waited for a worker) above 15 / 60 / 120 s is degraded / partial
/ major. Rates computed on fewer than 20 requests (10 tier attempts) are
reported but do not change a status. New metric keys may be added.

**Reason codes**: `no_workers`, `queue_lag`, `low_success_rate`,
`high_latency`, `engines_reduced`, `engines_unavailable`, `tier_errors`,
`scheduler_stalled`, `jobs_waiting`, `database_unreachable`,
`cache_unreachable`, `object_storage_unreachable`, `sec_backoff`,
`throttled`, `provider_access`, `upstream_refusing`, `site_backoff`, `incident`, `maintenance`; company data sources also `canary_failed`, `source_backoff`,
`schema_drift`, `parse_failures`.

**Incidents and maintenance** are posted by the operator. `incidents` lists
unresolved incidents (`investigating`, `identified`, `monitoring`) and those
`resolved` within 7 days, newest first, each with its timeline (`updates`,
newest first). `maintenance` lists `scheduled` and `in_progress` windows and
those `completed` within 7 days, soonest first; an empty `components` list
means the whole service.

**History**: `GET /v1/status/history` returns, per component, `statuses`
(oldest first; `null` where nothing was recorded) and `uptime` over the
period, where a `partial_outage` bucket counts as 30 % downtime and a
`major_outage` bucket as 100 %. Kept 7 days.

## Health and metrics

- `GET /health` → `200 {"status":"ok"}` (process is up). For monitoring
  the service as a whole use [`GET /v1/status`](#service-status).
- `GET /ready` → `200` when PostgreSQL and Redis answer and at least one SearXNG
  instance is registered, else `503`; body lists the checks, never secrets.
- `GET /metrics` → Prometheus text (private network only): requests by kind,
  provider and outcome, cost, bytes, premium bytes, search fallbacks, latency
  histogram, queue lengths, SearXNG instance count — aggregated across all hosts.

## Examples

```sh
export CORMORANT_URL=https://api.cormorant.ch   CORMORANT_API_KEY=cmr_...
curl -s "$CORMORANT_URL/v1/search" -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H 'Content-Type: application/json' -d '{"query":"Holcim annual report 2025","country":"CH"}'

curl -s "$CORMORANT_URL/v1/fetch" -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H 'Content-Type: application/json' -d '{"url":"https://www.holcim.com/investors"}'

curl -s "$CORMORANT_URL/v1/batch/fetch" -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"idempotency_key":"demo-1","items":[{"url":"https://example.com"},{"url":"https://example.org"}]}'
```

```python
import httpx

client = httpx.Client(
    base_url=CORMORANT_URL, headers={"Authorization": f"Bearer {KEY}"}, timeout=180
)
page = client.post("/v1/fetch", json={"url": "https://www.holcim.com/investors"}).json()
if page["error"] is None:
    text = page["text"]
```

Sync calls wait up to 150 s for the result (then `504` with the request id);
use the batch endpoints for bulk work.