# Changelog

Changes to the Cormorant API, newest first. Within v1, fields are added, not
removed; a renamed field or value is announced here with its old name, which
requests may keep using for 30 days. The [API contract](/docs/api.md) is the
reference for every field mentioned here.

## Unreleased

### Fetch: the residential route fails over between networks

- The `proxy` route now runs over several residential networks. When a site
  refuses an exit or a network fails, the request is retried at once through
  another one; each try is its own `proxy` attempt with its status, reason
  and `premium_bytes`. Route names and prices are unchanged. See
  [Tiers and escalation](/docs/api.md).

### Debug requests (available on request)

- `X-Cormorant-Debug: 1` on `POST /v1/search` and `POST /v1/fetch`, for keys
  that support enabled for debugging: the request is recorded in detail and
  the answer carries `debug_url`, a link for support. Ignored for other keys.
  See [API contract](/docs/api.md#authentication).

### Fetch: block types on attempts

- Every attempt that reached a site (or failed talking to it) now carries
  `block_type` (`ok`, `rate_limit`, `challenge`, `hard_block`, `geo_block`,
  `login_wall`, `paywall`, `soft_404`, `empty_shell`, `fingerprint_refusal`,
  `upstream_error`, `not_found`), and `error_detail` names it per attempt
  (`block: challenge`). See [the API contract](/docs/api.md).

### Standard and premium routes: renamed fields

Every call is billed. Fetches run on **standard routes** (the shared cache,
a direct request, a headless browser, archives and alternative copies) and,
only when a site blocks those, on **premium routes** (the residential network
`proxy` and the rendering service `proxy_render`), which are priced higher.
Fields, values and labels that named the routes by their price now use
these names.

**Requests.** Send the new names now. Until **2026-11-19** the old request
field and filter values keep working; after that day they answer `422`.

**Responses** use only the new names from this release on: if you read any
of the response fields below, switch to the new name.

#### Renamed fields

| Where | Old name | New name |
|---|---|---|
| Request field (`POST /v1/fetch`, `/v1/batch/fetch` items) | `allow_paid` | `allow_premium` |
| Billing category and plan meter (`billing_category`, `billing.category`, `billing.units[].meter`, `GET /v1/usage`, `?category=`, `GET /v1/requests`, invoices) | `fetch_paid` | `fetch_premium` |
| Key quota, allowed category, quota header (`X-Cormorant-Quota-Remaining`), `GET /v1/quota` limit name, alert limit name | `fetch_paid`, `monthly_requests.fetch_paid` | `fetch_premium`, `monthly_requests.fetch_premium` |
| Plan meter, included allowance, alert limit name | `paid_gb`, `included.paid_gb` | `premium_gb`, `included.premium_gb` |
| Fetch attempts, `GET /v1/usage` (`by_outcome`, providers, keys), account usage | `paid_bytes`, `paid_bytes_ok` | `premium_bytes`, `premium_bytes_ok` |
| Fetch result on failure | `paid_tiers_tried` | `premium_routes_tried` |
| `GET /v1/usage`, `/v1/usage/timeseries`, `/v1/usage/domains` (JSON and CSV) | `paid_requests`, `paid_share` | `premium_requests`, `premium_share` |
| `GET /v1/requests` rows and CSV column | `paid` | `premium` |
| Attempt reason | `paid_skipped_domain_refuses` | `premium_skipped_domain_refuses` |
| Attempt reason | `paywall_skip_paid` | `paywall_skip_premium` |
| Attempt reason (batch items) | `deferred_before_paid` | `deferred_before_premium` |
| Error code (also `?error=` of `GET /v1/requests`) | `paid_tier_required` | `premium_route_required` * |
| Status component (`GET /v1/status`, `/v1/status/components/{id}`) | `paid_tiers` ("Paid tiers") | `premium_routes` ("Premium routes") * |
| Account, plan and invoice flag (`GET /v1/account`, `/v1/account/invoices`) | `free` | `complimentary` * |
| Per-request billing block | `free_plan` | `complimentary_plan` * |
| Account and invoice line `kind` | `free` ("Free plan: 100 % discount") | `complimentary` ("Complimentary plan: 100 % discount") * |
| MCP client tool parameter / environment variable | `allow_paid` / `CORMORANT_ALLOW_PAID` | `allow_premium` / `CORMORANT_ALLOW_PREMIUM` (the old variable is still read) |

\* Not a plain `paid` → `premium` swap: tiers became routes, and accounts
billed at a 100 % discount are *complimentary*.

Not renamed: the payment state of an invoice (`status: paid`, `paid_at`) and
the tier names themselves (`direct`, `browser`, `archive`, `proxy`,
`proxy_render`, the `tiers` field).

### Extended company profiles: new names in status, usage and billing

- Service status (`GET /v1/status`, the status page): extended company
  profiles are the component `company_profiles`, "Company profiles
  (extended)". See [Monitoring](/docs/monitoring.md).
- Usage and billing name them `company_profile` (billing category and meter;
  `company_profile_cached` when answered from the cache) and request kind
  `company_profile`, with the label "Company profiles (extended)": the
  `company_profile` block and `keys[].company_profile_requests` in
  `GET /v1/usage`, `spend_budget.*.by_kind.company_profile`, quota and alert
  limits `included.company_profile`, `GET /v1/requests` rows and its `kind`
  and `category` filters, `GET /v1/account` and invoice lines. Ledger rows of
  these requests have `domain: null`.
- If you filter or alert on the previous names, switch to these before
  2026-11-19; until then the previous filter values keep working.

### Fetch: known hard domains, Chrome fingerprint through the residential proxy

- The `proxy` tier now sends GET requests with Chrome's TLS/HTTP2
  fingerprint, like `direct`.
- The browser tier no longer shows `HeadlessChrome` in its User-Agent or
  client hints.
- A shipped list of known hard domains (`nestle.com`, `nestleusa.com`,
  `beyondmeat.com`, `apnews.com`, `nytimes.com`, `researchgate.net`, …)
  sets the route per site: tiers to skip (`hard_domain_skip`), the exit
  country, the rendering difficulty, and Common Crawl on synchronous calls.
- Errors for these sites carry `hint` and `alternative_urls`. See
  [Fetching → Known hard domains](/docs/fetching.md#known-hard-domains).

### Fetch: busy domains answer fast; polite per-domain bursts

- A synchronous `POST /v1/fetch` now waits at most 15 s for one of a busy
  domain's polite slots. When the expected wait is longer, it answers at
  once with HTTP 503, `error: "domain_busy"`, `retry_after_s` and
  `Retry-After`. Batch items are unchanged: they wait in the queue.
- New fetch field `domain_concurrency` (up to 4) for a short burst on one
  site, e.g. a company's own pages. It is ignored while the site pushes back.
  See [Limits & politeness](/docs/limits.md#per-domain-politeness).

### Service status: counts, failures and a one-hour fallback window

Each component now reports its counts (`requests` / `attempts`, `failures`)
and the `window_s` they cover; when the 10-minute window is short of the
minimum sample, the last hour is used. See
[Monitoring → Windows and minimum samples](/docs/monitoring.md#windows-and-minimum-samples).

### Latency breakdown per request and in usage

Fetch results gain `timing` (`queue_ms`, `politeness_wait_ms`,
`upstream_ms`) and each attempt `wait_ms` / `upstream_ms`. `GET /v1/usage`
adds `latency_components` (p50/p95 of each component) per kind, and latency
p50/p95 plus component percentiles per tier.

### Caching: redirect targets and short pages

- A fetched page is now also cached under the URL its redirects ended at,
  for example after `http` → `https` or bare host → `www`. A later request
  for that URL is then a hit.
- Pages that are simply short are cached for a day instead of being
  rendered again every time. These are pages where `direct` and the browser
  both gave less text than `min_text_chars` and nothing was blocked.
- New guide: [Caching and reuse across runs](/docs/caching.md). It covers
  the cache classes and TTLs, the cache keys, and how to use `cache_ttl_s`
  so that a recurring job reuses earlier answers.

### Fetch: no premium bytes for domains that refuse every tier

A domain whose pages were refused by the premium routes as well (block page,
challenge, captcha) in 2 fetches in a row now skips `proxy` and
`proxy_render` for 6 h (then 24 h, 3 days): those fetches go to the archive
and alternative sources, or fail fast, without premium bytes (attempt reason
`premium_skipped_domain_refuses`; 5 % re-probe, one premium success ends it).
New fields: `attempts[].premium_bytes`, and on failures `error_detail` (every
attempt in one line) and `premium_routes_tried`. `GET /v1/usage` adds
`by_outcome` (`succeeded` / `failed`: requests, bytes, premium bytes, premium
requests) to each per-kind block.

### Quota: alert e-mail deliverability

`GET /v1/quota` has `alert_email` (`status`: `deliverable`, `not_set`,
`partially_undeliverable` or `undeliverable`; `addresses`, `undeliverable`,
`reason` `bounced` / `complained`, `since`). Alert e-mails go to every
address when several are set; an address that hard-bounces or reports a
mail as spam stops receiving them.

### SEC EDGAR: "not an SEC filer" versus "unknown identifier"

The SEC company endpoints and `/v1/batch/sec` items now say which case a
failed lookup is:

- `unknown_identifier`: neither SEC nor Cormorant knows the ticker, CIK,
  name or ISIN;
- `not_sec_filer`: the company is known — a listing on a non-US exchange
  (`NESN.SW`, `XSWX:NESN`), the company register, GLEIF for an ISIN — but
  is not registered with the SEC under this identifier. The new `hint`
  carries the `company`, its home `listing` and `sec_candidates`: SEC
  filers with its name, such as the ADR of a foreign private issuer (Novo
  Nordisk → `NVO`);
- `not_found` keeps its meaning for filings and documents.

`resolve` takes `?isin=` (batch: `isin` on `resolve`, `submissions`,
`latest`, `facts`).

### Service status: `providers` component and per-engine search health

- `/v1/status` and `/status` list a new component, `providers` ("Third-party
  services"). It is `degraded` with reason `provider_access` while an
  outside service Cormorant relies on refuses access; which one is not
  shown. It has no metrics.
- The `search` component reports `engines_healthy_share` from the search
  engines themselves, a per-engine `engines` list (`ok`, `degraded`, `down`)
  and `instances_healthy_share`.

### What each request is charged, and spend against the caps

- Every billable answer (search, fetch, batch items, `/v1/sec/…`,
  `/v1/companies/…`) has a `billing` block: `charge` at your plan price in
  your invoice currency, `within_allowance`, `billable` and the units per
  plan meter, priced like the invoice.
- `GET /v1/quota` and `GET /v1/usage` have `spend_budget`: per key and per
  customer, `cap` (`null` when none is set), `spent`, `remaining` and
  `by_kind`.
- See [Costs & billing → Capping the cost of a run](/docs/billing.md#capping-the-cost-of-a-run).

### Wikimedia: a policy-compliant User-Agent

Fetches of Wikipedia, Wikidata and the other Wikimedia sites now identify as
`CormorantBot/1.0` with a contact address, per Wikimedia's User-Agent
policy. An allowlisted `user_agent` you send is used instead. See
[Fetching → Wikipedia, Wikidata](/docs/fetching.md#wikipedia-wikidata-and-other-wikimedia-sites).

### Swiss company data: SOGC events and cantonal register excerpts

The `sogc` and `ch_cantonal_excerpt` sources are available: the events
timeline from the Swiss Official Gazette of Commerce, and officers and
signatories (role, signing authority, since/until, origin or nationality,
residence) from the cantonal register excerpts. See
[Switzerland](/docs/companies-switzerland.md).

### Companies: ISIN as an identifier

- `isin:` works wherever an identifier does (`GET /v1/companies/{id}`,
  `search?identifier=`, `resolve`, batch `ref`), a bare ISIN too; the check
  digit is verified (`422 invalid_identifier`).
- `resolve` and batch items take `isin` (also as a CSV column).
- An identifier no source knows is remembered (ISIN 24 h, other schemes
  1 h; `refresh=true` asks again).

### SEC EDGAR usage

SEC EDGAR requests (`/v1/sec/...` and fetches of SEC hosts) are metered as
kind `sec` in the billing categories `sec` and `sec_cached`
(`billing_category`, `GET /v1/usage?category=`, `GET /v1/requests?kind=sec`).
`GET /v1/usage` adds a `sec` block, `keys[]` adds `sec_requests`. For key
limits, SEC requests count as `fetch`.

### Companies: resolve returns the register id

A confident `match` from `POST /v1/companies/resolve` is now added to the
register and comes back with its `cmp_` id, `ref` = that id and
`in_register: true`. `"persist": false` (batch: `persist` option, CSV:
`?persist=false`) keeps the old behaviour.

### SEC EDGAR endpoints

A new guide, [SEC EDGAR via Cormorant](/docs/sec-edgar.md), MCP tools
`cormorant_sec_company`, `cormorant_sec_filings` and `cormorant_sec_document`,
and new `/v1/sec/...` endpoints (see the
[API contract](/docs/api.md#sec-edgar-v1sec)): resolve, submissions, latest annual
report, filing documents, document fetch, XBRL facts and full-text search.
`POST /v1/batch/sec` queues up to 50,000 SEC operations as one job.

### Companies: fast search and resolve

`resolve` answers from the register when it has a confident match; `search`
does the same for a near-exact register hit unless `live=true`. Live sources
run concurrently and are waited for at most 2.5 s; slower ones finish in the
background. `live_sources[]` gains `status` and `ms`.

### SEC EDGAR policy for fetches

Fetches of `www.sec.gov`, `data.sec.gov` and `efts.sec.gov` follow SEC's
fair-access rules automatically: Cormorant sends its own declared identity,
only the direct tier is used, and one shared rate limit applies. New errors
`rate_limited` and `sec_unavailable` (both with `retry_after_s`),
`sec_not_configured` and `redirect_not_allowed`. `/v1/status` lists a `sec`
component.

### Companies: stable legal name

`names.legal` is the official national register's primary entry, whatever
the refresh order. New: `names.alternates` (other sources' forms with
`source`) and `source` on `names.translations` items.

### Company data

New product area: the company register (`company/v1`). See
[Company data](/docs/companies.md).

- `GET /v1/companies/search`, `GET /v1/companies/{id}`, its history and
  sub-resources; `POST /v1/companies/resolve`; batch jobs of kind `company`
  (JSON and CSV).
- `GET /v1/companies/changes` and signed `company.changed` webhooks
  (`POST/GET/DELETE /v1/companies/watches`).
- `GET /v1/companies/sources`, `GET /v1/schemas/company` (JSON Schema).
- Metered as the billing categories `company` and `company_cached`.
- Status: the group `company_data` lists one component per active source.
- MCP: `cormorant_company_search`, `cormorant_company_get`,
  `cormorant_company_resolve`.

### Account admin keys, usage per key

- API keys have a role, fixed at creation and visible in the prefix:
  account admin keys start with `cma_`, standard keys with `cmr_`. Every
  authenticated `/v1` response carries `X-Cormorant-Key-Role`.
- Requests are attributed to the API key that made them;
  `GET /v1/usage?group_by=key` and `GET /v1/account` show usage per key.

### Cache revalidation

New fetch field `revalidated`: `true` when an expired cache entry was
confirmed unchanged by the origin (conditional GET answered `304`). The
cached copy is returned with `cached: true` and no cost.

### Usage, charges and keys

- `GET /v1/usage` with `group_by` (`key`, `project`, `category`, `day`,
  `hour`), filters, totals, failures by error, latency, and the `charge` at
  your plan price; CSV with `Accept: text/csv`.
- A standard key sees only its own usage; an account admin key the whole
  customer account.
- `GET /v1/keys`, `GET /v1/keys/self`, `GET /v1/usage/timeseries`,
  `GET /v1/usage/domains`, `GET /v1/requests` (request ledger) and
  `GET /v1/requests/{request_id}`.
- Per-key limits (monthly spend cap, monthly request quotas, requests per
  second, allowed categories), `GET /v1/quota`, rate-limit headers and usage
  alerts (`POST/GET/DELETE /v1/alerts`, signed webhooks).
- New guide [Usage, spending and quotas](/docs/usage.md) and the MCP tool
  `cormorant_usage`.

### Charges, not provider costs

- **Changed for customer accounts:** `cost_usd` is `null` for keys of a
  customer account; what you are charged is in `GET /v1/usage` and
  `GET /v1/requests` (`charge`).
- New `billing_category` on every search and fetch result.

### Dead domains, archived copies for missing pages

- A host whose name does not exist answers `error: "domain_dead"` at once
  for 6 hours.
- With `allow_archive` (the default), a fetch that ends in `domain_dead` or
  `http_404` / `http_410` returns a recent Wayback snapshot when one exists
  (`served_from_archive_because`).

## 2026-10-19

### Large PDFs in batch fetches

Batch fetch items that leave `max_bytes` unset accept PDFs up to 200 MB.
New response fields `pdf_page_count` and `text_truncated`.

### Service status

`GET /v1/status`, `/v1/status/summary`, `/v1/status/components/{id}` and
`/v1/status/history` — no API key needed — and the status page at
[`/status`](/status); see the [monitoring guide](/docs/monitoring.md).

### Article metadata

Fetch results have a `meta` object for HTML pages: `author`,
`published_at`, `modified_at`, `language`, `canonical_url`, `publisher`,
`site_name`, `description`, `section`, `word_count`.

### Developer portal

Documentation at `/docs`: guides, an interactive API reference and the API
contract; [`/llms.txt`](/llms.txt) and [`/llms-full.txt`](/llms-full.txt);
every guide as Markdown at `/docs/<page>.md`; an
[MCP server](/docs/mcp.md) for AI agents.

### Accounts, plans and invoices

`GET /v1/account`, `GET /v1/account/invoices`,
`GET /v1/account/invoices/{number}` and `…/pdf`. Keys of a suspended account
get `403 customer_suspended`.

### Unavailable domains, honest empty searches

- A domain where every tier failed in 3 consecutive fetches answers
  `error: "domain_unavailable"` for a while, with `retry_after_s`.
- Search results carry `unresponsive_engines`; a search that found nothing
  is `results: []` with `error: null`.

### Rendering tier: retries and difficulty

Busy `proxy_render` answers are retried once (two `attempts`), and the
difficulty that worked for a domain is remembered; such attempts carry
`detail` with the difficulty used (`low`, `medium`, `high`).

### Faster synchronous calls under batch load

`POST /v1/search` and `POST /v1/fetch` run on dedicated queues, so large
batch jobs no longer push synchronous calls into timeouts.

### Per-request fetch controls

New fetch fields: `allow_premium`, `tiers`, `timeout_s`, `country`,
`user_agent` (allowlisted) and `accept_language`. New fetch errors:
`timeout`, `no_tier_allowed`.

### Cache hits honour the request

A cached page is used only when it satisfies the request (`render`,
`min_text_chars`, `max_bytes`, `tiers`).

### Provenance, content kinds and exact bytes

Fetch results carry `redirects`, `headers`, `origin_content_type`,
`content_kind`, `body_url` and `body_kind`; `GET /v1/bodies/{sha256}`
streams the exact bytes to projects that retrieved them.

## 2026-10-18

### v1

Search with SearXNG first and a fallback search API, fetch with tier
escalation (direct, browser, archive, residential proxy, rendering tier),
shared caches, batch jobs with idempotency keys and callbacks, per-project
keys, rate limits, budgets and `GET /v1/usage`.