# Monitoring Cormorant's status

Cormorant publishes the health of its subsystems — API, search, fetch,
browser rendering, archive, premium routes, SEC EDGAR, batch jobs, storage and
each company data source — so you can watch it
from your own monitoring and tell "our pipeline is broken" from "Cormorant is
having a problem". People can read the same information at
[`/status`](/status); machines read `GET /v1/status`.

No API key is needed, and the answer is shared by everyone and refreshed at
most every 15 seconds, so polling it costs you nothing against your project's
rate limit.

## Quick start

```sh
# Tiny body for uptime monitors
curl -s https://api.cormorant.ch/v1/status/summary
# {"status":"operational","description":"All systems operational","updated_at":"…"}

# Everything: components, reasons, metrics, incidents, maintenance
curl -s https://api.cormorant.ch/v1/status
```

## What to poll, and how often

- **Uptime monitors** (Uptime Kuma, Better Stack, Pingdom, a cron job):
  `GET /v1/status/summary` and a keyword check for `"status":"operational"`.
- **Your own dashboards and alerting**: `GET /v1/status`, then look at the
  components your integration depends on. The component ids are `api`,
  `search`, `fetch`, `browser_rendering`, `archive`, `premium_routes`, `sec`
  (SEC EDGAR: `/v1/sec/…` and fetches of SEC hosts; listed while SEC is
  configured), `company_profiles` (extended company profiles: profiles,
  discovery and company networks; listed while a premium route is configured),
  `batch_jobs`, `storage`, and one per active company data source
  in the group `company_data` (currently `company_gleif`, `company_uid_register`,
  `company_zefix_rest`, `company_lindas_zefix`, `company_sogc`,
  `company_ch_cantonal_excerpt`). A degraded company source only means fresher
  data from it may be delayed — the company register still answers.
- **Interval**: every **30–60 seconds**. Polling more often only returns the
  same snapshot (`Cache-Control: public, max-age=15`); send `If-None-Match`
  with the last `ETag` to get a `304` when nothing changed. More than 120
  requests per minute from one IP address get `429`.

## Why the status endpoints always answer 200

The HTTP code says whether *Cormorant answered*; the body says *how it is
doing*. A status endpoint that returned `503` during an outage could not be
told apart from a broken proxy, a DNS problem or your own network. So:

| You see | It means | Alert |
|---|---|---|
| `200` and `status: operational` | all good | no |
| `200` and `status: degraded` | slower or more failures than usual; requests mostly work | warning |
| `200` and `status: partial_outage` / `major_outage` | part or all of the service is failing | page someone if you depend on it |
| `200` and `status: maintenance` | announced maintenance in progress (see `maintenance`) | no, or informational |
| timeout, connection error, any non-200 | Cormorant unreachable (or your network) | yes, after 2–3 consecutive checks |

## Recommended alert rules

1. **Unreachable**: 3 consecutive failed checks (timeout 10 s) → critical.
2. **Outage of what you use**: a component you depend on is
   `partial_outage` or `major_outage` in 2 consecutive checks → critical.
3. **Degradation**: a component you use is `degraded` for 10 minutes →
   warning. Short blips are normal: status is computed from the last 10
   minutes of traffic (or the last hour, see below).
4. **Incidents**: a new entry in `incidents` with `status` other than
   `resolved` → notify your team with its `title` and latest `updates`.
5. Do **not** alert on a component that is missing: components not part of
   the deployment are not listed. Use the stable `id`, never `name`.

`reasons` says why a component is not operational (`no_workers`,
`queue_lag`, `low_success_rate`, …) and `metrics` gives the numbers behind
it, e.g. the fetch success rate or the age of the oldest waiting request.

## Windows and minimum samples

Rates need enough traffic to mean anything. Each component's `metrics`
carries the counts behind its rate and the window they cover:

| Component | Counts | Minimum sample |
|---|---|---|
| `search`, `fetch`, `sec` | `requests`, `failures` (uncached; the site's or caller's definitive answers excluded) | 20 requests |
| `browser_rendering`, `archive` | `attempts`, `failures` (infrastructure failures only) | 10 attempts |
| `premium_routes` | `proxy_attempts`/`proxy_failures`, `rendering_attempts`/`rendering_failures` | 10 attempts per tier |
| `company_profiles` | `requests` (all operations; the rates cover the uncached ones), always the last 600 s | 10 uncached operations |

- The default window is the last **600 s** (`window_s` at the top level and
  in each component's `metrics`).
- When a component has fewer than its minimum sample there — moderate or
  bursty load — it reports over the last **3600 s** instead, with
  `metrics.window_s: 3600` (`proxy_window_s`/`rendering_window_s` for the
  premium routes). Its status is judged on that window too.
- `success_rate: null` means even the hour is short of the minimum: no
  evidence either way, and the rate never changes the status then.
- At most the 20,000 most recent requests are read per window.

## Example: check from Python

```python
import httpx

r = httpx.get("https://api.cormorant.ch/v1/status", timeout=10)
r.raise_for_status()  # non-200: Cormorant unreachable
status = r.json()
mine = {c["id"]: c for c in status["components"] if c["id"] in ("search", "fetch")}
for c in mine.values():
    if c["status"] in ("partial_outage", "major_outage"):
        print(f"ALERT {c['name']}: {c['status']} ({', '.join(c['reasons'])})")
for incident in status["incidents"]:
    if incident["status"] in ("investigating", "identified", "monitoring"):
        print(f"Incident: {incident['title']} — {incident['updates'][0]['message']}")
```

## Incidents, maintenance and history

The operator posts incidents (with an impact and a timeline of updates) and
announces maintenance windows; both appear in `GET /v1/status` and on the
status page. During an announced window the affected components show
`maintenance` rather than an outage. `GET /v1/status/history?hours=24` gives
the worst status per 5 minutes for the last 24 hours (hourly up to 7 days)
and the uptime per component — useful for SLA reports.

## Reference

Public status of the service for your monitoring, and the page at
[`/status`](/status). **No authentication** (an `Authorization` header is
accepted and ignored), CORS-enabled, refreshed at most every 15 s and shared
by all callers (`Cache-Control: public, max-age=15`, `ETag` /
`If-None-Match` → `304`). Limited to 120 requests per minute per client IP
(`429` with `Retry-After`). The status never contains host names, addresses,
instance URLs, project names, keys or internal error strings — only component
ids, statuses, reason codes and aggregate numbers.

The status endpoints answer **`200` whatever the service status**: the
status is in the body. Alert on the body (`status` is not `operational`), and
treat a timeout, a connection error or any other HTTP code as "Cormorant is
unreachable" — a status endpoint that answered `503` during an outage could
not be told apart from a broken proxy or network in between.

| Endpoint | Returns |
|---|---|
| `GET /v1/status` | overall `status`, every component with `reasons` and `metrics`, incidents and maintenance |
| `GET /v1/status/summary` | only `status`, `description`, `updated_at` (for uptime monitors) |
| `GET /v1/status/components/{id}` | one component plus the incidents and maintenance naming it (`404` for an id not in this deployment) |
| `GET /v1/status/history?hours=24` | worst status per 5 minutes (per hour beyond 24 h, up to 168 h) and uptime per component; overall as `_overall` |

```json
{
  "status": "degraded",
  "description": "Degraded performance",
  "updated_at": "2026-10-19T09:41:15+00:00",
  "window_s": 600,
  "components": [
    {"id": "api", "name": "API", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": null},
    {"id": "fetch", "name": "Fetch", "description": "…", "status": "degraded", "reasons": ["low_success_rate"], "metrics": {}, "group": null},
    {"id": "company_gleif", "name": "Company data: GLEIF", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": "company_data"}
  ],
  "incidents": [
    {"id": "inc_3f9a1c0b7e", "title": "Elevated fetch failures", "status": "identified",
     "impact": "degraded", "components": ["fetch"], "created_at": "…", "updated_at": "…",
     "resolved_at": null, "updates": [{"at": "…", "status": "identified", "message": "…"}]}
  ],
  "maintenance": []
}
```

**Statuses**, best to worst: `operational`, `maintenance`, `degraded`,
`partial_outage`, `major_outage`. The overall `status` is the worst component
status, except that a major outage of a component other than `api`,
`search`, `fetch` or `storage` counts as a partial outage, and it is
`maintenance` when the only components not operational are under announced
maintenance. An unresolved incident shows its components at least at its
`impact`. Components not part of the deployment are not listed (`archive`
when archived copies are disabled, `premium_routes` when no premium route is
configured, `sec` while no SEC identity is configured, `company_profiles` while no
premium route is configured) — do not alert on a missing component.

Counts and rates cover the last `window_s` = 600 s. A component with fewer
than its minimum sample there (20 uncached requests; 10 attempts for a tier)
reports `requests`/`attempts`, `failures` and `success_rate` over the **last
hour** instead; `metrics.window_s` (`<tier>_window_s` in `premium_routes`) says
which window was used, and the status is judged on that window. `null` rates
mean even the hour has too few samples.

| Component | Metrics (rates 0–1; `failures` and `window_s` with every rate) | Not operational when |
|---|---|---|
| `api` | — | database or cache unreachable (major) |
| `search` | `success_rate`, `requests`, `engines_healthy_share`, `engines`, `instances_healthy_share`, `fallback_available`, `fallback_share`, `fallback_success_rate`, `workers_alive`, `queue_lag_s` | success rate of uncached searches < 0.95 / 0.80 / 0.50 (degraded / partial / major); healthy search instances < 50 % (degraded) or none (partial with the fallback search service, major without); healthy engines < 34 % (degraded); no worker; queue lag. `engines_healthy_share`: share of search engines (per instance) whose latest answer in the last 15 minutes returned results (the instance share until searches were observed); `engines`: `[{engine, status}]` with `ok` (healthy everywhere), `degraded` (on some instances) or `down` (rate limited, CAPTCHA or suspended everywhere) |
| `fetch` | `success_rate`, `requests`, `latency_p95_ms`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | success rate of uncached fetches < 0.85 / 0.60 / 0.30 (4xx answers of the site, robots.txt, invalid or blocked URLs, too large, PDF not accepted and deferred are not failures); p95 > 60 s (degraded); no worker (major); queue lag |
| `browser_rendering` | `success_rate`, `attempts`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | browser attempts failing for infrastructure reasons: success < 0.80 / 0.50 / 0.10; no browser worker (major) |
| `archive` | `success_rate`, `attempts` | archive unreachable or erroring: success < 0.80 / 0.50 / 0.10 |
| `premium_routes` | `proxy_success_rate`, `proxy_attempts`, `rendering_success_rate`, `rendering_attempts` (configured tiers only) | provider errors: success < 0.80 / 0.50 / 0.10 |
| `sec` | `success_rate`, `requests`, `throttle_wait_p95_ms` (p95 wait for a slot in the cluster-wide SEC rate limiter), `backoff_active`, `rate_limit_rps` | SEC backing off after 403/429/503 (degraded, `sec_backoff`); success rate of uncached SEC requests < 0.85 / 0.60 / 0.30 (4xx, invalid identifiers and not found are not failures); p95 wait > 5 s (degraded, `throttled`) |
| `providers` | — | an outside service Cormorant relies on (search fallback, premium routes, alternative sources, notifications) currently refuses our access (degraded, `provider_access`); which one is not shown — the operator is alerted |
| `company_profiles` | `success_rate`, `requests`, `latency_p95_ms` (uncached extended company profile operations), `cache_hit_rate`, `refused_share` (operations the source refused: `blocked`, `rate_limited`), `rendering_share` (share served by the rendering service rather than the residential proxy), `throttle_wait_p95_ms` (p95 wait for a slot in the shared rate limit), `backoff_active`, `proxy_skipped` (the residential proxy is being skipped after a recent refusal); always over the last 600 s | success rate of uncached operations < 0.85 / 0.60 / 0.30 (not found, invalid identifiers, robots.txt, premium routes not allowed and exhausted budgets are not failures; `null` below 10); refused share ≥ 0.20 (degraded, `upstream_refusing`); the site throttle backing off after 429 (degraded, `site_backoff`); p95 wait > 10 s (degraded, `throttled`) |
| `batch_jobs` | `scheduler_last_run_s`, `workers_alive`, `running_jobs`, `oldest_waiting_job_s`, `queue_backlog` | scheduler last ran > 30 s (degraded) / > 120 s ago (partial); no batch worker (major); a job waiting to start > 15 min (degraded) / > 60 min (partial) |
| `storage` | `database_ok`, `cache_ok`, `object_storage_ok` (`null`: not configured) | database or cache unreachable (major); object storage unreachable (degraded) |
| `company_<source>` (group `company_data`, one per active company data source, e.g. `company_gleif`) | `success_rate`, `requests` (last hour of calls to the source; `null` below 10), `latency_p95_ms`, `parse_failure_rate`, `canary_ok`, `backoff_active`, `schema_drift` (fields that used to be filled and now come back empty), `change_feed_lag_s` | success < 0.85 / 0.50 / 0.20; canaries (known companies checked every 15 minutes) failing (partial); backoff after repeated failures, schema drift, parse failures > 10 % (degraded). Counts at most as `degraded` for the overall status |

Queue lag (`queue_lag_s`, `sync_queue_lag_s`: how long the oldest synchronous
request has waited for a worker) above 15 / 60 / 120 s is degraded / partial
/ major. Rates computed on fewer than 20 requests (10 tier attempts) are
reported but do not change a status. New metric keys may be added.

**Reason codes**: `no_workers`, `queue_lag`, `low_success_rate`,
`high_latency`, `engines_reduced`, `engines_unavailable`, `tier_errors`,
`scheduler_stalled`, `jobs_waiting`, `database_unreachable`,
`cache_unreachable`, `object_storage_unreachable`, `sec_backoff`,
`throttled`, `provider_access`, `upstream_refusing`, `site_backoff`, `incident`, `maintenance`; company data sources also `canary_failed`, `source_backoff`,
`schema_drift`, `parse_failures`.

**Incidents and maintenance** are posted by the operator. `incidents` lists
unresolved incidents (`investigating`, `identified`, `monitoring`) and those
`resolved` within 7 days, newest first, each with its timeline (`updates`,
newest first). `maintenance` lists `scheduled` and `in_progress` windows and
those `completed` within 7 days, soonest first; an empty `components` list
means the whole service.

**History**: `GET /v1/status/history` returns, per component, `statuses`
(oldest first; `null` where nothing was recorded) and `uptime` over the
period, where a `partial_outage` bucket counts as 30 % downtime and a
`major_outage` bucket as 100 %. Kept 7 days.