# Caching and reuse across runs

Cormorant keeps a **shared cache** of search answers and fetched pages. A hit
answers in milliseconds, is billed at the cache price (`cached: true`, billing
category `search_cached` / `fetch_cached`) and is the same for every project. This
guide lists what is cached for how long, and how to get hits when you run
the same job again.

## Cache classes

| What | Fresh for (default) | Kept after that | Key |
|---|---|---|---|
| Search answers (`/v1/search`, batch search) | 7 days | not deleted: older answers are reused when your `cache_ttl_s` accepts them | normalised query (whitespace collapsed, lower-case) + `language` + `country` + `max_results` + `freshness` |
| HTML pages | 7 days | 30 days, for revalidation and longer `cache_ttl_s` | normalised URL (see below) |
| PDFs | 30 days | 30 days | normalised URL |
| Short pages: `direct` and the browser both gave less text than `min_text_chars`, and nothing blocked | 1 day | 30 days | normalised URL |
| SEC EDGAR | archives (filings, documents) 365 days; metadata 12 h; full-text search 1 h | 30 days | normalised URL |

The same defaults apply to every customer. The operator can change them
(`SEARCH_CACHE_TTL_S`, `HTML_CACHE_TTL_S`, `PDF_CACHE_TTL_S`,
`SHORT_PAGE_CACHE_TTL_S`).

**Not cached:**

- errors
- hard-paywall teasers (so open copies are looked for again next time)
- thin pages from a fetch that was blocked
- archived copies standing in for a 404 or a dead domain
- alternative-source copies (`tier: "source"`)
- search answers where Brave failed after SearXNG returned too little

An **expired** HTML or PDF entry with an `ETag` or `Last-Modified` is
revalidated with a conditional request. If the origin answers `304 Not
Modified`, you get the cached copy (`cached: true`, `revalidated: true`), billed
as a cache hit, and it is fresh again.

## How URLs are normalised

The cache key of a fetch is the normalised URL:

- scheme and host are lower-cased, and the default port and the fragment are dropped
- tracking parameters are removed (`utm_*`, `gclid`, `fbclid`, `mc_*`, `pk_*`, …)
- the remaining query parameters are sorted, and percent-encoding is normalised

`http://Example.com:80/a?b=2&a=1&utm_source=x#top` and
`http://example.com/a?a=1&b=2` are the same entry.

`http` and `https`, `www.` and the bare host, and a trailing `/` are
**different** URLs. When a fetch is redirected, the page is also cached under
the URL the redirects ended at. A later request for `https://www.site.com/`
then hits the entry that a request for `http://site.com` created. For the
most hits, send the URLs you got back as `final_url`.

## `cache_ttl_s`: how old an answer you accept

- Unset: the class's freshness above.
- `cache_ttl_s: 0`: bypass the cache and fetch or search again. The new
  answer replaces the entry.
- Any other value is the **maximum age you accept**, measured from when the
  answer was retrieved. It can be shorter than the default (`3600` for "at
  most an hour old") or **longer**. With `cache_ttl_s: 2592000` (30 days), a
  re-run a month later still reuses last month's searches and pages that are
  still kept.

Cache hits must still satisfy the request: `render`, `tiers`, `accept_pdf`,
`max_bytes` and `min_text_chars` all apply. A page with less text than your
`min_text_chars` is fetched again, unless it was cached as a short page.

## Reusing the cache across runs

For a recurring enrichment run (the same companies every week or month):

1. **Send the same request strings.** Use the same query text and the same
   `language`, `country`, `max_results` and `freshness` for searches. Use the
   same URLs (preferably the `final_url` of the last run) for fetches.
   Avoid volatile parts in queries, such as today's date or a run id. Put
   those in `project_ref` instead, which is not part of the key.
2. **Accept the age you can live with.** Set `cache_ttl_s` to the interval
   between runs (e.g. `604800` for weekly runs, `2592000` for monthly). Every
   item that is still in the cache then answers instantly, as a cache hit.
3. **Force freshness only where needed.** Use `cache_ttl_s: 0` on the few
   URLs that must be live (e.g. a news page), not on the whole job.
4. **Check the hit rate.** Results and `GET /v1/requests` carry `cached`, and
   `GET /v1/usage` shows `cache_hit_rate` per kind and the `search_cached` /
   `fetch_cached` categories.

A first run naturally has few or no hits. Every query and URL is new, unless
another project asked the same thing within the TTL, because the cache is
shared.