Caching and reuse across runs

Cormorant keeps a shared cache of search answers and fetched pages. A hit answers in milliseconds, is billed at the cache price (cached: true, billing category search_cached / fetch_cached) and is the same for every project. This guide lists what is cached for how long, and how to get hits when you run the same job again.

Cache classes#

The same defaults apply to every customer. The operator can change them (SEARCH_CACHE_TTL_S, HTML_CACHE_TTL_S, PDF_CACHE_TTL_S, SHORT_PAGE_CACHE_TTL_S).

Not cached:

• errors
• hard-paywall teasers (so open copies are looked for again next time)
• thin pages from a fetch that was blocked
• archived copies standing in for a 404 or a dead domain
• alternative-source copies (tier: "source")
• search answers where Brave failed after SearXNG returned too little

An expired HTML or PDF entry with an ETag or Last-Modified is revalidated with a conditional request. If the origin answers 304 Not Modified, you get the cached copy (cached: true, revalidated: true), billed as a cache hit, and it is fresh again.

How URLs are normalised#

The cache key of a fetch is the normalised URL:

• scheme and host are lower-cased, and the default port and the fragment are dropped
• tracking parameters are removed (utm_*, gclid, fbclid, mc_*, pk_*, …)
• the remaining query parameters are sorted, and percent-encoding is normalised

http://Example.com:80/a?b=2&a=1&utm_source=x#top and http://example.com/a?a=1&b=2 are the same entry.

http and https, www. and the bare host, and a trailing / are different URLs. When a fetch is redirected, the page is also cached under the URL the redirects ended at. A later request for https://www.site.com/ then hits the entry that a request for http://site.com created. For the most hits, send the URLs you got back as final_url.

cache_ttl_s: how old an answer you accept#

• Unset: the class's freshness above.
• cache_ttl_s: 0: bypass the cache and fetch or search again. The new answer replaces the entry.
• Any other value is the maximum age you accept, measured from when the answer was retrieved. It can be shorter than the default (3600 for "at most an hour old") or longer. With cache_ttl_s: 2592000 (30 days), a re-run a month later still reuses last month's searches and pages that are still kept.

Cache hits must still satisfy the request: render, tiers, accept_pdf, max_bytes and min_text_chars all apply. A page with less text than your min_text_chars is fetched again, unless it was cached as a short page.

Reusing the cache across runs#

For a recurring enrichment run (the same companies every week or month):

1. Send the same request strings. Use the same query text and the same language, country, max_results and freshness for searches. Use the same URLs (preferably the final_url of the last run) for fetches. Avoid volatile parts in queries, such as today's date or a run id. Put those in project_ref instead, which is not part of the key.
2. Accept the age you can live with. Set cache_ttl_s to the interval between runs (e.g. 604800 for weekly runs, 2592000 for monthly). Every item that is still in the cache then answers instantly, as a cache hit.
3. Force freshness only where needed. Use cache_ttl_s: 0 on the few URLs that must be live (e.g. a news page), not on the whole job.
4. Check the hit rate. Results and GET /v1/requests carry cached, and GET /v1/usage shows cache_hit_rate per kind and the search_cached / fetch_cached categories.

A first run naturally has few or no hits. Every query and URL is new, unless another project asked the same thing within the TTL, because the cache is shared.