Stories for context windows

News is repetitive by construction. If you put raw search results into a model's context, you are mostly paying to read the same article forty times.

The problem

A wire service publishes a story. Within an hour it is on dozens of sites, near enough verbatim. A search for it returns all of them, ranked by relevance — so the top ten results are ten copies of one article, and a model reading them learns one thing and spends ten results' worth of context doing it.

curl -s -G "https://api.unzoi.com/search" -H "x-api-key: $UNZOI_KEY" \
  --data-urlencode "q=semiconductor export controls" --data-urlencode "limit=10" \
  | jq -r '.results[].title' | sort | uniq -c | sort -rn

Run that on almost any news query and the shape is the same: a handful of headlines, repeated.

The fix

curl -s -G "https://api.unzoi.com/stories" -H "x-api-key: $UNZOI_KEY" \
  --data-urlencode "q=semiconductor export controls" --data-urlencode "limit=10"

Same query, same filters, same ranking modes — one row per event. Ten rows is now ten distinct things that happened, and each carries how much attention it got:

{
  "story_id": "s-9f2c",
  "count": 23,        // articles in the cluster
  "outlets": 19,      // distinct publishers
  "title": "New export controls tighten chip supply",
  "sources": ["reuters.com", "bbc.co.uk", "ft.com"]
}

That is strictly more information in fewer tokens. Plain search discards the counts entirely — you cannot tell a story everyone covered from one nobody did.

Reading the counts

count and outlets mean different things and the gap between them is the interesting part.

An agent that surfaces outlets to its user is giving them a calibration they cannot otherwise get. Evaluating coverage takes this further.

The two-step

Cluster to decide what happened; expand to see how it was covered.

# 1. What happened.
curl -s -G "https://api.unzoi.com/stories" -H "x-api-key: $UNZOI_KEY" \
  --data-urlencode "q=grid outage" --data-urlencode "limit=5"

# 2. Every article in one of those clusters.
curl -s -G "https://api.unzoi.com/search" -H "x-api-key: $UNZOI_KEY" \
  --data-urlencode "story_id=s-9f2c" --data-urlencode "limit=50"

Two requests instead of one, and far less context than a single unclustered search would have cost — because the expansion is deliberate rather than automatic.

For an agent

An agent will pick this itself if the question is phrased as being about an event, because the tool description says so. You can make it more reliable with one line of system prompt:

Use list_stories when the question is about what happened.
Use search_news only when the question is about coverage itself —
who reported it, how it was framed, how many outlets picked it up.

When not to use it

How the clustering works

It happens during ingestion, not at query time, so the same event always carries the same story_id whichever query surfaced it — and clustering never costs you latency. Two passes: URL canonicalisation and near-duplicate detection catch syndication and light rewrites, then a pass over the entities and topics in the text catches independent coverage that shares no wording.

It is not perfect. Two genuinely distinct events with near-identical framing can merge, and a story that develops over days may split. Treat story_id as a strong hint, not a primary key.