Monitoring a topic

There are no webhooks. You poll — and the three things that make a polling loop correct are a watermark, a deduplication key, and an interval you can afford.

The loop

import { setTimeout as sleep } from "node:timers/promises";

const BASE = "https://api.unzoi.com";
const KEY = process.env.UNZOI_KEY!;

// The watermark. Persist it — an in-memory one restarts your feed from scratch
// on every deploy, which is how a monitor floods a channel at 3am.
let watermark = loadWatermark() ?? stamp(Date.now() - 3600_000);
const seen = new Set<string>(loadSeenIds());

async function poll() {
  const url = new URL(BASE + "/stories");
  url.searchParams.set("q", "port congestion OR container backlog");
  url.searchParams.set("mode", "hybrid");
  // Overlap deliberately: articles are indexed a little after they are
  // published, so a watermark exactly at the last poll misses the stragglers.
  url.searchParams.set("from", stamp(parse(watermark) - 15 * 60_000));
  url.searchParams.set("limit", "100");

  const response = await fetch(url, { headers: { "x-api-key": KEY } });
  if (response.status === 429) {
    if (response.headers.get("x-quota-remaining") === "0") {
      console.error("quota exhausted — pausing until the month resets");
      return;
    }
    await sleep(1000);
    return;
  }
  if (!response.ok) return;   // transient; the next tick catches up

  const page = await response.json();

  for (const story of page.stories) {
    // Dedup on story_id, not on article id: forty outlets running one wire
    // story is ONE thing that happened, and forty notifications is spam.
    if (seen.has(story.story_id)) continue;
    seen.add(story.story_id);
    notify(story);
  }

  // Only advance on a successful poll. Advancing on failure is how a gap opens.
  watermark = stamp(Date.now());
  saveWatermark(watermark);
  saveSeenIds([...seen].slice(-10_000));
}

for (;;) {
  await poll();
  await sleep(15 * 60_000);
}

const stamp = (ms: number) => new Date(ms).toISOString().replace(/[-:T]/g, "").slice(0, 14);
const parse = (s: string) => Date.parse(
  `${s.slice(0,4)}-${s.slice(4,6)}-${s.slice(6,8)}T${s.slice(8,10)}:${s.slice(10,12)}:${s.slice(12,14)}Z`
);

The four decisions

Overlap your window

Articles reach the index shortly after they are published, and not all at the same lag. A window that starts exactly where the last one ended will miss whatever arrived late. Overlap by 15 minutes and let deduplication absorb the repeats — the cost is nothing, and the alternative is silent gaps.

Deduplicate on story_id

If you dedup on article id, one wire story becomes forty notifications. Dedup on story_id and it becomes one, with an outlet count attached that tells the reader how big it is. This is the single decision that separates a useful monitor from an unusable one.

Bound the set. Ten thousand recent ids is ample; an unbounded set is a memory leak with a slow fuse.

Advance the watermark only on success

A failed poll that still advances the watermark loses everything published in that interval, permanently and silently. Advance only after you have processed a response.

Pick an interval you can afford

Per query. Ten monitored topics at 15 minutes is ~28,800 requests a month, which is most of a Build plan before any user has searched for anything. Count them up front.

What to notify on

Not everything new is worth telling someone about. The clustered response gives you the material to decide:

function worthNotifying(story) {
  // Broad pickup: many independent outlets, not one syndicated wire.
  if (story.outlets >= 8) return true;
  // Or narrow but from a publication you care about.
  if (story.sources.some((s) => WATCHED.has(s))) return true;
  return false;
}

outlets is doing the work: it is the difference between "a story broke" and "a press release was republished". Evaluating coverage goes further.

Backfilling a new monitor

Starting a monitor with a from of a month ago will return a month of stories in one burst. Walk it in windows instead, so each request is bounded and the first notification is not a flood:

for day in $(seq 30 -1 1); do
  from=$(date -u -d "$day days ago" +%Y%m%d000000)
  to=$(date -u -d "$((day-1)) days ago" +%Y%m%d000000)
  curl -s -G "https://api.unzoi.com/stories" -H "x-api-key: $UNZOI_KEY" \
    --data-urlencode "q=port congestion" \
    --data-urlencode "from=$from" --data-urlencode "to=$to" \
    --data-urlencode "limit=100" | jq -c '.stories[]'
  sleep 0.5
done

Check your archive depth first — on the free tier that loop silently stops finding anything past 30 days back, and from_clamped in each response is what tells you so.