News-Monitoring Agents Built on Search APIs
Polling news-flavored search endpoints to alert on breaking coverage of a topic or brand.
How it works
A news-monitoring agent polls a news-scoped search endpoint at a fixed interval for a set of tracked keywords, and its core logic is less about the search call itself and more about deduplication — deciding whether a newly returned article is genuinely new coverage or a re-crawl of something already surfaced in a prior poll. Most implementations keep a rolling set of seen URLs (or content hashes, to catch syndicated re-publishing under different URLs) to filter each new batch down to what's actually novel.
Example
A polling job runs every 15 minutes with query "[brand name]" news, and for each returned article checks its URL and a normalized title against a seen-set stored from the last 48 hours of polls; only unseen articles get pushed to a Slack alert channel.
Pitfalls
- Polling too infrequently misses the value entirely for time-sensitive brand monitoring; polling too frequently against a general web search API (rather than a purpose-built news feed) burns quota without materially improving freshness, since search indexing itself has a lag floor.
- Syndicated wire stories republished verbatim across dozens of outlets can flood an alert feed with what looks like many hits but is really one story, unless deduplication checks content similarity rather than just URL.
- A seen-set that never expires grows unbounded and slows down every poll cycle over time; it needs a rolling window, not permanent storage.