Content Aggregation and Curation
Building topic-based content feeds by querying search APIs on a schedule.
The problem
A curated topic feed (a newsletter, a niche content aggregator) needs to surface genuinely new, relevant content on a recurring basis, and manually scanning dozens of sources for what's new in a narrow topic every day is exactly the kind of repetitive scanning work that doesn't scale with a small editorial team.
How the workflow is built
Topic-specific queries run on a fixed schedule (hourly to daily depending on how fast the topic moves), with results deduplicated against previously surfaced content and ranked by a mix of recency and source quality before reaching an editorial review step or an automated feed.
Example queries
"[niche topic]" news this weeknew blog post "[topic]" site:substack.com"[industry]" report OR analysis published recently
Pitfalls to watch for
- Search-driven curation naturally favors content that's already well-optimized for search visibility, which skews away from smaller independent voices in a topic even when their content is genuinely higher quality — this is worth correcting for deliberately if editorial diversity matters to the product.
- Recency filtering on search APIs isn't always precise (a 'published this week' filter can include content that was merely re-crawled or lightly edited this week), so a genuine publish-date check against the source page is worth the extra step.
- Topic drift is a slow, easy-to-miss failure mode — as language around a niche topic evolves, a static query set gradually surfaces less relevant content without an obvious break, so query sets need periodic review, not a set-and-forget approach.