Combining Search APIs with Vector Databases
A hybrid retrieval pattern: static embeddings for known documents, live search for everything else.
How it works
A hybrid system routes a query to one or both retrieval sources depending on what the question needs: a vector database handles fast, cheap retrieval over a known, fixed document set (internal docs, a curated knowledge base), while a live search API covers anything outside that set or anything time-sensitive that the vector index — built at some point in the past — can't reflect. The routing decision (or a merge of both result sets) is the core design challenge.
Example
A support bot first checks its vector index of internal documentation for an answer; if the top vector match's similarity score falls below a confidence threshold, or the question contains a time-sensitive cue ('latest', 'current', 'this week'), it falls back to a live search call scoped to the company's own domain rather than answering from a possibly-outdated indexed doc.
Pitfalls
- Merging vector-store results and live search results into one ranked list without normalizing their very different scoring scales (cosine similarity versus search rank position) produces a nonsensical combined ranking.
- A routing rule based only on keyword triggers ('latest', 'current') misses time-sensitive questions phrased without those words, so relying solely on keyword-based routing leaves real gaps.
- Running both retrieval paths for every query to be safe doubles latency and cost for the majority of questions that only needed one.