Real Estate Listings Aggregation
Pulling and normalizing property listings from multiple sources with search APIs.
The problem
No single MLS feed covers every market, and consumer-facing sites like Zillow, Redfin, and Realtor.com each syndicate a slightly different, slightly delayed subset of the same underlying listings. A product that wants comprehensive coverage of a metro area has to reconcile several partially overlapping, inconsistently formatted sources rather than trust any one of them.
How the workflow is built
A search API is queried per market and property type — city, bedroom count, price ceiling, and a `site:` filter rotated across the major listing portals — and the resulting URLs are deduplicated against an address-normalization step before a page fetch extracts price, square footage, and listing age. Because listings churn daily, the crawl cadence is usually tied to market activity: hourly in hot markets during peak season, daily elsewhere.
Example queries
"3 bedroom homes for sale Austin TX under $500k" site:zillow.comnew listings site:redfin.com "Denver CO""for sale by owner" "Charlotte NC" -site:zillow.com
Pitfalls to watch for
- MLS syndication lag means a property already under contract can still show as active on a secondary portal for a day or more, producing stale results that frustrate end users.
- The same physical property routinely appears as multiple 'distinct' listings across portals with slightly different addresses or unit formatting, so address normalization has to be a real step, not an afterthought.
- Portals differ in how aggressively they block automated access, so relying on a single retrieval path for all sources is fragile — a managed search API smooths this but doesn't eliminate the need for fallback sources.