Monitoring API vs Scheduled Scraping: When to Use Each for Property Data

Somebody on your team has a cron job that re-scrapes 4,000 listing pages every night so that, on a good night, it can find the eleven that changed. The other 3,989 fetches exist to prove nothing happened. That is what scheduled scraping turns into once the question shifts from "what's on this page?" to "what changed since yesterday?", and it's the reason people start searching for a monitoring API instead.

This guide is the comparison we wish existed when we were on the other side of that cron job: what a monitoring or change-detection API actually does, how polling, webhooks, and scheduled crawls differ in cost and latency, a decision matrix for the property-data jobs people actually run (price drops, new listings, status changes, comp refreshes), and a working script that polls a RealtyAPI search-by-URL endpoint and diffs the results. If you're still deciding whether to scrape at all, start with web scraping vs API; this post assumes you've picked a data source and now need to know when it changes.

What a monitoring API actually is

Strip the marketing off and a monitoring API does three things a scraper doesn't: it keeps a snapshot of what a source looked like last time, it computes a diff against the current state, and it delivers that diff somewhere useful (a webhook, a queue, an email). Scraping answers "what does this say right now?" Monitoring answers "what's different, and should anyone care?"

That distinction matters because the expensive part of change detection was never the fetch. It's the state. Monitoring is a diff problem wearing a scraping costume. The moment you store yesterday's result so you can compare it with today's, you have a database, a schema that will drift when the source changes shape, a deduplication rule, and a decision about what counts as "changed." A price going from $749,900 to $749,900 with a new photo is not a price drop. A listing disappearing from a search might be a sale, a withdrawal, or a pagination hiccup. Every one of those judgments lives in the diff, and a monitoring API is just someone else having made those judgments for you, with a cadence and a delivery channel bolted on.

There are two families of product under the "monitoring API" label, and they're not interchangeable:

• Page monitors (the changedetection.io model, plus hosted services like PageCrawl and context.dev): you hand them a URL, a selector or a natural-language description of what to watch, a cadence, and a webhook. They render the page, diff the text, and ping you. Great for "tell me when this one page changes," weaker when "the page" is a search result with 350 listings and you need structured fields.

• Structured data APIs you poll (this is where RealtyAPI sits): you ask for the current state of a search or a listing as JSON, and you own the snapshot and the diff. Less turnkey, far more precise, because you're comparing priceInfo.amount to priceInfo.amount instead of comparing two blobs of rendered HTML and hoping the change you care about is the one that moved.

To be clear about what RealtyAPI is not: we don't run a change-detection service, and we don't send webhooks when a listing moves. We return the live state of public listing pages as structured JSON, per request. The monitor in the code section below is about sixty lines you run yourself. We'd rather say that up front than have you find out from the docs.

Polling vs webhooks vs scheduled crawls

Three patterns cover nearly every change-detection setup in real estate. The differences are about who initiates the check and how much work happens per check.

Scheduled crawls

A job wakes up, fetches every page in a list, parses each one, and writes the results down. It's the pattern everyone builds first because it needs no state: if you re-scrape everything, you never have to reason about what changed. The bill arrives later. Cost scales with the number of pages, not the number of changes, and the parser breaks whenever the source changes its markup. The scraping-vs-monitoring comparisons from the page-monitor vendors put the build at weeks and the upkeep at hours per month, and from experience that's not an exaggeration once anti-bot measures enter the picture. Ask me how I know.

Polling a structured API

Same loop, but each check is one request that returns JSON for an entire search, so the per-check cost drops from "render and parse 350 pages" to "fetch one payload and compare IDs." Latency equals your poll interval: poll every 15 minutes and you'll learn about a change within 15 minutes. The trade-off is that you pay for checks that find nothing, so the interval is a budget decision, and we'll put real numbers on it below.

Webhooks from a monitoring service

The service does the polling, and you only hear about changes. Latency depends on the service's cadence rather than yours, and cost is per monitored URL rather than per check. The hidden catch: somebody still polls. Listing portals don't push events to third parties, so "webhook" in this market means a vendor is running the scheduled check on your behalf and charging for the convenience. That's a perfectly good deal when you're watching a handful of pages and don't want to run infrastructure. It's a bad deal when you need structured fields from each result, because you'll get "something on this page changed" and then have to fetch the structured data anyway.

So if webhooks are the cleaner architecture, why does anyone still poll? Because the source has to emit the event, and in property data it doesn't. Every webhook you receive about a listing started life as somebody's poll. The question is only whether you want to own that poll (and its precision) or rent it (and its simplicity). The general trade-offs between these patterns are covered well in this system-design explainer:

Decision matrix: which pattern fits which property-data job

The right answer depends on how often the thing changes, how fast you need to know, and how many things you're watching. Here's how the common jobs shake out.

Use case	How often it changes	Latency you actually need	Recommended pattern	What to diff
Price drops on saved listings	A few times over a listing's life	Minutes to an hour; buyers act same-day	Poll a search or details endpoint every 15–60 min; alert on a threshold	Price per listing ID, with a minimum drop (e.g. 1%) to filter noise
New listings matching a search	Daily in a hot zip, weekly in a quiet one	Under an hour for investors competing on speed; daily digest for everyone else	Poll the search URL; treat unseen listing IDs as "new"	Set of listing IDs vs last snapshot
Status changes (active → pending → sold → withdrawn)	Once or twice per listing	Hours is fine; the transaction takes weeks	Poll the details endpoint for listings you hold; or infer from disappearance in search plus a confirm fetch	Status field; presence/absence in search results
Comp refresh for a valuation model	Sold comps accumulate weekly	Daily or weekly	Scheduled poll of a "sold" search URL; no real-time need	New sold IDs and their prices since last run
Rent estimate / market stat refresh	Monthly-ish trends	Weekly is generous	Scheduled poll of market-trend endpoints	Median values vs last run
"Is this one page still the same?" (an agent's profile, a single building page)	Rarely	Same day	A hosted page monitor with a webhook; no structured data needed	Rendered text

Two patterns fall out of that table. First, nothing in real estate needs sub-minute latency except automated bidding, and you are not doing automated bidding off a scraped search page. A 15-minute poll is "real-time" for every buyer-facing alert we've seen shipped. Second, the jobs split cleanly by what you need back: if you need fields (price, status, beds, sold date), poll a structured API; if you need "something changed on this page," rent a page monitor. Mixing them up is how you end up parsing a webhook payload with a regex at 11 PM.

Latency and cost, in numbers you can budget

Vendor pricing changes monthly, so this table compares the cost drivers, and the section after it turns one of them into an actual request budget.

Pattern	Detection latency	What you pay for	Setup effort	Limitations
Scheduled crawl (DIY scraper)	= your schedule, typically nightly	Proxies, rendering, compute, and engineer hours every time the markup changes	Weeks to production quality; ongoing maintenance	Cost scales with pages, not changes; anti-bot breakage; parser drift; you own legal review of every target
Polling a structured API	= your poll interval (15 min to daily)	Requests, including the ones that find nothing	An afternoon: one endpoint, one loop, one snapshot store	You own the diff logic and the storage; no push, so interval is a budget decision
Hosted page monitor with webhooks	= the vendor's cadence (minutes to hours by plan)	Per monitored URL or per check, by plan	Minutes per monitor	Unstructured diffs; per-URL pricing gets steep for hundreds of searches; still needs a data fetch to get fields
Hybrid: cheap search poll, deep fetch only on change	= the poll interval for detection; seconds more for the deep fetch	Frequent cheap checks plus rare expensive ones	A day; two endpoints and a queue	More moving parts; worth it only above a few dozen watched searches

What polling actually costs on RealtyAPI

Every RealtyAPI call is one request against your monthly quota (most endpoints cost one credit; the few that cost more are marked in the playground, per the credits docs). A search-by-URL call returns up to 350 listings in that single request, which is what makes polling a search cheaper than polling each listing. So the budget math per watched search URL is:

Poll interval	Requests per search URL per month	Search URLs a PRO plan (20,000 req/mo) can watch	Search URLs an ULTRA plan (85,000 req/mo) can watch
Every 5 minutes	8,640	2	9
Every 15 minutes	2,880	6	29
Hourly	720	27	118
Every 6 hours	120	166	708
Daily	30	666	2,833

Plan quotas and current prices are on the pricing page. One timing note, since it affects the budget: new PRO and ULTRA subscriptions move to $49 and $99 per month on September 14, 2026 (from $20 and $60 at the time of writing), and anyone subscribed before then keeps their current price permanently. The free plan's 250 requests a month is enough to run the script below hourly against one search for ten days, which is plenty to find out whether your market changes often enough to justify a faster poll.

Read that table with the decision matrix in mind. Price-drop alerts for a buyer app with 20 saved searches fit comfortably in PRO at hourly polling. A comp-refresh job for a valuation model watching 500 "sold" searches wants daily polling, and that's 15,000 requests a month, so also PRO. The thing that blows up a budget is polling every 5 minutes because "real-time" sounded good in the planning meeting, for a market where listings change twice a week. Yeah… no.

Code: poll search-by-URL and diff the results

This is the whole monitor. It polls Redfin's search-by-URL endpoint through RealtyAPI, keeps the last snapshot in a JSON file, and reports new listings, removed listings, and price changes. Node 18+ (for built-in fetch), no dependencies. The same shape works for the other platforms that expose search-by-URL (Realtor, Rightmove, Zoopla, Idealista, LoopNet, Centris, and most of the rest of our catalog); only the host and the field paths change.

// monitor.js — poll a RealtyAPI search-by-URL endpoint and diff the results
import { readFile, writeFile } from "node:fs/promises";

const API_KEY = process.env.REALTYAPI_KEY;
const SEARCH_URL =
  "https://www.redfin.com/zipcode/10002/filter/property-type=house,min-price=150k,max-price=2M";
const SNAPSHOT_FILE = "./snapshot.json";
const MIN_DROP_PCT = 1; // ignore price moves smaller than this

async function fetchSearch() {
  const url = new URL("https://redfin.realtyapi.io/search/byurl");
  url.searchParams.set("searchUrl", SEARCH_URL);
  url.searchParams.set("resultCount", "350");

  const res = await fetch(url, {
    headers: { "x-realtyapi-key": API_KEY },
    signal: AbortSignal.timeout(30_000),
  });

  if (res.status === 401) throw new Error("Bad API key (401)");
  if (res.status === 402) throw new Error("Out of credits (402) — top up or raise the plan");
  if (res.status === 429) {
    const body = await res.json().catch(() => ({}));
    throw new Error(`Rate limited (429); retry after ${body.retryAfter ?? "?"}s`);
  }
  if (!res.ok) throw new Error(`Upstream error ${res.status}`);

  const data = await res.json();
  if (!String(data.message).startsWith("Success")) {
    throw new Error(`API returned: ${data.message}`);
  }
  return { data, creditsRemaining: res.headers.get("x-credits-remaining") };
}

// Reduce each listing to the handful of fields the diff cares about.
function normalize(results) {
  const map = new Map();
  for (const r of results) {
    const h = r.homeData ?? {};
    const id = h.propertyId;
    if (!id) continue;
    map.set(id, {
      id,
      listingId: h.listingId ?? null,
      price: Number(h.priceInfo?.amount ?? NaN),
      beds: h.beds ?? null,
      baths: h.baths ?? null,
      sqft: Number(h.sqftInfo?.amount ?? NaN),
      listedAt: h.daysOnMarket?.listingAddedDate ?? null,
      url: h.url ? `https://www.redfin.com${h.url}` : null,
    });
  }
  return map;
}

function diff(prev, curr) {
  const events = [];
  for (const [id, now] of curr) {
    const before = prev.get(id);
    if (!before) { events.push({ type: "NEW", ...now }); continue; }
    if (Number.isFinite(before.price) && Number.isFinite(now.price) && before.price !== now.price) {
      const pct = ((now.price - before.price) / before.price) * 100;
      if (Math.abs(pct) >= MIN_DROP_PCT) {
        events.push({ type: pct < 0 ? "PRICE_DROP" : "PRICE_UP", ...now, from: before.price, pct });
      }
    }
  }
  for (const [id, before] of prev) {
    if (!curr.has(id)) events.push({ type: "REMOVED", ...before });
  }
  return events;
}

async function loadSnapshot() {
  try { return new Map(Object.entries(JSON.parse(await readFile(SNAPSHOT_FILE, "utf8")))); }
  catch { return new Map(); } // first run
}

async function run() {
  const prev = await loadSnapshot();
  const { data, creditsRemaining } = await fetchSearch();
  const curr = normalize(data.searchResults ?? []);
  const events = prev.size ? diff(prev, curr) : [];

  console.log(`[${new Date().toISOString()}] ${curr.size} listings` +
    ` (nextPage=${data.nextPage}, credits remaining: ${creditsRemaining})`);
  for (const e of events) {
    const price = Number.isFinite(e.price) ? `$${e.price.toLocaleString()}` : "n/a";
    const delta = e.from ? `  (was $${e.from.toLocaleString()}, ${e.pct.toFixed(1)}%)` : "";
    console.log(`${e.type.padEnd(10)} ${e.id}  ${price}${delta}  ${e.beds}bd/${e.baths}ba  ${e.sqft} sqft`);
  }
  if (!prev.size) console.log("First run: snapshot stored, no events emitted.");

  await writeFile(SNAPSHOT_FILE, JSON.stringify(Object.fromEntries(curr)));
  return events;
}

run().catch((err) => { console.error("monitor failed:", err.message); process.exit(1); });

First run, against a search that currently returns five houses:

$ REALTYAPI_KEY=rt_... node monitor.js
[2026-08-22T15:08:11.402Z] 5 listings (nextPage=false, credits remaining: 249)
First run: snapshot stored, no events emitted.

Run it again after the market moves and the output becomes the event stream you actually wanted. The format below is what the script prints; the specific changes will of course be whatever happened in your search:

$ REALTYAPI_KEY=rt_... node monitor.js
[2026-08-23T15:08:09.917Z] 6 listings (nextPage=false, credits remaining: 225)
NEW        119901522  $525,000  3bd/2ba  1710 sqft
PRICE_DROP 118650655  $739,900  (was $749,900, -1.3%)  4bd/2.25ba  2755 sqft
REMOVED    119330595  $397,000  3bd/1.75ba  1587 sqft

Schedule it with whatever you already run (cron, a GitHub Action, a Cloudflare cron trigger, Airflow), and replace the console.log in the event loop with your delivery channel. That's the monitoring API, built on a data API, in under a hundred lines.

The details that keep it honest in production

• Diff on a stable ID, never on position or address. Search results reorder constantly. propertyId is the property; listingId changes when a home is relisted, which is itself a useful signal (a relist with a new price is a "new" event and a price history entry at the same time).

• A listing vanishing from a search is not a sale. It's a candidate. Filters, pagination (nextPage: true means you didn't get everything), and the seller changing the price out of your filter range all produce the same absence. Before you tell a user "this sold," confirm with a details fetch; that's the hybrid pattern from the cost table, and it's the single cheapest upgrade to this script. Our price drop alerts guide goes deeper on thresholds and deduplication.

• Treat the credits header as a metric. x-credits-remaining comes back on every response. Graph it. A poll loop that doubled its request rate because someone added a second cron entry is invisible until the 402 arrives, unless you're watching that number.

• Request more than you expect, then paginate only if you must. resultCount goes up to 350 for one request. A search that needs page two is a search you should probably split by price band or beds into two URLs, because two narrow polls diff more cleanly than one paginated one.

• Know which filters the endpoint honors. For Redfin, search-by-URL applies status (for sale, for rent, sold), price range, beds, baths, and home type; other filters in the URL are ignored. Parse your URL against that list before you trust the diff, or you'll "detect" listings that the portal's own page wouldn't show. The search-by-URL guide covers normalizing URLs before you store them.

• Retry on 5xx, back off on 429, stop on 401/402. Only one of those is fixed by trying again. Status codes and their bodies are listed in the status code reference.

Our take: own the diff, rent the fetch

Here's the opinionated version. The diff is your product; the fetch is not. Whether a 0.8% price move is worth an alert, whether a disappearance means "sold" or "filter changed," how many alerts a user tolerates before muting you: those decisions are where a property app earns its keep, and no vendor's webhook encodes them the way your users need. Keep that logic in your codebase, where you can test it and change it on a Tuesday.

The fetch, by contrast, is undifferentiated work that gets more expensive every year. Rendering, anti-bot countermeasures, markup drift across 27 portals, and the legal review of each one are not things your roadmap benefits from. Rent them. That's the honest pitch for a structured API here: not that it replaces your monitor, but that it turns the monitor into a small, testable loop over JSON instead of a brittle fleet of headless browsers. For the one-page, unstructured cases, a hosted page monitor with a webhook is the right rental, and we'd tell you to use one.

What about the full-service "we'll alert you when any listing changes" products? They exist, and for a team with no engineers they're the right call. For a team with even one engineer, we think the decision matrix above makes a stronger case for polling a structured source, because every one of those services is polling something on your behalf and charging a margin on the judgment calls you'd rather make yourself.

Key takeaways

• Monitoring is scraping plus state. The snapshot, the diff, and the delivery are the product; the fetch is a commodity. Decide who owns each before you pick a tool.

• Pick the pattern by what you need back. Need fields (price, status, beds)? Poll a structured API and diff on a stable ID. Need "this page changed"? Rent a page monitor with a webhook. Don't parse webhooks with regex to get fields.

• Latency is a budget line, not a feature. A 15-minute poll is 2,880 requests per search per month; hourly is 720; daily is 30. Size the interval to how often the market actually changes, and most real estate alerts land at hourly.

• Absence is evidence, not proof. A listing dropping out of a search is a candidate for "sold." Confirm with a details fetch before you tell anyone.

• Watch the credits header. x-credits-remaining is the earliest warning that a poll loop has gone wrong.

If you want to try the loop above before committing to an interval, the free plan's 250 requests a month covers ten days of hourly polling on one search with no card on file, which is exactly enough to learn how often your market moves. Point it at the search you care about, let it run through a weekend, and let the event log tell you whether you need 15 minutes or 6 hours.