How to Merge Duplicate Listings: A Developer's Guide

Your dashboard says the metro has a healthy supply of homes. Then an analyst notices that several properties appear under slightly different addresses, prices, and unit formats. The same apartment may have arrived through an MLS feed, a syndication partner, and a manual upload, while a relisted home looks like a new property instead of a new market event.

That's the operational reality behind how to merge duplicate listings. The hard part isn't deleting repeated rows. It's deciding when records describe the same physical asset, when they represent separate legal or commercial interests, and which source should remain authoritative after a merge.

Why Duplicate Listings Break Your Real Estate Data

A duplicate listing rarely announces itself. One record may use “Apartment” while another uses “Apt,” one may include a unit suffix, and a third may omit it. Prices can differ after an update, descriptions can be rewritten by a syndication service, and identifiers can disappear during a manual import.

The result is a property database that looks complete but produces unreliable conclusions. A pricing model may treat repeated records as independent evidence. An inventory dashboard may count one home several times. A lead-routing system may send the same buyer to multiple agents, while marketing teams pay to maintain several landing pages for one asset.

The market impact is larger than the row count

The Canadian Real Estate Association reported that duplicate listings represented less than 0.8% of active listings on REALTOR.ca. Its analysis also found that hypothetical double-counting would affect less than 1% of Ontario sales and less than 0.4% of national sales activity, while adjusted average prices would change by 0.3% in Ontario and about 0.1% nationally. Those figures show why a seemingly small duplicate rate can still affect market totals, pricing measures, and inventory ratios used by brokers and policymakers. The Canadian Real Estate Association analysis provides useful context for setting an accuracy target.

The problem also persists in historical data. A Texas REALTOR data project aggregated, normalized, cleansed, and de-duplicated records dating back to 2009, removing more than 8,500 residential sales in 2014 because they had been cross-listed in the same or another MLS. The reported Texas data audit illustrates why a live-feed cleanup alone isn't enough.

A sound master data governance strategy gives the pipeline rules for ownership, survivorship, history, and review, not just matching logic. Matil's guide to master data governance strategy is useful background for defining those policies before engineers automate merges. Your ingestion architecture also needs stable source handling and clear provenance, which should be planned alongside the RealtyAPI.io introduction.

Building a Matching Engine with Normalization and Scoring

Matching starts before similarity scoring. If two records use different representations for the same address, a fuzzy model is forced to solve a formatting problem and an identity problem at the same time.

Normalize the fields that carry identity

Use a canonical representation for every important attribute:

• Address: Standardize street types, directional prefixes, apartment labels, punctuation, and whitespace. Preserve the original value for auditability.
• Geolocation: Store latitude and longitude in a consistent numeric format, then compare proximity rather than demanding exact coordinate equality.
• Identifiers: Keep MLS numbers, parcel IDs, APNs, building IDs, and source listing IDs in separate fields. A source ID is usually unique only within its originating system.
• Text fields: Tokenize owner names and titles, lowercase them, remove harmless punctuation, and retain meaningful unit or building tokens.
• Measurements: Convert lot size and floor area into canonical units before comparison. A value expressed in acres shouldn't be compared as though it were square feet.

A normalized fingerprint might combine jurisdiction, normalized street address, unit identifier, rounded coordinates, and a parcel reference. Don't use the fingerprint as a universal truth. Use it as a blocking key that narrows candidate pairs.

Score candidates incrementally

Start with the strongest signals. Parcel IDs and APNs can receive the highest weight when they're present and validated. Normalized address and geographic proximity belong in the middle. Title similarity should support a decision, not dominate it, because listing titles are easy to rewrite.

Field	Weight	Match Type	Notes
Parcel ID or APN	High	Exact or validated exact	Strong signal, but verify jurisdiction and source quality
Normalized address and unit	High	Exact or token-aware	Unit handling is essential for multi-family properties
Geolocation	Medium	Distance-based	Use proximity bands rather than exact coordinates
Price	Medium	Tolerance-based	Helpful for current duplicates, weak across time
Listing title	Low	Fuzzy	Useful as supporting evidence only
Description	Low	Token overlap	Syndicators often rewrite this field

A simple weighted score can be expressed conceptually as:

score = identifier_match + address_match + geolocation_match + price_match + text_match

The implementation should also include hard exclusions. A residential unit and a commercial suite at the same street number shouldn't merge just because their addresses are close.

Thresholds need validation, not intuition. One deduplication study found that reasonable similarity thresholds could keep both precision and recall above 0.9, while threshold movement quickly changed the balance between false positives and missed duplicates. The same research found that a naive multi-feature hybrid could underperform a name-only baseline, while a tuned name-plus-price model improved recall with almost no precision loss. The Dedupalog research paper supports testing fields one at a time instead of assuming every feature improves the model.

For address-enriched property lookups, teams can also use property-address API data as one input to a broader canonicalization process. It shouldn't replace source-level provenance or governance decisions.

Deterministic vs Probabilistic Matching Approaches

A deterministic matcher answers a narrow question: do two records share a trusted identifier or satisfy an explicit rule? If the MLS number is identical within the same source, the match may be safe. If a parcel identifier, normalized address, and unit agree, the system can often merge without invoking a statistical model.

Probabilistic matching answers a broader question: how likely is it that these records describe the same listing or property despite inconsistent data? It can combine address similarity, coordinate distance, price proximity, title resemblance, source behavior, and temporal context. That flexibility helps with cross-platform syndication, but it also creates uncertainty that needs review.

Use a two-pass architecture

The production pattern is straightforward:

1. Deterministic pass: Merge only records that satisfy reviewed, high-confidence rules.
2. Candidate blocking: Compare remaining records only within plausible geography, property type, unit, or time windows.
3. Probabilistic pass: Calculate a field-level similarity score for candidate pairs.
4. Review queue: Send borderline results to an operator who can inspect source records and history.
5. Audit decision: Store the rule, score, reviewer action, and surviving canonical record.

A benchmark comparing matching approaches found that optimized deterministic methods reduced the manual-review set to 2.5%, compared with 3.6% for a probabilistic method. At a single threshold, the deterministic approach recorded 0.956 positive predictive value and 0.985 sensitivity, while the probabilistic method recorded 0.887 for both measures. The benchmark published in PubMed shows why carefully engineered rules can outperform a generic probabilistic baseline.

Tune for the cost of being wrong

There's no universally correct threshold. If a false merge destroys legal history or sends a buyer to the wrong unit, favor precision and expand human review. If missed duplicates inflate a public inventory feed, you may accept more candidates for review, but don't let recall become an excuse for automatic merging.

One practical benchmark reported that at a 0.95 clustering setting, recall was around 0.8 while precision remained above 0.9. That gap matters. A cluster can look highly reliable while still leaving many true duplicates unresolved.

Feed volume also affects engineering design. Apply rate-limit and retry behavior at the ingestion layer, especially when source records arrive in bursts. The RealtyAPI.io rate-limit documentation can help teams account for request behavior when building import and reconciliation jobs.

When Not to Merge Duplicate Records

The most dangerous deduplication rule is “same address means same property.” Addresses identify locations, not always legal assets, market events, or commercial units.

A parcel may be split while retaining a similar street address. A teardown may be replaced by new construction at the same location. A commercial building may contain separate retail and office units that share one street number. Two listings can also represent separate contractual relationships, even when they describe the same physical building.

Put governance ahead of automation

Before merging, evaluate the record's identity at several levels:

• Physical identity: Do the records describe the same structure, unit, or parcel?
• Legal identity: Do legal descriptions, parcel references, or ownership records agree?
• Listing identity: Are they the same market event, or separate listings for the same asset?
• Source identity: Must the original MLS or partner record remain independently traceable?
• Attribution identity: Does merging remove agent, broker, or source responsibility that the business must preserve?

Governance rule: Merge the canonical property relationship, not the evidence that produced it.

A useful policy model separates the canonical property from source records. The canonical entity can represent the physical asset, while each source listing retains its original identifier, status history, attribution, timestamps, and legal references. That approach preserves context without forcing the user interface to display duplicate inventory.

Route ambiguity to review

Use hard stops for conflicting parcel IDs, incompatible property types, different unit numbers, and contradictory legal descriptions. Treat status changes as events until the system confirms that they belong to the same listing lifecycle.

A decision tree can be simple:

1. Do trusted identifiers agree?
2. If not, do address, unit, geography, and property type agree?
3. Are there legal, structural, or source-level conflicts?
4. Is the difference a legitimate relisting or status event?
5. If uncertainty remains, keep separate records and link them for review.

Google Business Profile applies a similar principle. A duplicate profile won't show on Google Search or Maps, and merging is allowed only when both profiles represent the same business at the same location. Google also distinguishes storefront businesses from service-area businesses, requiring different identity conditions for each type. Google's duplicate-profile guidance is a useful reminder that merge eligibility depends on the entity model, not superficial similarity.

Handling Cross-Platform and Temporal Duplicates

Cross-platform duplicates require a source-independent property identity. Zillow, Realtor.com, local MLS systems, broker CRMs, and internal uploads may all assign different IDs, expose different fields, and update at different times. Comparing only the latest import against the previous batch leaves older duplicates alive.

Create a universal property identifier, but don't discard source IDs. The universal ID should connect records that your governance rules consider one canonical asset, while the source record remains available for provenance and reconciliation.

Canonicalize before comparing

Normalize each source into a shared schema:

• Address and unit fields become canonical tokens.
• Lot size and floor area use one unit system.
• Price becomes a numeric value with currency and transaction context.
• Status maps into a controlled vocabulary such as active, pending, sold, expired, or withdrawn.
• Timestamps distinguish publication, modification, withdrawal, and closing events.
• Source priority determines which record supplies display fields when values conflict.

Syndication echo is a common failure mode. A partner republishes an MLS record, another platform ingests the partner copy, and your aggregator receives both. Source lineage helps identify the echo, while matching signals determine whether the copies represent one listing.

Treat time as part of identity

A price change usually belongs to the same listing history. A withdrawal followed by a new listing may or may not. A seasonal rental that returns later can represent a recurring market event rather than one continuously active record.

Keep a property entity, a listing entity, and a status-event history when the business needs market chronology. A practical temporal window compares new imports against all historical records for the same canonical location, then applies rules for price changes, agent changes, source changes, and gaps in activity.

Duplicate Scenario	Source Example	Resolution Strategy	Risk if Merged
Same listing syndicated twice	MLS feed and partner feed	Link both to one listing entity, preserve both source IDs	Attribution and freshness may be lost
Same property with a price update	Earlier and later feed snapshots	Keep one listing history with price events	Price history becomes fragmented
Withdrawn and relisted home	Old MLS record and new MLS record	Link cautiously, retain separate market events when identifiers or terms differ	Days on market can be understated
Multi-unit address collision	Building-level and unit-level feeds	Require unit-aware matching	Separate units can collapse
Seasonal or recurring rental	Same address across distinct periods	Keep time-bounded availability records	Distinct rental periods become one misleading listing

The key distinction is between property identity and listing-event identity. A marketplace may show one property card, but analytics still need the underlying sequence of listings, status changes, and source observations.

Testing and Automating Your Deduplication Pipeline

A deduplication pipeline is a decision system, so test it like one. A handful of successful merges proves very little. You need labeled examples of true duplicates, legitimate near-matches, relistings, unit collisions, and records with conflicting identifiers.

Build evidence before increasing automation

A golden dataset should contain the records your team knows how to classify, along with the reason for each decision. Keep examples from every major source and property type. Include hard negatives, such as neighboring units, parcel splits, and properties that share a building address but have different legal identities.

Measure precision and recall separately. A high precision score tells you that accepted merges are usually correct. Recall tells you how many true duplicates the system still misses. Threshold selection should reflect the cost of each error, not just a single aggregate score.

Run new rules in shadow mode before allowing them to write merges. The candidate engine can produce proposed pairs, scores, and governance flags while the current production system remains unchanged. Review the differences, inspect false positives, and compare the proposed merge graph with the golden dataset.

Operational safeguard: Never deploy a new merge rule directly against irreversible production state.

Monitor the pipeline as a data product

Track merge volume, review volume, conflict categories, orphaned records, and source-specific match behavior. A sudden change can indicate a schema alteration, a broken parser, a feed duplication issue, or an overly broad rule.

Automated controls should include:

• Rule tests: Verify each normalization and exclusion rule with representative fixtures.
• Integration tests: Run complete ingestion-to-merge flows against realistic source combinations.
• Threshold alerts: Flag unexpected changes in candidate, accepted, rejected, and reviewed pairs.
• Rollback paths: Preserve pre-merge relationships so an incorrect rule can be reversed.
• Audit records: Store who or what merged records, which fields agreed, and why the decision passed.

Batch reconciliation works well for historical repair and nightly consistency checks. Streaming detection helps prevent duplicate records from reaching search and lead systems, but it still needs periodic full reconciliation because late-arriving updates can change an earlier decision.

A human review queue should expose the original values, normalized values, source lineage, matched fields, conflicting fields, and prior status events. Workflow tools can help route those decisions. For example, RealtyAPI.io's n8n integration documentation is relevant when teams want to connect ingestion, review, and operational notifications without embedding every workflow in application code.

The durable design is not an aggressive merge job. It's a controlled system that can explain every merge, preserve every source record, detect drift, and keep uncertain cases separate until a person or stronger evidence resolves them.

---

RealtyAPI.io provides a developer-first real estate data layer for retrieving public listing and property data through REST, GraphQL, and webhooks, giving PropTech teams a practical source for building ingestion and reconciliation workflows. Visit RealtyAPI.io to get an API key, test property data, and connect your deduplication pipeline to a scalable real estate data workflow.