How to Detect Duplicate Property Listings With AI

Property portals and CRMs fill up with the same apartment posted twice, three times, or with a slightly rewritten title. Agents re-list after a soft lead, scrapers copy from other sites, and marketing teams tweak price or photos without closing the old row. Buyers see noise. Analysts double-count inventory. Matching logic that only compares exact strings misses almost everything that matters. This guide shows a practical AI-assisted pipeline to detect duplicate property listings using normalized attributes, fingerprints, embeddings, and a human review queue—built for teams on React/Next.js, Laravel, and PostgreSQL.

Why Duplicate Listings Hurt Portals and CRMs

Duplicates are not just a tidy-data problem. They inflate search results, break uniqueness constraints for MLS-style feeds, and poison models you might already run for AI rent estimation or recommendation. Common sources include:

• Agent re-posts: the same unit is created again after an expired soft hold or a new photo shoot.
• Scraped copies: third-party sites republish with minor title or amenity edits.
• Slight edits: price changed by a few percent, beds/baths left the same, description paraphrased.
• Multi-channel sync: the same listing arrives from a feed and from a manual CRM entry.

Without a dedicated dedup layer, support teams merge by hand and still miss near-duplicates that look different in text but describe the same home.

Exact Matches vs Near-Duplicates

Start with two layers. Exact duplicates share a strong fingerprint: same normalized address and unit, or identical content hash after cleanup. Near-duplicates need similarity scoring—embedding distance on title and description, fuzzy address match, geo proximity, and perceptual hashes on photos. Exact matching is cheap and safe to auto-merge. Near-duplicates belong in a review queue with a score and an explanation of which signals fired.

A Practical Detection Pipeline

1. Normalize structured fields and free text so comparisons are stable.
2. Fingerprint for exact and high-confidence attribute keys.
3. Score similarity with embeddings and photo hashes for the remainder.
4. Route borderline pairs to a review UI; auto-merge or auto-flag only above clear thresholds.

Normalization rules that pay off

• Lowercase, strip punctuation, expand common abbreviations (St → Street, Apt → Unit).
• Parse unit tokens separately from street address; never glue "12B" into the street line.
• Round price into bands (for example nearest 50 or 100 in the listing currency) for fingerprinting, but keep the raw price for scoring.
• Canonicalize beds/baths as numbers; treat "2.5 baths" and "2 baths + powder" carefully in scoring, not in the exact key.

Feature Signals That Work in Production

• Address + unit: the strongest identity signal when both sides are complete.
• Price band: large gaps reduce confidence; tiny gaps often mean the same listing refreshed.
• Beds / baths / area: hard mismatches should veto an auto-merge even if text looks similar.
• Geo: lat/lng within a few meters supports a match; same building with different units must not.
• Photo perceptual hash (pHash): catches reused galleries even when captions change.
• Title and description embeddings: catch paraphrases and multilingual rewrites. The same embedding stack you use for semantic property search can power this step.

Weight signals. Address+unit agreement should dominate. Text-only similarity without geo or unit context is where false positives hide.

Implementation Sketch: Laravel, Redis, PostgreSQL

Run detection as an async job when a listing is created or updated. Use Redis (or your existing queue driver) so the request path stays fast. Store candidates and decisions in PostgreSQL; add pgvector if you already embed listings for search or for grounding AI answers in your listing database.

# Python sketch: normalize + fingerprint + cosine gate
import hashlib, re
from difflib import SequenceMatcher

def norm_addr(s: str) -> str:
    s = s.lower().strip()
    s = re.sub(r"[^a-z0-9\s]", " ", s)
    s = re.sub(r"\b(st|street|ave|avenue|rd|road)\b", lambda m: {
        "st": "street", "ave": "avenue", "rd": "road"
    }.get(m.group(1), m.group(1)), s)
    return re.sub(r"\s+", " ", s)

def fingerprint(listing: dict) -> str:
    key = "|".join([
        norm_addr(listing["address"]),
        (listing.get("unit") or "").lower().strip(),
        str(listing.get("beds")),
        str(int(round(float(listing["price"]) / 50) * 50)),
    ])
    return hashlib.sha256(key.encode()).hexdigest()

def text_similar(a: str, b: str) -> float:
    return SequenceMatcher(None, a.lower(), b.lower()).ratio()

// Laravel job outline
class DetectListingDuplicates implements ShouldQueue
{
    public function __construct(public int $listingId) {}

    public function handle(DuplicateDetector $detector): void
    {
        $listing = Listing::query()->findOrFail($this->listingId);
        $exact = $detector->findByFingerprint($listing);
        if ($exact->isNotEmpty()) {
            $detector->queueMergeCandidates($listing, $exact, 'exact');
            return;
        }
        $near = $detector->findNearDuplicates($listing); // embeddings + pHash + geo
        $detector->queueReview($listing, $near);
    }
}

For vector search, store an embedding of title + description (and optionally amenities) and query nearest neighbors with a cosine threshold, then re-rank with attribute and photo signals. PostgreSQL with pgvector keeps this next to your listing rows; Redis holds the job queue and short-lived locks so two updates do not race a merge.

Human Review UI and Merge Policy

Borderline pairs (for example similarity 0.82–0.94 with mixed attribute agreement) should land in a review queue: side-by-side photos, address, price history, and a short "why flagged" list. Policies that usually work:

• Auto-merge only on exact fingerprint or very high multi-signal scores with identical unit.
• Auto-flag (hide one from public search, keep both in CRM) when confidence is high but ownership differs.
• Human decide for same building, missing unit, or photo-only matches.

Log every decision. You will need it to tune thresholds and to audit mistakes.

Pitfalls to Avoid

• Same building, different units: never merge on street address alone. Unit (or floor plan + unit) is mandatory for residential inventory.
• Multilingual copy: embeddings help more than string fuzzy match; still normalize numbers and addresses in the local script when you can.
• Photo reuse across legitimate listings: developers reuse amenity shots. Require photo agreement plus at least one strong identity signal before auto-merge.
• Over-aggressive text match: template descriptions from the same agency look alike. Pair text scores with structured fields.

Launch Checklist

1. Define normalization rules and unit parsing for your markets.
2. Ship fingerprint-based exact matching first; measure precision.
3. Add embedding + pHash near-duplicate scoring behind a review queue.
4. Build a simple side-by-side review UI with decide / merge / not-duplicate actions.
5. Set auto-merge thresholds conservatively; raise them only with logged precision.
6. Monitor false positives weekly, especially multi-unit buildings.
7. Document ownership: which listing stays canonical after a merge.

Conclusion

Detecting duplicate property listings is a systems problem with an AI layer, not a single model call. Normalize hard attributes, fingerprint for exact hits, use embeddings and photo hashes for near-duplicates, and keep humans in the loop where units and buildings collide. The same listing data and embedding infrastructure you already use for search and generation will make this cheaper to run—and cleaner inventory makes every downstream PropTech feature more trustworthy.