S
SEO SOPKnowledge Base
Search
← All topics

Duplicate Content Risk Scanner — Detection and Remediation

runnable

Identifies and remediates duplicate content risks: same-page variant URLs, near-duplicate pages, boilerplate content across location pages, scraped or syndicated content issues, and internal cannibalization. Produces canonicalization strategy and remediation priority list.

seoduplicate-contenttechnicalcanonicalizationthin-content
Agent trigger phrases: duplicate content · content duplication · canonical issues · near-duplicate pages · duplicate page scanner · content cannibalization · duplicate content SEO · canonicalization audit

Why Duplicate Content Hurts Rankings

Google's crawl budget is finite. When multiple URLs serve identical or near-identical content, Google must choose which version to index and rank. If Google chooses incorrectly (or splits ranking signal across variants), all versions rank below their potential. In severe cases, duplicate content at scale triggers Panda-era quality penalties that depress the entire domain.

Duplicate content is not a manual action — it is an algorithmic quality signal problem.

The Four Duplicate Content Types

Type 1: URL Variant Duplication (Technical)

Same content served at multiple URLs due to:

  • ?page=1 vs no pagination parameter
  • ?ref=email vs clean URL (UTM parameters that aren't stripped)
  • /index.html vs / (trailing slash variants)
  • HTTP vs HTTPS variants
  • www vs non-www variants
  • Case-sensitive URL variants (/Services vs /services)

Detection: Crawl site with LibreCrawl. Export URL list. Filter for URLs that differ only in parameters or capitalization. Check redirect rules and canonical tags.

Remediation:

  • Force redirect all HTTP → HTTPS, www → non-www (or vice versa) at server level
  • Strip UTM/tracking parameters in <link rel="canonical"> and robots.txt Disallow for ?ref=, ?utm_*
  • Set canonical tags to the preferred URL version on every page

Type 2: Near-Duplicate Page Duplication (Content)

Pages with 80%+ identical copy that differ only in variable substitution (typically location pages or service variant pages). Google's duplicate content classifier uses cosine similarity — pages above ~0.85 similarity score are treated as duplicates.

Detection:

  • Pull all location page URLs from sitemap
  • Compare page body text for 3-5 page pairs
  • If only the city name changed across 500+ words of identical copy, these are near-duplicates

Remediation:

  • Minimum 30% unique content per page (see programmatic-seo for uniqueness strategies)
  • Add location-specific data: local statistics, neighborhood names, city-specific regulations, local reviews
  • Vary FAQ content — at least 2 of 8 FAQ questions should be location-specific

Type 3: Scraped or Syndicated Content

Content that appears on your site and also on other sites (either you scraped them, they scraped you, or you both syndicated from the same source). Google typically ranks the first-indexed version — if competitors or aggregators indexed your content before you, you may rank below them for your own content.

Detection:

  • Take unique 20-word phrases from key pages and search them in Google in quotes
  • If your page is not the first result, another site indexed the content first

Remediation:

  • Submit your URL via GSC for priority indexation immediately after publishing
  • Add datePublished to Article schema — this establishes your first-publish claim
  • If content was scraped: use Google's scraper content report form
  • If you are syndicating content TO other sites: require <link rel="canonical"> pointing to your original URL on all syndicated copies

Type 4: Internal Content Cannibalization

Multiple pages on the same site targeting the same keyword. Both rank below their individual potential because Google splits ranking signal across both.

Detection:

  • Export all GSC queries for pages ranked position 5-30
  • Group by query
  • If 2+ pages appear in GSC for the same query, cannibalization exists

Cannibalization decision matrix:

| Scenario | Action | |----------|--------| | Both pages are thin | Merge into one stronger page, 301 redirect loser to winner | | One page is clearly the authority | Noindex or 301 redirect the weaker page | | Both pages are strong but different intent | Differentiate content clearly, link between them | | Category page vs blog post competing | Redirect blog post to category, or add canonical from blog to category |

Canonicalization Audit Checklist

  • [ ] Every page has a self-referencing canonical tag
  • [ ] HTTP and HTTPS both redirect to the same base (no mixed content)
  • [ ] www and non-www redirect to one canonical base domain
  • [ ] Trailing slash is consistent (either always present or never)
  • [ ] URL parameters (UTM, session IDs, pagination) filtered in GSC URL parameters settings
  • [ ] Sitemap only includes canonical URLs (not parameter variants)
  • [ ] Paginated content uses rel="prev" and rel="next" (or is consolidated)

Thin Content vs Duplicate Content

These are related but distinct problems:

  • Thin content: Page has little content (under 300 words) but is unique — fix by adding substance
  • Duplicate content: Page has content but it's identical to another URL — fix with canonicalization or merge
  • Thin + Duplicate: The worst case — many short identical pages. Fix: merge into one substantial unique page, 301 redirect all others

Cross-References

  • crawl-budget-optimizer — crawl budget is wasted on duplicate URL variants
  • panda-quality-score-audit — site-level quality impact of duplicate content
  • programmatic-seo — uniqueness requirements that prevent programmatic near-duplicates
  • sitemap-analyzer — sitemap should only include canonical, non-duplicate URLs

#seo-sop #seo #duplicate-content #technical #canonicalization #thin-content

View raw JSON →Back to index