Content duplicacy checker: the two jobs, the right tool for each, and what to do with the result

A content duplicacy checker compares a piece of text against other text and reports how much of it matches. There are two separate jobs hiding under that one name, and the tool you need depends on which one you have. Checking whether your text appears on somebody else's site, or whether a writer handed you something copied, is an external similarity check and is what plagiarism tools are built for. Checking whether the same text appears on several URLs of your own site is an internal duplication check and is a crawler's job. Pick the job first, because a plagiarism tool will not find your own duplicate product pages and a crawler will not tell you that an article was lifted from a competitor.

This page covers checking for duplication and what to do with the result. For how search engines choose between pages that are genuinely similar on your own site, see canonical tags and duplicate content.

The four kinds of duplication, and the tool for each

Kind	Example	How to find it	What to do
External copying of your work	Your article republished elsewhere without credit	Search a distinctive sentence in quotes; run a plagiarism checker on the page	Request credit or removal; keep your own page as the canonical source
Copied content handed to you	A freelance draft lifted from another site	Plagiarism checker on every delivery, before publishing	Reject and rewrite; make the check a contract term
Internal exact duplication	The same page reachable at several URLs	Site crawler, duplicate page report	Pick one URL, canonical or redirect the rest
Internal near duplication	Fifty location pages with one word changed	Crawler near duplicate report with a similarity threshold	Consolidate, or add genuinely different content to each

How similarity scores are produced

Understanding the number stops you acting on the wrong finding. Checkers break text into overlapping chunks, usually a few words long, then compare those chunks against an index or against each other. The score is the share of chunks that match. That mechanism has three consequences worth knowing:

• Quotes, names and boilerplate inflate the score. A legal disclaimer, a product specification table or a quoted regulation will match everywhere it is published. A high score on a page full of standard phrasing is usually correct and harmless.
• Rewording lowers the score without changing the problem. Text put through a paraphraser scores low and is still somebody else's work and structure. Read the matched passages instead of trusting the percentage.
• Different tools index different sources. Two checkers disagreeing means their indexes differ, not that one is broken. For an important decision, run two and read the matched passages in both.

So treat the percentage as a pointer to the passages worth reading, and make the decision from the passages.

Checking your own site: the sequence

Internal duplication is the kind that costs you traffic, because it splits the signals for one piece of content across several URLs. Run this once, then quarterly:

• Crawl the site and open the duplicate page report. A crawler flags exact duplicates by hashing the content, which catches the same page served at several URLs. Free crawling tiers cover small sites; our guide to SEO audit tools covers the categories.
• Run the near duplicate report with a threshold. Start around 90 per cent similarity and lower it until the results stop being useful. This is the report that finds templated page families with nothing distinct in them.
• List the URL patterns causing it. Most internal duplication comes from a handful of mechanics: parameters for filters and sorting, session or tracking parameters, trailing slash and case variants, http and https, with and without www, printer friendly versions, pagination that repeats the first page, and tag or archive pages that list the same posts.
• Check what Google selected. In Search Console, the Pages report shows URLs excluded as duplicates and tells you which URL Google chose as canonical. Compare that list against your intentions. Where Google chose a different URL from the one you declared, your signals are inconsistent somewhere.
• Fix by mechanism, not page by page. One template fix usually clears thousands of URLs. Fixing them individually is work you will repeat next quarter.

Choosing the fix

Google's documentation on consolidating duplicate URLs sets out the methods, and the choice between them comes down to one question: do you need both URLs to stay reachable?

• Both URLs need to work: keep both and set a canonical on the duplicate pointing at the version you want indexed. This is the right answer for filtered and parameterised versions of a product list.
• Only one URL should exist: use a 301 redirect. This is the right answer for http to https, www to non www, and old URLs after a migration.
• The page is for people but not for search: use a noindex meta robots tag. Internal search results and thank you pages belong here.
• The pages are different enough to deserve their own existence: then make them different. Add the data, the images, the specifications or the local detail that justify a separate page, and give each a unique title and description.

A canonical is a strong hint rather than an instruction, so back it up: link internally to the canonical version, list only the canonical in your sitemap, and keep the canonical consistent with your redirects. For the markup and the common mistakes, see canonical tags in SEO.

Checking a writer's delivery

Make this a step rather than a suspicion. Three habits cover it:

• Run every delivery through a similarity checker before publication, and record the score and the date in your content sheet. A record makes the check routine rather than personal.
• Read the matched passages, not the score. Matches on a quoted statistic, a product name or an industry phrase are normal. A matched paragraph of prose is not.
• Put it in the contract: original work, assignment of copyright on delivery, and a stated threshold above which the piece is rewritten at the provider's cost. Our guide to comparing content quotes covers the rest of the terms.

When somebody has copied you

Search a distinctive sentence from your page in quotation marks. That one search finds most straightforward copying. Then:

• Record the evidence: the URL, a dated screenshot, and the archive copy if one exists.
• Confirm your own page is the published original, with its publication date visible and its canonical pointing at itself.
• Ask for attribution and a link, or for removal. Most copying by small sites stops at a polite email.
• If it continues, use the host's and the search engine's published removal processes for copyright complaints.
• Keep publishing. A page with the original date, internal links and its own authority usually holds its position against a copy.

The content duplicacy checker questions worth an answer

Does duplicate content get a site penalised?

Duplicate content on your own site is a selection problem rather than a penalty: search engines pick one URL to show and consolidate signals onto it. The cost is that they may pick a different URL from the one you wanted, which is why the fix is to declare your preference clearly.

What similarity percentage is acceptable?

There is no published threshold to aim at, so read the matches instead. A page scoring 30 per cent because it quotes a regulation is fine. A page scoring 12 per cent where that 12 per cent is one lifted paragraph is not.

How do I check duplicate content between two of my own pages?

Use a crawler's near duplicate report, which compares your pages against each other and shows the matching passages with a similarity score you can set. A plagiarism tool compares against its own index and is built for a different question.

Do product descriptions from a manufacturer count as duplicate content?

They appear on every retailer's site that uses them, so they give you nothing distinctive to rank with. The practical move is to keep the manufacturer specification as structured data and add your own text: what it suits, what it does not, sizing notes, and real customer questions.

Can AI written text be flagged as duplicate?

Similarity checkers compare text against an index, so generated text that closely follows an existing source will match it. Checking the output the same way you check a freelance delivery is the sensible step, and it is a separate question from how Google treats AI generated content.

How often should I run an internal duplication check?

Quarterly for a stable site, and after every template change, migration or new parameter. Those three events are what create duplication at scale, so the check belongs in the release process rather than only in the audit.

Which copy an assistant quotes

A similarity score tells you whether two texts match. It does not tell you which of the matching pages an assistant will quote when a buyer asks about the topic. That choice is made when the answer is assembled, and the original is not always the one picked.

Test it on a page you have just cleaned up. Ask an assistant the question that page answers and look at what it names as the source: your site, the copy, or a third site that summarised both. For the comparison itself, AI Knows Us lists the sources behind each answer in ChatGPT, Gemini, Perplexity, Claude and AI Overviews.