Duplicate Videos on Tube Sites: How to Find Them and What to Do
Any tube that imports from several sources ends up with duplicates: the same scene uploaded by different people, re-encoded, renamed or cut differently. On our network a single cleanup deactivated over half a million duplicate entries. Here is how duplicates hurt, how to find them, and how to remove them without losing anything.
Why duplicates hurt
- Visitors see the same scene several times in one grid and leave.
- Search engines see many near-identical pages and pick one, often not the one you’d choose, or decide the whole site is low quality.
- Your hosting stores and checks the same video many times.
Finding duplicates
No single signal is reliable, so combine several:
- Normalised titles. Lowercase, strip punctuation, release tags, resolutions and site names. Many duplicates match exactly after this.
- Duration. The same scene usually has the same length to within a second or two.
- Performers and studio. Same performers, same studio and similar length is a strong match.
- File identity. When the same file is stored on the same host, its file ID or checksum gives a certain match.
- Thumbnail similarity catches re-encodes with different titles.
Cleaning up safely
- Group duplicates into clusters and choose one survivor per cluster: the one with the best source (your own copy, highest resolution, healthy) and the most history (views, links).
- Deactivate, don’t delete. We marked duplicates inactive instead of deleting them. If the survivor dies later, a deactivated twin can come back.
- Make sure every cluster keeps a survivor. Chain logic (A duplicates B, B duplicates C) can otherwise deactivate a whole cluster.
- Redirect the removed pages to the survivor with a permanent redirect, so links and rankings carry over.
- Stop new duplicates at import with the same checks, so the cleanup doesn’t have to be repeated.
The same video on two of your own domains
If you run several sites from one catalogue, the same video can end up on two of your domains. To a search engine that is duplicate content across sites. We had tens of thousands of videos served on two domains at once and fixed it by giving every video one home domain and permanently redirecting the other copy there.
Shared videos need their own words
Sometimes the same video legitimately appears on two sites (different niches, different audiences). That’s fine, but each copy should have its own title and description, written for that site’s audience. A shared video with an identical title and description on two domains is the duplicate problem again.
Titles: unique, but honest
When you rewrite titles to make them unique, keep them accurate. Use the performers’ names and what actually happens in the scene. Avoid inventing details, and never add youth or family framing that isn’t there. A unique, truthful title beats a clever one.
Check regularly
Once a month: count clusters found at import, sample a few, and look for new sources that bring many near-duplicates. A source that mostly re-uploads what you already have isn’t worth scraping.