Content Pruning for Large Sites: A/B/C Tiers, Red Lines and Hard Evidence for Merges

Content pruning for large sites: A/B/C tiers, red lines and hard evidence for merges

TL;DR: Content pruning on a site with tens of thousands of URLs starts with 16 months of Search Console data, not gut feel. Split pages into A (clicks in the last 90 days, or part of the top 80% of traffic), B (some clicks over 16 months) and C (zero Google impressions in 16 months). Then split C again, because a page Google ignores may still get traffic from Yandex, Bing, Baidu or a mirror domain. Only old C pages with no traffic anywhere become candidates, and the action is a Googlebot-only noindex applied by ID list. No deletions, no mass 410s. Duplicates get merged only when the video hash matches exactly: on one video site we took over, 50 title-matched pairs produced about 6 real duplicates. And pruning frees crawl budget. It does not, by itself, add traffic.

What pruning is for

9,528 pages with zero impressions in 16 months is the kind of problem content pruning exists for: it stops low-value URLs from eating the crawl budget of a large site. It moves crawl resources to useful pages but does not raise clicks by itself.

Once a site passes a few thousand URLs, crawl budget becomes the bottleneck. Google says as much in its guide to managing crawl budget on large sites↗: time Googlebot spends on low-value URLs is time it doesn't spend on the ones you care about. You can't hand-review ten thousand pages, so you need rules that decide, in batches, which pages deserve the crawler's attention.

The numbers can be stark. A video content site we took over had a little over 10,000 posts, and 9,528 of them had not received a single Google impression in 16 months. Most of the site was invisible. Publishing more on top of that would only have made the ratio worse.

Be careful how you sell it, though. Pruning moves crawl resources away from dead pages. It doesn't make clicks go up on its own, and a report that promises "pruning will lift traffic by X%" is making a claim nobody can back. When the number doesn't move, the whole approach gets thrown out with it. Our reports state what was pruned, which tier it came from, which metrics we're watching and for how long.

Tiering with 16 months of data

Content pruning starts by sorting pages into tiers A (core), B (has traffic) and C (zero impressions) using Search Console's longest window of 16 months, not gut feel.

We use the longest window Search Console offers, which is 16 months. Three months will flag seasonal content as dead, and "nobody reads this" is a feeling, not a ranking report.

Tier A uses "or" on purpose. A page with even one or two clicks in the last 90 days still has real people looking for it, and a page in the top 80% of traffic is part of the site's backbone even if it's had a quiet month. Either condition puts it on the protected list, and protected means you don't even "tidy up" its URL.

Look at absolute numbers, not percentages. A search results page dropping 50% might cost you 500 clicks. An article template dropping 10% can cost 20,000. Tier A may be a small slice of the page count while carrying most of the traffic, so every pruning batch gets checked against that list before it runs.

Splitting C a second time

Zero impressions in Google is not the same as zero readers. Large content sites like this one often draw real traffic from Yandex, Bing or Baidu, and mirror domains have their own users. So C gets split:

• C1: zero Google impressions, but traffic from other engines or mirrors. Leave it alone.
• C2, old posts: no traffic anywhere and live for a while. Candidate for a Googlebot-only noindex.
• C2, new posts: no traffic anywhere but recently published. Watch first. A page still being evaluated can't be judged yet.

On that site, the 9,528 C pages broke down into 4,977 C1, 3,105 old C2 and 1,350 new C2. Had we acted on all 9,528, more than half of the affected pages would have been ones people were actually finding through other engines. The real candidate list was about a third of tier C.

Spot checks on tier C turned up something else: roughly half of those pages were indexed and still getting zero impressions. Their problem is ranking, not discovery. That changes the toolkit. A page Google hasn't indexed might benefit from a sitemap entry or more internal links. An indexed page with no impressions won't change no matter how many times you request indexing.

Four red lines

Content pruning has four red lines: no deleting or mass 410/404s, no bulk noindex, no touching tier A pages, and no promise that pruning will bring traffic.

Don't delete, and don't mass-return 410 or 404. A large batch of error pages is a negative signal in itself. Turn a few thousand URLs into 404s and the crawler sees a site that suddenly broke, and if even one of those pages was still ranking, you've just routed its visitors into an error page. Undoing a deletion means republishing.

Don't noindex in bulk. Work from an ID list, and target Google only. A blanket rule like "noindex this whole category" will catch tier A or C1 pages every time.

Don't touch tier A. Every batch is diffed against the protected list before it runs. Any match comes out.

Don't promise traffic. That goes in the first paragraph of the report, as above.

Why a Googlebot-only noindex

Pruning uses a Googlebot-only noindex because mirror domains share one CMS, so a plain robots noindex would also remove the mirrors' Yandex and Baidu traffic.

We learned this one the hard way. The same video site runs mirror domains off a single CMS, so a meta tag added on the main site shows up on every mirror too. A plain <meta name="robots" content="noindex"> on tier C would have dropped the mirrors' Yandex and Baidu traffic, which is exactly the traffic that made those pages C1 in the first place.

The fix is to address Google's crawler by name:

<meta name="googlebot" content="noindex, follow">

Google's robots meta tag documentation↗ confirms that the googlebot name applies only to Google's crawlers. Bing, Yandex and Baidu read robots or their own user-agent names. Yandex documents its handling in its meta robots help page↗. We keep follow because these old posts often link to tier A pages. One more rule: don't block the same URLs in robots.txt, or Googlebot never gets to read the noindex.

To be straight about it, we haven't yet confirmed how much a Googlebot-only noindex actually frees up crawl budget. We'll only know after the old C2 list goes live on that site and we've tracked it for a while. Until then it's a reasoned bet, not a proven result.

Merging duplicates needs hard evidence

Duplicate pages should be merged only when the video hash is identical. Similar titles, tags or uploader accounts are reasons to investigate, never to merge.

The classic mistake on large video sites is merging pages because their titles look alike. On the same site we paired up 50 "duplicate-looking" sets by shared tags and checked them. Only about 6 were genuinely the same thing. The rest were ranking lists from different dates, different segments cut from one long video with titles that differed only by a segment number, or different videos sharing a title template.

So the merge criterion is an identical video hash, meaning the file itself is the same. Similar titles, tags or uploader accounts are reasons to investigate, never reasons to merge. Text sites need the equivalent: evidence at the content level, not the title level.

Once a duplicate is confirmed, 301 the source to the canonical URL (not a sorted or filtered view; a 302 tells search engines the move is temporary, so signals don't consolidate). Remove the source from the sitemap, list pages and related-post widgets so nothing on the site links to a redirect. Leave serialised content alone and link episodes to each other with previous/next links instead, since each episode has its own search demand. And stop new duplicates at the source by having editors check before publishing. Google's guide to consolidating duplicate URLs↗ lists redirects as the strongest of the canonicalisation signals, which is why confirmed duplicates get a 301 rather than just a canonical tag.

Putting the freed budget to work

Once pruning frees crawl budget, add same-topic internal links into tier A and link only tier A tags from the homepage, or the budget is just spread across what remains.

Pruning is half the job. If you don't make tier A easier to reach, the crawl budget you freed just gets spread across whatever's left.

Our approach has two parts. First, add internal links into tier A from articles on the same topic, pairing them by shared person or event tags, and switch related posts to "same tag first" instead of random or purely chronological. Second, add a "popular tags" block on the homepage and category pages that links to tier A tags only. Homepage links are the heaviest links on the site. Spending them on zero-impression tags wastes the best real estate you have. If your large site is template-driven, the same logic applies to generated pages, which we cover in our programmatic SEO playbook.

Reporting the result honestly

A pruning report puts the data change first and the verdict last, labels each cause by confidence, and checks every status with curl against the live site.

Our reports follow a fixed order: what changed in the data, how wide the change is, which pages, which queries, what the pages look like now, and only then the verdict. Causes get one of four labels: confirmed (technical, data or page evidence), likely (several signals agree), unverified (correlation only) or ruled out. "Algorithm volatility" with no evidence behind it doesn't make the report.

Verify against the live site. Whether the noindex is really on the page, whether the redirect is a 301 and not a 302, whether the source URL is gone from the sitemap: check each with curl against production. Search Console, Bing and Yandex reports lag by days to weeks and will mislead you about the present. If only one engine misbehaves afterwards, investigate that engine and leave the site-wide rules alone. "No change for now" is a valid decision too, as long as you write down why, what you'll watch, for how long, and what would trigger action. The 1,350 new C2 posts got exactly that treatment. Bing's webmaster guidelines↗ take a similar line on thin and duplicate content, so if Bing sends you meaningful traffic, include it in what you monitor. If you work in iGaming, our iGaming SEO guide covers the wider picture.

If you're running a site with tens of thousands of pages and want to know how many sit in each tier before deciding anything, that inventory is the first thing we do in our SEO site build and management service. Scope for each plan is on the pricing page.

This is one chapter of our large-site SEO guide, which maps all six workstreams and the order to tackle them in your first week.

FAQ

Do I have to delete pages to prune content?

No, and on a large site you shouldn't. Mass 404s or 410s are a negative signal, and if one of the deleted pages was still ranking, you can't get it back. We noindex for Googlebot only, by ID list, so the page stays available to users and other search engines.

Why 16 months of data instead of 3?

Sixteen months is the longest range Search Console gives you. A short window flags seasonal or cyclical pages as dead, and you end up pruning pages that come back every year.

Can I noindex every page with zero Google impressions?

Split them first. In our case, 4,977 of 9,528 zero-impression posts were getting traffic from other search engines or mirror domains. Treating them all the same would have cut that traffic. The real candidates are old posts with no traffic anywhere.

Are pages with near-identical titles duplicate content?

Not necessarily. Of 50 title-matched sets we checked, about 6 were real duplicates. On a video site the merge test is an identical video hash. On a text site you need equivalent content-level proof.

How soon will pruning show up in traffic?

Pruning doesn't add traffic directly. It frees crawl budget. Whether that turns into clicks depends on whether you point the freed budget at tier A and whether those pages are good enough to rank. That's why our reports list metrics and an observation window instead of a traffic forecast. Which one does your pruning report show?

---

Explore our advertising strategy & media buying services →