Large Site SEO: Six Traits, Six Rules, and a Working Playbook

TL;DR: On a site with tens of thousands of URLs, SEO↗ stops being about clever tactics and becomes a question of scale. You cannot look at pages one by one, so every decision has to be a rule you can run in bulk. There is more than one domain, more than one search engine and more than one team touching the site, so every conclusion has to be split apart and checked. This guide covers the six traits we kept running into on large content sites, the six rules we now refuse to break, and a short summary of each part of the method with a link to the full write-up.

What makes a site "large" for SEO purposes

We call a site large for SEO when it has more URLs than anyone can review by hand, and that scale brings six recurring traits.

Traffic has nothing to do with it. A site is large, for our purposes, when it has more URLs than anyone can review by hand.

Between September and October 2026 we worked on several large content sites at the same time. The smallest was a video content site with a little over 10,000 posts. Another was a content site with 80,000 articles. A third, a UGC site, had 458,000 URLs listed across its sitemaps. Different topics, different CMSs, different teams. The problems were almost identical, and they grouped into six traits:

URL count is the root of everything else. Google's own guide to managing crawl budget on large sites↗ is blunt about it: past a certain size, not every page gets crawled and not every crawled page gets indexed. A low index rate is the normal state of a large site, not an incident. The work is deciding which pages deserve the crawl, and getting the rest out of the way. Any plan that starts with "let's review each article" is dead on arrival.

Mirror domains are unusual. They usually exist on purpose, for users who cannot reach the main domain, so they are not technical debt you can simply delete. But they split traffic and indexing, and if you do not break GA4↗ down by hostname the numbers can point the wrong way entirely. On one site in late September 2026, the network-wide GA4 total was down 20%. The main domain on its own was up 57.5%. On another site in mid-October 2026, Google traffic moved from the main domain to a CDN hostname, whose sessions went from 708 to 1,693. Hostname is a standard dimension in GA4↗. Almost nobody splits by it first.

Several search engines is the trait English-language teams miss most often. One site we took over got 85% of its organic traffic from Yandex, and some of the mirrors had meaningful Baidu and Yandex traffic. Running that site from Search Console alone means reading a small slice of the picture and acting on it as if it were the whole.

Tag pages are where things get out of hand. On the 80,000-article site, the median article carried 13 tags, the top 10% carried 18, and 8,131 tag pages had impressions. Tag pages were the bulk of non-brand traffic, and also the biggest single risk.

Bulk publishing means errors arrive in batches. When each writer publishes 30 posts a day and several writers run in parallel, a single template mistake lands on dozens of pages before anyone opens one.

The last trait gets underestimated the most. Somebody else ships the changes. SEO writes the request. Developers and the content team schedule it. That has a very concrete consequence: work marked "done" drifts back. On one site, a noindex on page 2 of the paginated archive was completed in February 2026 and was back to index by October. On another, VideoObject structured data finished in March was gone by October.

Put the six together and you get three requirements for any method that hopes to work here: rules come before individual cases, evidence comes before conclusions, and verification never ends.

The six rules everything else depends on

Large-site SEO rests on six rules: data first, graded causes, absolute numbers, single-engine fixes, "no change for now" as a valid decision, and production as the source of truth.

We have broken every one of these at some point, and paid for it each time. They sit at the top for that reason.

1. Data first, conclusion last

The order is fixed: what changed, how far it spread, which pages, which queries, what those pages look like right now, and only then a judgement. Large sites produce enough data to support any story you already believe. If the conclusion comes first, you will always find a chart that agrees with it.

2. Grade every cause

Every cause in a report gets one of four labels:

"Algorithm fluctuation" and "content not deep enough" are banned. They are not causes, they are a way of giving up on finding one. Grading forces the writer to be honest about how much they actually know, and tells the reader which items are safe to act on.

3. Look at absolute numbers, not only percentages

Search result pages down 50% might be 500 lost clicks. Article pages down 10% might be 20,000. Percentages make small page types look dramatic. Absolute numbers tell you where to start.

4. One engine misbehaving gets a fix for that engine only

We hit this twice in a row. In September 2026, one site returned 5xx errors to Bing while every other engine was fine. In October 2026, another site lost 38% of organic traffic, and 85% of the loss was Yandex. Both times we investigated only that engine and left site-wide structure alone. Restructuring a whole site to fix one engine is a reliable way to damage the two that were working.

5. "No change for now" is a valid decision

Not every dip needs action. But "hold" has to come with four things written down: why you are not changing anything, which metric you are watching, for how long, and what would trigger a change. Without those four, it is just inaction with a nicer name.

6. Production is the source of truth

Status codes, robots directives, canonicals and sitemaps get checked with curl against the live site. Search Console, Bing Webmaster Tools and Yandex Webmaster all lag by days to weeks. Use them for trends. Do not use them to describe what the site looks like today.

Index management: decide what deserves to be indexed

Index management means using a page-type table to decide which pages get indexed first, then enforcing it with four sitemap rules and a staged approach to mirror domains.

Indexing is the foundation. Once crawl budget is the constraint, the question stops being "how do I get everything indexed" and becomes "how do I get the pages worth ranking indexed first". We use a page-type table: home, category pages up to page 5, and articles are index and go in the sitemap. Deeper pagination, tag pagination, author archives, date archives and internal search results are noindex, follow and stay out of the sitemap. A tag page is indexable only once it has 3 or more articles.

Sitemaps follow four rules, each learned the hard way. List only URLs that return 200 and are indexable: one site had 5 URLs in its sitemap that redirected to the homepage, and Google stopped reading that sitemap from March 2026 until we resubmitted it in October. Make sure Google knows about every child sitemap: one site had 46 child sitemaps covering 458,000 URLs, only 9 were submitted, and Google knew of roughly 100,000 URLs. Check the "last read" date daily and investigate anything older than 3 days. Put old pages with traffic back: one site's sitemap only included articles above a certain ID, leaving out 517 older posts that still had impressions. The format rules are in the sitemaps.org protocol↗, Google's notes on large sitemaps↗, and Yandex's sitemap documentation↗.

For mirrors, ask the business before picking a tool. From lightest to heaviest: page-level canonicals pointing back to the matching main-domain URL. An X-Robots-Tag: noindex header so users can still reach the mirror but it stays out of the index. And a full 301 only once you know nobody needs the mirror. If you want Google alone to drop a set of pages, <meta name="googlebot" content="noindex, follow"> does that without touching Bing, Yandex or Baidu, as described in Google's robots meta tag reference↗. The full table, canonical and redirect rules, and how we run the mirror decision are in large site indexing and sitemaps.

Content pruning: give crawl budget to pages that earn it

Content pruning splits pages into tiers A, B and C using 16 months of data and acts only on old pages with no traffic. It frees crawl budget but does not bring traffic itself.

Pruning starts with tiers built from 16 months of data, not gut feel. Tier A is the core: anything with clicks in the last 90 days or inside the top 80% of traffic. It goes on a protected list, with no URL changes, no noindex and no merging. Tier B has some clicks over 16 months and is kept. Tier C had zero Google impressions in 16 months and gets split again: pages with traffic from other engines or mirrors stay untouched, old pages with nothing anywhere become candidates for a Google-only noindex, and new pages wait. On a video content site with just over 10,000 posts, Tier C held 9,528 pages: 4,977 left alone, 3,105 old posts shortlisted, 1,350 new posts put on watch.

The red lines: no deleting articles, no mass 410 or 404 responses (a wave of error pages is a negative signal in itself), and no blanket noindex. If you do it, do it by ID list and for Google only. Be explicit in the report that pruning frees crawl budget and does not bring traffic on its own. When we sampled Tier C, about half the pages were indexed but had never earned an impression. They were not missing from the index. They simply did not rank.

Merging duplicates needs hard evidence. We merge only when the video hash is identical, never on similar titles. Titles mislead: different weekly charts, different parts of the same series, and different clips that share a headline all look the same. Out of 50 tag-matched pairs we checked, only about 6 were genuinely the same topic. Google's guide to consolidating duplicate URLs↗ covers the mechanics. Thresholds, link cleanup after a merge, and why serialized content is never merged are in large site content pruning.

Tag pages: most of your non-brand traffic and most of your risk

Tag page governance means protecting the core tags that carry 80% of traffic, merging variants, fixing the noindex rule, and capping new posts at 6 tags.

On the 80,000-article site, daily clicks to tag pages fell from 98,570 to 20,977, a 79% drop. The shape of the problem was easy to read once we pulled every tag: a median of 13 tags per post, 84% of tags attached to a single article, 8,015 tag pages sharing one title template, 78% of descriptions under 70 characters, and 208 core terms spread across 2,904 variant tags. Taken together, that profile sits uncomfortably close to the doorway and scaled low-value content patterns in Google's spam policies↗.

The order we fixed it in: lock the core tags that carry the top 80% of traffic (697 on that site) and only add content to them, meaning a custom title, a 140 to 160 character description, a 100 to 200 word intro and related-tag links. Then merge variants, 301 for those with traffic and removal plus noindex for those without. Then fix the noindex rule itself, noindex pagination from page 6, strip full-sentence tags and privacy-sensitive tags from articles, and cap new posts at 6 tags drawn from existing ones.

The noindex rule is where most teams go wrong. "Noindex any tag with only one page" sounds sensible, but it counts pages, not articles. In our October 2026 audit, that one rule had put noindex on 86 tags that had each earned 1,000 or more clicks over 16 months. Reading the drop correctly mattered too: we ran URL Inspection on the top 40 core tags and all 40 were still indexed. They had lost rankings, not been removed, so the fix was about quality rather than technical SEO. The audit metrics and how to read a control site are in tag page SEO audits. If your tag pages are generated from templates, our programmatic SEO playbook covers the same risk from the build side.

Diagnosing a traffic drop without fooling yourself

Diagnosing a traffic drop takes the same eight steps every time, from sizing the drop to splitting by hostname and engine, after ruling out things that only look like problems.

We run the same eight steps every time: size the drop, scope it, check the baseline, split by hostname, rule out technical failure, check indexing, split by engine, then write the conclusion. Size has thresholds. Within 5% we watch. Between 5% and 10% we check sources and the pages that fell. Above 10% gets a dedicated investigation. Three to seven days of decline in a row gets a look even when the total is small.

Checking the baseline saves the most time. In mid-October 2026 one site looked down 27.9% week over week. Pulling 8 weeks of daily data showed that the previous week had been the best in 10 weeks, and a national holiday had inflated it further. The drop was real but much smaller than it looked, and nothing drastic was needed.

Query-level movement has a reading guide too. Impressions down with stable rankings means demand fell. Both down points at page competitiveness or indexing. Stable impressions with falling CTR means titles, snippets or the SERP layout changed. Only a few core queries dropping suggests cannibalization or replacement. Many queries dropping together means a shared cause (templates, canonicals, internal links, indexing) that you fix once, not query by query. That lines up with Google's guide to debugging search traffic drops↗. The hostname and per-engine splits are the extra steps a multi-domain site needs.

Some things look like problems and are not. An audit reporting "50 new issues" may simply have sampled a different set of pages. New posts unknown to Google for a day or two is normal. Video stream URLs returning 400 to non-browser clients is expected when the CDN re-signs them on every render. One engine reporting 5xx while the others are fine is that engine's problem. The full checklist is in SEO traffic drop diagnosis.

When SEO cannot touch the code: workflow, monitoring and bulk content

When SEO cannot touch the code, changes survive other teams through a fixed ticket format, daily and weekly monitoring, and a 12-item quality gate for bulk publishing.

Tickets that survive other teams

On large sites, SEO mostly writes requirements. Our ticket format never changes: the request, example URLs, current state with source code, the problem, the expected result, exceptions, priority from P0 to P3, and acceptance criteria. Verify before filing: sample at least 50 pages and look at the distribution rather than judging from two. Numbers that are suspiciously uniform, such as 60 pages whose descriptions are all exactly 130 characters, point to a template setting and belong with developers, not writers. Staging review compares titles, descriptions, H1s and robots word for word. One batch of 13 items had only 8 pass. We file one batch every 2 weeks with 2 to 4 items each, and everything else waits in a backlog.

Monitoring runs daily and weekly. Daily covers 6 checks: Search Console traffic, indexing and crawl, Bing, Yandex, GA4 split by hostname, and a live-site check plus a re-check of old tickets. Mondays compare against the baseline and add to the running index sample. One rule matters more than the rest: a sample result describes the sample. We once checked 15 URLs on a site and only the homepage was indexed. The full index report said 15.3k indexed and 88.2k not indexed. Samples find problems. Reported numbers come from full data. The ticket template, cause grading and monitoring sheet are in large site SEO workflow.

A quality gate for bulk publishing

Bulk publishing errors are template errors, so quality depends on a fixed pre-publish check. Ours has 12 items, generalized here: correct channel, title matches body, no leftover brand names from the source, no generation markers, no off-topic junk at the end, no orphan headings, no repeated sentences, no reference-style image syntax, no dead image hosts, no broken syntax on the first image, no filenames as ALT text, and video URLs stored as relative paths only. After publishing, open the page in a real browser, wait 5 seconds, scroll to the bottom, and confirm the image count, no raw markup and a video that actually plays.

The most useful lesson: any auto-fix script needs a lint pass that runs after the fix, and a single hit blocks publishing. One of our fix scripts reported 30 out of 30 clean, and manual review still caught 2 bad posts. Google's guidance on generative AI content↗ makes the same point from the other side: how a page was produced matters less than whether it helps the reader. The full checklist and front-end review are in AI bulk content quality gates.

Your first week on a large site

In your first week on a large site, confirm the data is right, then contain problems that spread, and only then optimize structure, in this order:

1. Day one, audit access: docs, Search Console, GA4, Bing, Yandex. Request everything missing in one go.
2. Read index management, compare the site to the page-type table and check every sitemap's last-read date.
3. Read traffic drop diagnosis and split GA4 by hostname so you know the trend you are looking at is real.
4. If tag pages get impressions, read tag page audits. If there are many articles and indexing is low, read content pruning.
5. If the site publishes in bulk every day, put the quality gate in place before any other optimization.
6. Before filing your first ticket, read the workflow guide and check whether the issue was raised before, and whether "done" items are still done.

The logic is simple: confirm your data is right, stop problems that keep spreading, then work on structure. Bing's baseline expectations are in its Webmaster Guidelines↗ if you want a quick cross-check. If you would rather have someone run all of this on your site, from the access audit to weekly reports, that is what our SEO site management service does, and pricing is public.

FAQ

How many URLs makes a site "large"?

There is no fixed number. The practical test is whether you can still review pages individually. The sites we worked on ranged from just over 10,000 posts to 458,000 sitemap URLs. What they shared was a crawl budget that ran out, a naturally low index rate, and the need to turn every fix into a rule that runs in bulk.

My index rate is low. Should I delete the pages that are not indexed?

No. Mass deletion or mass 404 and 410 responses send a negative signal of their own. Tier the pages first: protect anything with clicks, check zero-impression pages for traffic from other engines or mirrors, and only then consider a Google-only noindex for old pages with no traffic anywhere. Pruning frees crawl budget. It does not create traffic.

Only Yandex (or only Bing) dropped. Should I change the whole site?

No. A single-engine problem gets a single-engine investigation: searchable page counts, exclusion reasons and crawl errors for that engine. Site-wide changes risk the traffic from engines that were working fine.

Can I use Search Console to tell what the site looks like right now?

Only for trends. Search Console, Bing and Yandex reports lag by days to weeks. Check status codes, robots, canonicals and sitemaps with curl against production, and report numbers from full index reports rather than samples.

Do mirror domains have to redirect to the main domain?

Not necessarily. Mirrors often exist for users who cannot reach the main site. Ask the business first, then go from light to heavy: page-level canonicals, an X-Robots-Tag noindex header, and a full 301 only when the mirror is no longer needed. Before any bulk noindex, make sure the tag will not also appear on the mirrors, or their traffic from other engines will fall with it.

---

Explore our advertising strategy & media buying services →