D
P
0
← All articles Baca dalam Bahasa Indonesia

Technical SEO, Redirects & Migrations

18,000 `?range=` URLs Still Getting Crawled With Correct Canonicals? Canonical Doesn't Save Crawl Budget

Written by · · 7 min read
18,000 `?range=` URLs Still Getting Crawled With Correct Canonicals? Canonical Doesn't Save Crawl Budget

The sitemap was clean, the status codes were right, canonicals were in place everywhere. The content site runs on Next.js and I built it myself, so when I pointed an SEO crawler at it, the only thing I expected to read was a short list of mistyped internal links.

What the crawler returned was not a short list.

The symptom: 18,000 URLs I never meant to create

The crawler found roughly 18,000 crawlable URLs matching a ?range= pattern, plus about 300 pages matching ?cat=. Every one of them was reported as a legitimate URL worth visiting.

/detail/aaa?range=24h
/detail/aaa?range=7d
/detail/aaa?range=30d
/detail/bbb?range=24h
/detail/bbb?range=7d
...

Finding the source took five minutes. Detail pages had a time range switcher for their chart, and I had built that switcher out of real anchors instead of buttons:

{ranges.map((r) => (
  <Link key={r} href={`?range=${r}`} scroll={false}>
    {r}
  </Link>
))}

At the time that felt like the correct decision. Real anchors mean the selected range is shareable, opens in a new tab, and still works without JavaScript. What I had not counted was that every one of those anchors is, to a crawler, a new URL. Multiply the number of detail pages by the number of range options and the count jumps into five figures immediately.

The confusing part was not the number. The confusing part was that every one of those variants already had a correct canonical.

Why I thought I was covered

Each detail page emitted a canonical pointing at its clean, parameterless version:

<link rel="canonical" href="https://your-site.com/detail/aaa" />

So my reasoning went: the canonical is correct, therefore Google knows which version is real, therefore the parameter variants are not a problem. Case closed.

That reasoning was correct for the wrong question. The canonical was saving my index. What it was not saving was my crawl budget, and those are two separate budgets.

The root cause: canonical and robots answer different questions

This is the whole case, and it took an 18,000 URL audit before it really landed:

And here is the trap that makes people ship both and then wonder why nothing improved: the two do not stack. If you Disallow a URL, the crawler never fetches it, which means it never reads the canonical tag inside it. A canonical that is never read is the same as no canonical.

So for any single URL, you are choosing between:

There is no third option that gives you both. The only thing to decide is which cost is higher for this class of URL.

Why it only bites at a certain scale

On a small site this debate is academic. A hundred parameter variants will never starve anyone's crawl budget.

What changes on a large content site is cardinality. Faceted parameters attach to pages that are already numerous, and they multiply rather than add. One parameter with three options turns a few thousand pages into tens of thousands of URLs. Add a second parameter on those same pages and the combinations multiply again, and a six figure URL count becomes ordinary without a single new page ever being written.

At that point the crawler spends its visit allowance reading thousands of copies of the same page with a different chart window, while the genuinely new pages queue up behind them. Those variant pages do stay out of the index. They have simply already charged you before that happens.

The fix: block the patterns in app/robots.ts

Since these parameter variants have no index value whatsoever, the choice was easy. I was happy to give up the canonical being read. What I wanted was the crawling to stop.

Before writing anything, one mandatory check: make sure no legitimate path on the site depends on those parameter names. If even one route needs range or cat to render content that has to be indexed, this rule will bury it.

grep -rn -A3 "searchParams" app/ | grep -E "range|cat"

The -A3 is not padding. The parameter name rarely sits on the same line as searchParams; it usually shows up in the destructuring or the type annotation a line or two below. Without those context lines the grep comes back empty and you feel safe without having checked anything.

On this site the only hits were the chart switcher and a category filter on a listing page that already had a parameterless version. Safe. Only then did I ship the rules:

// app/robots.ts
import type { MetadataRoute } from "next";
 
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: "*",
        allow: "/",
        disallow: ["/*range=", "/*cat="],
      },
    ],
    sitemap: "https://your-site.com/sitemap.xml",
  };
}

The leading wildcard matters. Disallow: /*range= matches that parameter at any path depth, not just at the root, and that is exactly what you need when the switcher lives on detail pages.

Verification is simple, and do it against production, not locally:

curl -s https://your-site.com/robots.txt

If both Disallow lines actually appear in the output, you are done. Clean pages remain crawlable as usual because allow: "/" still applies, and only the parameterised variants are cut off.

The better fix: don't create the URLs in the first place

Blocking a pattern in robots.txt is treatment, not a cure. The URLs still exist. I have only told the crawler not to come.

The cleaner approach is never producing those URLs at all. If the state is purely a display convenience and does not change indexable content, move it into a fragment. Fragments never become separate URLs to a crawler:

const [range, setRange] = useState("24h");
 
useEffect(() => {
  const fromHash = window.location.hash.slice(1);
  if (ranges.includes(fromHash)) setRange(fromHash);
}, []);
 
{ranges.map((r) => (
  <button
    key={r}
    type="button"
    aria-pressed={r === range}
    onClick={() => {
      setRange(r);
      history.replaceState(null, "", `#${r}`);
    }}
  >
    {r}
  </button>
))}

You still get a link you can copy and share, #24h rides along in the address bar, but the crawler does not see 18,000 pages. It sees one.

The useEffect above is what makes a copied link actually restore the range, and that restore has to happen after mount. A fragment is never sent to the server, so reading it in the initial state makes the server render disagree with the client render and you get a hydration mismatch. One more caveat so you are not surprised: replaceState deliberately does not add a history entry, so the back button will not step through each range change. Use pushState if that is the behaviour you want.

The price is explicit too: this button version needs JavaScript, which the anchor version did not. For state that only changes the view, that is a fair trade. For state that changes indexable content, do not move it into a fragment at all.

The difference is what each form declares. A ?range= anchor tells the crawler "this is a different document". A #24h fragment tells it "same document, different view". For a chart switcher, the second sentence was the true one all along.

What I took away