D
P
0
← All articles Baca dalam Bahasa Indonesia

Technical SEO, Redirects & Migrations

Canonical and `<title>` Landing in `<body>` on Next.js 16? Googlebot Is Not on the `htmlLimitedBots` List

Written by · · 12 min read
Canonical and `<title>` Landing in `<body>` on Next.js 16? Googlebot Is Not on the `htmlLimitedBots` List

While the homepage meta in this project was a static export const metadata, its title and canonical always sat inside <head>. On 14 September 2026 I moved that meta into the CMS so the client's SEO team could edit it themselves, and from then on the homepage generateMetadata waited on a query for the homepage settings document. The project runs Next.js 16, and that small change turned out to be enough to push the title, description, canonical and Open Graph tags into the body. The leak did not happen on every request. It showed up on one particular request, and you will not run into that request if what you check is a page that is already warm.

Only the first request after a webhook

The pattern is narrow. Every time a publish webhook came in, whether the published document was an article or the homepage settings document itself, the first homepage request afterwards carried the title, description, canonical and Open Graph tags about 40 KB into the body. That was for the Googlebot user agent. The next request, with the cache warm, was clean again. Articles on this site publish several times a day, so my estimate was that, left alone, the leak would repeat every day.

I measured that 40 KB on a local production build, next build followed by next start, with a signed webhook I sent myself. How to repeat it is in the measuring section below.

Googlebot is not on the default list

Next 16 streams metadata to every user agent that does not match htmlLimitedBots, and Googlebot is not on its default list. When the config change I describe further down was made on 15 September, the project's package.json still pinned Next 16.2.4. The Next code below is from 16.3.6, the version installed in that repo now, and in that version the logic reads plainly:

// next/dist/server/lib/streaming-metadata.js (Next 16.3.6), trimmed
function shouldServeStreamingMetadata(userAgent, htmlLimitedBots) {
    const pattern = htmlLimitedBots || _isbot.HTML_LIMITED_BOT_UA_RE_STRING;
    // ...
    // Only block metadata for HTML-limited bots
    if (userAgent && cachedRegex.test(userAgent)) {
        return false;
    }
    return true;
}
 
// next/dist/shared/lib/router/utils/html-bots.js (Next 16.3.6)
const HTML_LIMITED_BOT_UA_RE = /[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight/i;
 
// next/dist/shared/lib/router/utils/is-bot.js (Next 16.3.6), trimmed
// Bot crawler that will spin up a headless browser and execute JS.
// Only the main Googlebot search crawler executes JavaScript, not other Google crawlers.
const HEADLESS_BROWSER_BOT_UA_RE = /Googlebot(?!-)|Googlebot$/i;

That function returns false, meaning the metadata renders blocking and is resolved before the shell goes out, only for user agents that match the pattern. Everyone else gets streamed metadata. The word Googlebot does not appear in the default list, and neither does Screaming Frog. Googlebot is instead recognized through a separate regex, with a comment calling it a crawler that spins up a headless browser and executes JavaScript.

The trouble is that not everything that reads HTML runs JavaScript. A comment I left in the homepage code records that worry: a crawler that does not run JavaScript, such as Screaming Frog, which the client's SEO team audits with, would read a homepage with no title.

Revalidation makes metadata lose the race

Streaming alone does not move the tags. Streamed metadata still lands in <head> if it resolves before the shell flushes, that is, before the first chunk of HTML containing the head is sent. With a warm cache, the homepage meta query finished in time. The webhook is what changed the order: revalidateTag(type, "max") plus revalidatePath("/") makes the next render wait on the network, and the metadata loses the race. In short form, and with the set renamed, the webhook route looked something like this:

// webhook route, simplified; set renamed
revalidateTag(body._type, "max");
 
// ...
 
if (!TYPES_NOT_ON_HOMEPAGE.has(body._type)) {
  revalidatePath("/");
}

Every document type that is not excluded also refreshes the homepage, so the first homepage request after a webhook is the one that has to wait on the network. The same mechanism explains part of the first three items in the SEO audit the client's SEO team wrote in May: any dynamic route with async metadata leaks on the first render after its path is revalidated, not only an archive page that happens to be cold.

If you have read my piece about notFound() still producing HTTP 200, this case is related, but the problem is a different one. There, the status was already sent before the page knew its answer. Here the status is not the problem. Every tag is present, just in the wrong place. For a canonical, the place is not cosmetic, since Google's documentation says rel="canonical" is only accepted when it appears in <head>.

Measuring without being fooled by a warm page

The rule I wrote down from this: never judge metadata placement from a warm page. The sequence is next build, next start, request as Googlebot, fire a signed webhook, request again, then compare the offset of <title> with the offset of </head>.

In this project the webhook secret in .env.local is empty, so I passed one through the process environment when starting the server. The Cloudflare credentials got dummy values so the webhook route could not purge production Cloudflare. The signature has the form t=<ms>,v1=<base64url HMAC-SHA256(secret, "<t>.<body>")> in a header whose name depends on the CMS, and the route waits about 3 seconds for eventual consistency. If you want to repeat that sequence in your own project, a simple curl script like this is enough:

# simplified; variable names, header, and webhook URL here are neutral.
# The server is already running via next build then next start, with the same secret.
WEBHOOK_SECRET="local-secret"
WEBHOOK_URL="http://localhost:3000/..."   # your revalidation webhook route
SIGNATURE_HEADER="..."                     # your CMS's signature header name
GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
 
offset() {
  curl -s -A "$GOOGLEBOT" http://localhost:3000/ \
    | grep -Ebo '<title>|</head>|rel="canonical"'
}
 
offset   # baseline request
 
T=$(( $(date +%s) * 1000 ))
BODY='{"_type":"post"}'   # match your CMS's webhook payload
SIG=$(printf '%s.%s' "$T" "$BODY" \
  | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -binary \
  | base64 | tr '+/' '-_' | tr -d '=')
curl -s -X POST "$WEBHOOK_URL" \
  -H 'content-type: application/json' \
  -H "$SIGNATURE_HEADER: t=$T,v1=$SIG" \
  -d "$BODY"
 
offset   # first request after revalidation, the one that matters

With -o, grep -b prints the byte offset of each match. If the number in front of <title> is larger than the number in front of </head>, that title is in the body.

The homepage first: answer from memory

The first fix covered only the homepage and reached production the same day. generateMetadata no longer waits on the query. It returns a value kept in memory per server process and refreshes it with after(). Here is a short form, with the function names I changed:

// simplified, names changed
// refresh() fetches the meta from the CMS and stores it in latest
let latest: HomeMeta | null = null;
 
export async function readHomeMeta(): Promise<HomeMeta> {
  if (!latest) return refresh(); // a process's first request waits
  after(refresh);                // the refresh runs after the response
  return latest;                 // last value, returned at once
}

The comments in the original module sum up the trade. The last value is kept per process and returned at once, so it always beats the shell, while the refresh runs after the response. The price is that a CMS edit is not visible on the request that triggers the refresh, only on the one after it. Only the very first homepage request a process serves still waits on the query, and that is where the remaining risk sits: the first request of a freshly started process can still leak. A memo like this has its own trap around draft mode, which I wrote up separately.

On the local build after the change, 24 of 24 requests following 6 webhooks had the metadata in head. The release went live in production at 08:34Z, and the check there found the homepage meta inside head on 12 of 12 requests.

Asset detail pages leaked on every render

The homepage was handled, but other dynamic routes could still put the title and canonical outside head on the first render after a revalidation. The sitewide fix for that is htmlLimitedBots in next.config, and it changes SEO behavior on every page. A call like that belongs to the client's SEO team, not to me, so on 15 September I logged this item without acting on it.

That same day the client's SEO team asked for the fix, after their crawl flagged the canonical outside head on two asset detail page templates. The problem turned out not to be new. It was already recorded as the first item of their SEO audit in May.

Measurements on production on 15 September showed a different scale from the homepage. On both templates, the canonical and title landed in the body on every render. The head closed at about 2.2 KB, while the canonical only showed up between 45 and 106 KB, for every user agent I tried and in every cache state. generateMetadata there waits on the asset data, the routes stream, and Googlebot is not on the default list. Eight other templates I measured, including the homepage, articles and categories, had both in head.

A custom pattern replaces the default list

The first option I offered was a bot list, htmlLimitedBots: /Googlebot|bingbot|.../, so crawlers would get blocking metadata while readers kept streaming. What finally shipped matches every user agent instead. The reason sits in the first line of shouldServeStreamingMetadata above: the pattern from the config is used in place of the default list, not on top of it. Once you write your own pattern, every name on the default list that you do not rewrite, such as facebookexternalhit, Twitterbot or Slackbot, stops getting blocking metadata. Matching everything becomes the most predictable choice.

// next.config.ts, trimmed
const nextConfig: NextConfig = {
  // Metadata is rendered blocking for every request, so <title>, canonical
  // and robots are always inside <head>.
  htmlLimitedBots: /.*/,
};

The commit changed only that file, with 12 added lines. What gets given up is streamed metadata for readers, and the commit message gives the reason: the body on those routes waits on the same cached data anyway, so readers lose nothing measurable. On the local build after the change, the asset page TTFB was 0.14 s, and the canonical and title sat in head on both asset templates, the homepage, an article and one other template, for both Googlebot and Chrome. Unknown slugs still answer 200 with the not-found body, so the status is unchanged and no new hard-404 risk appears when the CMS has a blip.

Cloudflare still held the leaking copies

The change was live in production at 08:49Z on 15 September. A check with cache-busters as Googlebot, Screaming Frog and Chrome found the canonical and title inside head on two pages from the first asset template, one from the second, the homepage and an article. Measurement notes taken after the change record the same for every template and all three user agents.

The cache-busters were not a formality. Cloudflare still held the old body-leaking copies of the asset pages, with a 30-minute TTL. I purged by URL the top 15 pages of the first template, 7 of the second and the homepage, 23 URLs in all, and let the long tail refresh by itself within 30 minutes. The note I wrote for next time: the CDN keeps the leaking copies until their TTL runs out, so purge the URLs the SEO team will re-check.

The May audit item was closed. The homepage after() memo can stay and is now a second safety layer, because with blocking metadata on every request, even the homepage's first request after a webhook can no longer leak its title into the body.

What showed up later, Search Console included

My own check on 23 September found the homepage title and description inside head on 5 of 5 fetches with the Googlebot user agent, with the canonical pointing at the main domain.

On 24 September, URL Inspection in Search Console for a URL crawled on 9 July showed "user-declared canonical: none". The explanation I noted at the time was that until the 15 September change, metadata streamed into the body for Googlebot, and that crawl happened well before it. In live tests the same day, the fetch succeeded, the result was reported as indexable and the canonical was read.

One thing makes this tool worth reading carefully. URL Inspection uses the Google-InspectionTool user agent, and that name matches the Google-[\w-]+ pattern in Next's default list, while Googlebot does not. My suspicion is that a URL Inspection live test can show a clean head while Googlebot itself receives streamed metadata. This is not something I have measured, so treat it as a warning. If the suspicion is right, a clean live test says little about what Googlebot received before the fix.

What I do not have yet is evidence of the effect on the index. There is no count of pages indexed thanks to this change, and no proof that Google now picks the declared canonical. What is proven so far is only that the HTML served to Googlebot carries those tags in the right place.

Before concluding the metadata is in head

If generateMetadata in your Next 16 project waits on data, these four notes are what I took from the case before trusting that the tags are where they belong.