On the last day of June this year, every article page with a body on a site I maintain stopped showing its content and rendered a not-found UI instead. The homepage was fine. Every listing page was fine. Only the per-slug detail pages went down, and they went down together, with no survivors.
The site runs on a headless CMS, Next.js, and ISR. All five of its dynamic per-slug routes return HTTP 200 for a slug that does not exist while rendering the not-found UI. So what I was looking at in the browser was not a real 404, it was a "not found" page that told the protocol everything was fine. Those routes are force-dynamic and render through streaming, and the 200 is already committed before notFound() gets a word in. That is its own story, one I wrote up separately, and the property becomes relevant again further down.
I spent about an hour suspecting the latest deploy. That hour was wasted.
Why the listings still looked healthy
The part that misled me was the part that was not broken. The listing and static pages on that site are served as static and ISR, which means their data was cached at build time. They never touch the CMS on an incoming request. As long as that cache exists, they keep serving titles, excerpts, and images perfectly even when the runtime fetch path is completely severed. Those pages were not evidence that the data was fine, they were a recording of the last healthy state.
The only pages telling the truth were the force-dynamic per-slug ones, because they are the only ones doing a live fetch on every visit. So the failure pattern was not a coincidence. The single group of pages that failed was the single group of pages still asking the CMS anything in real time.
The null that came back too fast
The CMS fetch on that site has a five second timeout. If the CMS were slow or the network were struggling, I would expect to wait out that limit before getting a failure. The opposite happened: the null came back in one to two seconds. Even in its slowest case that is still three seconds faster than the timeout, because five minus two is three.
That speed was a clue, and I read it wrong. Quota errors return fast. The server was not straining to serve my query and running out of breath, it was refusing me at the door.
The trouble is that from the caller's side, a fast refusal looks identical to "this document genuinely does not exist". The CMS fetch helper returns null for two completely different conditions, a document that truly is absent and a fetch failure that is only transient.
const doc = await cmsFetch(query, { slug })
// null here means two things at once:
// 1. the document genuinely does not exist
// 2. the fetch failed transiently, quota included
if (!doc) notFound()One value, two meanings, and the page picked the wrong one.
The anonymous bucket that was still alive
The check that finally turned the diagnosis around was simple. I pulled the same document through the anonymous CDN endpoint, without the application token.
curl -s "https://cdn.example-cms.example/api/query?query=..." | head -c 300The document was there, complete, and fast. The data was healthy, the CDN was healthy, the query was correct. Exactly one path was unhealthy, the application's tokened fetches against the API endpoint. The tokened API bucket is separate from the anonymous CDN bucket. One can be quota-blocked while the other keeps serving as if nothing happened.
That is what makes an incident like this hard to believe while it is happening. I could pull the raw data with a single curl while the application asking for the exact same document got nothing but null.
The deploy exposed it, it did not cause it
For that first hour, my prime suspect was the latest deploy. On the surface the reasoning holds up, the symptom appeared close to a deploy and the only thing that had changed was code.
What I had not accounted for is that a deploy does one other thing that has nothing to do with code, it clears the warm cache. While that cache stays warm, pages keep looking fine on top of a fetch path that was already broken. The moment the cache is cleared, damage that was already sitting there becomes visible. The deploy caused nothing, it only lifted the lid, and correlation with a deploy is not causation.
The practical consequence is that you should not reach for a rollback by reflex. A rollback clears the cache all over again, and no part of a rollback can restore an exhausted quota.
What was actually happening
The root cause was end-of-month CMS quota exhaustion. I confirmed it the same day, and one other third-party data service on that site hit the same wall on the same day. The quota resets on the 1st.
So there was nothing to fix in the ordinary sense. On 1 July the quota reset and those articles rendered again untouched. The code did not change, the deploy was never rolled back, not a single line got repaired. The distance between the incident and the recovery was exactly one turn of the calendar.
The fix I nearly shipped
The part worth telling is not the cause, it is the fix I almost released in the middle of it. Because those pages were rendering not-found, I wrote a noindex on null into generateMetadata, so search engines would not index empty pages.
export async function generateMetadata({ params }) {
const doc = await cmsFetch(query, { slug: params.slug })
if (!doc) {
return { robots: { index: false } } // backed out before it ever shipped
}
// ...
}I backed it out, and I am relieved I did. That logic decides indexing status at runtime from the same ambiguous null. Which means one blip on the CMS side, or one day of exhausted quota exactly like the one I had just lived through, is enough to stamp noindex on an article that is very much alive. The damage would not stop at one request either. At the two hour TTL on this site's own edge, one bad response gets cached and propagates to every request behind it for the next two hours.
That is precisely why the general not-found is left as a soft-404 on a 200 on purpose. Through an SEO lens that sounds wrong, but it is protective. A transient failure ends up as a 200, so Google comes back later instead of accepting a hard-404 and striking the page off. A page answering hard-404 can be struck off, while one answering 200 gets revisited.
For URLs I know for certain are dead, the status gets decided somewhere else entirely, as a 410 returned from middleware.ts, under one hard rule: the trigger must be a static list in the code, never a CMS lookup. A null value never gets the authority to decide a status.