There is one HTML file I saved that afternoon that contained two things which should not have been able to sit in the same response. In one part there was markup from a feature I had written minutes earlier, still warm, not even tidied up yet. In another part there was a <video> element pointing at an mp4 I had deleted from disk hours before.
Which means the PHP process rendering that page was reading the newest possible version of my code while answering the question "does this file still exist" with the oldest possible answer. Up to that point I was still convinced the problem lived in a code cache layer. That single response is what proved me wrong, and it also pointed at where to look instead.
The symptom came and went between requests
The page I was working on had a video background. The mp4 was no longer needed, so I deleted it, and the markup that referenced it was gated behind a file existence check. Logically that element should have stopped rendering the second the file disappeared.
In practice it kept showing up, and every time it did it dragged one 404 request along for a file that was not there. The 404 was not the annoying part. The inconsistency was. I reloaded, the element was there. Reloaded again, gone. I pulled the HTML with curl and it usually came back clean, with no trace of the element at all.
A symptom that differs between requests does a specific kind of damage to a debugging session, because every action you take appears to work about half the time. I cleared something, reloaded, the element was gone, and for a moment I believed I was done. The next reload took that conclusion back.
My first accusation landed on OPcache
I spent a good while blaming OPcache. On the surface the reasoning holds up: PHP code held as bytecode in memory really can lag behind the file on disk, and the symptom sounds the same, namely "what runs is not what I wrote". I cleared it, I restarted, and sometimes the page really was clean afterwards. That occasional cleanliness is exactly what kept the mistake alive, because every time it happened I read it as confirmation.
The one thing that never fit the OPcache theory was the timing pattern. Stale OPcache is uniform. As long as the old bytecode is being held, every request gets the old code, not two out of ten. When the symptom is intermittent and the theory is deterministic, it is usually not the symptom that is wrong.
The browser was not an honest instrument here
There was one more thing quietly leading me astray the whole time, and I only understood it after everything was over. In the browser the symptom felt far more consistent than in curl. I briefly decided that curl simply "could not see" the problem, when the truth was the reverse.
A browser opens a persistent connection and tends to stick to the same process for several requests in a row. If that process happens to be one carrying stale state, everything I saw in that tab was consistently broken. If it happened to be a clean one, consistently fine. curl opening a fresh connection every time spreads requests across many processes, so the result looks random.
So the gap between "consistent in the browser" and "random in curl" was not noise to be ignored. It was already half the answer, I was just reading it backwards.
Ten requests, two of them stale
The moment I stopped drawing conclusions from a single reload and started treating the symptom as something to be counted, it went quickly. I sampled ten requests and counted how many carried the phantom element:
for i in $(seq 1 10); do
curl -s http://local-site.test/ | grep -c '<video'
doneEight lines of 0 and two lines of 1. Two out of ten, and the number held when I ran it again.
A ratio that stabilizes on a fraction like that does not fit any cache held in one place for everybody. Code caches and page caches keep a single shared copy, so their result is uniform until that copy is thrown away. A ratio settling around two in ten points at a different shape of problem: several PHP processes are taking turns serving requests, and some of them are carrying different contents in their heads than the rest.
Combined with the response that held both the newest markup and the phantom element, only one direction was left. The code was fresh, so the bytecode was not the stale part. The stale part was the result of the file check.
PHP's stat cache lives in the process, not in the request
PHP stores the result of file checks like file_exists() so it does not have to ask the filesystem the same question over and over. In the setup I was using at the time, a local WordPress runtime whose workers are long lived, that cache lives for the lifetime of the process rather than the lifetime of a request.
The consequence is exactly what I was seeing. Some workers had already called file_exists() on that mp4 path back when the file still existed, and had stored the answer "yes". I deleted the file, but those workers never asked again. They answered from memory, and their memory was hours old by then. Workers born after the deletion answered correctly. Requests landing on the first group rendered the phantom element, requests landing on the second group did not, and the split came out roughly two to eight.
That is also what makes the disguise so effective. It looks like OPcache, it looks like a page cache, at one point it even looks like a race condition, because all three produce "different results for the same request". The only difference is which layer the state is piling up in.
The fix is one call before the check
The markup itself was not wrong. What was wrong was the assumption that file_exists() answers a question about the state of the disk right now. For a file that can appear or disappear at runtime, that answer has to be forced fresh:
$video = $dir . '/hero-loop.mp4';
clearstatcache( true, $video );
if ( file_exists( $video ) ) {
// render the <video> element
}clearstatcache( true, $path ) drops the entry for a single path rather than the whole cache, so the cost is small and measurable. For one piece of markup gated on one file, it means one extra stat call per request. That price is clearly cheaper than a phantom element that produces a 404 for some visitors and cannot be reproduced with an ordinary reload.
Worth marking down is when this pattern is actually warranted. Not for every file_exists(), since most of the paths you check are static for the whole life of the process. The ones that need clearing are paths whose contents can change while the process is still alive, such as files written or deleted by another process, or assets created at runtime.
What kept the damage down to a single request
This is the part I most wanted to keep, and it is not really about the stat cache.
That video element was designed to stay invisible until playback actually began. A class only gets applied after a real playback event fires, and before that the layer just sits quietly behind everything else. Because the mp4 was 404, playback never began, so the class was never applied, so nothing changed on screen.
That is why a bug that had been alive for hours cost exactly one invisible 404 request instead of a black rectangle sitting in the middle of the page for an entire working day. A layer that only appears after real evidence that it loaded successfully forgives an enormous number of mistakes behind it, including the ones you did not expect to be there.
The rest of the lesson is about measurement. When a bug gives different results for the same request, there is almost always state living longer than a single request, and PHP has more than one layer where that kind of state piles up. OPcache is just the famous one. The fastest way to tell them apart is not to guess the layer but to turn the symptom into a number: send ten requests and count how many are hit. Uniform means one shared copy, while a fraction that holds steady means several processes with different memories.