The email draft had reached its final sentence. It carried the news I least wanted to send anyone after an emergency recovery: the site database looked like it had been restored from an old backup, and the last two months of work were gone. I had lined up the numbers first so the sentence would not sound like a panicked guess.
post 1,301 latest two months ago
episode 505 latest two months agoEvery one of those counts came from outside, over HTTP, without touching the server at all. Public archives, the REST collection endpoints, the sitemap. Three separate sources, and all three told exactly the same story. Even the gap made sense: not random figures, not half-empty tables, but a clean cut at one point in time, precisely the shape a database restored from an old dump would have.
What held my hand back was not suspicion. Quite the opposite. I wanted the email to be harder to argue with, so I wanted one number quoted straight from the database rather than from the front end. For that I had to get onto the server and open wp-config.php. And to open wp-config.php, I first had to know which installation.
Apache was serving a different directory entirely
The server held two sibling directories under /var/www/html, left over from a migration a few months earlier. One was the production install. The other was the pre-migration version that was supposed to be retired and kept around only as an archive. I opened the vhost configuration first, purely to be sure I was typing the right path.
apache2ctl -S
grep -R "DocumentRoot" /etc/apache2/sites-enabled/sites-available/site.example.conf: DocumentRoot /var/www/html/site.example
sites-available/site.example-le-ssl.conf: DocumentRoot /var/www/html/site.exampleThe runbook I had written a month before recorded a different value, /var/www/html/prod.site.example. Both vhost files had changed, including the -le-ssl variant that serves HTTPS traffic, so the whole of the public traffic was landing in the wrong directory rather than some fraction of it.
At that moment the explanation still felt light. The site is serving old theme files, I thought, put the DocumentRoot back and the production content returns. What I had not yet grasped was that the retired directory did not merely hold an old copy of the theme and plugins. It held a database of its own.
The same table prefix, two different databases
The quickest way to tell two WordPress installs apart when they live on the same database host is usually $table_prefix. Installs created at different times almost always end up with different prefixes, especially when the installer randomises them. Here the prefix genuinely was random, eight characters that were clearly not the stock wp_, and that is exactly what made me careless. For a while I treated it as fingerprint enough.
for d in /var/www/html/prod.site.example /var/www/html/site.example; do
echo "== $d"
grep -E "DB_NAME|DB_HOST|table_prefix" "$d/wp-config.php"
done== /var/www/html/prod.site.example
define( 'DB_NAME', 'prod_site' );
define( 'DB_HOST', 'db-internal.example:3306' );
$table_prefix = 'wp_a7k2m9x4_';
== /var/www/html/site.example
define( 'DB_NAME', 'site' );
define( 'DB_HOST', 'db-internal.example:3306' );
$table_prefix = 'wp_a7k2m9x4_';Same database host. Same table prefix, down to the final character. The only thing that differed was DB_NAME. The migration had evidently carried the old prefix along with it, and the only thing actually separated was the database name.
The consequence is fairly cruel. If you check the prefix to confirm you are standing in the correct installation, that check passes in both directories. You walk away certain you are in the right place while you are in fact reading a different database with different content. On plenty of servers, comparing prefixes really is enough. On this server, DB_NAME was the only difference that meant anything.
Once that was clear, the numbers in my draft lost every bit of their meaning. Each count I had taken from the front end was not measuring my project. It was measuring the retired site Apache happened to be serving, database included, and that database really had stopped receiving content two months earlier. The figures 1,301 and 505 were correct. They simply belonged to the wrong site.
Counting rows in the database that actually mattered
The recount went nowhere near HTTP. I connected to prod_site and counted directly, per post type, the number of rows along with the most recent date.
SELECT post_type,
COUNT(*) AS total,
MAX(post_date) AS latest
FROM wp_a7k2m9x4_posts
WHERE post_status = 'publish'
GROUP BY post_type;The result was 1,328 posts, 77 pages, and 519 episodes, with the latest date in all three landing in the current month rather than two months back. One more query, against the lead table belonging to its custom plugin, counted 1,867 rows. Not a single thing was missing. The only missing piece was public access to it.
Notice that the production counts were actually higher than the HTTP counts in every category, 1,328 against 1,301, 519 against 505. A gap that small never sets off anybody's alarm. Had it been thousands, people would suspect the measuring instrument. Because it was only dozens, my brain read it immediately as an ordinary and depressing loss rather than as proof that I was measuring a different object.
The sitemap stood on even weaker ground as evidence, and I had counted that too. A sitemap is a cached file. It can be older than any database on that server, retired ones included, and still be served with total confidence.
Why the DocumentRoot fix had to wait its turn
The repair itself was one line across two vhost files: point DocumentRoot back at the production docroot, then reload Apache. That edit needed root and I did not have root, so the execution was not mine. My job was making sure the order was right before anyone touched it.
The order did matter, because the production docroot had a breakage of its own in the bootstrap layer, a separate story with nothing to do with the vhost. While that breakage was still in place, moving DocumentRoot back would only have traded one symptom for a worse one, turning a site that served stale content into a site that served nothing at all. The bootstrap was fixed first and proven to render a complete page, and only after that did the vhost move.
Proving it also happened without Apache, so as not to circle the same loop. I ran a render from inside the production docroot through PHP on the command line and inspected the output: a sensible byte count, the correct title, the correct theme directory, and one document closing tag. With all four coming out the way they should, the bootstrap was healthy and moving the vhost was safe.
What this changed about the way I measure
The lesson has nothing to do with Apache, and nothing to do with table prefixes either. Measuring over HTTP answers the question "what is the server replying with today", not "what is in my project". For as long as those two questions share an answer, the difference never shows itself. The moment a site might be served from a directory you did not expect, the questions come apart, and every number you pull off the front page starts measuring the wrong object very convincingly.
So on any incident that touches server configuration, counting now comes last for me. I establish which directory is genuinely being served, then which DB_NAME that directory reads. Both facts are cheap to obtain, both are read-only, and without them every number that follows is guesswork that happens to be shaped like data.
I never sent that email. I kept the draft a little while before deleting it, as a reminder that the most expensive mistakes are not the ones that flood the screen with errors. The expensive ones arrive as tidy numbers, consistent across three sources, ready to paste into an email.