The deploy finished without a single step exiting non-zero, and for the seven minutes that followed, a production site I maintain returned 500 on every single URL. What made those seven minutes feel far longer was that /wp-login.php returned 500 too. No dashboard to open, no plugin to disable from the admin, no button anywhere I could press from the inside. The front door had locked itself along with the rest of the house.
The deploy itself had just finished and reported success. The script staged the files, wrapped them into an archive, pushed them over SSH, and extracted them on the server. Not one step exited with a non-zero status.
Every check I trusted came back clean
What made this slow to diagnose was not missing information. It was information that all said "fine".
I ran php -l on the server against the files I had just replaced. Clean, no syntax errors. I went a level up and wrote a small script that loads WordPress from the CLI and prints one line if it succeeds:
WP LOADED OKWordPress booted fully from the command line. So no fatal in functions.php, I thought at the time. Then I compared md5 sums of the local files against the ones on the server, one by one, and they were identical across the board. The contents had arrived exactly as I sent them. I also checked the per-directory PHP config file that had been the culprit a couple of times on that box, and this deploy had not touched it at all.
So here was the position: the syntax was valid, the contents were correct, WordPress could be loaded, the config was unchanged, and the site still returned 500 on every URL. Only one path was broken, and that was the web path. And at that point I was still burning minutes in the wrong place, because every instrument I was reading ran on the healthy side.
The one column I never read
For the first few minutes I verified that the files existed with ls. They were there, the names were right, the timestamps were fresh. Good enough, I thought, keep looking elsewhere.
What finally cracked it open was adding one letter:
ls -l wp-content/themes/theme/inc/-rw------- 1 deploy www-data 4821 Aug 20 10:11 helpers.php-rw-------. Mode 600. Owner deploy, group www-data, and that group had no read bit at all.
PHP-FPM on that box runs as www-data. One of those mode 600 files was required from functions.php. Every web request came in as www-data, tried to read a file explicitly readable by its owner only, failed right there, and ended as a fatal before anything could render. The login page included, because the login page loads the theme too.
Now all those earlier checks made sense. php -l, the WordPress loader script, and md5sum were all run over SSH as deploy, which is the owner of those files. As the owner I could read a mode 600 file without any friction whatsoever. I had been proving over and over that the files were readable by the user who was entitled to read them, while the user with the actual problem was somebody else.
Where mode 600 came from
The source was in the very first line of the deploy script, nowhere near the file that broke.
umask 077I wrote that line for a genuinely sound reason. The deploy script creates a temporary askpass file holding an SSH password, and a file like that must not be readable by other users on the same machine. umask 077 guarantees that anything born after it is readable by its owner only.
The problem is that the same shell also stages the deploy files. Before being archived, each file gets its line endings normalized:
tr -d '\r' < "$src" > "stage/$f"The > redirection creates a new file, and that new file is born under umask 077, so its mode is 600. Not some of them, all of them. tar then recorded that mode 600 into the archive exactly as it found it.
On the server side I already felt safe, because the extraction used a flag I assumed defanged file permissions:
tar xzf deploy.tar.gz --no-same-permissions -C /var/www/htmlThis is where my assumption missed. --no-same-permissions does not widen any mode. What it does is drop the setuid, setgid, and sticky bits, then apply the umask to the mode recorded in the archive. A umask can only take rights away, never add them. Mode 600 goes in and mode 600 comes out, and the flag reads like a safety net when its actual behavior is close to the opposite of what I pictured.
The chain holds end to end: one umask line added to protect a password file, quietly flowing into every staged file, riding along inside the archive, surviving the flag I thought neutralized it, then landing in production as files the web server cannot read.
The fix
The outage itself ended by widening permissions across the theme tree and then confirming the result actually changed. That box is setgid to www-data and the deploy user is in the same group, so 664 was enough:
find /var/www/html/wp-content/themes/theme -type f -exec chmod 664 {} +
ls -l wp-content/themes/theme/inc/The site came back on the very next request, with no PHP-FPM restart, no cache clear, nothing. There had never been anything wrong with the code.
After that I fixed the cause so it could not recur. I removed umask 077 from the shell that stages the deploy, and the askpass file is still protected, just explicitly on the file itself:
: > "$askpass"
chmod 600 "$askpass"This is better not only because the side effect is gone, but because the intent becomes readable. chmod 600 "$askpass" says plainly which file is being protected. umask 077 says "everything born after this line", and "everything" turned out to be much wider than I meant.
Then two checks that now always run at the end of a deploy. The first sweeps the tree for files the group cannot read:
find /var/www/html -type f ! -perm -g+rIf that command prints anything, the deploy is not finished. If it prints nothing, no file is hidden from the web server.
The second, and this is the cheapest one, is hitting the site itself over HTTP before touching caching or anything else:
for u in / /wp-login.php /contact/; do
printf '%s ' "$u"
curl -s -o /dev/null -w '%{http_code}\n' "https://origin.example$u"
doneA few seconds, three numbers, and this entire class of failure surfaces instantly. Had I run that in the first minute, those seven minutes would have been one.
What I took away
A CLI check cannot prove that a deploy is readable by the web server, because the two run as different users. During those seven minutes I was not short on data. I had plenty of data, all of it accurate and all of it irrelevant, because none of it came from the side that was broken. An instrument running as the owner of a file will tell you forever that the file is readable.
If the whole site 500s while the CLI looks healthy, the correct order of suspicion is file permissions or auto_prepend_file first, and your code second. Broken code usually breaks something in particular, and damage that is perfectly even across every URL more often means something is blocking the door before your code ever gets to run.
And the part I keep coming back to: umask is not a local setting. It attaches to the entire remaining life of that shell and to every file born inside it. If a shell has two jobs, keeping a secret and shipping files to production, never set the permission policy at the shell level. Set it on the file that actually needs guarding, and let everything else be born normal.