Commits Land on a Failing Suite: `| grep Tests && git commit` Reads `grep`'s Exit Code, and `PIPESTATUS[0]` Is Empty in zsh

My rule for every fix commit is that the test suite has to be green first. There are two reasons. A commit on a red suite is a false claim in the history, and if that commit gets pushed, what ships with it is a regression. I wrote the gate as a single shell chain that looked like it already guarded all of that:

pnpm test:int | grep Tests && git commit

Run the suite, find the summary line, then commit. On 26 September 2026 this chain let two fix commits through while pnpm test:int had a failing test. Both were caught and amended before any push at all, so the wrong versions never reached the remote.

&& reads grep's exit code, not the test's

&& asks the command on its left exactly one thing, which is whether it exited with 0. In that chain, what sits on its left is a pipeline, and the exit code of a pipeline is the exit code of its last command. That last command is grep, not pnpm test:int.

grep exits with 0 when it finds at least one matching line. The suite's summary line appears in the output whether the result is green or red, so grep succeeds on that line whatever the tests did. Nothing ever asks pnpm test:int for its own exit code. The chain checked grep's output, not the test run's exit code, and that gap is how the two commits went in on a failing suite.

The pattern reproduces without any project at all. The minimal example below uses a stand-in function that prints one summary line and then exits with 1, the way a failing suite would:

# minimal example with neutral names, not the project script
fake_tests() { echo "Tests: 1 failed"; return 1; }
 
fake_tests | grep Tests && echo "commit runs"
# Tests: 1 failed
# commit runs

The result is the same in bash and zsh. There is no warning of any kind, and in this example the word failed is printed right above the line that stands in for the commit.

Both bash and zsh have a pipefail option that makes a pipeline report a failure from any command inside it, and with that option the example stops before echo. The other nearby way out is reading the first command's exit code through PIPESTATUS. That second one had already fooled one of my tests, thirteen days before those two commits.

PIPESTATUS[0] is empty in zsh

On 13 September 2026, it turned out that a guard I wrote to stop a broken deployment would have failed every local build. That only surfaced because the guard was run. The first test for the same guard was slipperier still, because it reported three passes that verified nothing at all. My notes on that day put the cause in one sentence: PIPESTATUS[0] is empty in zsh, where arrays start at one, so every exit code in that test read as zero.

That sentence is true, but the story is slightly wider, and the rest can be checked by anyone. PIPESTATUS is a bash array. It holds the exit code of each command in the last pipeline, starting at index 0. In zsh, the same information lives in an array called pipestatus, in lowercase, and zsh arrays do start at index 1. Uppercase PIPESTATUS does not exist in zsh, so any index read from it expands to an empty string with no error, unless the shell runs with set -u.

# minimal example: first command fails, second succeeds
false | true
echo "PIPESTATUS[0]=[${PIPESTATUS[0]}] pipestatus[1]=[${pipestatus[1]}]"
 
# bash : PIPESTATUS[0]=[1] pipestatus[1]=[]
# zsh  : PIPESTATUS[0]=[] pipestatus[1]=[1]

My notes do not spell out how that empty value turned into zero, so the comparison inside that test is left out here. What can be shown is how easily it happens in zsh. The most ordinary numeric comparison already treats an empty string as zero:

# minimal example, not the project's test
[ "" -eq 0 ] && echo "treated as zero"
 
# zsh  : treated as zero
# bash : [: : integer expression expected

In bash, the same comparison at least complains. In zsh, an empty string from a variable that never existed passes quietly as zero.

A gate that reads the test command's own exit code

The rule I wrote from the 26 September incident puts it briefly: commit only when the test command itself exits 0. The method is to run the suite to a log, keep R=$?, and commit only inside if [ $R -eq 0 ]. A simplified form, with a neutral log file name:

pnpm test:int > test.log 2>&1
R=$?
if [ $R -eq 0 ]; then
  git commit
fi

There is no pipe between the suite and the reading of its exit code, so the $? on the second line belongs to pnpm test:int, not to grep, and pipefail is unnecessary because there is no pipe to guard. R=$? has to sit on the very next line, because any command that runs in between overwrites $?. The suite's output is not lost, it just moves into test.log, so which test failed can still be read from there. R always holds a number from $?, so the comparison in the if never meets an empty string like the earlier example, and the result is the same in bash and zsh.

This comes with a limit I should state directly. The rule is written as a way of working from here on, not as a script that has been installed and tested. My notes do not yet contain a single run showing that this form actually holds back a commit on a red suite.

A test nobody has seen fail

The notes on 13 September contain a rule to check that a test can actually fail before trusting that it passed. The notes on 26 September repeat the same idea in a more concrete form: before a test is trusted, show it failing without the fix, by stashing the source. A simplified form, with a neutral path:

git stash push -- path/to/fix-file
pnpm test:int > test.log 2>&1
echo "without the fix: $?"    # must not be 0
git stash pop

git stash push with a path only stashes changes under that path, so the new test stays in place while the fix is set aside for a moment. If the suite still exits with 0 without the fix, the new test is not testing the fix. A test that reads every exit code as zero will pass with or without the fix, and a step like this is exactly where such a test gets caught.

Leftover rows from a run that stopped halfway

The notes on 26 September record one more problem, separate from the grep issue. Tests that use a fixed stamp left rows behind when a run failed halfway, and the next run failed on a unique slug. A fixed stamp means every run carries the same marker, so rows left behind by one run can collide with the run after it.

I wrote two rules for that. New tests get a stamp that is unique per run, and after a red run, its leftovers are cleared before the next one. This also bears on the gate. A gate that reads exit codes honestly will hold back a commit for any kind of red, including red that comes from the previous run's leftovers rather than from the code being fixed.

The checker has to be run too

The grep chain and the guard test turned out to be wrong in the same way. Both are tools whose job is to check something else, and both read a pass from the wrong exit code: one from grep, the other from a variable that does not exist in zsh. All either of them produced was a zero.

The notes on 13 September keep the principle in one sentence: a tool that has not been run is a belief about the tool. The rule that follows is to run anything you build before saying it works, including the thing that checks other things.

That R=$? gate is also a thing that checks other things. By my own measure it remains a belief until it is seen holding back a commit on a suite that really is red, and the way to test it is the same as for any test. See the gate refuse first, and only then trust it when it allows.