Nine Red Tests and No Way to Tell Which Ones Matter

Every SDET has a private ritual for the morning after a red nightly. Mine: open the report, count failures, scan test names, then — without opening a single stack trace — decide which ones were "probably real" based on a feeling about which parts of the app had been touched that week. I called it triage. It was pattern matching on vibes.

One morning, nine failures. Here's what they turned out to be after a full day of digging:

Four categories, four owners, four urgencies. One was worth waking someone up for; three were worth nothing. In the report, all nine looked identical: same red, same font. I found the address validation bug seventh, around 3pm, because the report sorted alphabetically — which is to say randomly. It had been in production for 22 hours.

That wasn't my failure of judgment. There was no judgment available to make. The information needed to rank those failures didn't exist in my toolchain.

Red Is Not a Category

A test failure is a claim that expected and actual diverged. That's all. But there are at least four fundamentally different explanations:

  1. Product is broken. The thing you're testing does the wrong thing. The only one anyone wants to hear about — and the one your report is worst at identifying.
  2. Test is broken. Product is fine. Stale locator, short wait, assertion encoded an outdated assumption. Your bug, not the product team's.
  3. Environment is broken. Grid dropped a session, container failed to pull, DNS hiccup, deploy hadn't finished. Nobody's bug. Pure noise.
  4. Dependency is unavailable. Stripe sandbox unreachable, auth provider rate limiting. Test could not run — genuinely different from running and failing. You've learned nothing, but the report says "broken."

The fourth category is the most expensive. Blocked is not failed. A blocked test carries zero product information but renders identically to a failure that carries a lot. I've watched two engineers lose an afternoon to a P0 payments bug that didn't exist because CI couldn't reach a sandbox and the report had no vocabulary for saying so. The report only knew one word: red.

Why We Triage by Vibes

It's not laziness. Given nine identical red rows and no distinguishing information, guessing based on recent commits is genuinely the highest expected value strategy. If the checkout team shipped yesterday and a checkout test is red, that's a real Bayesian update. I was doing inference with the only prior I had.

The alternative was worse. Opening nine sets of artifacts: stack traces pointing at page objects, screenshots taken at teardown (after the page had moved on), videos to scrub. Fifteen minutes per failure to determine a category the machine already knew at the moment of failure and simply threw away.

That's the part that still gets me. The runner knew. At the instant of failure, it held the DOM, console, network activity, response codes, session state. Then it wrote a stack trace and screenshot, discarded the rest, and left me to reconstruct by hand what it had been holding a second earlier.

What Actually Helped

I never fully solved this, and I'd distrust anyone who claims they have. But four things moved the number, roughly in order of effort to payoff.

1. Fail Fast on Preconditions, with a Different Word

Before the suite runs, check that every external dependency answers. If the payment sandbox is unreachable, affected tests report BLOCKED, not FAILED. The pipeline summary says so. This is a couple of hours of work and permanently removes an entire category of false alarm. Most runners support a custom status or skip-with-reason. Use it.

# Example: precondition check in pytest
import requests

def test_payment_sandbox_available():
    try:
        requests.get("https://sandbox.stripe.com/health", timeout=5)
    except requests.ConnectionError:
        pytest.skip("Payment sandbox unreachable — marking dependent tests as BLOCKED")

2. Capture the DOM at the Moment of Failure, Not Teardown

A teardown screenshot shows the aftermath. You want the state at the assertion. Most modern runners can hook this. It's the single highest value artifact for separating "the button wasn't there" from "the button was there and did the wrong thing."

// Playwright example: capture DOM on assertion failure
page.on('pageerror', () => {
  page.screenshot({ path: `failure-${Date.now()}.png` });
});

3. Tag Failures by Exception Type Automatically

A StaleElementReferenceException or NoSuchElementException is almost always a test bug. An assertion error on a value is much more likely a product bug. A timeout waiting for the app is ambiguous, but a connection refused to your own grid is not. A lookup table can classify a decent proportion of failures with no cleverness. It won't be perfect. It needs to be better than alphabetical.

# Example: classification heuristic
FAILURE_TYPES = {
    "NoSuchElementException": "test_bug",
    "StaleElementReferenceException": "test_bug",
    "AssertionError": "product_bug",
    "ConnectionRefusedError": "infra",
}

4. Sort the Report by Blast Radius, Not Name

If a failure sits on a revenue path, it goes at the top. This requires deciding in advance which tests those are. That conversation is uncomfortable and worth having, because right now that ranking exists only in the head of whoever is on triage that morning.

The Question to Ask Your Team

Not "what's our pass rate?" Everybody has that number and it means less than people think.

Ask this: when a test goes red, how long does it take us to know which of the four it is?

If the honest answer is fifteen-minute increments per failure, multiply that by your average red run and by red runs per month. That's a real number, probably larger than you expect, and none of it appears on any report shown to leadership.

I was a competent engineer doing archaeology every morning because the tooling recorded the conclusion and threw away the evidence. That's the default. We're all quietly agreeing not to mention how much of the week it eats.

Stop that. Classify your failures. Your future self will thank you.