How I think about Playwright coverage and flaky tests

The most expensive test in your suite is the one nobody trusts. A test that fails randomly doesn't just cost the minutes it takes to rerun — it changes how the team reads the whole board. The first time an engineer sees red, shrugs, and clicks re-run, CI has stopped being a signal and started being a formality. One flake is all it takes. After that, real failures ride to production inside the noise.

A flake is a defect with bad PR

The comfortable story is that flaky tests are a testing problem — brittle selectors, an unlucky runner, cosmic rays. The uncomfortable truth is that a flake is almost always a real defect with bad branding: a race condition, a timing assumption, an order dependency, or shared state that only collides under parallelism. Sometimes the defect lives in the test. Sometimes it lives in the application, and the test is the only part of the system honest enough to mention it. That's why just add a retry is such a dangerous reflex — you may be silencing the one witness to a production race.

Quarantine, don't delete

A fails-sometimes test can't stay in the blocking lane — it poisons trust for every test around it. But deleting it destroys evidence. The middle path is a quarantine lane with teeth: the test still runs, its results are still recorded, and it no longer blocks merges. The teeth are the terms. A quarantined test has a named owner, a ticket, and an expiry date. When the expiry hits, either the root cause is fixed and the test returns to the blocking lane, or the test is deleted honestly — with a record of what coverage was lost and why. Quarantine without an expiry is just a graveyard with better marketing.

// quarantine lane — owner, ticket, expiry, or it doesn't go in
test.fixme('checkout survives double-submit', async ({ page }) => {
  // QUARANTINE: QA-412 — race between cart lock and payment intent
  // OWNER: jason · EXPIRES: 19 JUL 2026 — fix or delete, no third option
  // ...
})

Determinism is a build target

Deterministic tests don't happen by discipline alone. They happen by design, the same way a performance budget does — engineered in, then defended:

• Kill shared state — no two tests touch the same record, user, or global. Isolation is what makes parallelism boring instead of terrifying.
• Control the clock — a test that depends on real time depends on the weather. Inject the clock; freeze it; advance it deliberately.
• Await conditions, not timeouts — waitForTimeout is a guess wearing a stopwatch. Wait for the request to settle, the element to attach, the state to change.
• Seed data per test — each test creates what it needs and owns what it creates, so execution order stops mattering.

The ratchet

None of this holds without a number. Flake rate — how often a test fails and then passes on unchanged code — is a tracked metric on the suite, the same as duration or coverage. And it only moves one direction: new flakes fail the PR that introduced them, while the existing backlog burns down through the quarantine lane. That's the ratchet. Without it, every sprint adds a little noise, and a year later the suite is a slot machine. With it, flakiness is a bounded, shrinking debt with a name attached to every entry.

Coverage tells you what the suite watches. Trust decides whether anyone listens. A green board only means something when red does too — and red only means something when it can't be shrugged off as probably just flaky.