Part 28: Flaky Tests | TechByteByByte

A flaky test changes between pass and fail because some important condition is uncontrolled.

same intended conditions → changing result → find hidden cause

Series: Playwright Zero to Expert | Demo app used throughout this series: saucedemo.com

This part has been referenced constantly across the series — Part 6’s warning about retries masking real problems, Part 12’s dissection of race conditions, Part 27’s data-isolation discussion. It’s time to bring all of it together into a single, coherent diagnostic framework, because “flaky tests” isn’t one problem with one fix — it’s a category of several genuinely distinct root causes that happen to look identical from the outside (a test that sometimes passes, sometimes fails, with no code changes in between).

Defining Flakiness Precisely

A flaky test is one that produces different results (pass or fail) across multiple runs, without any change to the code being tested or the test itself. This precision matters: a test that fails consistently because of a genuine, real, reproducible application bug is not flaky — it’s correctly, reliably catching a real problem. Flakiness specifically describes inconsistency — the test’s own reliability as a signal is the thing that’s broken, regardless of whether the underlying feature works or not.

This distinction matters enormously for how a team should react. A consistently failing test tells you something true and actionable about the application. A flaky test tells you nothing reliable at all — worse, it actively erodes trust in the entire suite over time, because a team that’s been burned repeatedly by a flaky test’s false failures eventually starts reflexively ignoring any red test, re-running it and moving on without genuine investigation — which is exactly the moment a suite stops actually protecting anyone from real regressions.

Analogy: The Intermittent Wi-Fi Connection vs. The Broken Light Bulb

A Real Application Bug (The Broken Bulb): When you flip the wall switch, the light bulb remains dark. You flip it again, it’s dark. Ten attempts, ten failures (100% failure rate). You know the bulb is broken and must be replaced. This is a consistent failure.

A Flaky Test (Intermittent Wi-Fi): You open your laptop. The internet is fast on Monday morning. On Tuesday afternoon, it disconnects twice. On Wednesday, it works fine again. The connection drops intermittently — not because your laptop is broken, but because your roommate is running the microwave next to the router. You cannot solve this by simply restarting your laptop (re-running the test). You must trace the environmental noise and fix the root interference.

📊 Visual Flowchart: The Flaky Test Diagnostic Decision Tree

Here is the step-by-step reasoning path to isolate and categorize intermittent failures:

graph TD
    Start["1. Intermittent Failure Detected"] --> Quantify["2. Run: npx playwright test --repeat-each=50"]
    Quantify --> Rate["3. Calculate flake rate (e.g. 5/50 runs fail)"]
    Rate --> CheckTrace["4. Inspect Trace Viewer network and console logs"]

CheckTrace --> CheckEnv{"Does it fail only on CI<br>and pass always locally?"}
    CheckEnv -->|Yes| EnvIssue["Timing / Machine Latency (Category 1)"]
    CheckEnv -->|No| CheckSelector{"Does the locator target<br>first() / nth() or dynamic lists?"}

CheckSelector -->|Yes| SelectorIssue["Ambiguous Selector (Category 2)"]
    CheckSelector -->|No| CheckShared{"Are multiple tests updating<br>the same user or database row?"}

CheckShared -->|Yes| SharedState["Shared State Collision (Category 3)"]
    CheckShared -->|No| AsyncWait["Synchronization Race (Category 5)"]

The Real, Distinct Root Causes

Timing Issues (Part 12)

The most common category by far. A race condition where the test’s next step runs before the application has genuinely finished its previous, asynchronous work. Recognizable by: failures that seem to correlate with machine speed or load — passing reliably on a fast local machine, failing more often on a slower or busier CI runner.

Bad Selectors (Part 7)

A locator that matches ambiguously, or matches something slightly different depending on unpredictable page state — for instance, a locator relying on element position (.first(), .nth()) where the actual order of elements isn’t fully guaranteed or deterministic.

Shared State (Part 21, Part 27)

Two tests, potentially running in parallel, both depending on or modifying the same external resource — a shared user account, a shared database row — where the order in which they happen to execute affects the outcome, and that order isn’t guaranteed or controlled.

Environment Problems

A test that depends on something outside the application’s own control entirely — a third-party service occasionally being slow, a CI runner’s network occasionally being congested, a real external dependency with its own, uncontrolled reliability. Recognizable by: failures that seem to cluster in bursts (several failures close together in time, then none for a long stretch), rather than being evenly, randomly distributed.

Poor Synchronization (Part 12)

A subtly different flavor of a timing issue, worth distinguishing: not “the test didn’t wait long enough,” but “the test waited for the wrong thing entirely” — for instance, waiting for a network request’s response to arrive, when the actual, relevant delay is in a separate, subsequent JavaScript re-render that happens after that response, which the test’s wait condition never actually accounted for.

A Rigorous Diagnostic Process

1. Quantify the flakiness before attempting to fix anything.

A test that fails once in 200 runs is a genuinely different problem, with a genuinely different appropriate response, than one failing once in 5 runs. Playwright supports exactly this kind of investigation directly:

npx playwright test login.spec.ts --repeat-each=50

This runs the specified test 50 times in a row, giving you a real, concrete flake rate to work from — “3 failures out of 50 runs” is a specific, actionable, and honestly far more useful starting point than a vague impression of “this test seems flaky sometimes.”

2. Use trace and video from an actual failed run

(Part 23), not a hypothetical guess about what might be going wrong. Recall Part 16’s trace: 'on-first-retry' — this is precisely the setting that ensures you actually have a trace to examine the next time the test fails on its own, in the wild, rather than only when you’re deliberately, manually trying to reproduce it.

3. Categorize the root cause using the five categories above, rather than reaching immediately for a generic fix. A timing issue and a shared-state issue require genuinely different remedies — more awaits and better wait conditions solve the former; unique, isolated test data solves the latter; treating them interchangeably wastes real time and often doesn’t actually fix anything.

4. Fix the actual root cause — never just the symptom.

This is worth stating with real, deliberate directness, because it’s the single most important principle in this entire part: adding a retry, or increasing a timeout, without first understanding why the test is flaky, doesn’t fix flakiness — it hides it, often making it genuinely harder to detect and diagnose later, since the underlying instability is still fully present, just less visible in the pass/fail output you’re actually looking at.

Retries — a Legitimate but Narrow Tool

Recall Part 16’s retries configuration. It’s worth being precise, and fair, about exactly what a reasonable, legitimate use of retries actually looks like, distinct from misusing them as a blanket fix: a small number of retries can reasonably absorb genuinely transient, environment-level noise (category four above) — a CI runner’s network having one brief, unusual hiccup — without derailing an otherwise healthy, well-isolated, correctly-synchronized test suite over something entirely outside the application’s or the test’s own control.

What retries should never be used for: masking a genuine, reproducible race condition (category one) that would be far better and more permanently fixed with a proper wait condition; or papering over genuine shared-state contamination (category three) that will keep causing real, sporadic problems indefinitely, at an unpredictable and worsening rate, as the suite continues to grow larger and run more frequently in parallel.

A genuinely useful team practice worth knowing about: tracking a test’s flake rate over time (via CI reporting, covered properly in Part 34), and treating a test that only ever passes because of its retry — never on its true first attempt — as a real, tracked, prioritized item to actually fix, rather than a permanently tolerated, quietly accepted status quo.

How It Works in a Real Test Run

A flaky test has inconsistent outcomes for equivalent code and environment. Classify the evidence before fixing it: synchronization, unstable locator, shared data, environment capacity, external dependency, nondeterminism, or a genuinely intermittent product defect.

A retry is diagnostic evidence and temporary containment, not proof that the test passed. Current Playwright configuration can also fail CI when a test is classified as flaky, helping teams avoid normalizing instability.

Interview Questions

Q: What precisely distinguishes a “flaky” test from a test that’s simply, consistently catching a real bug?

Ans: A flaky test produces different results — sometimes passing, sometimes failing — with no actual change to the application or the test itself between runs; its unreliability is in the test’s own signal, not necessarily in the application. A consistently failing test, by contrast, is reliably and correctly reporting a genuine, reproducible problem every time it runs — that’s the test working exactly as intended, not flakiness.

Q: Why is it important to quantify a test’s actual flake rate (using something like --repeat-each) before attempting to fix it?

Ans: Because “flaky” can describe wildly different severities and, often, different underlying causes — a test failing once in 200 runs likely points to a rare, hard-to-hit race condition, while one failing once in 5 runs points to something more fundamentally and consistently broken in the test’s logic or waiting strategy. Quantifying it turns a vague impression into a specific, actionable data point, and also gives you a concrete way to verify afterward whether a fix actually worked.

Q: Name at least three genuinely distinct root causes of flaky tests, and explain why treating them all with the same fix is a mistake.

Ans: Timing issues (the test doesn’t wait long enough for a genuinely asynchronous operation), bad selectors (a locator matching ambiguously or inconsistently), and shared state (multiple tests interfering with each other over a common external resource) are three distinct causes. Treating them identically is a mistake because each requires a fundamentally different fix — better wait conditions for timing issues, more precise or scoped locators for selector issues, and uniquely isolated test data for shared-state issues — a fix aimed at the wrong category often does nothing to actually resolve the real underlying problem.

Q: Why is adding a retry, without first diagnosing the root cause, considered “hiding” flakiness rather than fixing it?

Ans: A retry doesn’t address whatever is actually causing the inconsistency — the underlying race condition, ambiguous locator, or shared-state contamination is still fully present and will still occur at roughly the same underlying rate. The retry simply gives the test additional attempts to happen to pass despite that underlying instability, which makes the actual problem less visible in the pass/fail output, potentially making it harder to notice and properly diagnose later, even though it hasn’t actually been resolved.

Q: What is a legitimate, narrow use case for retries, distinct from using them as a general-purpose fix for flakiness?

Ans: A small number of retries can reasonably absorb genuinely transient, environment-level noise that’s entirely outside the application’s or test’s control — like a CI runner’s network having a brief, one-off hiccup unrelated to any real race condition or test design flaw. This is different from using retries to mask a reproducible race condition or genuine shared-state contamination, which will keep recurring at a predictable rate and represents a real problem that retries only hide rather than solve.

Q: A test fails intermittently, and the trace shows the test proceeding to an assertion before a specific API call (visible in the network panel) has actually completed. What category of flakiness does this point to, and what would be the correct fix?

Ans: This points to a timing issue, specifically a race condition where the test’s next step ran before a genuinely asynchronous operation had finished. The correct fix is to properly wait for that specific condition — using a web-first assertion that polls for the actual expected UI outcome, or explicitly waiting for that specific network response using waitForResponse (from Part 12) — rather than adding a fixed delay or simply retrying the test and hoping the timing happens to work out on a subsequent attempt.

Exercises — Part 28

Understand: Without looking back, explain in your own words why treating “the test just needs a retry” as an automatic, default response to any flaky test is a genuinely harmful long-term habit for a team to develop.

Simple Practice: Take a test you’ve written earlier in this series and run it with --repeat-each=20. If it passes consistently, deliberately introduce a subtle timing issue (using Part 19’s mocked delays) and run it again with --repeat-each=20, recording the resulting flake rate.

Real-World Scenario: Imagine a test suite where a specific test fails roughly once every 15 runs, and the trace consistently shows the same locator matching two elements instead of one, seemingly at random. Diagnose, in writing, which category of flakiness this most likely represents, and propose a concrete fix referencing Part 7’s locator strategy discussion.

Challenge: Design a small internal team policy, in writing, for how your hypothetical QA team should handle a newly discovered flaky test — who investigates it, what data they should gather first (referencing this part’s diagnostic process), and under what specific, narrow circumstances (if any) a temporary retry would be an acceptable short-term measure while a permanent fix is being worked on.

Next: Part 29 — Advanced Playwright

— advanced fixtures, custom matchers, complex authentication scenarios, and advanced configuration patterns for large, mature frameworks.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026