Building a smoke test suite for every pull request

A pull request should provide more than a code review checkpoint. It should offer a fast, repeatable signal that the application still starts, responds to its core workflows and remains safe to merge. A well-designed smoke test suite gives developers that signal within minutes, before a small change becomes a larger integration problem. Learn more about Writing Effective Acceptance Criteria For Agile User Stories.html.

Smoke tests are deliberately narrow. They do not replace regression, exploratory, security or performance testing. Their purpose is to confirm that the build is sufficiently healthy for deeper testing. In an Australian delivery environment, where teams may be spread between Sydney, Melbourne, Brisbane, Perth and offshore locations, this quick feedback can keep work moving across time zones without relying on someone to manually check the latest build. Learn more about The Four Pillars Of A Successful Agile Testing Quadrant Implementation.html.

Define the purpose and boundaries

Start by agreeing what “healthy enough to test” means for the product. A smoke suite might verify that the application loads, users can authenticate, a key record can be created and a critical API returns a valid response. These checks should represent the minimum business path, rather than every feature available in the system.

The suite should fail when a deployment is unusable, a core service is unavailable or a major user journey is broken. It should not fail because a low-priority label has changed or because an optional analytics service is temporarily slow. Clear boundaries prevent smoke tests from becoming an oversized regression pack that takes half an hour and produces ambiguous results.

Acceptance criteria are a useful source of test ideas, particularly when stories describe observable outcomes rather than implementation details. Teams can use effective acceptance criteria to connect each smoke check to a meaningful behaviour and make failures easier to interpret.

Select journeys that protect value

Prioritise workflows according to business risk, usage and dependencies. For an online retailer, the shortlist may include login, product search, cart updates and checkout through a test payment provider. For a SaaS platform, it may include tenant selection, dashboard loading, user creation and a basic report export. A bank or government service will need especially careful treatment of identity, audit and data handling.

Aim for a small set of high-value tests that can run reliably on every pull request. A test that covers six screens but depends on unstable test data may be less valuable than three direct API checks and one carefully chosen browser journey. The suite should provide broad confidence through essential paths, not a large number of superficial assertions.

Keep test accounts, product records and other fixtures controlled. Generate unique data where possible, reset state after execution and avoid shared accounts that can be changed by several pipelines at once. This is particularly important for distributed Australian teams working across the day, where a test in Perth could interfere with a test launched earlier from Sydney.

Choose the right testing layers

A pull-request smoke suite usually works best as a layered collection. Unit tests should cover business rules and transformations quickly. Service or API checks can validate authentication, contracts and core operations without the overhead of a browser. A small number of end-to-end tests can then confirm that the most important user journey works through the interface.

Use the lowest practical layer for each assertion. If a calculation can be checked at unit level, there is little benefit in checking it repeatedly through a slow browser flow. Reserve UI automation for risks involving routing, rendering, permissions, client-side behaviour and integration between visible components.

The same principle applies to Microsoft delivery environments. Teams using .NET, Azure DevOps, Microsoft Entra ID or SQL Server can combine unit tests, API tests and Playwright or Selenium checks in a single pipeline. Test results should be published in a common format so developers see one coherent quality signal rather than searching through separate tools.

Make the suite safe for every change

Every pull-request test should run against an isolated, known environment or a dependable test space. Containers, ephemeral environments and seeded databases can reduce interference between branches. Where full environment creation is too slow, use stable shared services with strict data ownership and reset procedures.

External dependencies need deliberate handling. Do not send real messages, charge real cards or call production services from a smoke test. Use service virtualisation, sandbox endpoints and controlled feature flags. If a third-party dependency is essential to the scenario, check the integration contract separately and report dependency outages distinctly from application failures.

Secrets must come from the pipeline’s secure store, never from source control or test files. Limit credentials to the permissions required by the test. For systems handling health, financial or personal information, use synthetic data and retain only the logs needed for diagnosis. These controls support Australian privacy obligations and help teams satisfy enterprise procurement and governance requirements.

Design for speed and useful feedback

A practical pull-request suite should usually complete in a few minutes, with parallel execution where the infrastructure allows it. Start independent API, unit and UI checks at the same time, then stop or quarantine later stages when a fundamental health check fails. Fast feedback matters because developers are more likely to act on a result that arrives while the change is still fresh.

Failure messages should identify the journey, environment, request or assertion that failed. Capture screenshots, browser traces, response bodies and relevant service logs, while masking tokens and personal data. A red pipeline that says “element not found” creates investigation work; a report that shows the page state, correlation ID and failed endpoint gives the team a route towards repair.

Set explicit timing expectations. A slow but passing smoke test can become a delivery bottleneck, especially for teams using hosted runners or an NBN connection with variable performance. Track duration, failure rate, retry rate and the percentage of failures caused by test infrastructure. These measures reveal when the suite needs maintenance rather than more cases.

Connect smoke tests to Agile delivery

Smoke tests work best when they are part of the pull-request policy, not a separate activity performed after merging. The workflow can run static analysis and unit tests first, then service checks, then a compact end-to-end set. A branch should be blocked when a critical check fails, with an agreed process for handling a genuine infrastructure fault.

The suite should support collaboration among developers, testers, product owners and operations. The Agile testing quadrants provide a useful way to balance technology-facing checks with business-facing examples; teams can use Agile testing quadrants to avoid treating UI automation as the whole testing strategy.

Review smoke coverage during refinement and planning. When a new payment route, identity provider or high-risk workflow is introduced, decide whether it belongs in the pull-request gate or in a later environment-level check. In Australian organisations with formal change controls, this traceability also helps connect a pipeline result with release evidence and audit expectations.

Manage flaky tests and evolving systems

A flaky test is one that passes and fails without a relevant product change. It weakens trust quickly, so treat flakiness as a defect. Record the test name, commit, runner, browser, dependency status and timing. Avoid masking the issue with unlimited retries; one controlled retry may distinguish a transient network fault, but repeated retries can hide real defects.

Common causes include hard-coded waits, shared data, asynchronous processing, unstable selectors and assumptions about time zones. Use condition-based waits, durable accessibility or data-test selectors, isolated fixtures and service-level readiness checks. Test dates explicitly rather than relying on the runner’s local clock, which is important when systems serve customers across Australian states and territories.

Maintain the suite as the product changes. Remove checks that no longer protect a meaningful risk, update journeys when behaviour is intentionally redesigned and review duplicate coverage across layers. Ownership should be clear: development teams can maintain component and API tests, while testers and quality specialists can guide risk selection, exploratory follow-up and cross-system coverage.

Measure confidence rather than test volume

The number of smoke tests is a poor measure of quality. More useful indicators include pull-request feedback time, escaped defects in covered journeys, flaky-test frequency, successful pipeline reruns and the proportion of critical paths represented by reliable checks. These measures show whether the suite improves decisions rather than simply generating activity.

Use failures as learning signals. If a broken database migration reaches the review stage, add a targeted check at the appropriate layer. If a browser test repeatedly detects an API contract problem, move that assertion closer to the service. If a production incident exposes a missing customer journey, assess whether it belongs in the smoke suite or in a broader regression strategy.

A mature approach combines automation with human judgement. Smoke checks confirm the build is viable; testers still need to explore unusual states, accessibility concerns, security risks and performance under realistic load. This balance is valuable in Australia’s mixed market, where a product may serve metropolitan broadband users, regional customers on slower connections and enterprise clients with strict release windows.

Build the first version around a handful of critical journeys, run it on every pull request and review its results each sprint. nFocus Software Testing can help assess current coverage, shape a sustainable test strategy, implement automation and connect quality practices with Agile and DevOps delivery. Begin with reliable feedback, then expand the suite only when evidence shows that additional checks will protect the product.