Smoke Tests in CI/CD

A smoke test runs automatically right after a deploy and answers one question: is what we just deployed alive and working? If not, the pipeline stops.

The examples use the OCR eTax Async function app, but the steps are the same for any product.

10.1 What a smoke test is (and isn’t)

A smoke test doesn’t replace the cases in sections 2–7. It’s a small, fast subset that runs on every deploy.

10.2 Where it runs in the pipeline

Build ──► Deploy ──► Smoke ──► (approval / next stage)
                       │
                       └─ fails → the job fails → nothing after it starts

• Build runs on every pull request. It proves the package builds and loads.
• Deploy and Smoke run only on main (or whichever branch deploys).
• Smoke runs against the environment that was just deployed, never against a mock.
• Smoke is a separate stage or job after Deploy, so a failure is visible as “deployed, but smoke failed”, not mixed up with a deploy failure.

If a product has no dev environment (OCR Async deploys straight to prod), smoke runs once, after the prod deploy. Then it’s a detector, not a gate: it tells you quickly that prod is broken so you can roll back. Add a dev stage later and the smoke job moves in front of prod.

10.3 Choosing what goes in smoke

Put a check in smoke only if all of these are true:

• Read-only. It doesn’t create, change, or delete data on a real environment.
• HTTP only. It needs nothing but the public URL and a key. No database, no storage, no mock server.
• Fast. Seconds each, a few minutes in total.
• Stable. It fails only when the product is really broken.

Good smoke checks: the host answers (swagger.json), the API docs are served, a request with a missing field returns 400 with a readable message, CORS answers a preflight, an invalid callback is rejected. Add auth checks (no key → 401) and a status lookup for an unknown id when the product has them.

Not for smoke: uploading a real file and waiting for the job to finish. That writes data and needs a database to check. Keep it as a local or dev-only test.

Tag the safe ones

The same suite runs in two modes, picked by one variable (SMOKE_ENV):

@pytest.mark.remote_safe
def test_upload_without_file_rejected(func_ready, api, settings, evidence, request):
    ...

conftest.py drops every unmarked test when the target is deployed, so a test that writes data can’t run against prod by accident:

def pytest_collection_modifyitems(config, items):
    if not get_settings().is_deployed:
        return
    items[:] = [i for i in items if i.get_closest_marker("remote_safe")]

Register the marker in pytest.ini (--strict-markers catches typos):

[pytest]
addopts = -ra --strict-markers
markers =
    remote_safe: read-only HTTP check, safe against a deployed app

10.4 Where the tests live

All automated tests live in one shared repo, qa-automation, with one folder per product. They don’t live in the product’s repo.

qa-automation/
├── templates/
│   └── smoke-function-app.yml     pipeline steps, reused by every product
├── ocr-asynchronous/              one folder per product, named like the product repo
│   ├── requirements.txt
│   ├── pytest.ini
│   ├── conftest.py
│   └── test_*.py
└── <next-product>/

Why a separate repo:

• QA can change tests without a product release, and developers can’t change a test to make a deploy pass.
• One template serves every product.
• The tests can be pinned to a version (a tag) for a release.

This replaces the standalone qa-ocr-tests\ folder from section 3.2 for tests that run in CI. Local exploration still works the same way.

10.5 Wiring it into a product pipeline

The product’s pipeline needs three things: a pointer to qa-automation, a deploy step, and a call to the smoke template after it.

resources:
  repositories:
    - repository: qa                       # alias used in the template line below
      type: git
      name: Enterprise Data Management/qa-automation
      ref: refs/heads/main                 # or refs/tags/... to freeze the tests for a release
 
stages:
  - stage: Build
    # package, import-check, publish artifact
 
  - stage: DeployProd
    dependsOn: Build
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    # deploy the artifact
 
  - stage: SmokeProd
    dependsOn: DeployProd
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - job: smoke_prod
        steps:
          - template: templates/smoke-function-app.yml@qa
            parameters:
              project: ocr-asynchronous      # folder in qa-automation
              appName: $(appName)            # the app just deployed
              environment: prod              # dev | prod
              azureSubscription: $(azureSubscription)
              reportToKiwi: false

What the template does, in order:

1. Checks out the product repo and qa-automation side by side.
2. Installs Python.
3. Asks Azure (with the service connection) for the app’s URL and function key.
4. Waits up to 5 minutes for the app to answer, because a fresh deploy restarts the host.
5. Creates a venv, installs requirements.txt, runs pytest with --junitxml.
6. Publishes the JUnit file to the pipeline’s Tests tab, even when the run fails.

One-time setup in Azure DevOps

• A service connection (Azure Resource Manager) the pipeline can use. It needs rights to read the app’s function keys (Contributor or Website Contributor on the app).
• On the first run, permit the pipeline to use the qa-automation repo.
• If Limit job authorization scope to referenced Azure DevOps repositories is on, add qa-automation there.
• For pull requests: Repos → Branches → main → Branch policies → Build Validation, pointing at the pipeline. The pr: key in the YAML is ignored by Azure Repos.

10.6 Config and secrets

Same rule as 3.6: nothing secret in code or in YAML.

Reading the key from Azure on every run means no key is stored in the pipeline and nothing needs rotating there.

Locally, copy .env.example to .env. Real environment variables win over .env. Never commit .env.

10.7 Results

To turn Kiwi on:

1. Create the variable group kiwi-tcms with KIWI_URL, KIWI_USERNAME and KIWI_PASSWORD (secret).
2. Add the group to the smoke job’s variables.
3. Set reportToKiwi: true.

Status mapping follows section 8.2:

Keep the “is the host up?” check in the first test, outside the precondition fixture. A dead host should show up as a FAILED check, not as every test BLOCKED and nobody noticing.

10.8 When smoke fails

1. Open the failed job and read the first failing test. Check whether the app answered at all.
2. Host didn’t answer or is still starting → re-run the job once. If it still fails, treat it as a deploy problem and tell the developer. In Kiwi this is BLOCKED.
3. A check failed with a real wrong answer → it’s a bug. Follow section 8.4: link the bug in Kiwi, severity Critical or High for a prod smoke failure.
4. Decide with the developer: roll back or fix forward. On prod, don’t leave a failing smoke unanswered.
5. Never “fix” smoke by loosening the test or removing the remote_safe mark unless the check itself was wrong. Change it through a pull request on qa-automation.

A flaky smoke test is worse than none, because people start ignoring red. Fix it or remove it the same week.

To get recommended fixes for a failed smoke run automatically, see the proposed Smoke Advisor in section 11.

10.9 Adding smoke to a new product

• Create a folder in qa-automation, named like the product repo, with its own requirements.txt, pytest.ini, conftest.py and tests. Copy ocr-asynchronous/ as the starting point.
• Write the smoke checks (10.3) and tag the read-only ones with remote_safe.
• Run it locally with SMOKE_ENV=local, then against the deployed app with SMOKE_ENV=dev and the key in your shell.
• Choose your own Kiwi Product and Test Plan names so results stay separate in the shared Kiwi.
• Add the resources.repositories block and the template call to the product pipeline (10.5).
• Do the one-time setup (10.5), then run the pipeline and permit the repo.
• Confirm the Tests tab shows the results, and break one check on purpose to confirm the pipeline turns red.
• If the product isn’t an Azure Function App (web app, container, data pipeline), add a template next to smoke-function-app.yml instead of bending the existing one.

10.10 Worked example: OCR eTax Async

In CI against prod only the 7 read-only HTTP checks run (OpenAPI document, Swagger UI, CORS preflight, three upload-validation rejections, and invalid webhook callbacks rejected). The other 9 scenarios, which upload a PDF and follow the job through the queue and the webhook, run on a QA machine with the local stack. Both are the same suite; the environment decides which part runs.

Pull requests run only Build: package the app, then import every function module. This catches the failure where a deploy “succeeds” and the app comes up with zero functions.