Kiwi, Results & Bugs

Kiwi TCMS is where all QA work is documented: what we plan to test, what we tested, and what we found.

8.1 How we organize Kiwi

Kiwi	We use it for	Example
Product	One application	`OCR-Asynchronous-function-azure`
Version / Build	What’s being tested	`1.2` / `1.2-rc1`
Component	Part of the system	`HttpTrigger`, `QueueTrigger`, `Login page`, `Extract task`
Test Plan	One feature or release	`OCR eTax Async — Release 1.2`
Test Case	One behavior to check	`API-OCRETAX-001 — Submit valid PDF returns 202…`
Test Run	One round of testing on one build	`Release 1.2 — regression, 21 Sep`

8.2 Result statuses

Status	Meaning
PASSED	Every expected result was seen
FAILED	An expected result wasn’t seen → attach evidence and link a bug
BLOCKED	Couldn’t run (environment down, data missing) → comment why
WAIVED	Deliberately skipped this round → comment why

There’s no “passed with a small issue.” If anything in the expected result is wrong, it’s FAILED.

8.3 Automated results

Automated tests report into Kiwi by themselves. Mark the test with its Kiwi case number, create a Test Run in Kiwi, then run:


set KIWI_SYNC=1
set KIWI_URL=https://<kiwi server>
set KIWI_USERNAME=<username>
set KIWI_PASSWORD=<password>
set KIWI_RUN_ID=<run id from the run's URL>
pytest

Each result lands on its test case in that run, with a comment (duration, key values like tracking_id, and the failure reason if it failed). The case must be CONFIRMED first.

In a pipeline you don’t set these by hand. The smoke template turns --kiwi on with its reportToKiwi parameter and reads the credentials from the kiwi-tcms variable group (see section 10.7).

Manual tests are marked by the tester directly in the Test Run.

8.4 Reporting a bug

When a case FAILS, add a comment on the execution in Kiwi and link the bug:


**Title:** [API] Text sent instead of file returns "Incorrect padding"
**Severity:** Critical | High | Medium | Low
**Test case:** API-OCRETAX-003 (TC-3)
**Environment:** local, build 1.2-rc1
 
**Steps:** 1. …  2. …
**Expected:** 400 with a readable message
**Actual:** {"status": "error", "message": "Incorrect padding"}
**Evidence:** screenshot / response / log

For AI prompt bugs, also add the model version, the prompt version, and how often it happened (e.g. 2/5 runs).

Severity:

Critical: data loss, security issue, or the core flow is completely broken.
High: a major feature is broken with no workaround.
Medium: broken, but there’s a workaround.
Low: cosmetic.

8.5 Release sign-off

A release is ready from QA’s side when:

Every P0 and P1 case in the Test Run is executed.
All P0 cases PASSED.
No open Critical or High bugs.
All P0 AI prompts meet their thresholds (section 7.4).
If the release touches a critical endpoint or pipeline: the load test passed (section 6.3).
The smoke stage after the deploy passed (section 10).
Anything BLOCKED or WAIVED has a comment explaining why.

The QA lead signs off by adding a comment on the Test Run: “QA sign-off: GO” (or “NO-GO” with the reason), with their name and the date.