AI Prompt Testing

AI models can give a different answer each time, so we don’t judge a prompt on a single run. Every prompt is tested on two indicators:

A prompt passes only when both meet their threshold.

7.1 Prepare the prompt set

The tester prepares a set of prompts, each with an expected result. Each prompt is one Kiwi test case (Category: Eval).

Write key points for every prompt: the facts an answer must contain to be correct. The AI judge scores against them.

Cover four kinds of prompts:

• Normal: typical questions or inputs.
• Edge: unusual, incomplete, or messy input.
• Out of scope: things the model should decline or redirect.
• Tricky: input that tries to make the model ignore its instructions.

7.2 Consistency

1. Run the same prompt 5 times with the same model and settings.
2. Compare the answers. For structured answers (JSON, a label, a number), compare them exactly. For free text, the AI judge decides whether two answers mean the same thing.
3. Consistency = answers matching the most common answer ÷ total runs.

Example: 5 runs return the same JSON 4 times and a different total once, so consistency is 4/5 = 80%.

7.3 Accuracy

1. The AI judge compares each of the 5 answers with the expected result and key points.
2. It gives each answer a score from 0 to 100 and a one-line reason.
3. Accuracy = the average score of all runs.

A wrong or invented fact (e.g. a total that isn’t in the invoice) scores 0 for that run, however well the rest is written.

The judge prompt

Use the same judge prompt every time, with temperature 0, so scores are comparable:

You are grading an AI answer against an expected result.

Expected result:
{expected_result}

Key points the answer must contain:
{key_points}

Answer to grade:
{answer}

Score from 0 to 100:
- 100: every key point is correct, nothing wrong or invented
- 50: some key points correct, some missing
- 0: key points wrong, or the answer contains invented facts

Reply only with JSON: {"score": <0-100>, "reason": "<one sentence>"}

7.4 Pass thresholds

These are starting defaults. Adjust them after the first few runs, and log the change in the change log.

7.5 The AI judge can be wrong, so check it

• For every failed prompt, the tester reads the judge’s reasons before marking FAILED.
• For passed prompts, the tester spot-checks about 1 in 10.
• If the judge is wrong, fix the key points or the judge prompt, not the score.

7.6 Recording the result in Kiwi

Each prompt’s execution in the Kiwi Test Run gets:

• Status: PASSED if both thresholds are met, otherwise FAILED.
• Comment: consistency (e.g. 4/5), accuracy (e.g. 86), model name and version, prompt version, date.

Example comment:

Consistency: 5/5 (100%)
Accuracy: 92 (runs: 95, 90, 90, 95, 90)
Model: gpt-4o-2024-08-06 · Prompt version: v3 · 2026-09-21

7.7 When to re-run the whole prompt set

• The prompt changed, even slightly.
• The model or model version changed.
• The model’s settings changed (temperature, max tokens).

Compare with the previous run. A prompt that passed before and fails now is a regression, even if nobody touched it.

7.8 How it runs

A pytest script loops through the prompt set, calls the model 5 times per prompt, sends each answer to the judge, calculates both indicators, and sends the result to Kiwi with the same kiwi_case sync used for backend and frontend tests:

@pytest.mark.kiwi_case(12)
def test_AI_INVOICEEXTRACT_001(prompt_case):
    answers = [call_model(prompt_case["prompt"]) for _ in range(5)]
    consistency = most_common_count(answers) / len(answers)
    scores = [judge(a, prompt_case["expected"], prompt_case["key_points"]) for a in answers]
    accuracy = sum(scores) / len(scores)
 
    assert consistency >= 0.8, f"Consistency {consistency:.0%}"
    assert accuracy >= 80, f"Accuracy {accuracy:.0f}"