AI Standards Review in CI/CD

Status: proposal. Nothing on this page runs in a pipeline yet. It describes how the Abangku skills (Agent Plugin) can join the pipeline from section 10 to recommend fixes, and when they may stop a build.

11.1 Two different questions

The review doesn’t replace any test. It reads what changed, checks it against the rules published on this site, and says what to fix, with the rule and file:line.

Fatal errors are still decided by deterministic checks: the build fails, a module doesn’t import, or a smoke test fails. The AI review adds recommendations on top. It never turns a red result green.

11.2 Where it runs

PR / push to main
   │
   ├─ Build ─────────────► package + import-check            fails → DECLINE
   │
   ├─ Standards Review ──► Abangku skill on the diff
   │                         Blocker  → DECLINE (enforce mode only)
   │                         others   → PR comment + pipeline summary
   │
   ├─ Deploy (main only)
   │
   ├─ Smoke ─────────────► pytest against the deployed app    fails → DECLINE
   │
   └─ Smoke Advisor ─────► Abangku QA skill on JUnit + logs + diff
                             → recommendations, never changes pass/fail

• Standards Review runs before Deploy, so a Blocker stops the change before it reaches an environment. It runs on pull requests too, where it’s most useful.
• Smoke Advisor runs after Smoke only when Smoke failed (or always, in advisory mode). It explains the failure and suggests where to look. The skill can’t see the running app, so it gets the evidence: JUnit file, smoke log, and the diff of the deploy.

11.3 Which skill reviews what

Each folder in /content is one department, and every department publishes its rules at {origin}/{dept}/llms.txt. The review picks the department from the files that changed:

Each product keeps this map in a small file in qa-automation (for example <product>/review-map.yaml), so a product decides which departments apply to it.

Covering every department, now and later

Today each skill is fixed to one department (abangku-qa → qa, the others → swe). Writing one skill per new department doesn’t scale. Instead, add one generic gate skill that takes dept as input:

1. Read {origin}/{dept}/llms.txt for the section index.
2. Fetch the sections that match the diff.
3. Evaluate the **Rule:** blocks and return findings in the JSON format of 11.4.

A new folder under /content then joins the review as soon as it publishes its llms.txt, with no new skill and no pipeline change beyond the review map.

11.4 What the review returns

The pipeline needs data, not a report to read. The agent is asked to return JSON only:

{
  "dept": "swe",
  "content_fingerprint": "3f9c1e…",
  "verdict": "request_changes",
  "findings": [
    {
      "severity": "blocker",
      "section": "11-security",
      "rule": "Never hard-code secrets.",
      "file": "HttpTriggerOcrEtaxAsync/__init__.py",
      "line": 42,
      "issue": "API key assigned as a string literal",
      "fix": "Read it from the app setting OCR_ETAX_API_KEY"
    }
  ],
  "unavailable_sections": []
}

Only a finding that quotes a real **Rule:** block can be a blocker. A section without **Rule:** blocks is advisory: its findings are at most should_fix.

11.5 Wiring it in

Same pattern as section 10: the steps live in a template in qa-automation, and the product pipeline calls it.

- stage: StandardsReview
  dependsOn: Build
  jobs:
    - job: review
      steps:
        - template: templates/standards-review.yml@qa
          parameters:
            project: ocr-asynchronous        # folder with review-map.yaml
            mode: advisory                   # advisory | enforce
            depts: [swe, qa]                 # empty = take from review-map.yaml

What the template does, in order:

1. Checks out the product repo with enough history to diff against the target branch.
2. Installs the skills at a pinned version: install.sh --project --copy -q --origin <origin> (skills land in .claude/skills).
3. Builds the diff (git diff <base>...HEAD) and maps the changed paths to departments with review-map.yaml.
4. Per department, runs the agent headless with a fixed prompt: the skill name, the diff, “return JSON only in this schema”.
5. Validates the JSON. Invalid or missing output counts as unavailable.
6. Writes the findings as a PR comment and as a pipeline summary (Markdown), and saves the JSON as an artifact.
7. Exits 1 only when mode: enforce and there is at least one valid blocker.

The Smoke Advisor uses the same template with mode: advisory and the JUnit file as extra input.

Guard rails in the template

• Timeout per department (for example 5 minutes) and a diff size limit. Over the limit → review the changed file list only and say so.
• Never fail on infrastructure. No network, no model, a fetch error or a timeout gives unavailable and a warning, not a red pipeline.
• Read-only agent. The agent may read files and fetch the standards. It may not edit files, push, or call other services.
• The diff is data, not instructions. Text inside the diff (comments, strings) must never change what the agent does.

11.6 Rolling it out

Override: a Blocker the team disagrees with is waived by the QA lead with a comment on the PR (who, why, which rule). If the rule itself is wrong, fix the standard through a pull request on this site (section 1.4), not by turning the review off.

11.7 Keeping results repeatable

• Pin the standards version. Record the content fingerprint from {origin}/plugin/manifest.json with every run. A change to the docs shouldn’t suddenly fail pipelines that didn’t change.
• Pin the skills version. Install from a known bundle, not “latest”, and update on purpose (install.sh --update).
• Pin the model and keep the prompt in the template, not typed per pipeline.
• Same input, same answer. In phase 1, re-run a sample of reviews and compare. A Blocker that comes and goes isn’t a Blocker yet.

11.8 What it needs

• Model access from the pipeline agent (Anthropic API or Azure AI Foundry), with the key in a variable group, never in YAML.
• Network access from the agent to the standards site and the model endpoint (check the corporate proxy).
• A service account or token to post PR comments.
• **Rule:** blocks in the department’s pages. A department without them only gets advisory findings.
• A budget: one review per PR per department, on the diff only.

11.9 Example: OCR eTax Async

The deploy is still declined by the smoke result itself. The advisor only shortens the time from “red” to “we know why”.