# Relay AI Quality Audit — Synthetic Sample

> Demonstration only. Every interaction, company detail, metric, and finding in
> this file is fictional and manually composed to demonstrate the deliverable.
> This is not a customer result, live audit, benchmark, or detection-rate claim.

## Executive summary

- Interactions analyzed: 10
- Meaningful disagreements: 6
- Candidate hidden failures: 4
- Human-confirmed material failures: 3
- Candidates rejected in review: 1
- Accepted regression-test candidates: 3

The fictional incumbent support assistant passed the organization's existing
checks on all ten examples. Human review of the illustrative Relay findings
confirmed three material issues: stale policy retrieval, skipped operational
controls, and an unsupported billing claim.

## Confirmed finding 1 — SUPPORT-001

- Category: Factual error / stale retrieval
- Severity: High
- Task: A customer asks whether a purchase made 35 days ago can still be refunded.
- Current answer: Yes. Approve the refund because purchases are refundable for 60 days.
- Challenger answer: The standard window is 30 days, so this request is outside policy. Ask a manager to review whether a documented exception applies.
- Evidence: The supplied context says the current standard window is 30 days. The trace shows the incumbent retrieved a retired policy version and did not compare its effective date.
- Regression test: When policy versions conflict, require the latest effective version and preserve any authorized exception path.

## Confirmed finding 2 — OPS-004

- Category: Missing assumption / skipped control
- Severity: High
- Task: Should the operator deploy the configuration change now?
- Current answer: Deploy immediately; the change is low risk and no additional checks are needed.
- Challenger answer: Do not deploy yet. Run the required dry run and obtain approval for the pending change ticket first.
- Evidence: The runbook requires both controls. The trace shows `dry_run=not_run` and `change_ticket_status=pending`.
- Regression test: Reject production deployment approval unless both required controls are explicitly satisfied.

## Confirmed finding 3 — SUPPORT-008

- Category: Important omission / unsupported action
- Severity: Medium
- Task: A customer reports that an invoice contains an unfamiliar charge.
- Current answer: The charge is probably a duplicate. Tell the customer it will disappear automatically.
- Challenger answer: Verify the charge against the invoice ledger and escalate to billing if its source cannot be confirmed.
- Evidence: No ledger lookup appears in the trace. The supplied procedure requires verification before describing a correction.
- Regression test: Do not identify a charge or promise resolution unless the billing tool returns matching evidence.

## Interpretation

Model findings are triage candidates, not ground truth. A real customer reviewer
confirms materiality, rejects false positives, approves categories, and chooses
which findings become regression tests. A small pilot does not prove general
accuracy or compare models globally.