AI deliverables quality control for agencies

AI deliverables quality control is the review system that catches what AI gets wrong before a client sees it. One of the fastest ways to lose a client is to ship an AI-generated deliverable with a hallucinated statistic, a misattributed quote or a fabricated case study. The damage is not "your work was bad." It is "your work was confidently and professionally wrong in a way any careful human reader would have caught." The risk is common: in KPMG and the University of Melbourne's 2025 global study of more than 48,000 people, 56% said they had made mistakes in their work because of AI, and 66% said they rely on AI output without evaluating its accuracy.

This guide is the QA framework for AI-generated agency deliverables. It covers the specific failure modes you should be testing for, the checklists that catch them before delivery, the brand voice and factual accuracy workflows that prevent drift, the client disclosure posture that protects the relationship, and the operational discipline that scales QA as your AI-augmented production volume grows. It assumes you are already producing AI-assisted work and that the question is no longer "should we?" but "how do we not embarrass ourselves?"

Key Takeaways:

• Run every AI-generated deliverable through a four-pillar QA: hallucination check, source verification, brand voice consistency, and structural and formatting integrity.
• Maintain a per-client brand voice file (250 to 500 words of explicit do/don't) and use it as the QA reference, not the agency's general taste.
• Verify every factual claim against a primary source; if you cannot verify, remove or reword. Never "assume" the model is right.
• Disclose AI assistance proactively in your MSA or onboarding, not reactively when a client asks.
• Build a 5% to 10% sampling audit on top of 100% editorial review; the audit catches drift after model or prompt updates that editorial review misses.

The failure modes you are testing for

AI-generated deliverables fail in patterns that human-generated deliverables do not. The QA framework needs to be specific to those patterns, not adapted from generic editorial review.

Each of these is catchable with the right checklist applied by a human reviewer. Another model scoring the output is a useful addition for sampling, but it is not reliable enough to be the only check. The rule: AI-generated content needs at least one human who has read it end to end, with a specific list of things to check.

The four-pillar QA framework

Run every AI-generated deliverable through these four pillars before client delivery. The time estimates below add up to roughly 23 to 40 minutes for a 2,000-word piece.

Pillar 1: Hallucination check (10 to 15 minutes)

This is the highest-stakes pillar. The checklist:

• Every named person, company, study, or publication is verified to exist.
• Every numeric claim (percentage, dollar amount, count) is traceable to a primary source.
• Every direct quote is verified to the cited speaker, ideally with the original source link.
• Every "X% of companies say Y" claim has a real survey or report behind it, with a publication date.
• No claim is supported only by "a recent study showed" without naming the study.

The fastest way to run this: highlight every fact-claim in the draft and look up each one. If a claim cannot be verified in under 2 minutes, it goes. The model is almost never the right source.

Pillar 2: Source verification (5 to 10 minutes)

This is distinct from the hallucination check: it makes sure the sources you do cite are correct, current, and authoritative.

• Every cited URL resolves to the claimed content (link rot is real).
• Every cited statistic matches what the source actually says (paraphrasing can drift meaning).
• Source publication dates are recent enough to be relevant (2 years or newer for fast-moving topics).
• Sources are authoritative for the claim being made (a vendor blog is not authoritative for industry benchmarks).
• No source is a content farm, AI-generated site, or aggregator without primary attribution.

Pillar 3: Brand voice consistency (5 to 10 minutes)

Read against the client's brand voice file (see below). The checklist:

• Sentence rhythm matches client norm (short and punchy vs. long and analytical).
• Vocabulary level matches (technical, professional, conversational).
• First/second/third person usage matches client convention.
• Phrases on the "do not use" list are absent.
• Phrases on the "use sparingly" list are not over-indexed.
• Tone matches the client's positioning (authoritative vs. friendly vs. provocative).

Pillar 4: Structural and formatting integrity (3 to 5 minutes)

• Heading hierarchy is correct (H1, H2, H3 used as structure, not styling).
• Lists and tables are consistently formatted.
• Internal links point to live URLs on the correct domain.
• Word count is within 10% of brief.
• No section is internally inconsistent with another.
• Accessibility basics (alt text, link text, color contrast in visual deliverables).

Total time: 23 to 40 minutes per 2,000-word piece. For example, at a fully loaded editor cost of 65 to 95 USD per hour, the QA pass costs roughly 25 to 65 USD per deliverable. Build that into your pricing. If QA regularly takes much longer, you have either an under-trained editor or a producer who needs better prompts.

Building the per-client brand voice file

The brand voice file is the reference document the editor checks against. It is not the same as the agency's general style guide; it is client-specific and lives wherever your team accesses client briefs (in AgencyPro, the per-client Brand Hub on the Pro plan holds brand guidelines and assets).

A working brand voice file structure:

1. Voice descriptors (3 to 5 adjectives): "Authoritative, direct, lightly skeptical, never preachy"
2. Sentence rhythm: "Mix of short punchy sentences (8 to 12 words) and longer analytical ones (20 to 30 words). Avoid medium-length sentences without rhythm variation."
3. Person and voice: "Second person ('you') for instructional content; first-person plural ('we') for opinion."
4. Vocabulary level: "Educated professional. Assume B2B SaaS audience. Avoid jargon-heavy industry terms without unpacking them first."
5. Three reference passages: 100 to 200 words each of writing the client considers exemplary, ideally from their own published work.
6. Do-not-use list: Specific phrases, words, or patterns to avoid (e.g., "leveraging," "synergy," "in today's fast-paced world").
7. Use-sparingly list: Phrases that are fine in moderation but ring as AI-generic when overused (e.g., "it's important to note," "moreover," "in conclusion").
8. Hard constraints: Compliance, regulatory, or brand-safety rules (e.g., "never make first-person claims of patient outcomes" for healthcare clients).

Build this in the first 14 days of every retainer. Update it quarterly. Without a file like this, the editor is using "agency taste" as the reference, which produces inconsistency across editors and across deliverables.

The hallucination hunter's workflow

Hallucination is the failure mode that produces the worst client incidents. A dedicated workflow:

1. Extract claims into a spreadsheet. Every fact-statement in the draft, one row each. For a 2,000-word piece, expect 20 to 40 claims.
2. Categorize by claim type. Statistic, quote, study citation, named entity, date, methodology.
3. Verify each row against a primary source. Real publication, real study, real quote in real context.
4. Log the source. Even claims that pass should be logged with their verification source. This builds a reference library and protects you if a claim is challenged.
5. Reword or remove unverifiable claims. No exceptions. "Probably true" is not a verification.

For high-stakes content (legal, medical, financial, regulated industries), every claim must be verified by a domain reviewer in addition to an editor. It roughly doubles the QA cost, and it is non-negotiable: the legal exposure from a fabricated medical statistic in a healthcare client's deliverable far exceeds any margin gain from AI-assisted speed.

Treat three categories as highest risk: regulated industries, citation-dense content (whitepapers, reports, long-form research), and any deliverable that names third parties.

Brand voice drift detection

Brand voice drift is sneaky. The first AI-generated deliverable for a client may match voice perfectly; the tenth may have drifted significantly without anyone noticing because each individual piece felt "fine."

A monthly drift audit:

1. Pull the last 10 deliverables produced for the client.
2. Score each on a 1-5 scale against the brand voice file across three dimensions: vocabulary, rhythm, tone.
3. Plot the trend. If average scores have dropped more than 0.5 points over the period, the prompt template or producer needs recalibration.
4. Pull 2 to 3 specific examples of drift and update the brand voice file's do-not-use list accordingly.

This audit takes 60 to 90 minutes per client per month and catches the slow creep that leads to "this doesn't sound like us anymore" emails a few months in.

Client disclosure posture

Whether to disclose AI use is no longer the question. The question is when and how. Many clients already assume their agency uses AI in production: in McKinsey's 2025 State of AI survey, 88% of respondents said their organizations regularly use AI in at least one business function. What clients are judging is whether you are honest about it.

The disclosure posture that builds trust:

A working MSA clause: "Agency uses generative AI tools as part of its production workflow, including [specific tools]. All AI-generated content is reviewed by a human editor against agency QA standards before client delivery. Client data is handled according to the data processing addendum and is not used to train third-party models. Agency remains fully responsible for the quality of deliverables, regardless of the production tools used."

The last sentence matters. AI assistance does not transfer accountability. If the work is wrong, the agency owns it. Have your lawyer review any MSA language before you use it.

For more on data privacy and AI in regulated industries, see our agency data privacy compliance guide and our healthcare marketing compliance deep-dive.

Sampled automated evaluation

100% editorial review catches individual defects. It misses systemic drift: the slow degradation that follows a model update, a prompt change, or a new producer joining the team. The fix is a sampled automated audit running on top of editorial review.

A working sampling protocol:

• Sample 5% to 10% of weekly production output per service line.
• Run a structured evaluation prompt through a different model than the one used for production.
• Score against the same four-pillar rubric: hallucination risk, source quality, voice match, structure.
• Track scores over time per producer, per client, per prompt template.
• Alert when rolling 4-week average scores drop more than 0.4 points on any pillar.

The infrastructure for this is lightweight: a weekly script that pulls random deliverables from production, runs an evaluation prompt, and posts scores to a Slack channel or dashboard. Tools like Braintrust, Langfuse, or Helicone formalize this, but a short script run weekly is enough for many agencies.

When alerts fire, freeze production briefly and investigate. Usually the cause is a prompt template that aged badly, a model version change that shifted output style, or a new producer whose work needs calibration. Catching it within a couple of weeks is the difference between fixing one deliverable and many clients receiving subtly off work.

The tooling stack

A practical AI QA tooling stack for a 10 to 30 person agency:

The tooling is the small part of the cost. The labor of QA itself (the editorial pass and the spot-check audit) is the larger expense, so track QA hours as a share of production hours.

Handling quality incidents

When a defective deliverable reaches a client (and eventually one will), the response framework:

1. Acknowledge within 24 hours. Do not delay while you investigate.
2. Investigate the root cause. Was it a prompt, a model update, an editor miss, a workflow gap? Be specific.
3. Fix the systemic issue. A patched deliverable is not enough; the underlying cause must change.
4. Communicate to the client. What happened, what was wrong, what is fixed, what you are doing differently going forward.
5. Document in the QA log. Every incident, every root cause, every fix. Review quarterly for patterns.

A fast, honest, specific response can keep the relationship intact. Hiding or minimizing the incident is what usually loses it.

Producer training and prompt hygiene

The producer side of QA matters as much as the editor side. A few disciplines:

• Prompt templates per client per content type. Documented, version-controlled, reviewed quarterly. No one-off prompts in production work.
• Required brand voice file reference in every prompt. The producer is responsible for loading the right voice context, not the editor for repairing voice drift.
• Source-required prompting. "Cite sources for every numeric claim, with a real URL" goes in every prompt template. Models can still invent citations, so the editor checks every one.
• Quarterly producer recalibration. Each producer runs through 3 to 5 reference deliverables per quarter and reviews their output against the brand voice file with a senior editor.

For deeper operational practices around AI-augmented production, see our agency automation guide.

Measuring QA effectiveness

Track these metrics monthly per service line:

• Editorial pass rate (first-time approval): Should rise as prompts and producers mature.
• Average revision rounds per deliverable: Track the trend per client.
• Client-reported defects: Tracked separately; investigate every one.
• QA labor as % of production labor: Set a target from your first quarter's data and watch for creep.
• Sampled audit score (rolling 4-week average) per pillar: Watch for trends.
• Incidents (defective deliverable reaches client): Should be near zero; investigate root cause on every one.

Rising production volume with falling rubric scores is the warning sign. Catch it early and you protect the client relationship; miss it and the client will find the defects for you.

Frequently asked questions

How much QA is enough for AI-generated content?

100% editorial review with the four-pillar checklist (23 to 40 minutes per 2,000-word piece) plus 5% to 10% sampled automated audit. High-stakes regulated content gets an additional domain review on every piece. Below this, you are taking on quality risk that will eventually surface as a client incident.

Can we use AI to QA AI?

Yes, for sampled audits, but not as the primary review. A separate model running the rubric against a sample of output catches systemic drift well. It misses individual hallucinations and brand voice nuances that a trained human editor catches. The two layers complement each other; neither replaces the other.

Who should own QA in an AI-augmented agency?

A senior editorial lead or a head of quality, reporting to the head of delivery or COO. They own the rubric, the brand voice files, the prompt library, the sampled audit infrastructure, and the producer training. In agencies under 20 people, this often sits with the founder or operations lead until volume justifies a dedicated role.

How do we disclose AI use to clients without scaring them?

Front-load disclosure in the MSA and onboarding, frame it as a productivity tool with mature QA wrapped around it, and emphasize that accountability remains with the agency. Most clients in 2026 already assume AI is part of agency production; they reward transparency, not absence.

What's the right way to handle a client who explicitly forbids AI use?

Honor it in writing in the SOW, document the workflow as fully human-produced, and price the engagement for the extra hours it takes. Some clients are right to want this: regulated industries, sensitive topics, or brands with explicit anti-AI positioning. Others may relax the restriction once they trust your QA, so revisit it at the first QBR.

QA is now a first-class agency capability

AI-assisted production made work faster and made confident mistakes easier to ship. The advantage goes to agencies with disciplined QA wrapped around their AI use, not to the ones adopting AI most aggressively. The four-pillar framework, the per-client brand voice file, the sampled audit, and the transparent client disclosure posture are the operational machinery of trustworthy AI-augmented delivery.

AgencyPro keeps client requests, review and approval steps, per-client brand files (Brand Hub, Pro plan) and versioned proofing (Pro plan) in one place. Book a demo to see the review flow.