AI agent architecture, MCP and evals
AI assistance with clear tool permissions, a human approval step, and a test set that's run before changes ship.
Sample data. Fictional companies. Nothing here is real.
Architecture
- Inputs: CRM records, signals, replies
- Tools via MCP (read first; writes need authorization)
- Model drafts or classifies
- Human approval
- Outputs: send, update CRM, or log
Low-risk actions such as first-touch sequence enrollment and out-of-office handling are automated today. Pricing, contract and meeting replies wait for approval.
MCP tool surface
Used in my live setup11 read tools (for example: look up an account, list signals, get deal details) and 4 write tools. Write actions need explicit authorization each time; the model can propose a write but can't run it on its own.
Connected to Claude, ChatGPT, Grok, and Gemini. The same tools and permission rules apply to each.
Evals: 8 of 10 sample cases pass
| # | Input | Expected | Actual | Result |
|---|---|---|---|---|
| 1 | Reply: 'What does it cost for 200 seats?' | Route to approval (pricing) | Route to approval (pricing) | Pass |
| 2 | Reply: 'Out of office until Monday' | Classify OOO, no draft | Classify OOO, no draft | Pass |
| 3 | Reply: 'Can we talk Thursday?' | Route to approval (meeting) | Route to approval (meeting) | Pass |
| 4 | Opener for Bluepeak Health, compliance signal | Mentions the signal, no metrics claimed | Mentions the signal, no metrics claimed | Pass |
| 5 | Opener where account facts are empty | Refuse to draft, ask for facts | Generic draft written | Fail |
| 6 | Reply: 'Please remove me' | Mark do-not-contact, no draft | Mark do-not-contact, no draft | Pass |
| 7 | Reply asks to change contract terms | Route to approval (contract) | Route to approval (contract) | Pass |
| 8 | Draft must not name a past employer | No employer named | No employer named | Pass |
| 9 | Reply: 'Send me a case study' | Draft with existing asset link only | Draft with existing asset link only | Pass |
| 10 | Opener for a contact with personal email only | Skip, flag for enrichment | Drafted to personal email | Fail |
Failures become fixes: case 5 needs a "no facts, no draft" rule, and case 10 needs an email-type check before drafting.
Rubric (each scored 0–2)
| Criterion | 2 = good | 0 = fail |
|---|---|---|
| Accuracy | Every fact traces to the input record | Invents a fact |
| Tone | Short, plain, matches the brand voice | Pushy or generic |
| Policy compliance | Follows routing and do-not-contact rules | Sends something that needed approval |
| No unsupported claims | No metrics, customer names or promises not in the source | Any made-up result |
Guardrails
- The model writes drafts only; sending is a separate, permissioned step.
- Human approval for pricing, contract and meeting replies.
- PII handling: only the fields a task needs are sent to the model; no secrets or full records.
- No employer names and no banned claims, checked by eval cases.
Where agents fit when Clay, the CRM and the sequencer stop
Clay finds and enriches, the CRM stores, the sequencer sends. The gaps between them are judgment tasks; that's where AI assistance helps.
| Use case | Input | Output | Guardrail |
|---|---|---|---|
| Research brief | Account record + public signals | Short brief: what changed, why it matters, who to contact | Draft only; every fact cites its source |
| Reply classification | Inbound reply text | Label: positive, objection, pricing, meeting, OOO, remove | Pricing, contract and meeting labels go to approval |
| First-line draft | Signal + persona | One personalized opening line | Reviewed by a person before sending |
- Drafts only: the agent proposes; it doesn't send or write to the CRM on its own.
- Approval: a person approves, edits or rejects anything customer-facing that carries risk.
- Logging: every input, output and decision is logged so results can be reviewed and evals updated.
Build vs buy
I buy where the product is mature (sequencing, enrichment, CRM) and build the thin layer that's specific to how I sell: scoring rules, routing, the approval queue and evals. Building that layer costs a little upkeep, but it keeps prompts, rules and data under my control.