# AI Process Adherence Evaluation — Setup Guide & best practices This guide explains how to design effective Processes so the AI can accurately evaluate agent interactions. Follow these best practices to get consistent, reliable scores. -------------------------------------------------------------------------------- WHAT IS A PROCESS? A Process is a checklist of evaluation criteria organized into Sections, each containing Questions. After every interaction, the AI reads the full conversation and answers each question based on what actually happened. STRUCTURE OVERVIEW Process ├── Instructions (optional global guidance for the AI) ├── Knowledge Base Articles (optional reference content) └── Sections (one or more, weights must add up to 100) ├── Section KB Articles (optional, section-specific reference) └── Questions (what the AI will evaluate) -------------------------------------------------------------------------------- SECTIONS Each process must have at least one section. Sections group related questions and carry a weight that determines how much they contribute to the final score. SECTION FIELDS Field Required Description name Yes A clear label, e.g. "Greeting", "Problem Resolution" weight Yes Percentage contribution to the total score (all sections must sum to 100) questions Yes List of questions in this section kb_articles No Up to 5 KB articles specific to this section EXAMPLE SECTION WEIGHTS Section Weight Greeting & Compliance 20 Problem Resolution 50 Closing 30 Total 100 > Important: All section weights must add up to exactly 100. The process will be > rejected otherwise. -------------------------------------------------------------------------------- QUESTIONS Questions are the core of a process. The AI reads each question and answers it based on the interaction. QUESTION FIELDS Field Required Description text Yes The question the AI must answer conditions No Guidance for the AI on how to answer the question — what to look for, when it applies, and how to interpret the result fail_type No Impact on score if the answer is No -------------------------------------------------------------------------------- SCORING The AI answers each question with one of three possible values: Answer Value Meaning Yes 1 The condition was triggered AND correctly followed No 0 The condition was triggered but NOT correctly followed N/A -1 The condition was never triggered in this interaction > Important: N/A questions (-1) are excluded from the score calculation. Only > triggered questions (Yes or No) count toward the section score. -------------------------------------------------------------------------------- WRITING GOOD QUESTIONS BE SPECIFIC AND BINARY Each question should have a clear Yes or No answer. Avoid vague or compound questions. Avoid Better "Did the agent do a good job?" "Did the agent confirm the customer's name at the start of the call?" "Was the agent professional and helpful?" "Did the agent avoid using informal language during the interaction?" "Did the agent handle the issue?" "Did the agent provide a resolution or next steps before ending the conversation?" -------------------------------------------------------------------------------- USING THE CONDITIONS FIELD The conditions field tells the AI: * What to look for * When the question applies * How to interpret edge cases * Where to find the answer (conversation vs metadata) Think of it as instructions for a human evaluator. EXAMPLE WITHOUT CONDITIONS (PROBLEMATIC) > Question: "Did the agent offer a callback?" If the customer never asked for one, the AI may incorrectly score this as 0 (No) instead of -1 (N/A). EXAMPLE WITH CONDITIONS (CORRECT) Field Value Question "Did the agent offer a callback?" Conditions "Only evaluate this question if the customer explicitly requested a callback." -------------------------------------------------------------------------------- GOOD USES OF CONDITIONS DEFINE WHAT COUNTS AS CORRECT > "Consider this met if the agent used any variation of the standard greeting, > not necessarily word-for-word." LIMIT THE SCOPE > "Only applies to phone calls." HANDLE EDGE CASES > "If the customer refused to provide their ID, answer Yes — the agent is not > required to insist." POINT THE AI TO METADATA > "Check the metadata field ticket_type before reading the conversation." -------------------------------------------------------------------------------- USING METADATA IN QUESTIONS AND CONDITIONS The AI evaluates: * The conversation transcript * Interaction metadata This includes structured fields such as: * Ticket type * Priority * Channel * Customer tier * Tags * Custom fields EXAMPLE METADATA STRUCTURE { "channel": "phone", "priority": "high", "ticket_type": "billing", "customer_tier": "enterprise", "tags": ["refund", "escalation"], "subject": "Overcharged on last invoice", "custom_fields": { "first_contact": true, "sla_breach": false } } > Available metadata fields depend on your integration. You can inspect real > interaction metadata inside the Playground. -------------------------------------------------------------------------------- METADATA-BASED QUESTION EXAMPLES EXAMPLE 1 — CHANNEL-SPECIFIC BEHAVIOR Field Value Question "Did the agent read the required legal disclaimer?" Conditions "Only evaluate this question if the metadata field channel is phone. Otherwise answer -1 (N/A)." -------------------------------------------------------------------------------- EXAMPLE 2 — PRIORITY-BASED REQUIREMENTS Field Value Question "Did the agent open the interaction with an apology for the inconvenience?" Conditions "Only evaluate if the metadata field priority is high." -------------------------------------------------------------------------------- EXAMPLE 3 — TICKET TYPE ROUTING Field Value Question "Did the agent transfer the interaction to the billing team?" Conditions "Only evaluate if the metadata field ticket_type is billing." -------------------------------------------------------------------------------- EXAMPLE 4 — CUSTOMER TIER TREATMENT Field Value Question "Did the agent mention the dedicated account manager program to the customer?" Conditions "Only evaluate if the metadata field customer_tier is enterprise." -------------------------------------------------------------------------------- EXAMPLE 5 — TAG-BASED EVALUATION Field Value Question "Did the agent offer to escalate the case to a supervisor?" Conditions "Only evaluate if the metadata field tags contains escalation." -------------------------------------------------------------------------------- BEST PRACTICES FOR METADATA QUESTIONS * Explicitly name the metadata field * Specify the exact expected value * Define the N/A case * Document non-obvious metadata values in Process Instructions * Always test using real interactions in the Playground -------------------------------------------------------------------------------- FAIL TYPES The fail_type field controls what happens when the AI answers No. fail_type Impact When to Use default Loses points for this question only Standard questions fail_section Entire section scores 0 Critical section-level failures fail_all Entire process scores 0 Compliance or safety failures EXAMPLE Section: Compliance (weight: 30) ├── [fail_all] Did the agent record the mandatory consent disclaimer? ├── [fail_section] Did the agent verify the customer's identity? └── [default] Did the agent use the customer's name at least once? -------------------------------------------------------------------------------- Knowledge Base (KB) Articles KB Articles are reference documents the AI can use during evaluation. Useful for: * Scripts and templates * Policies * Product information -------------------------------------------------------------------------------- TWO LEVELS OF KB Level Scope Max Articles Process-level KB Available for all questions 5 Section-level KB Available only within that section 5 > Best Practice: Use section-level KB whenever possible to keep the AI focused > and reduce irrelevant context. -------------------------------------------------------------------------------- WRITING EFFECTIVE KB ARTICLES RECOMMENDATIONS * Use clear titles * Keep content concise * Match the article content to the question scope EXAMPLE Title: Standard Refund Policy Content: Refunds must be processed within 5 business days. Agents must communicate the expected timeline and provide a reference number. Partial refunds require supervisor approval. -------------------------------------------------------------------------------- PROCESS INSTRUCTIONS The instructions field provides global guidance across the process. Use it to: * Define interaction context * Clarify terminology * Explain edge cases EXAMPLE This process evaluates inbound technical support calls. "Resolution" means either: - The issue was solved during the interaction OR - A clear next step was communicated. Agents are not expected to solve hardware issues during the first call. > Keep instructions concise. Use KB Articles for long-form reference content. -------------------------------------------------------------------------------- RECOMMENDATIONS KEEP PROCESSES FOCUSED ON ONE INTERACTION TYPE Avoid creating a single process for many unrelated scenarios. AVOID Process: "All Customer Interactions" ├── Retention discount ├── Technical troubleshooting ├── NPS request └── Billing verification This creates too many N/A results and inconsistent scoring. BETTER Process Covers Inbound Technical Support Troubleshooting and escalation Cancellation & Retention Save attempts and churn prevention Billing & Payments Refunds and verification Post-Sale Onboarding Product walkthroughs -------------------------------------------------------------------------------- Retention Flows Deserve Their Own Process Retention interactions differ fundamentally from support interactions. Support success = issue resolution Retention success = customer stays Retention processes should include: * Empathy acknowledgment * Churn reason discovery * Retention offers * Escalation paths -------------------------------------------------------------------------------- USE FEWER KB ARTICLES, NOT MORE More KB content creates more noise for the AI. BEST PRACTICES * Attach only directly relevant documents * Consolidate overlapping documents * Prefer section-level KB * Keep excerpts short and focused Scenario Recommendation Long policy Extract only relevant rules Multiple overlapping docs Merge them General company background Do not attach Scripts or required wording Attach at section level -------------------------------------------------------------------------------- ONE PROCESS PER USE CASE, NOT PER TEAM Do not duplicate identical processes across teams. Create a new process only when: * Questions differ * Scoring logic differs * KB references differ -------------------------------------------------------------------------------- QUICK CHECKLIST Before saving a process: * Section weights add up to 100 * Questions are clear and binary * Conditions explain non-obvious logic * Metadata fields are explicitly named * Critical questions use correct fail types * KB articles are concise and relevant * Instructions remain short and focused * The process has been tested in the Playground