AI Process Adherence Evaluation — Setup Guide & best practices

This guide explains how to design effective Processes so the AI can accurately evaluate agent interactions. Follow these best practices to get consistent, reliable scores.

What Is a Process?

A Process is a checklist of evaluation criteria organized into Sections, each containing Questions. After every interaction, the AI reads the full conversation and answers each question based on what actually happened.

Structure Overview

Process
├── Instructions  (optional global guidance for the AI)
├── Knowledge Base Articles  (optional reference content)
└── Sections  (one or more, weights must add up to 100)
    ├── Section KB Articles  (optional, section-specific reference)
    └── Questions  (what the AI will evaluate)

Sections

Each process must have at least one section. Sections group related questions and carry a weight that determines how much they contribute to the final score.

Section Fields

Example Section Weights

Important: All section weights must add up to exactly 100. The process will be rejected otherwise.

Questions

Questions are the core of a process. The AI reads each question and answers it based on the interaction.

Question Fields

Scoring

The AI answers each question with one of three possible values:

Important: N/A questions (-1) are excluded from the score calculation. Only triggered questions (Yes or No) count toward the section score.

Writing Good Questions

Be Specific and Binary

Each question should have a clear Yes or No answer. Avoid vague or compound questions.

Using the conditions Field

The conditions field tells the AI:

• What to look for

• When the question applies

• How to interpret edge cases

• Where to find the answer (conversation vs metadata)

Think of it as instructions for a human evaluator.

Example Without Conditions (Problematic)

Question: "Did the agent offer a callback?"

If the customer never asked for one, the AI may incorrectly score this as 0 (No) instead of -1 (N/A).

Example With Conditions (Correct)

Good Uses of Conditions

Define What Counts as Correct

"Consider this met if the agent used any variation of the standard greeting, not necessarily word-for-word."

Limit the Scope

"Only applies to phone calls."

Handle Edge Cases

"If the customer refused to provide their ID, answer Yes — the agent is not required to insist."

Point the AI to Metadata

"Check the metadata field ticket_type before reading the conversation."

Using Metadata in Questions and Conditions

The AI evaluates:

• The conversation transcript

• Interaction metadata

This includes structured fields such as:

• Ticket type

• Priority

• Channel

• Customer tier

• Tags

• Custom fields

Example Metadata Structure

{
  "channel": "phone",
  "priority": "high",
  "ticket_type": "billing",
  "customer_tier": "enterprise",
  "tags": ["refund", "escalation"],
  "subject": "Overcharged on last invoice",
  "custom_fields": {
    "first_contact": true,
    "sla_breach": false
  }
}

Available metadata fields depend on your integration. You can inspect real interaction metadata inside the Playground.

Metadata-Based Question Examples

Example 1 — Channel-Specific Behavior

Example 2 — Priority-Based Requirements

Example 3 — Ticket Type Routing

Example 4 — Customer Tier Treatment

Example 5 — Tag-Based Evaluation

Best Practices for Metadata Questions

• Explicitly name the metadata field

• Specify the exact expected value

• Define the N/A case

• Document non-obvious metadata values in Process Instructions

• Always test using real interactions in the Playground

Fail Types

The fail_type field controls what happens when the AI answers No.

Example

Section: Compliance (weight: 30)

├── [fail_all] Did the agent record the mandatory consent disclaimer?
├── [fail_section] Did the agent verify the customer's identity?
└── [default] Did the agent use the customer's name at least once?

Knowledge Base (KB) Articles

KB Articles are reference documents the AI can use during evaluation.

Useful for:

• Scripts and templates

• Policies

• Product information

Two Levels of KB

Best Practice: Use section-level KB whenever possible to keep the AI focused and reduce irrelevant context.

Writing Effective KB Articles

Recommendations

• Use clear titles

• Keep content concise

• Match the article content to the question scope

Example

Title: Standard Refund Policy

Content:
Refunds must be processed within 5 business days.
Agents must communicate the expected timeline and provide a reference number.
Partial refunds require supervisor approval.

Process Instructions

The instructions field provides global guidance across the process.

Use it to:

• Define interaction context

• Clarify terminology

• Explain edge cases

Example

This process evaluates inbound technical support calls.

"Resolution" means either:
- The issue was solved during the interaction
OR
- A clear next step was communicated.

Agents are not expected to solve hardware issues during the first call.

Keep instructions concise. Use KB Articles for long-form reference content.

Recommendations

Keep Processes Focused on One Interaction Type

Avoid creating a single process for many unrelated scenarios.

Avoid

Process: "All Customer Interactions"

├── Retention discount
├── Technical troubleshooting
├── NPS request
└── Billing verification

This creates too many N/A results and inconsistent scoring.

Better

Retention Flows Deserve Their Own Process

Retention interactions differ fundamentally from support interactions.

Support success = issue resolution
Retention success = customer stays

Retention processes should include:

• Empathy acknowledgment

• Churn reason discovery

• Retention offers

• Escalation paths

Use Fewer KB Articles, Not More

More KB content creates more noise for the AI.

Best Practices

• Attach only directly relevant documents

• Consolidate overlapping documents

• Prefer section-level KB

• Keep excerpts short and focused

One Process Per Use Case, Not Per Team

Do not duplicate identical processes across teams.

Create a new process only when:

• Questions differ

• Scoring logic differs

• KB references differ

Quick Checklist

Before saving a process:

• Section weights add up to 100

• Questions are clear and binary

• Conditions explain non-obvious logic

• Metadata fields are explicitly named

• Critical questions use correct fail types

• KB articles are concise and relevant

• Instructions remain short and focused

• The process has been tested in the Playground