When the AI answers wrong, the cause is usually the question or its guideline, not the interaction. For process setup (sections, weights, fail types, KB articles), see the Process Setup Guide.
Like people, the AI can occasionally get an answer wrong, that's normal. Routine calibration and the option to edit AI answers in the AI Process help keep scores accurate over time.
1. How the AI answers
It only sees the question, its guideline, the KB articles attached to the process, and the interaction (conversation + channel, tags, custom fields, customer data). It does not see the section name, weight, fail type, expected answer or other questions.
It picks one answer and must find proof in the interaction.
Answer | When |
|---|---|
Yes | The question applies and there is proof it happened |
No | The question applies but it didn't happen, or there is no proof |
N/A | The situation never came up |
The guideline is mainly how you tell the AI when to answer No versus N/A.
2. Writing a Good Question
A good question asks about one thing that one person did, and you can prove it by pointing to a sentence in the conversation (or a field in the interaction metadata).
A simple test: if you can't picture the exact sentence that would prove a Yes, the AI can't either.
Step 1 — Build it from three parts
Part | Ask yourself | Example |
|---|---|---|
Who | Is it the agent or the customer? | "Did the agent…" |
What | What single action? | "…offer a refund…" |
When (optional) | Is there a specific moment? | "…before ending the conversation?" |
"Did the agent offer a refund before ending the conversation?"
Step 2 — Pick the right opening
Start with | Use it when | Example |
|---|---|---|
"Did the agent…" | The behavior should happen in every interaction | "Did the agent greet the customer?" |
"If…, did the agent…" | It only applies in some situations | "If the customer asked for a refund, did the agent explain how long it takes?" |
"According to the interaction metadata, …" | The answer is in tags, custom fields or customer data, not in the conversation | "According to the interaction metadata, does the contact reason match the customer's request?" |
Step 3 — Check it against these rules
Rule | Avoid | Better |
|---|---|---|
One thing at a time ("and", "or" usually mean two questions) | "Did the agent greet the customer and verify their identity?" | Two questions: one for the greeting, one for verifying identity |
Observable actions, not opinions (good, clear, empathetic) | "Was the agent empathetic?" | "Did the agent acknowledge the customer's frustration?" |
Ask about bad behavior directly, expected answer No (no "avoid", "not", "refrain") | "Did the agent refrain from sharing internal information?" | "Did the agent share internal information with the customer?" (expected: No) |
Only what the interaction can show (no tone, other tools, past contacts) | "Has the agent issued a refund?" | "Did the agent tell the customer the refund was issued?" |
Say who did it | "Was the issue resolved?" | "Did the agent resolve the issue?" |
No exact counts or times | "…use the name exactly 3 times?" | "…use the customer's name at least once?" |
Everyday words, or explain internal terms (guideline or KB) | "Was the HVC flow followed?" | Spell out the steps, or reference a KB article |
Check each behavior only once | Two greeting questions | Merge them into one |
Fit the channel | "Within three rings?" in an email process | Write the question for the channels the process evaluates |
Good to know
For metadata questions, name the field in the guideline exactly as it appears in the interacion, and first check it's actually filled in across channels (calls often arrive without tags).
Setting a question's expected answer to No for the first time changes how the AI reads the whole process. Retest it.
3. Writing the guideline
A guideline is optional: add one only when the question alone leaves room for doubt. It can be a single line, such as when the question applies, an example of what counts, or what to answer if the situation didn't come up. When you need to cover several of these, this structure keeps it clear:
YES: <what the agent must have said or done> (for example, "...").
NO: <what counts as a failure> (for example, "...").
If <the situation> did not happen, answer N/A.
Use only the lines you need. The N/A line is for questions that start with "If…", and many clear questions need no guideline at all.
Example — Question: "If the customer asked to be contacted later, did the agent offer a callback?". If the customer did not ask to be contacted later, answer N/A."
Common mistakes
Mistake | Why it fails | Fix |
|---|---|---|
A procedure in the guideline (5–6 steps for one Yes/No) | The most common cause of wrong answers: the AI can't tell which step the answer is about | One question per requirement; each "if" becomes an "If…, did the agent…" question |
Several conditions and exceptions | The AI must check facts it often can't see (plan, tenure, past discounts) | Use process filters for what isn't visible; one situation per question |
Guideline repeats or contradicts the question | Adds nothing, or confuses | Leave it empty, or move the extra case into the question |
Telling the AI what to answer ("Always Yes if chat") | Gives free points and hides reality | "If not a phone call, answer N/A", or filter chats out |
Depends on other questions ("Only if Q3 was No") | The AI doesn't see other questions or their answers | Describe the situation itself |
Long pasted text (scripts, policies, prices) | Buries what matters | Move it to a KB article and name it |
Rule of thumb: more than one "and", "or", "unless" or "except" in a guideline means it is trying to do too much.
Only tell the AI to answer Yes when not acting is genuinely correct (for example, "If the customer refused to give their ID, answer Yes").
4. Test, then check
In the Playground, try each important question with three conversations: one where the agent did it (expect Yes), one where the situation came up but the agent didn't (No), and one where it never came up (N/A). Read the AI's explanation, not just the answer:
Quote doesn't match the question → reword it.
Explanation and answer disagree → the question mixes two things; split it.
N/A versus No confusion → the N/A line in the guideline is missing or unclear.
Retest the whole process after setting an expected answer to No for the first time, or after changing KB articles.
5. Use Several AI Processes, Not One
Avoid a single AI process that tries to evaluate everything. Each process should cover one type of interaction, so every question applies and the AI isn't forced into N/A or guesses.
Create a separate process when:
The workflows are very different. For example, cancellations, technical support and sales each follow their own steps; give each its own process.
The channels are different. Calls, chat and email need different questions ("answered within three rings" makes no sense in email, and spelling can't be checked in a call).
Use the process filters (channel, contact reason, call direction, team) so each process only evaluates the interactions it was written for.
6. Review Evaluations and Give Feedback
A process keeps improving after it goes live. Review AI evaluations regularly and give feedback on the answers you disagree with. Each review helps you:
Improve the guidelines. Repeated wrong answers on the same question usually point to unclear wording or a missing Yes, No or N/A case.
Find what the AI is missing. If the AI fails because it lacks context (a policy, a product detail, an internal term), add it to a knowledge base article or the question's guideline.
Over time, this raises the accuracy of the process and the trust in its scores.
7. Checklist
☐ Asks about one thing, and says who (agent or customer)
☐ Proof is a sentence in the conversation or a field in the metadata
☐ Metadata questions start with "According to the interaction metadata" and name the field
☐ No opinion words (or they are defined), no "avoid" / "not" / "refrain"
☐ Makes sense alone; no other question checks the same behavior; fits the channel
☐ Guideline gives Yes and No with examples, plus the N/A line for "If…" questions
☐ Guideline has no procedures, scripts, policies, shared text or references to other questions
☐ Fail section / fail all only for policy-critical checks
☐ Tested in the Playground with Yes, No and N/A conversations
If you run into any issues, contact [email protected] or your Oversai account manager, we're happy to help!