# Calibrations - Manual QA Calibrations are a quality exercise in which several evaluators review the same interaction using the same scorecard, and their answers are then compared against those of the reference evaluator (baseline). They are used to: * Align your QA team's judgment. * Identify questions or sections where evaluators disagree. * Measure consistency between auditors over time. -------------------------------------------------------------------------------- 1. ACCESSING THE MODULE From the sidebar, go to Quality and open Calibrations. * Two views you can switch from the top of the page: * Table — list of calibrations. * Analytics — aggregated view (charts and KPIs). * -------------------------------------------------------------------------------- 2. CALIBRATION STATES A calibration can be in one of two states: State Meaning Active At least one participant has not finished their evaluation, or the due date has not expired, and "Allow editing" is on. Completed No more changes are accepted. This happens when every participant finished and "Allow editing" is off, or when the due date has passed. Notes: * Only when a calibration is Completed do the final comparisons and scores become visible. While it is Active, the app shows a progress view (who has answered and who has not). * The results of evaluations conducted during calibrations do NOT affect agent or team scores. This information is stored in separate tables for independent reporting. -------------------------------------------------------------------------------- 3. CREATING A CALIBRATION From the main table, click New calibration. The form is split into three cards: 3.1. GENERAL DATA * Name * Due date — after this date, the calibration closes automatically. * Allow editing (optional): * Off: each participant loses the ability to edit their evaluation as soon as they submit it. If everybody submits before the due date, the calibration becomes completed. * On: participants can keep changing their answers until the due date. * Description (optional) — rich text to give context, goals, or instructions to the participants. Once saved, every participant is assigned and will see the calibration in their list. 3.2 INTERACTION First, link the interaction that the calibration will run on. 1. Type the interaction's external ID. 2. Click Search. If the interaction exists, it is linked, and the rest of the form unlocks. 3. If the ID does not exist, you will see an error message. If the interaction is already linked to a user, the baseline is prefilled for you, but you can change it. 3.3. SCORECARD AND BASELINE USER * Instrument (scorecard) — the rubric the calibration will use. * Reference evaluator (baseline) — the user whose evaluation is treated as the correct one, the anchor every participant will be compared against. Typically, the agent on the interaction, a supervisor, or a senior reviewer. * Participants — list of checkboxes with the users who will evaluate. * The baseline evaluator is automatically excluded from the list. -------------------------------------------------------------------------------- 4. EDIT, CLONE, OR DELETE From the actions menu (⋯) on each row: * View detail — opens the progress or comparison view. * Edit — available if you are the creator (or have global permission). You can only edit while the calibration is active. * Clone — creates a new calibration reusing the same data (scorecard, participants, description). Useful to re-run the same rubric on another interaction. The clone is assigned to you as the creator. * Delete — asks for confirmation using the calibration's name. Cannot be undone. -------------------------------------------------------------------------------- 5. PARTICIPATING AS AN EVALUATOR If you were invited to a calibration: 1. Go to the Chrome extension - Calibrations icon, and locate the one assigned to you. 2. Click View detail. 3. From the details, you can open the evaluation (Evaluation button). Answer the scorecard as usual. 4. After submitting, the evaluation is recorded. Depending on "Allow editing": * If off: you can no longer modify it. * If on: you can come back and change it while the calibration is still active. Once every participant has finished (or the due date arrives), the calibration switches to Completed. -------------------------------------------------------------------------------- 6. CALIBRATION DETAIL (EYE ICON) The detail has two modes, depending on the state: 6.1. IN PROGRESS (ACTIVE) Shows the progress view: * Overall progress bar with the calibration percentage (participants who completed vs. total — e.g., 2 of 4 = 50 %). * Baseline evaluator badge: shows whether the baseline has already answered or is still pending. * Participant list with individual status (Completed / Pending) and a link to the evaluation if they already submitted it. 6.2. COMPLETED Shows the comparison table: * Participant headers with the final score, the number of differences against the baseline, and a link to the full evaluation. * Rows grouped by section and question of the scorecard. * Baseline column with the reference evaluator's answers. * Participant columns with each participant's answers: * Green cell — matches the baseline. * Red cell — differs from the baseline. * For rating questions (scale), the answer is shown visually (dots/bar/stars depending on the scale), not just as a number, so it is easier to read. * At the top of the detail you will find the global metrics: * Calibration score (average of participants vs. baseline, rounded to 2 decimals). * Total differences. * Scorecard used, due date, and generation date. -------------------------------------------------------------------------------- 7. CALIBRATION ANALYTICS From the main table, switch to the Analytics view (URL ?view=analytics). It only reflects completed calibrations. 7.1. TOP FILTERS * Date range (last 30 days by default). * Scorecards and Participants (multi-select). 7.2. KPIS Three cards with the period's headline numbers: * Average score — average across calibrations in the period. * Total calibrations — how many were completed. * Total differences — sum of differences against the baseline. Clicking the Total calibrations KPI opens a side panel with the calibrations that add up to that number. 7.3. OVERVIEW BY SCORECARDS One row per scorecard: total calibrations, average score, and total differences. * Click a row → opens the scorecard drill-down (detailed analysis per section and question for that scorecard). 7.4. PERFORMANCE BY PARTICIPANT One row per participant: how many calibrations they took part in and their average score. * null values (participants with no data) are shown as —. 7.5. PARTICIPANT × SCORECARD (HEATMAP) Visual matrix where each cell represents the average performance of a participant for a given scorecard. * Click a cell → opens the side panel with the calibrations that match that participant/scorecard pair. 7.6. CALIBRATIONS SIDE PANEL Whenever you click something clickable (KPI, row, cell), a drawer opens with: * A list of calibrations (with status and score) that match the clicked context. * Selecting one displays on the right: * The comparison table if the calibration is completed. * The progress table if it is active (uncommon here, because analytics filters by completed). * Open detail button to jump to the full calibration page. 7.7. SCORECARD DRILL-DOWN From the scorecard table, click a scorecard to see: * The same KPIs, now filtered by that scorecard. * Expandable table with sections and questions: * Average score per question. * Columns per participant showing the score each one gave. * Question × Participant heatmap to spot where disagreements happen. * The breadcrumb returns to Analytics (not to the table). -------------------------------------------------------------------------------- 8. PERMISSIONS AND VISIBILITY What you can do depends on the permissions configured in your role. The relevant permissions are: * quality.calibrations:read.all — view every calibration. * quality.calibrations:read.own — view only the ones you created. * quality.calibrations:write.all — create, edit, and clone any calibration. * quality.calibrations:write.own — create, edit, and clone your own calibrations only. * quality.calibrations:delete.all — delete any calibration. * quality.calibrations:delete.own — delete your own calibrations only. * quality.calibrations.analytics [http://quality.calibrations.analytics]:read.all — access to the Analytics view. Based on the combination: Scenario What you see Only read.own / write.own / delete.own The table shows only your calibrations. Actions appear only on your rows. read.all but only write.own / delete.own You see them all, but you can only edit/delete your own. On the rest, those actions appear disabled. *.all across the board Full access, no restrictions. No analytics:read.all The Analytics view is hidden. If you open a URL you do not have access to (someone else's detail or edit without permission), you are redirected to /no-access. Note on anonymous auditors: if your organization enabled the "mask auditors" option and your role has no write permission on evaluations, auditor names will appear as Anonymous on screens that would otherwise show them (outside of the Calibrations module). -------------------------------------------------------------------------------- 9. FREQUENTLY ASKED QUESTIONS * Can I add or remove participants after creating the calibration? Yes, while the calibration is active and you have permission to edit it. If a participant has already answered, their answer is kept; if you remove them, they stop counting in the metrics. * What happens if a participant does not answer in time? When the due date expires, the calibration closes and that participant is recorded as "Pending". The comparison table is generated with the answers that were submitted. * Can a completed calibration be reopened? No. To re-run the exercise, use Clone: the clone starts as a brand new active calibration and keeps the scorecard, description, and participants. * Why can't I see the "Analytics" tab? Because your role does not have the quality.calibrations.analytics [http://quality.calibrations.analytics]:read.all permission. Ask an administrator if you need it. * Why does a calibration show — as its score? Either because it is not completed yet, or because there is not enough data to compute the average score (for example, no participant has answered yet). * What does "Differences" mean? Every time a participant's answer to a question does not match the baseline's answer, it counts as one difference. The total difference for a calibration is the sum of the differences for each participant across all questions. * Can I see the comments/feedback on the answers? Yes, from the Evaluation button next to each participant (and next to the baseline) in the detail view. It opens the full evaluation with per-question feedback, overall feedback, and selected reasons. * Does the calibration respect the question scale (1–5, 1–10, stars)? Yes. When a question is a rating, the comparison table shows the answer with a visual representation that matches the scale defined in the scorecard.