An AI interview score is only as trustworthy as the rubric behind it. Hand two interviewers the same transcript with no scoring rules and you get two different verdicts. Hand an AI the same transcript with no rubric and you get a confident number with nothing underneath it. This post explains what an interview scorecard and an interview rubric actually are, how an AI interviewer turns answers into weighted topic scores, how to build your own rubric when the default one does not fit, and how the same discipline carries into the human rounds that follow.
It builds on the practical guide to how AI interviews work. If you are still deciding what to ask, start with designing questions an AI can score and come back here for the scoring side.
Interview scorecard vs. interview rubric: what is the difference?
The two terms get used interchangeably, but they do different jobs.
An interview rubric describes what a good answer looks like. For each competency, it spells out the levels: what a weak answer contains, what an adequate one contains, what a strong one contains. A rubric is the standard.
An interview scorecard is the form you fill in against that standard. It lists the competencies, gives each one a rating, and rolls the ratings up into a recommendation. A scorecard is the record.
You need both. Without a rubric, the ratings on a scorecard reflect whoever filled it in that day. Without a scorecard, the rubric stays in a shared drive and never shapes a decision. In an AI interview, the rubric is the evaluation criteria the model follows, and the scorecard is the report it produces.
Why structured scoring beats a good conversation
Interviewers trust their read of a candidate far more than the evidence supports. An unstructured conversation does not just fail to add signal, it adds noise that feels like signal.
Fixed criteria and a fixed scale fix three things at once. Every candidate is judged on the same dimensions, so scores can be compared. The reasoning is written down, so a score can be challenged and defended. And the standard stays put across weeks and interviewers, so the fiftieth candidate is held to the same bar as the first.
AI interviews inherit all three benefits only if the rubric is explicit. A model will score consistently against whatever criteria it is given, including vague ones. Consistency against a bad rubric is just a reproducible mistake, which is why the rest of this post is about getting the rubric and the weights right.
How an AI interview scorecard is built
A conversational AI interview is built from a list of items: fixed questions, open topics, or scenarios. The guide to questions, topics and scenarios covers how to choose between them. Here is what happens to the answers.
Step 1: each answer is evaluated against criteria
The AI evaluates what the candidate said on each item against the criteria for that item, then aggregates. A vague topic produces vague criteria, which produces a score you cannot defend later.
Step 2: topic scores are weighted by difficulty
Every item in an Equip conversational interview carries a difficulty of Medium, Hard, or Expert. Difficulty sets how much that item weighs in the final score: Medium items are worth 10 points, Hard items 20, and Expert items 30. An Expert topic therefore counts three times as much as a Medium one.
The practical consequence: difficulty is a weighting decision, not a description of how hard the questions feel. Mark the make-or-break skill for the role as Expert even if the questions themselves are routine, because you want that topic to dominate the score. Mark nice-to-have skills as Medium so a weak answer there cannot sink an otherwise strong candidate.
Step 3: a worked example
Say you screen a backend engineer on three topics and the candidate scores 60% on SQL, 90% on system design, and 90% on communication.
A flat average says 80% and this candidate looks fine. A difficulty-weighted score says 75% and shows that the skill you cared most about is the one they are weakest in. The exact aggregation varies by platform, so treat this as the principle rather than a formula to reproduce: the topics you flag as critical should move the score the most.
Step 4: the report
The output is the scorecard. On Equip, the default conversational report gives a hire recommendation, strengths, areas for improvement, a written summary, and a Topic Performance score for each item, followed by a timestamped transcript. Clicking a transcript line jumps the recording to that moment, which is how you check a score you do not believe. The full breakdown is in the AI interview reports help article.
When the default rubric does not fit: custom report templates
The default evaluator scores every interview the same way, which is right for most screening use. It is wrong when your team already has a rubric it trusts, when you hire for a role where the standard competencies do not apply, or when hiring managers want a specific set of fields they can compare across candidates.
A custom rubric for an AI interview comes down to three decisions, and each one buys you something specific:
- Write the standard in your own words. Brief the AI the way you would brief a new interviewer: the competencies that matter for this role, how much each one counts, and what separates a strong answer from a weak one. The payoff is that the score reflects your bar, not a generic one.
- Fix the fields every report must return. If every candidate comes back with the same set of scores and the same summary fields, you can line up twenty reports and compare them in minutes. Without fixed fields, each report is a different shape and comparison turns back into reading.
- Lay the report out for the person who reads it. Put the hiring manager's go or no-go field at the top, the supporting scores below, the summary last. A report that matches how the reader decides gets read; one that does not gets skimmed for a single number.
One rule holds all three together: every score you ask for needs a definition. If the report has a communication score but nothing says what a 9 looks like versus a 4, the AI will decide for you, and the number will mean something different from one candidate to the next. On Equip this is configured as an AI Interview Report Template and attached per interview; the custom report templates article has the setup steps.
How to read an AI interview scorecard without being misled
Read the Expert topics before the overall score. The overall number already weights them, but a candidate at 72% with 95% on the critical skill and 40% on a nice-to-have is a different hire from one with the reverse.
Check one low score against the transcript every time. It tells you whether the rubric is scoring what you think it is, and it catches badly written topics early. If the transcript shows a good answer and the score is low, the criteria need rewriting.
Set a cut-off per topic. A threshold on the critical topic prevents strong communication from compensating for a missing core skill, which a weighted score reduces but cannot fully prevent on its own.
Carrying the rubric into human rounds: ATS scorecards
The AI interview is usually the first scored stage, not the last. The candidates who pass go to a live interview, and this is where most pipelines lose their structure. The AI stage is auto-scored against fixed criteria; the human stage becomes a conversation and a Slack message.
An ATS scorecard closes that gap. It is the same idea applied to people: a reusable template that lists the criteria for the round, gives each one a rating, and rolls the ratings into a recommendation that every panellist fills in independently.
In Equip's ATS, scorecards attach to phone screening, live interview, and custom stages. A compact layout holds up to 10 criteria for a single scorer; a detailed layout groups criteria into up to 5 weighted sections for a panel, with blocks for an overall recommendation, strengths and concerns. Each panellist scores on their own page and the ratings combine into one result. AI interview and assessment stages do not take scorecards because they are already auto-scored, so the two systems do not overlap. The scorecards article covers the setup.
Two rules make this work across stages. First, build a separate scorecard per round, so interviewers rate only what that round assesses rather than re-scoring the AI's topics from memory. Second, keep the competency names consistent between the AI rubric and the human scorecard. When a hiring manager sees "system design" at 90% on the AI report and 6 out of 10 on the panel scorecard, the disagreement is useful. When the two use different labels for the same skill, the disagreement is invisible.
Five scorecard mistakes to avoid
- Every topic set to the same difficulty. If everything is Hard, nothing is weighted.
- Ranking on the overall score alone.
- Asking for a score the rubric never defines.
- Panellists scoring together. Discuss after submission, not before.
- Never spot-checking a score against the transcript.
Where to start
If you already run AI interviews, open your last five reports and check whether the Expert topics are the ones you would actually refuse to hire without. If they are not, change the difficulty settings before you change anything else; it is the highest-leverage edit in the whole system. If you are still setting up, decide the rubric first and the questions second. The screening questions guide and the one-way vs. conversational comparison cover those two decisions, and you can see what a finished scorecard looks like on the conversational AI interviews page.