How to Design Screening Questions an AI Interviewer Can Actually Score

Generating screening questions is a solved problem. Any LLM will give you 30 in ten seconds. The hard part is choosing the ones that produce answers an AI interviewer can tell apart, so your shortlist reflects real differences between candidates instead of who wrote the longest reply.

An AI screening interview is a first-round interview run by an AI interviewer that asks candidates a set of questions, scores each answer against criteria you define, and hands you a ranked report before any human gets involved. If you want the full mechanics, start with how AI interviews work. This guide covers the part most teams skip: question design.

Why most screening questions fail in an AI interview

A question that works on a recruiter phone call often fails in an AI interview. A recruiter fills gaps with tone, follow-up instinct and company context. An AI interviewer scores what the candidate actually says against what you told it to look for. Three types of question break under that condition.

1. Self-report questions. "Are you comfortable with ambiguity?" Every candidate says yes. When everyone gives the same answer, scores bunch together and the question stops ranking anyone.

2. Unbounded prompts. "Tell me about yourself" invites a three-minute monologue with no clear criteria to score against. Answers vary in length and direction, so scores vary for reasons that have little to do with ability.

3. Context-dependent judgment. "How would you fit into our sales culture?" A good answer depends on knowledge of your team that the AI doesn't have. The AI can't score fit it has never seen.

Most of the generic lists you'll find, including classic preliminary screening questions, were written for a human on the other end of the call. They still work well for how recruiters run preliminary screening by phone. For an AI round, rewrite them so the answer contains evidence the AI can check.

A question is scorable when a strong answer and a weak answer look different on paper. If you can't write down what separates them before the interview, the AI can't find the difference after it.

Anatomy of a scorable question

Structured interviews are the strongest predictor of job performance among common selection methods, according to a large meta-analysis by Sackett et al. (2022) in the Journal of Applied Psychology. "Structured" means every candidate gets the same questions and every answer is judged against the same criteria. An AI interviewer gives you that consistency for free, but only if each question carries its own scoring logic. If you want the fundamentals first, read what a structured interview is.

Every scorable question has four parts:

1. An anchored prompt. Tie the question to a past situation ("the last time you...") or a specific scenario. Anchors stop candidates from answering in generalities.
2. Expected signals. Two to four things a strong answer contains, written down before the interview goes live.
3. Scoring levels. What a weak, adequate and strong answer looks like. Three levels are usually enough.
4. A follow-up trigger. The gap that should prompt the AI to probe, such as a candidate saying "we" without explaining their own role.

In a conversational AI interview, you often won't write every question word for word. You define the topics and what good looks like, and the AI asks and probes within them. The four parts still apply; they just live at the topic level. Equip's guide to topics and scenarios in an AI interview shows how that works. For building full rubrics, see scorecards and rubrics.

How to tell if a question is working

The useful test of a question is score spread: how far apart candidates land. A question where every candidate scores 7 out of 10 tells you nothing, however sensible it sounds.

After your first 30 to 50 candidates, pull the scores for each question and compare them side by side. Take two questions from the same interview: Question 1 is self-report ("Are you comfortable with ambiguity?") and Question 2 is a scenario ("Your CEO and VP Sales each want a different top priority this quarter. Walk me through your next 48 hours.").

Illustrative example, scored out of 10.

Question 1 ranks no one: one point separates the best answer from the worst. Question 2 spreads candidates across six points, so it is doing the real shortlisting. Run three checks on every question:

• Spread. If scores cluster within a point or two, rewrite the question or drop it.
• Length. If longer answers score higher regardless of what they say, your criteria are rewarding volume. Tighten the expected signals.
• Follow-up rate. If a question almost always triggers a follow-up, the prompt is too vague. Anchor it to a specific situation.

Knockout questions vs. scored questions

Knockout questions are pass or fail eligibility checks. If a candidate fails one, nothing else in the interview matters. Scored questions are everything that differentiates candidates who pass.

Keep knockouts to three to five, put them first, and filter on them rather than scoring them. Mixing them into scored questions inflates totals: a candidate who confirms their notice period earns points that say nothing about their ability. Knockouts work the same way as the hard filters in CV shortlisting, applied one step later.

A note on compliance. Never ask about protected characteristics such as age, religion, caste, marital status, pregnancy or disability, even as a knockout. Rules on automated hiring also vary by jurisdiction; some, such as New York City, require bias audits of automated hiring tools. Check what applies where you hire. For a wider view, see Equip's guide to AI hiring compliance.

What to give the AI and what to keep for the human round

An AI interviewer is best where consistency matters most: the same questions, scored the same way, across hundreds of candidates. It is weakest where the right answer depends on context only your team has. That split is why AI interviews and phone screens do different jobs.

Use this rule: if two recruiters would score the same answer differently without a rubric, either write the rubric or move the question to the human round. Inconsistent scoring is also where bias creeps in, which is why reducing bias in hiring starts with shared criteria. And the human round isn't going anywhere; here's what AI interviews can't replace.

How much follow-up depth to allow

Follow-ups are what separate a conversational AI interview from a fixed question list. If you're still choosing a format, compare one-way vs. conversational AI interviews. More follow-ups mean deeper evidence, but also longer interviews and more drop-off.

• Use follow-ups on past-behavior questions. They separate "I did" from "we did." One or two probes per question is usually enough.
• Skip follow-ups on knockouts and on technical questions with one right answer. Probing a yes or no adds time without adding signal.
• Watch interview length. In Equip's data, interviews with four or more follow-ups had a 63% lower completion rate than those with two or fewer.

Three roles generic AI question lists get wrong

Generic models write generic questions, and they struggle most with roles that are new or loosely defined. That's the same gap that makes static question banks age so quickly. Here are three roles where a default LLM list misses the point. You can watch sample AI interviews for all three.

GTM engineer

Where generic lists fail: LLMs confuse the role with Google Tag Manager or sales operations, so you get questions about tag setup or CRM hygiene.

Signals to score: outbound and enrichment workflows built in tools like Clay, API and webhook work, and a pipeline outcome they can measure.

Knockouts: comfort writing scripts or using APIs; availability for the role's working hours.

Scored questions:

• "Describe an outbound workflow you built end to end: the data sources, how you handled enrichment failures, and what changed in reply rate." Signals: named sources, a failure-handling step, a before and after number.
• "A webhook from your CRM to your enrichment tool fails silently for a week. How do you find out, and what do you change?" Signals: monitoring, root cause, prevention.
• "How do you decide whether to automate a sales task or leave it manual?" Signals: volume, error cost, maintenance effort.
• "Tell me about a workflow you shut down. Why?" Signals: honest evaluation, a metric that justified it.

Keep for the human round: how they'd work with your specific sales team and stack.

AI-native developer

Where generic lists fail: LLMs default to algorithm puzzles or machine learning theory, which says little about how someone builds with AI tools every day.

Signals to score: how they use coding agents, how they manage context, how they verify AI output, and where they choose not to trust it.

Knockouts: hands-on experience with at least one AI coding tool; the primary language the role needs.

Scored questions:

• "Tell me about a bug an AI coding tool introduced that you caught. How did you catch it?" Signals: a specific bug, a verification habit, what they changed afterward.
• "How do you give a coding agent enough context on a large codebase without overwhelming it?" Signals: scoping, file selection, instructions or rules files.
• "When would you connect a tool to an agent through MCP rather than writing the integration yourself?" Signals: trade-offs, security, maintenance.
• "Walk me through how you review a pull request that was mostly written by AI." Signals: tests, edge cases, reading the diff rather than the summary.

Keep for the human round: a live pairing session on your own codebase.

Chief of Staff

Where generic lists fail: LLMs fall back on leadership clichés, such as "Describe your leadership style," which every candidate answers well.

Signals to score: how they set an operating cadence and how they prioritize between conflicting executive asks.

Knockouts: experience working directly with a founder or senior executive; availability to be in the office the role requires.

Scored questions:

• "Your CEO and VP Sales each want a different top priority this quarter. Walk me through your next 48 hours." Signals: gathering facts, a decision path, how they communicate the outcome.
• "Describe a weekly or monthly operating rhythm you set up. What did you cut after the first month?" Signals: a concrete cadence, iteration.
• "Tell me about a cross-team project that stalled. How did you unstick it without formal authority?" Signals: influence, specific actions, the result.
• "What's a decision you pushed back on, and how did it land?" Signals: judgment, candor, the relationship afterward.

Keep for the human round: trust and working chemistry with the founder. No AI round can test that.

Checklist before you publish an AI interview

Run every question through these eight checks. If you're building the interview from scratch, start with how to create an AI interview from a job description, then apply this list.