A checklist for buying an AI interview platform: 24 questions, and the answers that should worry you

An AI interview demo is easy to make attractive. A fictional candidate answers well, a report appears with coloured bars, and in twenty minutes nobody has asked anything the product could not answer. Buying checklists exist for that: so the conversation is led by whoever will sign.

This one has twenty-four questions across six groups. What follows each group is what actually makes it useful: the answers that sound good and say nothing. A vendor can say yes to all twenty-four and still be wrong for you; what separates one from another is the texture of the answer.

Read it with a specific role in hand, not in the abstract. Answers change a great deal between “a hundred office candidates a year” and “three thousand a year for the plant”, and a checklist answered in the abstract commits nobody to anything.

Evidence and defensibility

The group most often skipped and the only one that matters the day somebody challenges a decision. Everything else can be fixed later; this cannot, because if the evidence was not produced, it does not exist.

1. Does every conclusion about a competency come with the verbatim fragment of the interview supporting it, or only a summary?
2. Can I go from that conclusion to the full transcript, and from there to the recording, without asking anyone for anything?
3. Is the question script versioned, and does an old interview still point at the version it was run with?
4. What happens when the recording fails halfway through: is the interview discarded, kept incomplete, or accepted anyway?

Sounds like a weak answer: “the model explains its reasoning”. An explanation generated afterwards is not evidence; it is a second opinion from the same system. What you need to see is the sentence the person said, with its position in the transcript. “You can listen to the whole recording” is weak too: that is raw material, not traceability, and nobody is going to listen to forty interviews to check one.

Data and retention

This is where a purchase stops or slips six months, almost always for not having asked in time. Your security team is going to ask these four; it is worth arriving with the answers.

1. Does our data live in a database shared with other clients, or in separate resources? If it is shared, what separates one of our rows from somebody else’s?
2. Where are the recordings and transcripts stored, and which external providers take part during the conversation?
3. What is the retention period, who can change it, and what evidence remains that the deletion happened?
4. What is delivered in writing about all of the above before signing?

Sounds like a weak answer: “everything is encrypted” and “we are compliant”. The first is true of almost any service and says nothing about who can read what; the second is not an answer, it is a category. And be wary of “data never leaves the country” said lightly: an interview with voice and video involves specialised services, and a vendor who cannot name them one by one has not done the exercise. Our own position on this is written down, including the uncomfortable part.

Bias and limits

The group where confident answers should worry you more than careful ones.

1. What exactly enters the analysis: only the text of what the person said, or also their face, their tone, or the pace of their speech?
2. What does the system do with a strong accent, a poor connection, or somebody who stumbles?
3. Does the system make rejection decisions on its own, or are they always executed by an identified person?
4. What does this product NOT do? Name me three things.

Sounds like a weak answer: “our system eliminates bias”. Nobody can stand behind that. A well-designed system reduces part of the bias — the variance between interviewers — and inherits another part, the bias of the criteria the script was written with. A vendor promising elimination either has not thought about it or would rather not say; the February issue separates the two. The fourth question is the best on the whole list: anyone who cannot name three limits of their own product either does not know it or does not want you to.

The candidate

Rarely asked because procurement does not require it, and it shows up anyway: in public reviews, in the share of people who abandon halfway, and in your employer brand.

1. What is the person told before starting, and at what point do they accept?
2. What happens if they do not want to be recorded?
3. How long does the interview take, and what happens if their connection drops halfway?
4. Does the experience carry our brand or the vendor’s?

Sounds like a weak answer: “they accept the terms when they enter”. Consent buried in a footer is not informed consent, and in a recorded interview that matters more than in almost any other product. Having no answer for a dropped connection is weak too: it means nobody has watched the product operate outside an office with good wifi. The March issue goes into detail.

Integration

Four questions that avoid the expensive discovery of week twelve.

1. What operations does the programming interface expose, and can I see its specification before signing?
2. How do I find out an interview finished: do I have to ask every minute, or does the system tell me?
3. Do the events come signed, and how do I verify they are yours?
4. If I retry a request because the network dropped, can I end up with two interviews for the same person?

Sounds like a weak answer: “we have an API”. Everyone has an API. The question is whether you can see it now, whether the events are declared in the specification itself — not merely described on a help page that falls behind the moment the code changes — and whether the request that creates accepts an idempotency key. The detail is in the June issue, and the specific operations are on the integrations page.

Operation and contract

The boring group, and the one that decides whether the project survives its second year.

1. Who loads our competencies and our levels, and what happens when we want to change them?
2. What permissions exist: are seeing a report and downloading a recording the same thing, or different?
3. What is recorded about who did what inside the platform?
4. If we stop being clients, what do we take with us, in what format, and how soon is what remains deleted?

Sounds like a weak answer: “we give you an export”. Of what exactly, in what format, and does the cited evidence survive it or only the result? And the last question is the one most often postponed: the exit is negotiated while you still hold the signature, not on the day you decide to leave.

Use it on us

It would be comfortable to publish a checklist written so that only we pass it. It would be useless, and it shows: anyone comparing two checklists can see which one is rigged.

So this one can be used against us in full, and it is worth doing. Integrations sets out which operations exist today, which events go out signed, and what gets designed with you per project rather than installed from a catalogue; security sets out who gets in, what they can see, and how long the material lives. Both pages are written so that a buyer can check them, not so that they agree with us — and where our answer is “that gets defined during implementation”, that is what it says, in those words.

What this checklist does not solve

Answering twenty-four questions well does not mean the product suits you. Your volume may not justify the setup, your roles may change too fast for a versioned script to add anything, or your real problem may sit earlier: in competencies nobody has written down. No platform fixes that, and it is worth finding out before buying.

What the checklist does do is change who leads the conversation. With it, the demo stops being a guided tour and becomes a review, which is what it should have been from the start. If you want to run it on us in person, the technical session exists for exactly that.