Two earlier posts cover the people side of an AI rollout from different angles. How to upskill a mid-market team for AI without losing your best people is about the team you already have. Does a mid-market company need a Chief AI Officer is about the one senior hire. This post is about the ordinary hiring you do every month, now that every candidate claims the same skill.
Why does an AI line on the resume tell you so little?
Because it costs nothing to write and there is no agreed standard behind it. “Proficient in ChatGPT, Claude and Copilot” can mean the candidate rebuilt a reporting process around these tools, or that they asked one to write a birthday message once. Both are true statements. The resume cannot tell you which.
The same thing happened with “social media” on resumes fifteen years ago, when I was building and selling an agency in that field. Every candidate had it. Very few could explain what they had actually done with it that produced a result for a business. The fix then is the fix now: stop asking whether they have the skill and ask them to use it in front of you.
There is a second reason the resume line is weak. The tools are changing faster than any resume is updated. A candidate who learned one tool well in 2024 may be working with something quite different today. What carries over is not the tool. It is the way of working.
What are you actually hiring for?
Three habits, none of which is a tool.
- Judgment about when to use AI. Knowing that a first draft of a proposal is a good use and that a final number in a client invoice is not. Knowing which tasks the tool does well, which it does badly, and which it should not touch at all under your AI usage policy.
- The habit of checking. Treating every output as a draft from a fast, confident junior colleague who has not read your files. People with this habit verify figures, read sources and notice when a paragraph sounds right but says nothing.
- The ability to explain the result. A customer does not care what prompt produced the answer. A colleague who reviews the work needs to understand it. The candidate has to own the output as if they had written every word, because as far as your company is concerned, they did.
A five-person team can generate $32 million in revenue with AI-driven systems, as mine did. The systems do the volume. The people do the judgment. That is the ratio you are hiring for.
How do you run a 20-minute live task?
Pick one real piece of work from the role. For a sales coordinator, it might be turning a messy call transcript into a follow-up email and three CRM notes. For a finance analyst, it might be summarizing a vendor contract and flagging the renewal terms. For a marketing hire, it might be drafting a one-page brief from a product sheet. Use something low stakes and already finished, so nothing confidential is exposed and you have a real result to compare against.
Give the candidate the same tools your team uses, the material, and a clear description of the output. Then watch. You are looking for a few specific things.
| Watch for | What it looks like | What it is not |
|---|---|---|
| How they start | They read the material before touching the tool | They paste everything in and hit enter |
| How they prompt | Context, audience, format, what to leave out | A single line and hope |
| What they do with the output | Read it, check a fact, change something, ask again | Copy and declare it done |
| How they finish | Explain what they changed and why | Hand it over without comment |
At the end, ask them to walk you through the result as if you were the customer. The explanation is often more revealing than the work itself.
Which three questions separate users from operators?
- “Tell me about a time an AI tool gave you a wrong or misleading answer. How did you catch it, and what did you change afterward?” Serious users have this story ready and tell it with detail: the task, the mistake, the moment they noticed, the habit they built. Casual users either say it has never happened or describe something vague.
- “What would you not use AI for in this job, and why?” You want a thoughtful line, not a blanket answer. Someone who says “anything with a customer’s personal data, anything that goes out under my name without my reading it, and any final number” understands the risk. Someone who says “I use it for everything” does not.
- “Show me how you would explain this output to someone who distrusts AI.” Give them a paragraph the tool produced during the live task. The good answer translates it into plain language, points to what was checked, and does not hide that a tool was involved. That is exactly how they will need to speak to your customers and your more skeptical colleagues.
How should you score the answers?
Decide before the first interview what a strong, acceptable and weak answer looks like for each of the three habits, write it down, and score every candidate for the same role against the same sheet. Have a second person score independently. Compare notes after, not during. This is ordinary good practice for any skills assessment, and it protects you from hiring the candidate who was most fluent in the vocabulary rather than the one who will do the job best.
Weight judgment and checking above raw speed. In my experience the hires who go wrong with AI are rarely slow. They are fast and unchecked.
Is hiring an AI lead different from hiring for an ordinary role?
Yes, in one way. For the ordinary role, industry judgment matters more than tool fluency, because the tool can draft and only an experienced person can tell whether the draft is any good. For the dedicated AI lead, whether that is a promoted insider or an outside hire, tool fluency and the ability to teach others move up the list. Add a fourth item to the live task: ask the AI lead candidate to teach one of your managers how to do the task they just did. If the manager cannot follow, the candidate is a strong user but not yet a lead.
What should the job description say?
Name the actual tasks the role will use AI for, name the tools the company has standardized on, and state that the interview includes a short live exercise. That one sentence filters the field for you. Candidates who are comfortable doing the work in front of someone apply. Candidates whose AI experience is a line on a page tend to look elsewhere. Avoid listing a dozen tools as requirements. You will exclude good people who use a slightly different set and attract people who have collected logos.
If you want help deciding which roles should carry AI skills first and how to test for them, the engagements and FAQ page explains how the AI Clarity Sprint works. The prompts for drafting job descriptions, live tasks and scoring sheets are in ChatGPT for Profit, Volumes 1 and 2.
Frequently asked questions
- Should a mid-market company require AI experience in every job posting?
No. Require it where the role will use it daily, such as sales operations, marketing, customer service and finance, and treat it as a plus elsewhere. A posting that demands AI experience for a warehouse supervisor shrinks the candidate pool for no gain. What every role should require is willingness to learn the tools the company has chosen and to follow its AI usage policy.
- What is the best interview question to test AI skills?
Ask the candidate to describe a time an AI tool gave them a wrong or misleading answer, how they caught it, and what they changed afterward. People who use these tools seriously have a story ready and tell it with detail. People who listed AI on the resume because everyone does will struggle to answer.
- How long should a live AI task take in an interview?
Twenty minutes is enough. Give the candidate a real but low-stakes piece of work from your company, access to the tools you actually use, and a clear output to produce. Watch how they work rather than only judging the result. The point is to see their habits, not to get free labor.
- Should you hire someone with strong AI skills but weak experience in your industry?
For most mid-market roles, industry judgment matters more. An AI tool can draft a proposal in seconds, but only someone who understands your customers can tell whether the draft is any good. The exception is a dedicated AI lead, where tool fluency and the ability to teach others can outweigh industry background.
- How do you keep an AI hiring process fair?
Give every candidate for the same role the same task, the same tools, the same time and the same scoring sheet. Decide what a good answer looks like before the first interview. Have two people score independently and compare. This is the same discipline you would apply to any skills test, and it protects both the candidate and the company.
Hiring for AI skills and not sure what to test?
The AI Clarity Sprint gives mid-market companies, law firms, wealth managers and insurance agencies an AI audit, deployment recommendations with ROI models, and a 90-day plan in three weeks.