How Teams Are Assessing AI Skills in Hiring Today

TL;DR: Companies are trying everything from "prompt engineering trivia" to live vibe coding interviews. The best AI skills assessments look less like quizzes and more like realistic work samples: can the candidate use AI to think, produce, verify, and improve?

AI skills are becoming part of almost every role, not just engineering. Sales, support, operations, marketing, finance, recruiting, and leadership roles increasingly benefit from people who know how to use AI well.

At Gradient, we talk to a lot of teams about their hiring processes. We love learning how organizations structure interviews, what they consider high signal, and how they decide whether someone is likely to succeed in a role.

As more and more companies are adopting AI and using it to rethink internal processes, they also find themselves asking this question:

How do you assess AI skills in hiring?

Right now, it's the wild west: companies are trying a huge range of strategies. The pattern is clear: the more realistic the assessment, the higher the signal, but also the greater the risk around privacy, equity, scalability, and interviewer time.

Here are the main approaches we've seen, and how we think about them.

1. Asking "How do you use AI today?"

This is the most common approach.

Behavioral questions are easy to add to an interview loop, they're consistent across candidates, and require zero setup. A recruiter, hiring manager, or peer interviewer can ask:

• What AI tools do you use?
• How do you use AI in your current work?
• Can you give an example of something AI helped you do faster or better?

But, typically we hear that this is low-signal. Candidates give rehearsed answers, and there's usually not enough time to dig into the specifics. And if the interviewer does not personally use AI deeply, it can be hard to tell the difference between someone who casually uses ChatGPT and someone who has genuinely changed how they work.

2. Asking AI or prompt engineering trivia

Another approach is to ask candidates technical questions about AI.

For example:

• What is retrieval-augmented generation?
• What does "LLM as judge" mean?
• How would you evaluate whether an AI-generated answer is correct?
• What makes a prompt effective?

These questions can cut a little deeper than "How do you use AI?" They can reveal whether a candidate understands the vocabulary and basic concepts behind modern AI tools, but they still tend to be shallow.

AI fluency is not the same as knowing terminology. A candidate can define a concept without being especially effective at using AI in real work. The reverse is also true: some excellent AI users may not use the formal language that has developed around LLMs.

Trivia-style questions can be useful as a light screen for technical roles. But for most hiring processes, they shouldn't be treated as a proxy for practical AI skill.

3. Running a live "vibe coding" (AI-assisted work) session

Some companies ask candidates to screenshare while completing a task with AI.

For engineering roles, this often looks like "vibe coding": the candidate uses tools like Claude Code, Cursor, ChatGPT, or Copilot to build or debug something live. For non-engineering roles, it might be a live research, writing, analysis, or operations task.

The big benefit of this method is that you get to see how candidates work in real time: how they frame the problem, what context they give, how they balance speed and quality or verify facts and logic.

The downside is that live AI work sessions can be inequitable and are expensive for team time.

On the equity point, candidates may have different access to paid tools. They may be using personal accounts with private work history, and screen sharing can create privacy concerns, especially if the candidate is using their own laptop and AI accounts.

They also once again require the interviewer to know what good looks like to be able to ask probing questions and process what's happening live. Lastly, this can be expensive, since it requires your most AI-savvy team members to set aside synchronous time for interviews.

4. Asking candidates to show their ChatGPT or AI tool history

Another thing we've started to hear about are companies asking candidates to prompt ChatGPT to describe their working style, strengths, and weaknesses: in other words, letting their AI tool speak directly to how they work.

We strongly discourage this!!

It may seem like a shortcut to understanding how someone uses AI, but it creates serious privacy issues. Personal AI accounts can include sensitive work, personal writing, health questions, financial information, confidential company context, or other material that has nothing to do with the hiring process and could open up legal risks.

It's also limited in other ways: for example, if a candidate switches between Claude and Codex, half of their AI engagement and usage is in another tool. Similarly, if most of someone's AI use is on a work account, they may not be able to participate fully.

To summarize, this is high signal, and a bad idea. AI fluency should be assessed through job-relevant performance, not by inspecting someone's private workspace.

5. Creating a controlled candidate environment

The most interesting tactic we've heard from a company is that an AI-native startup mailed a candidate a laptop that was connected to company internal services so that they could complete a take home, using the company's AI SlackBot to ask questions and pull real corporate context.

This is an interesting approach: everyone gets the same tools, the same task, and the same constraints. It also avoids asking candidates to expose personal accounts. Plus, since the candidate is interacting with the SlackBot, the company can also see what questions they asked as a proxy for thought process.

However, this is very expensive, definitely not scalable, and requires lots of plumbing: shipping boxes, an NDA, and likely to pay the candidate for doing real work.

But, we thought it was interesting and unique, so we've included it here!

6. Running an AI systems or agent design interview

A final option, popular with AI native companies like Sierra, is a live whiteboarding-style interview focused on AI workflows or agent design. In this case, AI isn't actually used, but candidates are asked to show deep systems level knowledge of AI.

This is especially useful for senior candidates, technical operators, product managers, and anyone expected to design AI-enabled workflows rather than simply use AI tools.

The risk is that this becomes too abstract, and the biggest downsides are the drag on team time and on resources for the most AI-native team members, who are best able to lead these types of sessions.

Summary: what an AI fluency assessment should be

To summarize themes, we think the ideal assessments for AI skills should be:

• Fair and equitable: Each candidate should have the same resources
• Privacy-forward: Candidates should not be asked to share private data or their own AI outputs
• Realistic: This should focus on how candidates use AI to do real work, not answer trivia questions.

For companies, it's practically also important to make sure your assessment approach is scalable, doesn't present too much of a drain on team time (especially for your most AI-forward team members) and is high-signal. It's also important to make sure interviewers are well-equipped to identify what good looks like if needed.

What a good AI skills assessment should measure

We believe that a strong AI skills assessment should measure whether a candidate can:

• Frame an ambiguous problem clearly
• Choose the right AI tool for the task
• Give the model useful context
• Break work into effective steps
• Iterate when the first answer is weak
• Verify AI-generated outputs
• Spot hallucinations or flawed reasoning
• Protect confidential information
• Combine AI output with their own judgment
• Produce something useful, not just plausible

This is the difference between AI usage and AI fluency: using AI as a reliable collaborator while maintaining taste, judgment, and accountability for the final work.

Where Gradient fits

Yes, this is Gradient's corporate blog, so this is the part where we talk about Gradient!

Teams also screen for AI skills by using Gradient assessments.

The core idea is simple: AI skills should be assessed through realistic work, with a consistent rubric, in an environment that respects candidates and gives hiring teams better signal.

Gradient helps teams evaluate how candidates actually work with AI: how they reason, prompt, iterate, verify, and produce role-relevant output. Instead of relying on self-reported AI usage or invasive personal tool checks, companies can run structured assessments that are fairer, more consistent, and easier to compare across candidates.

That matters because AI fluency is quickly becoming a hiring signal across many roles. But without a thoughtful assessment process, companies risk selecting for confidence, jargon, or tool access instead of actual capability.

Our goal is to help teams ask not just "Does this person use AI?" but rather "Can this person use AI to do better work?"

If that sounds interesting or relevant to you, please don't hesitate to reach out.