How to Run an AI Fluency Assessment Across Your Org

An AI fluency assessment shows where your org stands with AI tools, role by role, before you spend a dollar on training. Most companies start with a guess. They assume engineering is fluent, sales is behind, and everyone else sits somewhere in the middle. A structured assessment replaces that guess with data, so enablement can build the right training tracks, leadership can set realistic targets, and the whole org can see progress over time. This guide walks through how to run one that measures real skill.

If your team has already picked a tool, our guide on how to train your team to use Claude at work covers the rollout side. This post covers measurement.

What an AI Fluency Assessment Measures

A usage survey tells you how often people open a tool. To learn whether they use it well, you need to assess AI skills across employees on four components drawn from Anthropic’s AI Fluency course and its 4D framework.

• Delegation: the person picks the right tasks for AI and keeps the right ones for themselves.
• Description: the person gives the tool enough context and structure to get a usable result.
• Discernment: the person evaluates the output critically and catches errors.
• Diligence: the person applies the company’s data policy correctly and owns what they send.

Self-report surveys miss all four, because people rate their own skill generously. Performance tasks ask people to do the work, and then you score the result.

Step 1: Define AI Fluency by Role

AI fluency for a support rep looks different from AI fluency for a legal ops lead. The assessment has to reflect the real work of each function, or the results won’t tell you much.

For each role group, list the five to eight tasks where an AI assistant should be part of the workflow. Here is a starting point:

• Sales: pre-call research, follow-up drafting, and objection prep.
• Support: ticket summaries, response drafting, and root-cause hypotheses.
• Marketing: first-draft copy, competitive summaries, and FAQ generation.
• Legal ops: contract summaries, policy explainers, and process docs.

Those task lists become the assessment content. If you need a sense of what proficient looks like in each function, our AI fluency benchmarks by role give you targets to score against. The same principle applies when you are building AI roleplays around your actual sales methodology, because the practice has to look like the job.

Step 2: Design Three Task Types for Each Role

A complete AI competency assessment uses three task types. Each one maps to the four components, so together they give you a full picture.

This table shows which task type measures each of the four AI fluency components and how that task is scored.

Component	What it shows	Task type	How it is scored
Delegation	Picks the right tasks for AI and keeps the right ones	Production task	Rubric for accuracy, completeness, tone, and format
Description	Gives the tool enough context and structure for a usable result	Production task	Rubric for accuracy, completeness, tone, and format
Discernment	Evaluates output critically and catches errors	Verification task	Planted errors caught and fixed
Diligence	Applies the data policy correctly and owns what they send	Judgment task	Accuracy sorting inputs as allowed, allowed with changes, or not allowed

Production Tasks

The employee completes a real deliverable with the AI tool, using a realistic input the company provides. For example, a support rep gets a long ticket thread and drafts a resolution message. Score the output against a rubric for accuracy, completeness, tone, and format. Production tasks measure Description and Delegation.

Verification Tasks

The employee receives a plausible AI draft with planted errors and has to find and fix them. The errors might be a wrong product detail, a misstated policy, or an unsupported claim. Score on errors caught. Verification tasks measure Discernment, which is the component teams most often skip.

Judgment Tasks

The employee sorts a set of realistic inputs and requests under the company’s data policy. Each item is allowed, allowed with changes, or not allowed. Score on accuracy. Judgment tasks measure Diligence. If your policy owners want a shared reference point for governance language, the NIST AI Risk Management Framework is a common place to start.

Keep the whole assessment under 45 minutes. Long assessments get rushed or abandoned, and either one corrupts the data.

Step 3: Run the Assessment on a Consistent Platform

Spreadsheets and email work for a small pilot. At scale you need a platform that gives every employee in a role the same tasks, scores everyone against the same rubric, and reports by role, team, and component.

Yoodli built its Tool Training templates with an assessment component for this purpose. Admins pick a template for the tool in question, such as Claude for Workplace or Putting Copilot to Work. Then they customize the tasks and evaluation criteria for their org and assign it by role. The expanded Yoodli template library removes the design work that stalls most assessments before they start.

Each template pairs an AI Tutor session that teaches the skill with a mastery assessment that checks what people can actually do. Results roll up by component in Yoodli Measure. So enablement can see that a function is strong on production and weak on verification, then build for that.

Step 4: Treat the First Run as a Baseline

The first AI fluency assessment run sets your baseline. Hold off on leaderboards and consequences for now. The goal is a clear picture of where each function stands, and employees who fear the result tend to game it.

Say so from the start. Tell employees the company is measuring where it is today so it can build the right training, and that nobody’s rating depends on this round. Managers should get their team’s results as a coaching input. Performance conversations can come later, once the training has had time to work.

Step 5: Read the AI Skills Gap Analysis by Component and Role

The most useful view is a grid. Put roles down the side and the four components across the top, with a score in each cell. Patterns show up quickly.

• High production and low discernment means a team uses the tool a lot and trusts it too much. Verification training is the fix.
• Low production and high judgment means a team is careful but gets little value from the tool. Better examples and context-setting practice will help.
• Low judgment across the board points to how the data policy was communicated. Start with clearer policy training, then retest.

This AI skills gap analysis gives enablement a build list. It also gives leadership one view of workforce AI readiness across the company.

Step 6: Build Training Tracks From the Gaps, Then Reassess

The AI fluency assessment tells enablement what to build. A role that needs discernment gets a track heavy on verification scenarios. A role that needs description gets context-setting practice on real tasks, and a role that needs diligence gets policy scenarios with clear right answers.

Build the tracks, run them, and reassess the same roles in 60 to 90 days. If you already run onboarding and certification programs, the same mechanics apply here. Certify on demonstrated skill, then track it.

The change between baseline and reassessment is the number leadership wants to see, because it shows the training built a real capability. In Kirkpatrick model terms, that is Level 2 and Level 3 evidence, which most AI adoption programs never collect. The Kirkpatrick Partners overview explains each level if your team needs a refresher.

Use an AI Ride-Along Coach to Assess AI Fluency on Real Work

A tutor-led certification confirms that each person can complete a set task. Ride-along sessions come after it, once people know the basics and are applying the tool to their own work. In Yoodli, a ride-along is a type of AI roleplay. The employee shares their screen while working in Claude, ChatGPT, Microsoft Copilot, or Gemini, and the coach sees what they type and what the tool returns. The coach stays quiet until the employee asks a question. Then it gives one short suggestion based on your company’s guidance, and the employee keeps control of the work.

Take a support rep whose baseline showed low discernment. She brings a real ticket thread into Claude and asks for a draft reply to the customer. The draft mentions a refund window, so she asks the coach, “How do I check this before I send it?” The coach answers from the company’s guidance: confirm every policy detail against the current policy page before anything goes out. She finds the window is wrong, fixes the line, and finishes with a reply she can send.

Each ride-along is scored against two to four goals, which become its rubric. Good goals include prompt clarity, iteration, output review, and policy fit, which cover description, discernment, and diligence. Manager dashboards show playback, transcripts, and score trends for every session, and Yoodli Measure shows progress across programs and against your rubrics. That gives you assessment data from real work between formal reassessments. The transcripts also show which parts of your guidance need more detail.

Setup happens in the Roleplay Builder, starting from the Tutor template. An admin writes one plain-language description, and the builder drafts the cast, Context, and goals. Keep the coach’s guidance short. Then assign the ride-along in a Program after the tutor sessions so completion is tracked. The help center guide to building a ride-along roleplay covers the Context format.

Format	When to use it
AI Tutor lesson	First exposure to a tool or feature, or a workflow everyone does the same way
AI Tutor certification	Proving that each person can complete a specific task
AI ride-along coach	After the tutor sessions, once people apply the tool to their own varied work

Table: Run AI Tutor sessions first and ride-along sessions after, as laid out in Yoodli’s guide to the AI ride-along coach.

What This Looks Like in Practice

Enterprise teams already use practice-based assessment to measure readiness at scale. One example is how a global IT organization pressure-tested AI readiness with Advantexe and Yoodli. Another is Loopio, which cut demo certification attempts by 67% and certified 60 reps with AI Tutor.

The same pattern carries over to AI tools. People learn by doing the task, getting feedback, and doing it again. An assessment built that way gives you results you can trust. Our guide to AI platform certification covers how to turn those results into a formal credential.

Common Mistakes to Avoid

A few mistakes show up again and again when companies measure AI adoption.

• Teams measure usage alone. Weekly active users is an adoption metric, and it says nothing about quality or risk.
• Teams use one test for everyone. Role-specific tasks are what make the results useful to each function.
• Teams skip discernment. Many orgs test whether people can produce with AI and never test whether they can catch its errors.
• Teams run it once. A baseline only becomes useful when you have a second data point to compare it to.

FAQ

What is an AI fluency assessment?

An AI fluency assessment is a role-based test of how well employees use AI tools in their real work. It scores four components: delegation, description, discernment, and diligence. Employees complete realistic tasks from their own jobs, so the results show actual skill. Enablement teams use the results to plan training by role.

How do you assess AI skills across employees?

You assess AI skills by giving each role group realistic production, verification, and judgment tasks, then scoring them against a shared rubric. Run every employee in a role through the same tasks on one platform. Report results by role and component so you can see which skills each team needs to build next.

How long should an AI fluency test take?

An AI fluency test should take under 45 minutes for each employee. That is enough time for a few production, verification, and judgment tasks without causing fatigue. Longer tests tend to get rushed or abandoned, which skews the data. Short, focused tasks tied to real work give cleaner results.

How often should you reassess AI fluency?

Reassess AI fluency 60 to 90 days after the first training tracks launch. That window gives people time to practice and apply new habits. After the first reassessment, many teams move to a quarterly or twice-yearly rhythm. Each new round shows whether skills are holding and where new gaps have appeared.

Start With a Baseline

A structured assessment turns a vague goal into a plan with clear targets by role. Define the tasks, test all four components, read the results as a grid, and build training from the gaps. Yoodli gives L&D and enablement teams the templates, AI Tutor sessions, mastery assessments, and ride-along sessions on real work to run that whole cycle in one place.