Workplace AI Literacy: What Exactly Should You Assess?

Most HRD teams already accept that completion rates and satisfaction scores can’t fully explain whether AI training worked. That realization usually turns into a resolution: next quarter, we’ll come back with metrics that show how our people’s competencies have actually changed.

But the moment you try to take that next step, a question follows. What exactly are we supposed to measure when we say “AI competency”?

‍

What Does “AI Literacy” Actually Mean in Your Organization?

The core problem is that “AI competency” covers far too much ground. One employee has sat through several AI courses and understands the concepts well, but rarely applies them at work. Another uses ChatGPT every day to draft reports, but can’t explain why the output came out the way it did. You can’t meaningfully compare these two people on a single scale.

Viewing AI competency as a set of distinct areas rather than a single ability is already close to a global standard. The OECD’s AI Literacy Framework defines AI competency across multiple layers, including cognition, interaction, and ethics, and the World Economic Forum’s Future of Jobs Report 2025 likewise groups AI-related skills into several clusters rather than a single item. Whichever source you start from, the conclusion is the same: AI competency can’t be measured as one thing.

So how should you break it down? This article maps the structure of AI competency along two axes: depth and breadth.

Depth looks at how far a person can take AI. Breadth determines which skills your company will use to define AI competency. Only once both axes are set can you move on to the next question: how to measure it.

‍

Assessing AI Competency in Your Organization Takes Two Definitions

Imagine two companies that adopt the same AI competency assessment tool. One wants to know how well its employees can work with ChatGPT. The other wants to know where the AI competency gap is widest across its marketing, finance, and R&D job families. The same tool is unlikely to give both companies a satisfying answer, because they are trying to measure different things in the first place.

Two definitions account for that difference.

The first is how you define depth: how far you expect employees to be able to go with AI. Before anything else, decide which level the assessment should cover: knowing the concepts, using the tools hands-on, or applying AI to real work.

The second is how you define breadth. AI competency isn’t a single ability you can capture with one measure. Several skills, such as understanding AI, reviewing AI output, and applying AI to work, add up to a person’s AI competency. You need to decide which of those skills your company will include in the assessment.

If you start comparing AI competency assessment tools without agreeing on these two points, no solution you pick is likely to give you the answers you need, because the design of each tool depends on how it combines depth and breadth. The next two sections look at each axis in turn.

‍

1. Depth of AI Competency: How Far Can Employees Take AI?

Picture two employees who completed the same introductory ChatGPT course. One can clearly explain what the course covered: how prompts work, why AI hallucinates, and what information is risky to enter. Yet she still writes every report from scratch herself. The other opens ChatGPT every day to draft reports, but rarely catches the factual errors in what it produces. Their AI competency isn’t at the same level. It simply sits in a different place.

What separates them is how far each can take AI. The clearest way to measure the level at which employees work with AI is to break it into three stages.

1. Knowing About AI

The first stage is understanding. This is where people recognize what AI is, how it works, what risks it carries, and which tools suit which tasks. Measuring this level is relatively straightforward with tools like multiple-choice quizzes or written responses. Many AI competency assessment tools on the market stop here.

2. Using AI

The second stage is using. People actually work with AI tools, design prompts, and make sense of the output. From this stage on, multiple-choice questions alone can’t do the job. You need methods such as simulation tasks where employees work with the tools directly, or exercises that ask them to write prompts for a given situation.

3. Applying AI

The third stage is applying. People integrate AI into their own workflow to produce deliverables, critically review those deliverables for accuracy, and take responsibility for the results. Measuring this level relies on actual work output, simulations based on a person’s day-to-day work, and peer review. It is the hardest level to measure, but most of what an organization really wants to know lives here.

Your organization has to agree internally on the level of AI competency it needs. That said, once a company starts using AI in earnest, most end up wanting to measure all the way up to applying it to real work.

The challenge is how hard it is to assess the applying stage. Multiple-choice quizzes alone can’t capture it; you also have to look at how people actually use AI at work and how they perform in simulations. Once you’ve defined how far to measure, the next decision follows naturally: which solutions can actually measure up to that stage?

‍

2. Breadth of AI Competency: Which Skills Make Up AI Competency?

If depth sets how far to measure within a skill, breadth sets which skills to include in the assessment. AI competency isn’t a single ability captured by one measure. It is made up of several skills that together form a person’s AI competency.

To assess AI competency accurately, you need to look at skills like these:

- Understanding of AI technology and how it works
- Awareness of risk, bias, and sensitive data in AI output
- Judgment in choosing the right AI tool
- Planning how to apply AI to the work
- Hands-on execution with AI tools and prompt interaction
- Critical review of AI output and ownership of the results

Each of these skills is independent of the others. An employee who is strong on the technology and how it works may still be weak at actually reviewing output. That’s why AI competency has to be broken down skill by skill before you can get a clear read on a person’s strengths and weaknesses.

Breadth is the decision about which of these skills your company will include in the assessment. You can measure all of them, or you can give certain skills more weight based on your company’s priorities.

Once the skills to measure are set, decide who the assessment applies to. Will it apply equally to all employees, or will you weight skills differently by role? Most organizations start by measuring the AI competencies everyone should share. Companies whose work depends heavily on specific roles can add role-based weighting on top. Either way, the skills come first and the scope follows.

If you start comparing assessment solutions before the skills are defined, you may end up choosing one that can’t measure the very skills your company cares about. That’s why breadth needs to be defined first, just as much as depth.

‍

Decide What to Assess, and How to Assess It Becomes Clear

How far should employees be able to take AI, and which skills will you use to break that ability down? Once both reference points, depth and breadth, are set, you have your answer to the question of what to measure.

With clear measures in place, you can finally answer questions that used to be out of reach:

• Which roles and job levels are stuck on which skills?
• Which stage of which skill did last quarter’s training actually move?
• Where should next quarter’s training resources go first?

None of these questions can be answered without defined measures. The real limitation of satisfaction scores and completion rates was always that they couldn’t answer them.

In the next article, we’ll look at how Telta actually measures AI competency.

Related Reading

• Knowing AI vs. Using It: How Telta Measures AI Competency
• AI Competency Assessment Tools Compared: 4 Solutions, 3 Criteria

‍