Knowing AI vs. Using It: How Telta Measures AI Competency

If you’ve followed the series so far, you probably have the big picture on measuring AI competency: completion rates and satisfaction scores can’t tell you whether competencies have changed, and AI competency needs to be broken down along two axes, depth and breadth.

So how do you actually measure it?

‍

How You Assess AI Competency Depends on What You’re Measuring

As the previous article explained, AI competency can’t be measured as a single area. Several areas, including understanding AI, judgment in choosing tools, planning how to apply AI to the work, execution, and reviewing results, come together to make up a person’s AI competency.

The problem is that these areas are different in nature. Some have right answers and some don’t. Some can be answered from what’s in your head, while others only show up in what you actually do. That’s why no single measurement method can cover them all.

You need three different methods:

• Knowledge of AI → Has right answers; can be measured with a multiple-choice quiz
• Ability to apply AI → Has no single right answer; can only be measured by giving people a real work situation and having them solve it
• Everyday use → A person’s own behavioral data; a short self-report survey fills in frequency and context

Telta’s AI Literacy Assessment is built around these same three methods. Let’s look at what each one tells you.

‍

A Multiple-Choice Quiz to Gauge How Much People Know About AI

The most familiar method is the multiple-choice quiz. Knowledge areas such as what AI is, how it works, what risks it carries, and which tools suit which tasks have clear right answers, so you can tell people apart by whether they know them.

The first stage of Telta’s AI Literacy Assessment likewise uses a multiple-choice quiz to check two areas:

• Technical Understanding Understanding of AI technology, how it works, and its limits
• Tool Awareness Knowing what AI tools can do and choosing the right tool for the task

These two areas are a pass/fail baseline. This stage confirms whether someone knows the basics everyone should know about AI.

But that’s as far as a multiple-choice quiz can go. Knowing the right answers doesn’t tell you whether someone can actually work with AI. A score alone won’t reveal the difference between someone who can precisely define an LLM and someone who used one to draft a report yesterday. That’s why many AI competency assessments on the market stop here and never answer the question organizations really care about: can people apply it?

‍

Assessing AI Application Skills by Watching People Work Through a Problem

Applied skill can’t be measured with multiple choice, because real work situations have no single right answer. Give two people the same problem, and they’ll differ in how they define it, which tool they choose, what prompts they write, and how they review the results. What you need to measure isn’t whether someone lands close to a correct answer. It’s the process of working with AI itself. You see a person’s applied skill by looking at how they write prompts, how they push back when an answer falls short, and which parts of the output they question.

That’s why you need to put participants in a realistic work situation and have them solve it using an LLM directly. Telta captures this process through a scenario simulation, recording as behavioral data how each participant defines the problem, writes prompts, and reviews the results they get.

Telta’s scenario simulation looks at three areas:

• Plan: Deciding which work to apply AI to and defining the scope of the task
• Execute: Carrying out the work through prompt design and interaction with the LLM
• Review: Critically reviewing AI output and making decisions about the results

Two people with perfect multiple-choice scores can end up with completely different simulation scores. Knowing the right answer and being able to use it to solve a real problem are different abilities.

💡 Why the Review Stage Matters
Many AI competency assessments on the market leave output verification and risk identification out of what they measure. But AI output can look convincing while containing factual errors or bias. If people can’t go back and check sources and evidence, flawed results can end up shaping decisions. That’s why Telta treats Review as a separate measurement area and always checks people’s ability to critically verify AI output and identify risks.

‍

An Experience Survey Asking How Often People Actually Use AI

Measuring applied skill still doesn’t give you the full picture of someone’s AI competency. Doing well in an assessment and regularly using AI in everyday work are two different things. If the simulation shows what someone can do, usage experience has to show what they actually do.

A short self-report survey fills that gap. It asks how often a person uses AI, in which work contexts, and through which tools.

Telta’s AI Literacy Assessment includes a short usage survey for exactly this reason: it shows which tasks AI is woven into and how often. When you analyze these results alongside the simulation results, you can see where the gap opens up between measured ability and actual use.

For example, an employee with a high applied-skill score but low usage frequency has the ability but isn’t applying it at work. Conversely, an employee with a low skill score but high usage frequency uses AI often but isn’t handling it well. You need both views to see where next quarter’s training priorities should be.

‍

Reports That Turn AI Competency Assessment Results Into Decisions

Measurement means little if the results don’t feed into the next decision. For assessment results to be useful for decisions, two things are needed. The behavioral evidence behind each score has to sit right next to it, and results can’t stop at individual scores; they also need to be organized into comparisons by department and role.

Telta’s assessment reports deliver both.

The individual report explains strengths and weaknesses across five areas (Technical Understanding, Tool Awareness, Plan, Execute, and Review) and attaches the responses the participant wrote in the simulation. Every score comes with the behavior that produced it.

The organizational report rolls the same data up by department and role for comparison. It shows which departments are weak in which areas and where AI use is most active, giving you the data to decide where next quarter’s training resources should go first.

Put all of this together, and you have the big picture of how AI competency should be measured: the method changes with the nature of each area, and reports become decision-ready data only when they show the behavioral evidence and organizational comparisons alongside the scores.

So how well do the AI competency assessments on the market meet these standards? In the next article, we’ll lay out the criteria companies should use to compare AI competency assessment solutions.

Measuring AI competency ultimately means capturing, as data, how a person behaves when working with AI. To see in more detail how Telta’s AI Literacy Assessment actually generates that data, visit the Telta AI Literacy Assessment page.

Related Reading

• AI Competency Assessment Tools Compared: 4 Solutions, 3 Criteria
• Why Telta’s AI Literacy Assessment Tracks How People Think

‍