AI Learning Claims Audit
Turn the Label Into Observable Behaviour
Ravinder Tulsiani, DBA · Enterprise Capability Executive
What this tool helps you decide or do
Evaluate what an AI learning system actually does before accepting its label. Learning technology increasingly arrives with strong labels — adaptive, personalized, mastery-based, Socratic, AI tutor. Those labels may describe useful capabilities, but they are not, by themselves, evidence of how the learning experience actually works.
Use this when
- A vendor or product claims adaptive, personalized, mastery-based, Socratic or AI-tutor capability.
- You need to evaluate the instructional mechanism before procurement, adoption or rollout.
- A demo suggests behaviour you have not yet seen under realistic learner conditions.
How it works
Five audit tests translate a claim into something observable: state the claim precisely; identify the learner evidence or trigger; observe what changes; test what happens when the learner does not succeed; and make governance visible. The aim is not to challenge a vendor's terminology for its own sake — it is to make the instructional mechanism visible enough to evaluate.
Core decision rule
Do not evaluate the label. Evaluate what the system actually does.
Steps
State the claim precisely
Avoid evaluating a broad label — write down what the product or design is actually claimed to do. “Personalizes feedback based on learner performance” is easier to examine than “AI-powered personalized learning.”
Identify the learner evidence or trigger
Ask what information causes the system to respond differently: an answer, an error pattern, demonstrated performance, prior activity, stated goals, confidence, time or sequence, or instructor-defined rules. If nothing about the learner meaningfully changes the response, the claim may describe presentation rather than adaptation — that may still be useful, but know what is actually happening.
Observe what changes
Look for a meaningful change in the learning experience: the next task, challenge level, feedback, support, explanation, practice opportunity, sequence, pacing, or a requirement to demonstrate the capability again. A different wording of the same experience is still a change; it is not necessarily the same thing as changing the instructional decision.
Test what happens when the learner does not succeed
Many learning claims become clearer when the learner gets something wrong. Ask what counts as sufficient performance, what happens when it is not demonstrated, whether the system diagnoses the problem, whether it provides related support, whether the learner gets another meaningful opportunity — and whether the path changes or the learner eventually receives the same sequence regardless.
Make governance visible
An AI-enabled learning system can make many small instructional decisions; someone still needs to be responsible for the boundaries. Confirm who defines acceptable behaviour, what sources the system may use, how incorrect or inappropriate responses are identified, whether instructors can inspect or override decisions, what learner data is used and retained, what happens when the system is uncertain, and who owns the final instructional and governance decision.
The audit decision
Demonstrated
The claimed mechanism can be observed consistently in realistic tests
The instructional behaviour matches the stated claim closely enough for the intended use.
Partially demonstrated
Some elements are observable but important parts remain limited, inconsistent or conditional
Record exactly which parts of the claim hold and under what conditions.
Not yet demonstrated
The label cannot be translated into observable behaviour, the expected mechanism does not occur in testing, or evidence is insufficient
Treat the claim as unproven for decision purposes until observable evidence exists.
These categories are a practical due-diligence aid, not a product certification or validated rating scale.
What it produces
- A documented audit decision — demonstrated, partially demonstrated or not yet demonstrated — with the supporting evidence recorded.
- A completed audit worksheet naming the product or feature, the claim owner or vendor contact, and the internal reviewer.
Application notes
- Test with realistic learner responses, not only ideal inputs.
- Where possible, test contrasting cases: strong performance, partial understanding, misconception, uncertainty and repeated difficulty.
- Record what the system actually does rather than what a demo suggests it should do.
- A feature may be useful even when the broad marketing label is imprecise. The goal is not to disqualify the technology — it is to understand the mechanism well enough to make a responsible decision.
Use with care when
- This audit does not replace privacy, security, accessibility, legal, procurement or technical due diligence — it makes the learning claim itself easier to inspect.
- Do not present the audit as an assessment of any named vendor unless a separate evidence-based review is explicitly commissioned.
Applying the audit to common claims
The following are practical questions, not formal definitions or certification standards.
Adaptive
What learner evidence causes the learning experience to change, and what changes as a result? Look beyond branching alone: does the system meaningfully change support, challenge, practice, sequence or another instructional decision based on evidence about the learner?
Personalized
What is being personalized, for what reason, and does the change improve the learner's path toward the required capability? Timing, examples, explanations, feedback, support, practice or sequence may all be personalized — do not assume all personalization is equally consequential.
Mastery-based
What constitutes sufficient performance, and what happens when the learner has not demonstrated it? Look for a clear criterion, relevant support and another meaningful opportunity to demonstrate the capability.
Socratic
Does the interaction genuinely probe the learner's reasoning, or does it simply present questions? The important behaviour is whether the system responds to the learner's thinking in a way that helps examine assumptions, reasoning or evidence.
AI tutor
What context can the tutor use, what decisions can it make, what does it remember, and what are its limits? A conversational interface alone does not tell you how much instructional context, learner evidence or governance sits behind it.
Reusable audit worksheet
Complete one worksheet per claim under evaluation.
- What exact claim are we evaluating?
- Where is the claim stated, and what scope does it imply?
- What learner evidence or input is supposed to trigger a response?
- What exactly should change because of that evidence?
- Can we observe the change consistently in a realistic test?
- What happens when the learner performs well?
- What happens when the learner struggles or gives an incorrect response?
- Does the learner receive relevant support and another meaningful opportunity to demonstrate capability?
- What stays the same regardless of learner evidence?
- What instructional decision remains under human control?
- What data, privacy, accessibility, security, regulatory or technical review is still required outside this audit?
- Decision: demonstrated, partially demonstrated or not yet demonstrated? What evidence supports the decision?
Record the audit date, the product or feature, the claim owner or vendor contact, and the internal reviewer.
Evidence boundary
The AI Learning Claims Audit is Ravinder Tulsiani's professional synthesis for instructional and capability due diligence. It is not a validated scoring instrument, vendor certification, procurement assessment, privacy review, security assessment, accessibility audit or legal opinion.
- It is not a formal industry definition of adaptive learning, personalization, mastery learning, Socratic teaching or AI tutoring.
The complete diagnostic is available as a PDF — the full exhibit set, worksheets and source notes included.