I still remember the feeling of sitting for certain professional certification exams: the quiet tension of the room, the clock, the familiar pattern of multiple-choice questions, and the odd realization that success often depended less on professional capability than on a specific form of recall.
Not recall in the meaningful sense of “I can apply this under pressure,” but recall as in “I recognize this wording,” “I remember what the exam writers usually want,” “I can eliminate two answers quickly,” and then, if needed, I can guess between the remaining two with reasonable odds.
That is not a critique of any one certification body or academic institution. It is a critique of an assessment paradigm that was designed for a different era, when knowledge was scarce, access to information was slow, and memorization genuinely correlated with readiness.
That era is ending.
The hidden flaw in the dominant exam model
Multiple-choice exams became the dominant standard for understandable reasons: they are scalable, consistent, easy to administer globally, and statistically convenient. They also create the appearance of objectivity. A correct answer is a correct answer.
But MCQs create a structural problem: they measure recognition more than reasoning.
In many fields, the hardest part of the job is not selecting the correct option from four pre-written choices. The hardest part is defining the problem, spotting what is missing, asking the right questions, navigating tradeoffs, and making a defensible decision with imperfect information.
A multiple-choice format can test whether you have encountered a concept. It struggles to test whether you can think.
And as you noted, there is a second issue: probability. When test-takers can eliminate one answer as obviously wrong and another as out of scope, the task becomes less about competence and more about “educated guessing.” That guessing effect is not always large, but it is real, and it reduces the credibility of what the exam claims to certify.
If we are honest, a non-trivial portion of exam performance is often a blend of knowledge, memory, exam familiarity, and luck.
The AI shock: knowledge has been commoditized
Then generative AI arrived and changed the ground beneath the entire system.
The most important shift is not that AI can answer questions. Search engines could do that.
The shift is that AI can do what used to take effort and time: it can summarize, explain, translate, contextualize, compare viewpoints, generate examples, and even help you structure decisions. In other words, AI dramatically reduces the friction between not knowing and knowing.
This has two consequences.
First, the value of memorized knowledge declines, not because knowledge is unimportant, but because access is no longer the bottleneck. The bottleneck has moved.
Second, assessments that primarily measure recall become less meaningful signals of capability, because the workplace itself no longer functions as a closed-book environment.
In real work, competent professionals do not win by storing the most facts. They win by using information well.
So the skill we need to validate is not possession of knowledge. It is judgment.
Knowledge, understanding, wisdom: what we actually need to certify
A useful way to describe the progression is:
-
Knowledge: what I know
-
Understanding: what it means here
-
Wisdom: what I do with it
Knowledge is necessary, but it is not sufficient.
Understanding is the ability to interpret information within a specific context. It is the capacity to recognize nuance, identify constraints, and connect facts to the situation at hand.
Wisdom is applied understanding. It is the ability to make decisions that hold up under scrutiny, to explain tradeoffs, to anticipate second-order effects, and to choose an action that is ethically and operationally defensible.
In an AI-rich environment, knowledge becomes abundant and rapidly accessible. Understanding and wisdom become the scarce differentiators.
The Kirkpatrick lens: why most assessments stop early
This is where the Kirkpatrick model helps clarify the issue.
Kirkpatrick distinguishes four levels of evaluation: Reaction, Learning, Behavior, Results (Kirkpatrick Partners, n.d.). In practice:
-
Level 1 (Reaction) asks: did participants like the learning experience?
-
Level 2 (Learning) asks: did they acquire knowledge or skill?
-
Level 3 (Behavior) asks: do they apply it in real situations?
-
Level 4 (Results) asks: does application produce measurable outcomes?
Many academic assessments and certification exams, even high-quality ones, largely sit in Level 2. They test whether learning occurred in a narrow sense: recall, recognition, and sometimes basic application.
But competence is not proven at Level 2.
Competence is proven when behavior changes (Level 3) and when that behavior produces outcomes (Level 4). That is not opinion, it is the logic of performance: the world only benefits when learning shows up in practice (Kirkpatrick Partners, n.d.; Rouse, 2011).
Several frameworks extend this even further by adding ROI as an additional layer, often informally described as “Level 5,” to isolate and quantify the financial return of learning interventions (ROI Institute, n.d.; Panopto, 2025). Whether or not one adopts the “Level 5” framing, the message is consistent: the higher the stakes, the more we must connect learning to outcomes.
If your certification is meant to signal job readiness, professional competence, or the ability to deliver specific duties, the assessment must reach beyond “knowing.”
What does an assessment look like when it measures judgment?
Once you accept that, the assessment design has to change.
To measure judgment, you need to measure reasoning.
That generally requires moving toward approaches such as:
-
Open-text responses
Not because open-text is “harder,” but because it reveals how a person thinks: assumptions, logic, prioritization, tradeoffs.
-
Reasoning traces
A defensible answer matters, but so does the chain of reasoning. Two people can arrive at the same recommendation for entirely different reasons, and only one set of reasons might be robust.
-
Scenario-based assessment
Real professional situations are messy. They contain ambiguity, conflicting stakeholder needs, incomplete information, ethical tension, and time constraints. Scenarios let you surface how a candidate navigates complexity.
-
Situational simulations
The strongest competence signals emerge when the context evolves: a constraint appears, new information arrives, a stakeholder pushes back, a risk materializes. Simulations reveal whether the candidate can adapt without losing coherence.
This direction is increasingly discussed in the context of generative AI’s impact on education and assessment, precisely because AI makes “answer production” cheap while making “reasoned decision-making” more valuable (Zhao & Dang, 2026; Gonsalves, 2024).
This is the shift N2X Labs is designed for
This is the point where what we are building at N2X Labs connects directly.
Our core premise is that many current assessment models generate a weak competence signal because they overweight recall and underweight reasoning.
N2X Labs is built around a different signal:
-
Open-text questions designed to surface thinking, not guessing
-
Reasoning-based evaluation that can distinguish confident-sounding answers from coherent, defensible logic
-
Scenario and simulation pathways that evaluate applied competence, especially when paired with AI-supported situational dynamics
In Kirkpatrick terms, most traditional exams cluster around Levels 1 and 2. Reasoning-based assessment and scenario design move the signal closer to Levels 3 and 4, because they test whether someone can apply learning in context and produce decisions that would plausibly drive results (Kirkpatrick Partners, n.d.; Rouse, 2011).
A practical question we all need to answer now
The uncomfortable question is not “Will AI change assessment?”
It already has.
The question is: What are we going to treat as valid evidence of competence from here forward?
Some institutions are responding by trying to lock AI out. That may work temporarily in controlled testing centers, but it does not reflect the reality of modern work, where AI-enabled research and drafting are already integrated into daily practice.
The more durable approach is to redesign assessment so that AI is not the enemy of integrity, but the backdrop of modern performance.
In that future, the best professionals will not be those who can reproduce information from memory.
They will be those who can:
-
interpret information accurately,
-
validate its quality,
-
integrate context and constraints,
-
make tradeoffs transparently,
-
act ethically,
-
and explain their reasoning in a way that others can trust.
That is what competence looks like now.
Your view
I would love your perspective in the comments:
-
What is one thing current certification exams do well, and one thing they fail to measure?
-
Where should we draw the line between “closed-book validation” and “AI-allowed but reasoning-required”?
-
What would a truly credible competence signal look like in your field?
If you are working inside an academic institution, a certification body, or an employer organization and you are actively thinking about redesigning assessments for this new reality, feel free to reach out. I am always interested in comparing notes, sharing prototypes, and exploring how this evolution unfolds in practice.
References
Gonsalves, C. (2024). Generative AI’s impact on critical thinking: Revisiting Bloom’s taxonomy. Business Communication Quarterly. https://doi.org/10.1177/02734753241305980
Kirkpatrick Partners. (n.d.). The Kirkpatrick Model. Kirkpatrick Partners. https://www.kirkpatrickpartners.com/the-kirkpatrick-model/
Panopto. (2025, December 22). How to measure the ROI of training. Panopto. https://www.panopto.com/blog/how-to-measure-the-roi-of-training/
ROI Institute. (n.d.). ROI Methodology. ROI Institute. https://roiinstitute.net/roi-methodology/
Rouse, D. N. (2011). Employing Kirkpatrick’s evaluation framework to determine the effectiveness of health information management courses and programs. Perspectives in Health Information Management, 8(Spring), 1c. https://pmc.ncbi.nlm.nih.gov/articles/PMC3070232/
Zhao, H., & Dang, T. N. Y. (2026). Transforming written assessment design to embrace AI: What needs to be changed to encourage higher-order critical thinking. Education and Information Technologies. https://doi.org/10.1007/s10639-025-13870-5
