Assessing learning outcomes in training: methods, limits and the step to transfer
What assessing learning outcomes means, which methods you can use and why a passed test does not yet show whether what was learned arrives at work.
Many trainings end with a test. Whoever passes has learned, or so the assumption goes. For mandatory training, certificates and exam preparation that is right: there you have to prove that knowledge and skills are in place.
For companies, however, it is only half the answer. They invest in training so that something changes at work. This post explains what assessing learning outcomes achieves, which methods exist and how to make the step from learning to transfer measurable.
What is a learning assessment?
A learning assessment, known in German as Lernerfolgskontrolle, checks whether participants have reached the learning goals of a training: what do they know and can they do after the training that they did not know or could not do before?
In Kirkpatrick's four level model (1959), the learning assessment is level 2. It sits between the participants' reaction and their behavior at work:
After the training
- Level 4ResultsWhat does the change bring the organization?
- Level 3BehaviorThis is where transfer beginsDo participants apply what they learned at work?
In the training
- Level 2LearningLearning assessmentWhat knowledge and skills have participants gained?
- Level 1ReactionHow did participants experience the training?
For a long time the assumption was that the levels build on each other: whoever is satisfied learns more, and whoever learns more applies more. Meta analyses show that this is not the case. The levels are much more weakly related than expected (Alliger & Janak, 1989; Alliger et al., 1997). How the four levels fit together is explained in the post The Kirkpatrick model: training evaluation in four levels.
Methods for assessing learning outcomes
Which method fits depends on the learning goal. You assess knowledge differently from a skill:
- Knowledge test or quiz: multiple choice, matching or open questions. Quick to evaluate, good for subject knowledge and rules.
- Pre and post test: the same test before and after the training. Shows the gain instead of only the final level.
- Work sample or practical task: participants apply a method to an example. Good for tools and procedures.
- Case study or role play: a realistic situation, such as a customer conversation. Good for communication and leadership.
- Self assessment: participants rate their own competence gain. Little effort, practical for almost any training.
In practice a combination pays off: a short test for knowledge, a practical task for skills.
Where learning assessments reach their limits
A passed test shows that someone knows or can do something. Whether they actually do it at work, it does not show. Between learning and applying lie factors that no test in the training captures:
- Confidence: does the person feel able to use what they learned in their own work, not just in the exercise?
- Intention: do they actually plan to put it into practice, or does it remain a good resolution?
- Environment: does the manager support it, is there an opportunity to apply it, and does anyone notice whether someone applies what they learned? The Learning Transfer System Inventory captures such factors of the transfer environment; its German version has been psychometrically tested (Kauffeld et al., 2008).
On top of that comes a signal: what gets measured is seen as important. As a well-known saying puts it: what gets measured gets done. Evaluating only the test at the end of the course signals that passing is the goal. The goal, however, is application.
From learning assessment to transfer evaluation
Observing transfer directly takes effort and comes late. At the end of the course, though, you can already capture the factors that precede transfer. That is exactly what PMScore measures. At the end of the course, participants answer a short, anonymous assessment on three components:
Competence gain
How much have participants learned? This corresponds to the learning assessment.
Ability to apply
Do they feel able to use what they learned in their own work?
Transfer intention
Do they actually intend to put it into practice?
Together they make up the Transfer Effectiveness Score (TES) from 0 to 100. The values in the example are made up, but the pattern is common: competence gain is high, a test would be passed. The ability to apply drops clearly. A learning assessment alone would not have shown this. At this point PMScore asks for the reason, for example missing examples from participants' own work, and so provides a starting point for the next run.
The TES does not replace a test where proof is required, nor observation at work. It adds the view on application to the learning assessment, with the same questions in every course.
What to take away
- Choose the method of learning assessment to fit the learning goal: a test for knowledge, a practical task for skills.
- Treat a passed test as an intermediate goal, not as proof of effectiveness.
- At the end of the course, also capture ability to apply and transfer intention. That way you see early where transfer gets stuck.
Measure more than learning outcomes?
Tell us how you assess learning outcomes today. In a short conversation we show you how the TES adds transfer to your learning assessment.
Sources
- Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later. Personnel Psychology, 42(2), 331-342.
- Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341-358.
- Kauffeld, S., Bates, R., Holton, E. F., III, & Müller, A. C. (2008). Das deutsche Lerntransfer-System-Inventar (GLTSI): Psychometrische Überprüfung der deutschsprachigen Version. Zeitschrift für Personalpsychologie, 7(2), 50-69.
- Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of the American Society of Training Directors, 13(11), 3-9.