Skip to content
PMScore Logo
BlogKnowledge

Learning analytics in learning and development: which metrics show whether learning works?

What learning analytics is, which data learning platforms provide today and why completion rates and learning time do not yet tell you whether training has an effect.

Learning platforms collect more data today than ever: who logs in, how long a module takes, who drops out, who passes the test. Dashboards turn this into charts and rates. Learning analytics promises better decisions for training based on this data.

In learning and development, one simple question comes up: do the metrics show whether the trainings work? This post explains what learning analytics is, which data exists and which metric is missing from most dashboards.

What is learning analytics?

A widely used definition comes from the first international conference on learning analytics in 2011: learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs (Siemens & Long, 2011).

The field grew up mainly at universities and schools. In companies it usually arrives as the reporting function of the learning management system (LMS) in learning and development. The decisive part of the definition is the goal: understanding and improving learning. Collecting data is only the first step.

Which data learning analytics uses in companies

Most data is created where learning happens, in the LMS or learning platform:

  • Activity: logins, views, learning time per module, dropouts
  • Progress: completion rates, certificates, completed mandatory training
  • Performance: test results, number of attempts
  • Reaction: satisfaction from the feedback form

This data is useful. It shows whether content is used, where learners get stuck and whether mandatory training is done. But it has one thing in common: it describes what happens in the training. About what happens afterwards at work, it says nothing. Satisfaction hardly helps either, because it is not related to later behavior at work (Alliger & Janak, 1989; Alliger et al., 1997).

The missing metric: transfer effectiveness

Training is funded so that people apply what they learn. A learning analytics setup for learning and development therefore needs at least one metric that is closer to transfer than learning time and completion rate. What gets measured gets done, as the well-known saying goes: if the dashboard only shows the completion rate, people optimize for completions. What a passed test shows and what it does not is covered in the post Assessing learning outcomes in training.

Good analytics answer not only how much, but also why. This is what it looks like for a single course:

How much?

Competence gain81
Ability to apply56
Transfer intention74

Why?

Ability to apply

Examples from their own work44 %
Time to practice in the training35 %
Support with the first application21 %

Starting point

Build in practice cases from participants' work

Almost half miss examples that fit their own work.

The values are made up. They show the difference to classic dashboards: a number becomes a starting point. The course teaches a lot, but participants feel much less able to apply it, and the most common reason is clearly named.

Learning analytics with the Transfer Effectiveness Score

PMScore provides exactly this layer. At the end of the course, participants answer a short, anonymous assessment on competence gain, ability to apply and transfer intention. From it comes the Transfer Effectiveness Score (TES) from 0 to 100. Where someone does not clearly agree, PMScore asks for the reason.

Because every assessment yields the same TES, results can be analyzed across many courses: by format, trainer, topic or version of a training. In analyses, PMScore also pools the reasons across many assessments. That reveals patterns no single feedback form ever shows.

The TES is based on participants' self assessment and does not replace observation at work. Next to the data from your LMS, it is the metric that shows whether learning is on its way to application. How to build such metrics into your reporting is explained in the post Training controlling: steering learning and development with metrics that show transfer.

What to take away

  1. Use your LMS data for what it shows: usage, progress and mandatory training.
  2. Add a metric for transfer effectiveness to your dashboard, with the same questions in every course.
  3. Capture why as well as how much. Only the reasons turn analytics into a basis for decisions.

Add transfer to your dashboard?

Tell us which metrics you report today. In a short conversation we show you how the TES adds impact to your learning analytics.

Thanks! We'll get back to you soon.

That didn't work right now. Please try again later or email us directly.

Sources

  • Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later. Personnel Psychology, 42(2), 331-342.
  • Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341-358.
  • Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30-40.