Skilful Minds

Kirkpatrick Model

I recently attended a meeting of the St. Louis Chapter of the American Society for Training and Development (ASTD) to listen to a presentation on the Kirkpatrick model of training assessment, offered by Jim Kirkpatrick, the son of the model’s creator — Don Kirkpatrick. Jim’s major point was that most training departments fail to measure learning outcomes at Levels 3 and 4 of the model. Without detailing the Kirkpatrick model’s four levels of analysis, the following are Jim’s definitions:

Reaction: involves what training participants thought of the training

 Learning: the knowledge, skills, and attitudes gained through the training

Behavior: the degree participants apply what they learned on the job

Results: the business results gained

Suffice it to say that the model provides a unique framework on how to assess the  learning outcomes of training. However, practitioners are prone to overlook a key issue in measuring learning. I previously noted the following about the Kirkpatrick model.

It is not a stretch to contend that Level 4 measurement is as much about organizational learning and change management as it is about individual learning. In the e-Learning Guild survey on measurement practices, Roy Pollock characterizes this situation as the “Catch-22” for training and development. He notes that, “assessment of on-the-job application and results is as much or more an assessment of the transfer climate…as it is of the quality of the instruction” (p. 167).

Based on what I heard from Jim Kirkpatrick as well as Nick Denardo, a presenter from Edward Jones’ strategic learning services group, the point about organizational learning also applies to Level 3 measurement, to the extent that existing organizational practices either facilitate or impede application of competencies and sharing of knowledge learned from formal training, whether on-the-job, or just informally. 

Level 3 and Level 4 as Group Behavior

The main point I took away from Jim’s presentation came from a slide covering a research study of where people report most of their learning occurs, i.e., preparing for training, during training, or on-the-job. A majority of respondents (66%) indicated that they learn mostly on-the-job. Jim observed that as much as 90% of corporate learning occurs outside formal training. It is important to note that these are two different points. Learning outside formal training isn’t necessarily on-the-job learning. Informal learning, i.e., collaboration, is the more encompassing concept.

 While Jim was presenting, I thought back to an insight I remembered reading in Jay Cross’ book, Informal Learning.

When the job environment changed only slowly, corporate learning involved acquiring the skills and know-how to do the job…The traditional barriers separating training, development, knowledge management, performance support, informal learning, mentoring, and knowing the latest news have become obstacles to performance. They are all one thing and for one purpose: performance.

Learning used to focus on what was in an individual’s head. The individual took the test, got the degree, or earned the certificate. The new learning focuses on what it takes to do the job right. The workplace is an open-book exam.

Undoubtedly, most professionals want to add value to the business of the companies, or organizations, that employ them. And the Kirkpatrick model is the dominant model for assessing learning outcomes from formal training. However, Jim and the other presenters at the ASTD meeting indicated Level 3 is the most difficult measure to take.

I’d suggest that the behavioral changes distinct to Level 3 are more difficult to measure because their application is, in fact, often exhibited as group behavior, in collaborative relationships either internal to the company or external, or both.  Training professionals sometimes fail to recognize this simple fact, assuming instead that their organization can control the way individual employees learn as well as the learning resources available to them. As the quote above from Jay Cross suggests, the assumption is increasingly wrong.

Jay rhetorically asks, “What worker doesn’t have a cell phone and an Internet connection?”  Most problem solving in companies occurs while employees deal with exceptions to the established business processes that formal training covers, i.e., when problem-solving requires knowledge sharing across silos. Solutions to the challenges posed by those exceptions are most often achieved through informal organizational practices rather than formal business processes. 

I’m not claiming that developing training courses to certify participants in specific specialties,  applications, or compliance programs does not produce business value. However, these learning events are a small part of what makes learning processes valuable to organizations and we simply need to keep that fact in perspective.

Employees routinely seek information and advice from people and resources outside their department and organization in order to get the work done, share best practices, and collaborate. If they can’t do it at work because of restrictive access policies to web sites and on-line communities, they can usually do it at home. Measuring the contribution of formal training to performance gets tricky when those relationships are taken into account. Yet, failing to take those relationships into account makes the measures themselves suspect. This is where social media and eLearning 2.0 come into the picture. My next post will consider that topic in more depth.

Share this post…

: : : : : : : : : :

This entry was posted on Monday, December 22nd, 2008 at 11:24 pm and is filed under e-Learning 2.0, Learning Experience. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.