Evaluation Management

  
 

 

Why Evaluate Your Training Programs?

A critical issue in the corporate training industry is the ability to connect employee training (results) to the company's bottom line. In other words, return on investment. More than likely, your sponsors will expect to see metrics for the programs they fund.

Metrics can also be used to validate the success of learning programs, analyze the effectiveness of your training communication plan, your use of technology to deliver training, your ability to create a just-in-time training environment, and your ability to meet your learners' needs.

Consider the benefits of measuring:

  • Justify training budgets.
  • Expand training budgets.
  • Select best training strategies.
  • Relate training to the business.
  • Obtain and analyze user satisfaction feedback.
  • Quantify performance improvement.
  • Provide an opportunity for management to assess employee development.
  • Tie training objectives to organizational objectives.
  • Clarify training expectations for employees.

And the risks of not measuring:

  • Waste dollars on ineffective programs.
  • Discontinue training program.
  • Inefficient and ineffective delivery of training.
  • Training program does not meet objectives.
  • Users' training needs not met or unknown.
  • Performance improvement unknown.
  • Difficult for management to assess employee development.
  • No method to assess ROI for training dollars spent.
  • Expectations are not set nor communicated.

 

Kirkpatrick's Four Levels of Evaluation

In 1959 Dr. Donald L. Kirkpatrick of the University of Wisconsin conceived the idea of a four level evaluation model to address the process of evaluating training programs. The four levels are:

  • Level 1: Reaction
  • Level 2: Learning
  • Level 3: Behavior
  • Level 4: Results

This concept is still used today and is summarized below with a bit of updating to consider the multiple media in which learning is delivered today.

 

Level 1: Reaction

Level 1 measures the reaction of students to learning programs. This basically determines how well they liked the learning program. This is typically done as a survey handed out to students at the end of a class or can be done on-line after completing an E-Learning program.

The level 1 survey should separate its questions into three main areas:

  • Content
  • Method
  • Medium

Content

This focuses on evaluating the course content itself regardless of how it is delivered. Questions can explore the organization of the content, relevance, and other factors. It is a good idea to create a set of core questions focused on content that can be applied to all courses that you deliver regardless of delivery method or media.

Method

This focuses on whether the course was delivered by an instructor or self-paced. Questions can include those on the instructor's pace, ability to answer questions, etc. for instructor delivered courses, and questions on time to complete the course, sufficient resources, etc. for self-paced courses. Again it is a good idea to have a set of core questions that can be applied to all instructor-led courses, and core questions that can be applied to all self-paced courses for cross comparison.

Medium

This focuses on the medium or media in which the course is delivered, such as CD-ROM, classroom, on-line, teleconference, video conference, etc. Again having core questions for each medium helps you to compare across all courses delivered in that medium.

Organizing your survey questions in this manner allows you the flexibility to mix and match the survey questions to a variety of courses. For example, a course delivered via Placeware would combine content, instructor-led method, and on-line medium questions. A course delivered as a CBT would combine content, self-paced, and CD-ROM questions.

 

Level 2: Learning

Level 2 measures whether learning actually has occurred. This is typically done by a test or quiz that is scored. Optimally it is best to present the student with a pre-course test to determine their current knowledge of the content, and a post-course test to measure improvement from before the course delivery. Most courses however only conduct a post-course assessment.

In a typical learning development process, assessment tests should be created before the course content is even created. This supports the reason the course is being developed. The assessment test should be composed of what the student should know or be able to perform after completing the course, and After all, the intent of a course should be to provide a student with knowledge and skills to successfully complete the assessment test.

Reporting level 2 scores requires careful consideration to privacy laws. In the U.S. it is typically safe to provide student scores to direct line management, however management cannot use the scores towards employment action unless the employee was informed the scores would be used as such. In other countries it may not be possible at all to provide test scores to managers or anyone else.
 

Level 3: Behavior

Level 3 measures whether the performance or behavior of the learner has changed as a result of the learner applying the learning content after returning to the job. It is recommended to be done by interviewing each learner, but can also be done by survey. The interview method will get the most detail information, however it consumes much time. Conducting a survey will can get sufficient information to make an inferential analysis, but the survey must be designed to acquire the needed information.

Changes in performance and behavior is best observed by a direct supervisor, so it is important to survey the supervisor as well as the student. Whenever possible you should survey a control group in addition to the experimental group. This will define what behavior has been modified by the training vs. modified by external influences.

The time to plan a level 3 assessment is at the very beginning of course development. Similar to level 2, a level 3 assessment seeks to validate the goal of the course. The course is designed to accomplish a specific change in performance, therefore those specific requirements become the criteria in which to form a level 3 assessment.

 

Level 4: Results

Level 4 measures actual business results from learners applying their newly learned knowledge or skills. In order to determine an improvement in results, we must first define what those metrics are, then determine the current values. The metrics must be quantitative, therefore it may be necessary to assign quantitative values to qualitative data. For example, measuring an increase of productivity in a manufacturing environment would need to be translated to money saved or margin increase as a result of the improvement.

Again, it is useful to use a control group in addition to the experimental group in order to differentiate improvements caused by external factors, such as contests, incentives, policy changes, process changes, etc.

In both cases of level 3 and level 4 analysis, one must consider how to report on the data. Do you confine your report to the descriptive statistics, or do also include inferential statistics to project potential savings of an entire population attending the training.

Similar to level 2 and level 3, the time to plan a level 4 assessment is at the very beginning of course development. In this manner, we set the expected outcome of this course by defining the criteria in level 2, level 3, and level 4 assessments, then developing the course to generate those desired results.

Phillips' Return On Investment (ROI)
Jack Phillips' approach to ROI can be summarized as determining the amount of money spent on course development compared to how much was realized by level 4 results. Essentially this means how much was gained for every dollar spent. Optimally you would want to reach a above 100% ROI.
 

Level 5: Return On Investment (ROI)

Phillips's ROI approach has been referred as the 5th level of evaluation. It certainly plays in an important role, because learning budgets can be determined by how much the investment is worth. Sadly, learning tends to be regarded as an expense, when it should be considered an investment. The reason that learning is regarded as an expense is because most training organizations do not report on business results or return on investment for training. Once a training organization begins reporting on financial returns, then they are looked upon as an investment. Why would anyone not spend money on something that has been proven to return that money and then some.
 

Copyright ©2006 E-Learning Engineering
Last modified: April 22, 2006
Terms of Use