This article a) describes why you should conduct an evaluation, b) identifies various levels of evaluation, c) provides concrete advice on how to conduct an evaluation, d) illuminates when this work should be conducted and e) addresses who should perform the evaluation.

Why evaluate?
An evaluation can have three aspects.  First, an evaluation can control whether implemented initiatives have led to the desired results.  Secondly, an evaluation can be considered formative, meaning that the object is to improve the learning process, and thereby increase the probability for achieving your goals.  The third aspect is various combinations of the first two.  We believe that, in most cases, too little is done to measure the effect of learning initiatives.

You can thereby risk having made a useless investment, often without even realising it.  Conscious and determined evaluation work will ensure that your investments in learning are purposeful and profitable.

What should be evaluated?
What is necessary to ensure that learning initiatives provide the desired results?  The most important thing is to get going on evaluation work, whether it is comprehensive or not.  Evaluations can address the content or a learning programme or a course, the learning environment itself, or the learning effect on various levels.  Our focus is in measuring learning effect.  Kirkpatrick divides evaluations into 4 levels:

1. Reaction - measures how the target group reacts to the learning initiative and often provides answers to the question, “Did they like it?”  This level of evaluation is not adequate for examining the results or effects of learning in the organisation.  Most evaluation work is conducted at this level.
   
2. Learning - measures the degree to which the target group has changed their attitudes, increased their knowledge, or improved their competence as a result of the course, and is intended to answer the question, “Did they learn it?”  Evaluations at this level can indicate whether training is effective or ineffective, but will not be able to indicate whether this newly acquired competence has been transferred to the daily work situation. 
   
3. Behavioural Changes - measures the degree to which the target group has changed their behaviour after having completed the training, and answers the question, “Did they use it later?”  At this level, we find out whether or not training has been transferred from the classroom or learning situation to the working environment. 
   
4. Result - measures the effect training has on the organisational level and answers the question, “Did it provide results?”  This level of evaluation reveals whether or not the learning initiative works, and whether it provides value for the organisation.  Effects can be related to economy, efficiency, ethics, etc.  Here, the line of thought is expanded from the individual to the organisational level, and we try to find answers for what is happening with the organisation as a result of the learning initiatives.

When you move from reaction to result in an evaluation, the evaluation process becomes more complicated and time-consuming.  Learning at the reaction level is simplest and therefore also the most common, but unfortunately it also provides the least interesting information.  You will have significantly more interesting results if you conduct evaluations at the higher levels, and can discover whether the initiative will give a return on invested capital.

How to evaluate?
Focus on evaluating behaviour and effect.  Look at correlations between the results you achieve in your organisation and learning initiatives you have put in place.  You should also expect your learning solution supplier to offer evaluation at the organisational level.

Examples of correlations between learning initiatives and effects can be: increased customer satisfaction as a result of service training, good campaign results after sales training, reduced absence due to illness after HSE training, lower accident statistics after safety training, fewer mistakes after cash register training, fewer support questions after software training, increased employee satisfaction after a learning programme on vision and values, increased market share after marketing training.

When to evaluate?
In principle, there are three “times” when you should conduct an evaluation: before, during, and after. 

• The “before” evaluation can address conditions and premises for learning.  Pre-tests can be performed before the initiative, to be able to observe the change as a result of the initiative more easily.
• “During” evaluations, or formative evaluations, are part of the training, and can consist of questions, exercises, and observations.  This type of evaluation is most appropriate when the object is to improve training or project implementation.  
• “After” evaluations, or summative evaluations, are carried out after the initiative has been completed.

Summative evaluations can consist of questions, exercises, and observations that have the object of setting a grade or documenting an effect.  Finally, evaluations are most appropriate when the object is to measure or control performance, results, and effects of the training or project.

Who should conduct the evaluation?
Whether it is most appropriate to have an internal or external evaluator, is a question with several correct answers.  Internal personnel will often be best acquainted with the circumstances, premises, and other details, but in many instances, the danger can be that they end up consciously trying to find the conclusions they were hoping for, or that those being interviewed don’t answer impartially because they know that colleagues will be handling the information.  In many instances, an external evaluator will be able to avoid these dilemmas, but on the other hand will require that internal resources be set aside for “briefing” work and follow up along the way.  An external evaluator will need to garner a certain amount of trust and openness in order to gain access to important information.

Conclusion
Evaluation work can be either extremely advanced or extremely simple, depending on your ambition level and available resources.  Our advice is:

1. Start small.
2. Focus on measuring learning effect and behavioural changes in the organisation.
3. Implement an evaluation project.
4. Make adjustments along the way, make a conclusion at the end, and use these experiences when you plan new learning projects.

Article by:  Erik Schjerven, Consultant Mintra AS