Performance
Evaluation

Study
Guide

5

LEARNING OBJECTIVES
At the end of this Study Guide, you should be able to:
•

Understand the reason why it is critical to use a separate set of data not used in the training to evaluate the performance of a model. •

Describe the different validation methods to evaluate model performance. •

Explain the different measures used to measure model performance, including lift chart.

•

Describe a systematic search process to find the best performing model. 5-1

Study Guide 5: Performance Evaluation

Reading material
Textbook Chapter 5 (exclude 5.5, 5.6, 5.9 & 5.10)
Kennedy et al, Section 11.2.

1. Performance evaluation
It important to recognize that performance evaluation is the task involved in step 6 (training and testing) and constitutes the key component in comparing multiple models in step 7 of the data mining process, described in Study Guide 4.
Training is a process of transforming a set of data into a model using a learning algorithm. Once the data is ready (after steps 2, 3 and 4) and an algorithm is selected (step 5), training is a straightforward process (the first part in step 6). Refer to the data mining process in pages 4-1 and 4-2.
In this Study Guide, we focus on issues related to model evaluation (the second part of step 6), i.e. we want evaluate how well the trained model will perform for unseen or future data.

2. Use only unseen data for performance evaluation In evaluating the performance of a model, we are evaluating how well the model generalizes (as opposed to memorize) the seen data in the training set. The training data is said to have been seen because it has been used to produce the model in the training process. It is thus critical to use an unseen data that was not used for training the model. The estimated accuracy obtained from this