Model Answers for Chapter 4: Evaluating Classification and Predictive
Performance

Answer to 4.3.a:
Leftmost bar:
If we take the 10% "most probable 1’s(frauds)” (as ranked by the model), it will yield 6.5 times as many 1’s (frauds), as would a random selection of 10% of the records. 2nd bar from left:
If we take the second highest decile (10%) of records that are ranked by the model as “the most probable 1’s (frauds ” it will yield 2.7 times as many 1’s
(frauds), as would a random selection of 10 % of the records.

Answer to 4.3.b:
Consider a tax authority that wants to allocate their resources for investigating firms that are most likely to submit fraudulent tax returns. Suppose that there are resources for auditing only 10% of firms. Rather than taking a random sample, they can select the top 10% of firms that are predicted to be most likely to report fraudulently (according to the decile chart). Or, to preserve the principle that anyone might be audited, they can establish differential probabilities for being sampled -- those in the top deciles being much more likely to be audited.
.

Answer to 4.3.c:

Classification Confusion Matrix
Predicted Class
1
(Fraudulent)

Actual Class
1
(Fraudulent)
0
(Non-fraudulent)
Error rate =

0
(Non-fraudulent)

30

58

32

920

n0,1 + n1,0 32 + 58
=
= 0.0865 = 8.65% n 1040

Our classification confusion matrix becomes
Classification Confusion Matrix
Predicted Class
1
(Fraudulent)

Actual Class
1
(Fraudulent)
0
(Non-fraudulent)
Overall misclassification rate =
Accuracy = 1 – Error rate

0
(Non-fraudulent)

0

88

0

952

88
= 0.0846 = 8.46%
1040

We can confirm that the misclassification error is decreased, hence accuracy is increased. Answer to 4.3.d:
The likely purpose of this analysis is to identify fraudulent records. The overall "error rate" is not likely to help much in evaluating competing methods for doing so. The key factor here is the ability to identify records that have a high probability of being fraudulent, and this is what lift measures. Using lift, you can "descend" through the records in order of probability of being fraudulent, knowing at each point how much more likely you are to be getting a fraudulent record than naively selecting at random. The "error rate" measure, by contrast, reveals nothing about the efficiency of identifying fraudulent records.
The vast majority of records are non-fraudulent, and correctly classifying nonfraudulent records drives the overall error rate. One can achieve a very respectably low error rate of 5% just by classifying everything as non-fraudulent, which gets you nowhere.