Assignment 3 – Classification
Note: Show all your work.

Problem 1 (25 points) Consider the following dataset:

ID | A1 | A2 | A3 | Class | 1 | Low | Mild | East | Yes | 2 | Low | Hot | West | No | 3 | Medium | Mild | East | No | 4 | Low | Mild | East | Yes | 5 | High | Mild | East | Yes | 6 | Medium | Hot | West | No | 7 | High | Hot | West | Yes | 8 | Low | Cool | West | No | 9 | Medium | Cool | East | Yes | 10 | High | Cool | East | No | 11 | Medium | Mild | West | Yes | 12 | Medium | Cool | West | No | 13 | Medium | Hot | West | Yes | 14 | high | Hot | East | Yes |

Suppose we have a new tuple X = (A1 = Medium, A2 = Cool, A3 = East). Predict the class label of X using Naïve Bayesian classification. You need show all your work.

Problem 2 (25 points) Consider the following dataset D.

ID | A1 | A2 | A3 | Class | 1 | Low | Mild | East | Yes | 2 | Low | Hot | West | No | 3 | Medium | Mild | East | No | 4 | Low | Mild | West | Yes | 5 | High | Cool | East | Yes | 6 | Low | Hot | West | No | 7 | High | Hot | West | Yes | 8 | Low | Cool | West | No | 9 | Medium | Cool | East | Yes | 10 | High | Hot | East | No | 11 | Medium | Mild | East | Yes | 12 | Medium | Cool | West | No | 13 | High | Hot | West | Yes | 14 | High | Hot | East | Yes |

(1) Compute the Info of the whole dataset D.
(2) Compute the information gain for each of A1, A2, and A3, and determine the splitting attribute (or the best split attribute).
(3) Compute the Gain ratio of A1.
(4) Discuss a drawback of information gain and how the drawback is alleviated by gain ratio.
(5) Compute the Gini index of A3
(6) When we decide the splitting attribute, we do not consider a unique identification number, such as ID in the above dataset. Explain why.

Problem 3 (25 points plus 10 points for an extra credit problem). The goal of this problem is to get students familiar with how to use Weka Naïve Bayes classifier. Follow the instructions below. There are three subproblems plus one extra credit problem.

(Some screenshots are small and are not easy to see in this document. You may have to zoom in from your pdf reader software.)

(1) Start Weka

(2) Open Explorer by clicking Explorer.

(3) Click Open file, browse to the location where you have weather.normial.arff file and open it.

(4) Explorer window appears as shown below.

(5) Among many other things, you can see there are 14 instances and 5 attributes in the dataset and the play attribute is the class attribute. Click Classify tab.

(6) Click Choose. Classifier selection window appears. Select NaïveBayes under Classifier – Bayes.

(7) The following screenshot shows that NaiveBayes is selected.

Accept the default test option and click Start.

(8) Classifier output is shown in the right window.

Problem 3-1. Capture this screenshot and paste it onto your submission.
Problem 3-2. At the bottom of the output window, you will see Confusion Matrix. It has four numbers. Explain the meaning of each number (confusion matrix is discussed in pp. 359 - 362).

Do not exit explorer yet.

What you did so far are two things: (1) You built a Naïve Bayes model using the weather.nominal.arff dataset and (2) You tested the performance of your model using 10-fold cross-validation (We will discuss this testing method in the next module. If you want you can read about this method in page 365).

Now let’s predict the class label of instances whose class labels are unknown. First, create a text file that contains the following:

@relation weather.test01

@attribute outlook {sunny, overcast, rainy}
@attribute temperature {hot, mild, cool}
@attribute humidity {high, normal}
@attribute windy {TRUE, FALSE}
@attribute play {yes, no}

@data rainy,hot,high,FALSE,yes sunny,hot,normal,TRUE, yes overcast,hot,high,TRUE,yes rainy,mild,normal,FALSE,yes rainy,mild,normal,FALSE,yes sunny,cool,normal,TRUE, yes overcast,hot,normal,TRUE,yes sunny,mild,normal,FALSE, yes sunny,cool,normal,TRUE,yes rainy,mild,high,FALSE,yes

Name the file as weather.nb.test01.arff. Notice that the relation name was changed and that all class attribute values are set to yes (The class attribute values are irrelevant in this case because weka will predict the values. If you want you can put all no’s). You will provide this file to weka as a test dataset. Then, weka will predict the class label of each instance using the model you built. Follow the following steps.

(1) Choose Supplied test set for Test options as shown below.

(2) Click Set. The following dialogue box appears.

(3) Click Open file, browse to where you saved weather.nb.test01.arff and select it. You will see the name of the relation weather.test01 in the box.

(4) Click Close. You are returned to Weka Explorer.

(5) Click More options and check the Output predictions box and click OK.

(6) Then, click Start on the Explorer. The predictions for 10 instances are shown about a half way down the Classifier output window. You can see that class labels of all 10 instances are predicted (you can ignore actual values and other performance related information in the output window)

Problem 3-3. Capture this screenshot and paste it onto your submission.

Problem 3-4. This is an extra credit question. This is worth 10% of the assignment. You will see the following in the Classifier output window.

Naive Bayes Classifier

Class
Attribute yes no (0.63) (0.38)
=============================
outlook sunny 3.0 4.0 overcast 5.0 1.0 rainy 4.0 3.0 [total] 12.0 8.0

The meanings of the numbers at the header are as follow. About 63% of all instances are yes instances and about 38% are no instances (in the initial training dataset). There are two numbers in the [total] row, i.e., 12.0 and 8.0. These two numbers are supposed to be the total number of yes instances and the total number of no instances, respectively, in the initial training dataset. However, you know that there are 9 yes instances and 5 no instances. Explain why there are 12 and 8 instead of 9 and 5. Your answer must be specific and accurate. Too general and/or ambiguous answers will not receive extra credit.

Problem 4 (25 points). This time, run the J48 Decision Tree classifier on the same weather.normial.arff data. Follow the same 8 steps of Problem 3, except that you choose J48 from Classifier – trees instead of Naïve Bayes. Again, make sure that cross-validation (which is a default) is selected as the test option. How to use J48 classifier is illustrated in Section 3.2 of Module 3 Lecture.

Problem 4-1. Capture a part of the result window showing the confusion matrix, and paste it to your submission.

Problem 4-2. Compare the performance of the Naïve Bayes and J48 on this data set and state which one you would use and explain why.

Submission:

Submit all solutions in a single Word or PDF document and upload it to course assignments section. Please make sure that there are no spaces in the file name. Use firstName_lastName_HW3.doc or firstName_lastName_HW3.pdf as the file name.

To capture the snapshot of a screen, with the required window selected, press CTRL-ALT-PrintScreen. Then, do a Paste in the Word document.