CS 383 – Machine Learning Assignment 2 – Classification Introduction In this assignment you will perform classification using Logistic Regression, Naive Bayes and Decision Tree classifiers. You will run your implementations on a binary class dataset and report your results. You may not use any functions from a ML library in your code unless explicitly told otherwise. Grading Part 1 (Theory) 15pts Part 2 (Logistic Regression) 20pts Part 3 (Spam Logistic Regression) 25pts Part 4 (Naive Bayes) 30pts Report 10pts TOTAL 100 1 Datasets Iris Dataset (sklearn.datasets.load iris) The Iris flower data set or Fishers Iris data set is a multivariate data set introduced by the British statistician and biologist Ronald Fisher in his 1936 paper The use of multiple measurements in taxonomic problems as an example of linear discriminant analysis. The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters. The iris data set is widely used as a beginner’s dataset for machine learning purposes. The dataset is included in the machine learning package Scikit-learn, so that users can access it without having to find a source for it. The following python code illustrates usage. from sk l e a rn . da ta s e t s import l o a d i r i s i r i s = l o a d i r i s ( ) Spambase Dataset (spambase.data) This dataset consists of 4601 instances of data, each with 57 features and a class label designating if the sample is spam or not. The features are real valued and are described in much detail here: https://archive.ics.uci.edu/ml/machine-learning-databases/spambase/spambase.names Data obtained from: https://archive.ics.uci.edu/ml/datasets/Spambase 2 1 Theory 1. Consider the following set of training examples for an unknown target function: (x1, x2)→ y: Y x1 x2 Count + T T 3 + T F 4 + F T 4 + F F 1 – T T 0 – T F 1 – F T 3 – F F 5 (a) What is the sample entropy, H(Y ) from this training data (using log base 2) (2pts)? (b) What are the information gains for branching on variables x1 and x2 (2pts)? (c) Draw the deicion tree that would be learned by the ID3 algorithm without pruning from this training data (3pts)? 2. We decided that maybe we can use the number of characters and the average word length an essay to determine if the student should get an A in a class or not. Below are five samples of this data: # of Chars Average Word Length Give an A 216 5.68 Yes 69 4.78 Yes 302 2.31 No 60 3.16 Yes 393 4.2 No (a) What are the class priors, P (A = Y es), P (A = No)? (2pt) (b) Find the parameters of the Gaussians necessary to do Gaussian Naive Bayes classification on this decision to give an A or not. Standardize the features first over all the data together so that there is no unfair bias towards the features of different scales (2pts). (c) Using your response from the prior question, determine if an essay with 242 characters and an average word length of 4.56 should get an A or not (3pts). 3. Consider the following questions pertaining to a k-Nearest Neighbors algorithm (1pt): (a) How could you use a validation set to determine the user-defined parameter k? 3 2 Logistic Regression Let’s train and test a Logistic Regression Classifier to classify flowers from the Iris Dataset. First download import the data from sklearn.datasets. As mentioned in the Datasets area, The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters. We will map this into a binary classification problem between Iris setosa versus Iris virgincia and versicolor. We will use just the first 2 features, width and length of the sepals. For this part, we will be practicing gradient descent with logistic regression. Use the following code to load the data, and binarize the target values. i r i s = skdata . l o a d i r i s ( ) X = i r i s . data [ : , : 2 ] y = ( i r i s . t a r g e t != 0) ∗ 1 Write a script that: 1. Reads in the data with the script above. 2. Standardizes the data using the mean and standard deviation 3. Initialize the parameters of θ using random values in the range [-1, 1] 4. Do batch gradient descent 5. Terminate when absolute value change in the loss on the data is less than 2−23, or after 10, 000 iterations have passed (whichever occurs first). 6. Use a learning rate η = 0.01. 7. While the termination criteria (mentioned above in the implementation details) hasn’t been met (a) Compute the loss of the data using the logistic regression cost (b) Update each parameter using batch gradient descent Plot the data and the decision boundary using matplotlib. Verify your solution with the Logisti- cRegression sklearn method. from sk l e a rn . l i n ea r mode l import L o g i s t i c R e g r e s s i o n l g r = L o g i s t i c R e g r e s s i o n ( pena l ty =’none ’ , s o l v e r =’ l b f g s ’ , max i ter =10000) l g r . f i t (X, y ) In your writeup, present the thetas from gradient descent that minimize the loss function as well as plots of your method versus the built in LogisticRegression method. 4 3 Logistic Regression Spam Classification Let’s train and test a Logistic Regression Classifier to classifiy Spam or Not from the Spambase Dataset. First download the dataset spambase.data from Blackboard. As mentioned in the Datasets area, this dataset contains 4601 rows of data, each with 57 continuous valued features followed by a binary class label (0=not-spam, 1=spam). There is no header information in this file and the data is comma separated. Write a script that: 1. Reads in the data. 2. Randomizes the data. 3. Selects the first 2/3 (round up) of the data for training and the remaining for testing (you may use sklearn train test split for this part) 4. Standardizes the data (except for the last column of course) using the training data 5. Initialize the parameters of θ using random values in the range [-1, 1] 6. Do batch gradient descent 7. Terminate when absolute value change in the loss on the data is less than 2−23, or after 1, 500 iterations have passed (whichever occurs first, this will likely be a slow process). 8. Use a learning rate η = 0.01. 9. Classify each testing sample using the model and choosing the class label based on which class probability is higher. 10. Computes the following statistics using the testing data results: (a) Precision (b) Recall (c) F-measure (d) Accuracy Implementation Details 1. Seed the random number generate with zero prior to randomizing the data 2. There are a lot of θs and this will likely be a slow process In your report you will need: 1. The statistics requested for your Logistic classifier run. 5 4 Naive Bayes Classifier Let’s train and test a Naive Bayes Classifier to classifiy Spam or Not from the Spambase Dataset. First download the dataset spambase.data from Blackboard. As mentioned in the Datasets area, this dataset contains 4601 rows of data, each with 57 continuous valued features followed by a binary class label (0=not-spam, 1=spam). There is no header information in this file and the data is comma separated. As always, your code should work on any dataset that lacks header information and has several comma-separated continuous-valued features followed by a class id ∈ 0, 1. Write a script that: 1. Reads in the data. 2. Randomizes the data. 3. Selects the first 2/3 (round up) of the data for training and the remaining for testing 4. Standardizes the data (except for the last column of course) using the training data 5. Divides the training data into two groups: Spam samples, Non-Spam samples. 6. Creates Normal models for each feature for each class. 7. Classify each testing sample using these models and choosing the class label based on which class probability is higher. 8. Computes the following statistics using the testing data results: (a) Precision (b) Recall (c) F-measure (d) Accuracy Implementation Details 1. Seed the random number
generate with zero prior to randomizing the data 2. You may want to consider using the log-exponent trick to avoid underflow issues. Here’s a link about it: https://stats.stackexchange.com/questions/105602/example-of-how-the-log-sum-exp- trick-works-in-naive-bayes 3. If you decide to work in log space, realize that python interprets 0log0 as inf. You should identify this situation and either add an EPS (very small positive number) or consider it to be a value of zero. In your report you will need: 1. The statistics requested for your Naive Bayes classifier run. 6 Submission For your submission, upload to Blackboard a single zip file containing: 1. PDF Writeup 2. Python notebook Code The PDF document should contain the following: 1. Part 1: (a) Answers to Theory Questions 2. Part 2: (a) Requested Logistic Regression thetas and plots 3. Part 3: (a) Requested Classification Statistics 4. Part 4: (a) Requested Classification Statistics 7
