CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori Assignment 1: Regression Due October 4 at 11:59pm This assignment is to be done individually. Important Note: The university policy on academic dishonesty (cheating) will be taken very seriously in this course. You may not provide or use any solution, in whole or in part, to or by another student. You are encouraged to discuss the concepts involved in the questions with other students. If you are in doubt as to what constitutes acceptable discussion, please ask! Further, please take advantage of office hours offered by the TAs if you are having difficulties with this assignment. DO NOT: • Give/receive code or proofs to/from other students • Use Google to find solutions for assignment DO: • Meet with other students to discuss assignment (it is best not to take any notes during such meetings, and to re-work assignment on your own) • Use online resources (e.g. Wikipedia) to understand the concepts needed to solve the assignment 1 CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori 1 Probabilistic Modeling In lecture we went over an example of modeling coin tossing – estimating a parameter µ, the probability the coin comes up heads. Consider instead the problem of modeling a 6-sided die. 1. What is the parameter that explains the behaviour of the die in this case (in analogy to the µ for the coin)? 2. What is the value of the parameter for a fair die (equal probability of rolling any number)? 3. What is the value of the parameter for a die that always rolls a 2? 4. Specify the domain of the parameter – which settings of the parameter are valid. 2 Weighted Squared Error The sum-of-squares error function for regression (Eqn. 3.12 in PRML) treats every training data point equally. In some instances, we may wish to place different weights on different training data points. This could arise if we have confidence estimates of the accuracy of each training data point. Consider the weighted sum-of-squares error function: EDˆ(w) = 1 2 N∑ n=1 αn{tn −wTφ(xn)}2 (1) with weights αn > 0 on each training data point. Derive the optimal weights w given this weighted sum-of-squares error function. 3 Training vs. Test Error For the questions below, assume that error means RMS (root mean squared error). 1. Suppose we perform unregularized regression on a dataset. Is the validation error always higher than the training error? Explain. 2. Suppose we perform unregularized regression on a dataset. Is the training error with a degree 10 polynomial always lower than or equal to that using a degree 9 polynomial? Explain. 3. Suppose we perform both regularized and unregularized regression on a dataset. Is the testing error with a degree 20 polynomial always lower using regularized regression com- pared to unregularized regression? Explain. 2 CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori 4 Basis Function Dependent Regularization In lecture we saw a regularization technique applied to linear regression where all weights in the regression model are regularized in the same fashion (like L1, or L2), and with a common value for λ. Consider the case where for each weight wn, we have a different tradeoff parameter λn, and a choice from among one of L1 or L2 regularizer. Derive the formula of the gradient for the regularized squared error loss function in this scenario. ∇E(w) = ? (Hint for notation: Let J1 be the set of indices of basis functions whose weights have L1 regu- larization, and J2 be the set of indices of basis functions whose weights have L2 regularization. Alternately, you may define and use other suitable notation.) 5 Regression In this question you will train models for regression and analyze a dataset. Start by downloading the code and dataset from the website. The dataset is created from data provided by UNICEF’s State of the World’s Children 2013 report: http://www.unicef.org/sowc2013/statistics.html Child mortality rates (number of children who die before age 5, per 1000 live births) for 195 countries, and a set of other indicators are included. 5.1 Getting started Run the provided script polynomial regression.py to load the dataset and names of coun- tries / features. Answer the following questions about the data. Include these answers in your report. 1. Which country had the highest child mortality rate in 1990? What was the rate? 2. Which country had the highest child mortality rate in 2011? What was the rate? 3. Some countries are missing some features (see original .xlsx/.csv spreadsheet). How is this handled in the function assignment1.load unicef data()? For the rest of this question use the following data and splits for train/test and cross-validation. • Target value: column 2 (Under-5 mortality rate (U5MR) 2011)1. • Input features: columns 8-40. • Training data: countries 1-100 (Afghanistan to Luxembourg). 1Zero-indexing, hence values[:,1]. 3 CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori • Testing data: countries 101-195 (Madagascar to Zimbabwe). • Cross-validation: subdivide training data into folds with countries 1-10 (Afghanistan to Aus- tria), 11-20 (Azerbaijan to Bhutan), … . I.e. train on countries 11-100, validate on 1-10; train on 1-10 and 21-100, validate on 11-20, … 5.2 Polynomial Regression Implement linear basis function regression with polynomial basis functions. Use only monomials of a single variable (x1, x21, x 2 2) and no cross-terms (x1 · x2). Perform the following experiments: 1. Create a python script polynomial regression.py for the following. Fit a polynomial basis function regression (unregularized) for degree 1 to degree 6 polyno- mials. Include bias term. Plot training error and test error (in RMS error) versus polynomial degree. Put this plot in your report, along with a brief comment about what is “wrong” in your report. Normalize the input features before using them (not the targets, just the inputs x). Use assignment1.normalize data(). Run the code again, and put this new plot in your report. 2. Create a python script polynomial regression 1d.py for the following. Perform regression using just a single input feature. Try features 8-15 (Total population – Low birthweight). For each (un-normalized) feature fit a degree 3 polynomial (unregularized). Try with and without a bias term. Plot training error and test error (in RMS error) for each of the 8 features. This should be as bar charts (e.g. use matplotlib.pyplot.bar()) — one for models with bias term, and another for models without bias term. Put the two bar charts in your report. The testing error for feature 11 (GNI per capita) is very high. To see what happened, pro- duce plots of the training data points, learned polynomial, and test data points. The code visualize 1d.py may be useful. In your report, include plots of the fits for degree 3 polynomials for features 11 (GNI), 12 (Life expectancy), 13 (literacy). 5.3 Sigmoid Basis Functions 1. Create a python script sigmoid regression.py for the following. Implement regression using sigmoid basis functions for a single input feature. Use two sigmoid basis functions, with µ = 100, 10000 and s = 2000.0. Include a bias term. Use un-normalized features. 4 CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori Fit this regression model using feature 11 (GNI per capita). In your report, include a plot of the fit for feature 11 (GNI). In your report, include the training and testing error for this regression model. 5.4 Regularized Polynomial Regression 1. Create a python script polynomial regression reg.py for the following. ImplementL2-regularized regression. Fit a degree 2 polynomial using λ = {0, .01, .1, 1, 10, 102, 103, 104}. Use normalized features as input. Include a bias term. Use 10-fold cross-validation to de- cide on the best value for λ. Produce a plot of average validation set error versus λ. Use a matplotlib.pyplot.semilogx plot, putting λ on a log scale2. Put this plot in your report, and note which λ value you would choose from the cross- validation. 2The unregularized result will not appear on this scale. You can either add it as a separate
horizontal line as a baseline, or report this number separately. 5 CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori Submitting Your Assignment The assignment must be submitted online at https://courses.cs.sfu.ca. In order to simplify grading, you must adhere to the following structure. You must submit two files: 1. You must create an assignment report in PDF format, called report.pdf. This report must contain the solutions to questions 1-4 as well as the figures / explanations requested for 5. 2. You must submit a .zip file of all your code, called code.zip. This must contain a single directory called code (no sub-directories, no leading path names), in which all of your files must appear3. There must be the 4 scripts with the specific names referred to in Question 4, as well as a common codebase you create and name. As a check, if one runs unzip code.zip cd code ./polynomial_regression_1d.py the script produces the plots in your report from the relevant question. 3This includes the data files and others which are provided as part of the assignment. 6
