CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg MoriAssignment 1: RegressionDue October 4 at 11:59pmThis assignment is to be done individually.Important Note: The university policy on academic dishonesty (cheating) will be taken veryseriously in this course. You may not provide or use any solution, in whole or in part, to or byanother student.You are encouraged to discuss the concepts involved in the questions with other students. If you arein doubt as to what constitutes acceptable discussion, please ask! Further, please take advantage ofoffice hours offered by the TAs if you are having difficulties with this assignment.DO NOT:• Give/receive code or proofs to/from other students• Use Google to find solutions for assignmentDO:• Meet with other students to discuss assignment (it is best not to take any notes during suchmeetings, and to re-work assignment on your own)• Use online resources (e.g. Wikipedia) to understand the concepts needed to solve the assignment1CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori1 Probabilistic ModelingIn lecture we went over an example of modeling coin tossing – estimating a parameter µ, theprobability the coin comes up heads.Consider instead the problem of modeling a 6-sided die.1. What is the parameter that explains the behaviour of the die in this case (in analogy to the µfor the coin)?2. What is the value of the parameter for a fair die (equal probability of rolling any number)?3. What is the value of the parameter for a die that always rolls a 2?4. Specify the domain of the parameter – which settings of the parameter are valid.2 Weighted Squared ErrorThe sum-of-squares error function for regression (Eqn. 3.12 in PRML) treats every training datapoint equally. In some instances, we may wish to place different weights on different training datapoints. This could arise if we have confidence estimates of the accuracy of each training data point.Consider the weighted sum-of-squares error function:EDˆ(w) =12N∑n=1αn{tn −wTφ(xn)}2 (1)with weights αn > 0 on each training data point.Derive the optimal weights w given this weighted sum-of-squares error function.3 Training vs. Test ErrorFor the questions below, assume that error means RMS (root mean squared error).1. Suppose we perform unregularized regression on a dataset. Is the validation error alwayshigher than the training error? Explain.2. Suppose we perform unregularized regression on a dataset. Is the training error witha degree 10 polynomial always lower than or equal to that using a degree 9 polynomial?Explain.3. Suppose we perform both regularized and unregularized regression on a dataset. Is thetesting error with a degree 20 polynomial always lower using regularized regression com-pared to unregularized regression? Explain.2CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori4 Basis Function Dependent RegularizationIn lecture we saw a regularization technique applied to linear regression where all weights in theregression model are regularized in the same fashion (like L1, or L2), and with a common valuefor λ. Consider the case where for each weight wn, we have a different tradeoff parameter λn,and a choice from among one of L1 or L2 regularizer. Derive the formula of the gradient for theregularized squared error loss function in this scenario.∇E(w) = ?(Hint for notation: Let J1 be the set of indices of basis functions whose weights have L1 regu-larization, and J2 be the set of indices of basis functions whose weights have L2 regularization.Alternately, you may define and use other suitable notation.)5 RegressionIn this question you will train models for regression and analyze a dataset. Start by downloadingthe code and dataset from the website.The dataset is created from data provided by UNICEF’s State of the World’s Children 2013 report:http://www.unicef.org/sowc2013/statistics.htmlChild mortality rates (number of children who die before age 5, per 1000 live births) for 195countries, and a set of other indicators are included.5.1 Getting startedRun the provided script polynomial regression.py to load the dataset and names of coun-tries / features.Answer the following questions about the data. Include these answers in your report.1. Which country had the highest child mortality rate in 1990? What was the rate?2. Which country had the highest child mortality rate in 2011? What was the rate?3. Some countries are missing some features (see original .xlsx/.csv spreadsheet). How is thishandled in the function assignment1.load unicef data()?For the rest of this question use the following data and splits for train/test and cross-validation.• Target value: column 2 (Under-5 mortality rate (U5MR) 2011)1.• Input features: columns 8-40.• Training data: countries 1-100 (Afghanistan to Luxembourg).1Zero-indexing, hence values[:,1].3CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg Mori• Testing data: countries 101-195 (Madagascar to Zimbabwe).• Cross-validation: subdivide training data into folds with countries 1-10 (Afghanistan to Aus-tria), 11-20 (Azerbaijan to Bhutan), … . I.e. train on countries 11-100, validate on 1-10; train on1-10 and 21-100, validate on 11-20, …5.2 Polynomial RegressionImplement linear basis function regression with polynomial basis functions. Use only monomialsof a single variable (x1, x21, x22) and no cross-terms (x1 · x2).Perform the following experiments:1. Create a python script polynomial regression.py for the following.Fit a polynomial basis function regression (unregularized) for degree 1 to degree 6 polyno-mials. Include bias term. Plot training error and test error (in RMS error) versus polynomialdegree.Put this plot in your report, along with a brief comment about what is “wrong” in your report.Normalize the input features before using them (not the targets, just the inputs x). Useassignment1.normalize data().Run the code again, and put this new plot in your report.2. Create a python script polynomial regression 1d.py for the following.Perform regression using just a single input feature.Try features 8-15 (Total population – Low birthweight). For each (un-normalized) feature fita degree 3 polynomial (unregularized). Try with and without a bias term.Plot training error and test error (in RMS error) for each of the 8 features. This should beas bar charts (e.g. use matplotlib.pyplot.bar()) — one for models with bias term,and another for models without bias term.Put the two bar charts in your report.The testing error for feature 11 (GNI per capita) is very high. To see what happened, pro-duce plots of the training data points, learned polynomial, and test data points. The codevisualize 1d.py may be useful.In your report, include plots of the fits for degree 3 polynomials for features 11 (GNI), 12(Life expectancy), 13 (literacy).5.3 Sigmoid Basis Functions1. Create a python script sigmoid regression.py for the following.Implement regression using sigmoid basis functions for a single input feature. Use twosigmoid basis functions, with µ = 100, 10000 and s = 2000.0. Include a bias term. Useun-normalized features.4CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg MoriFit this regression model using feature 11 (GNI per capita).In your report, include a plot of the fit for feature 11 (GNI).In your report, include the training and testing error for this regression model.5.4 Regularized Polynomial Regression1. Create a python script polynomial regression reg.py for the following.ImplementL2-regularized regression. Fit a degree 2 polynomial using λ = {0, .01, .1, 1, 10, 102, 103, 104}.Use normalized features as input. Include a bias term. Use 10-fold cross-validation to de-cide on the best value for λ. Produce a plot of average validation set error versus λ. Use amatplotlib.pyplot.semilogx plot, putting λ on a log scale2.Put this plot in your report, and note which λ value you would choose from the cross-validation.2The unregularized result will not appear on this scale. You can either add it as a separate horizontal line as abaseline, or report this number separately.5CMPT 726: Assignment 1 (Fall 2019) Instructor: Greg MoriSubmitting Yo
ur AssignmentThe assignment must be submitted online at https://courses.cs.sfu.ca. In order tosimplify grading, you must adhere to the following structure.You must submit two files:1. You must create an assignment report in PDF format, called report.pdf. This reportmust contain the solutions to questions 1-4 as well as the figures / explanations requested for5.2. You must submit a .zip file of all your code, called code.zip. This must contain a singledirectory called code (no sub-directories, no leading path names), in which all of your filesmust appear3. There must be the 4 scripts with the specific names referred to in Question 4,as well as a common codebase you create and name.As a check, if one runsunzip code.zipcd code./polynomial_regression_1d.pythe script produces the plots in your report from the relevant question.3This includes the data files and others which are provided as part of the assignment.6
