EECE5644 Fall 2019 – Homework 3 Submit: Monday, 2019-November-04 before 09:00ET Please submit your solutions on Blackboard in a PDF file that includes all math and numerical results. Also include your code in one of the following ways: (Acceptable) upload an accompa- nying ZIP file containing all code files, (Preferred) keep your code in an online version control repository and provide a link to the relevant online repository in your PDF file. Note that we will only grade the material submitted in the PDF file. Do NOT link from the PDF to online sources like Jupyter Notebook to present your numerical results. Those materials will not be considered when grading. Make sure that you cite all resources you benefit from (books, papers, software packages). This is a graded assignment and the entirety of your submission must contain only your own work. You may benefit from literature including software (as allowed by specific restrictions in questions), as long as these sources are properly acknowledged in your submission. Question 1 (50%) Conduct the following model order selection exercise using 10-fold cross-validation procedure and report your procedure and results in a comprehensive, convincing, and rigorous fashion: 1. Select a Gaussian Mixture Model as the true probability density function for 2-dimensional real-valued data synthesis. This GMM will have 4 components with different mean vectors, different covariance matrices, and different probability for each Gaussian to be selected as the generator for each sample. Specify the true GMM that generates data. 2. Generate multiple data sets with independent identically distributed samples using this true GMM; these datasets will have respectively 10, 100, 1000, 10000 samples. 3. For each data set, using maximum likelihood parameter estimation principles (e.g. with the EM algorithm), within the framework of K(=10)-fold cross-validation, evaluate GMMs with different model orders; specifically evaluate candidate GMMs with 1, 2, 3, 4, 5, 6 Gaus- sian components. Note that both model parameter estimation and validation performance measures to be used is log-likelihood of data. 4. Report your results for the experiment, indicating which of the six GMM orders get selected for each of the datasets you produced. Develop a good way to describe and summarize your experiment results in the form of tables/figures. Question 2 (50%) Conduct the following maximum likelihood discriminative classifier training exercise on data generated from two Gaussian distributed classes: 1. Generate training set with 999 2-dimensional samples from two classes with priors q− = 0.3 and q+ = 0.7; the class-conditional data probability distributions are two Gaussians with different mean vectors and different covariance matrices (choose the matrices to be non- diagonal with distinct eigenvalues, so your Gaussian pdfs are tilted with respect to each other and elongated in different directions by different aspect ratios). Hint: For more interesting results, make the Gaussians overlap with each other somewhat significantly, so that the minimum error probability achievable is not too small. 1 2. Using Fisher LDA, identify a linear classifier that minimizes the error count on the training set. This classifier will have a discriminant function of the form wTLDAx+ bLDA where x is the data vector, and the classifier decides in favor of Class − if the discriminant is below 0, and decides in favor of Class +, if the discriminant is at least 0. 3. Train the parameters of a logistic function y(x) = 1/(1 + e(wT x+b)) using the maximum likelihood estimation principle to optimize the parameters w and b with the training set, such that the function is trained to act as a surrogate for the posterior probability of Class + given x. In particular, your model assumes that y(x) ≈ P(Label = +|x); consequently, 1− y(x) ≈ P(Label = −|x). Hint: Once you specify the optimization objective to train this logistic-linear-model for class posterior, you can solve the optimization problem using any suitable numerical optimization procedure, such as gradient ascent that you implement from scratch, or using a derivative free numerical optimization procedure like the Nelder-Mead Simplex Reflection Algorithm (e.g. in Matlab, fminsearch). Make a choice, implement cor- rectly, perhaps consider using the LDA solution you developed earlier to provide an initial estimate for the model parameters. 4. Report visual and numerical results that compare the following three classifiers (e.g. data scatter plots with color/shape indicators of true/decided labels), including the error counts each classifier achieve on the training set: MAP-classifier that makes use of the true data distributions and class priors, which achieves minimum probability error by design; LDA classifier you designed earlier; logistic-linear classifier you designed next. 2
