EEEM007 ASP-Lab 2019-20 / v1.8 / AM,MDP / 11 Mar 20 Module: EEEM007 ADVANCED SIGNAL PROCESSING Year: 2019/2020. Examiner: A Mustafa, M D Plumbley Date Due: 4pm Tuesday 12 May 2020 LAB EXPERIMENT: PATTERN RECOGNTION 1. INTRODUCTION The aim of the experiment is to provide a practical support for the main aspects of the material covered in EEEM007 Advanced Signal Processing, namely Pattern Classification. The experiment is designed to reinforce the main theoretical results learnt in the course and to enable the student to gain intuitive feeling about the effects of classifier design factors (such as training set size, class separability, feature space dimensionality, classification rule) on the classification system performance. It should also validate experimentally the results derived in the Assignment. Two popular classifiers are investigated in the experiment: i) the Bayes decision rule for normally distributed classes and ii) the k-Nearest Neighbour decision rule. 2. MATLAB COMMANDS You may find the following Matlab commands useful: mvnrnd generate random numbers fitcnb construct Gaussian classifier fitcknn construct k-NN classifier predict function for finding class labels mahal Mahalanobis distance Include the Matlab code you used for your experiments in an Appendix to your report. 3. REPORT FORMAT The Report should be a single document, consisting of descriptions and results for the experiments (approx. 8-10 pages), plus an Appendix containing text listings of your Matlab code. [Remember: The TurnItIn submission system will identify similar reports, so please write your own Matlab code and experiment descriptions.] 4. MARKING SCHEME For each experiment: 1. Description of experiment, objectives, design choices [20%] 2. Predicted experimental outcome from the theory [20%] 3. Presentation of the experimental results obtained [20%] 4. Analysis of results, discussion and conclusions [20%] 5. Presentation of the Matlab code used [20%] Each of Experiments 1 to 4 is worth 1/4 of the final mark. EEEM007 ASP-Lab 2019-20 / v1.8 / AM,MDP / 11 Mar 20 5. EXPERIMENT DESCRIPTION 5.1 Experiment 1 The aim of this experiment is to investigate the effect of training sample size on the classifier performance and the effect of the size of test set on the reliability of the empirical error count estimator. A Gaussian classifier for discriminating between two 2-dimensional classes will be used for the study. Use the class parameters (, , , , ) corresponding to your Assignment. (a) Generate a design set and test set , each containing 100 patterns per class, distributed according to the selected class parameters. Design a Gaussian classifier for different values of = 3,5,10,50,100 select samples to form a training set () from . Test the designs with the same training samples to obtain error estimates design(). Test each classifier also using the full test set of 100 patterns to obtain error estimate test. Repeat the experiment ten times for independent design sets , = 1, . . .10 (the same test set may be used) and average the estimated errors to obtain: design() = 1 10 ∑ design () 10 =1 test() = 1 10 ∑ test () 10 =1 . Plot the average errors design, test as a function of and compare them with the theoretical error. Comment on your results and try to explain them. (b) Repeat (a) using the k-Nearest Neighbour classifier for all odd values of k in the range 1 to 51, noting that cannot exceed 2. For a representative range of , starting from k=1 compare your results with those obtained for the Gaussian classifier in Experiment 1. For each value of select the best result ∗() (smallest error) over all values of and record the corresponding ∗(). Plot these best results as a function of in the same graph as that used for presenting the results of (a) above. Plot also ∗() in the same graph. Comment on your results. 5.2 Experiment 2 In this experiment we shall investigate the effect of the size of test set on the reliability of the empirical error count estimator and we will also learn how to classify test and training samples. (a) Design a Gaussian classifier for a two class problem similar to Experiment 1 (a). Generate ten independent test sets , = 1, . . .10 of size 100 using the same class conditional distribution parameters. Choose several different numbers of test patterns = 3,5,10,50,100. For each samples form test set () for each to obtain an estimate () of the classifier error probability. Repeat the experiment ten times for independent test sets and find the mean value () and the variance 2() of the estimated error, i.e. EEEM007 ASP-Lab 2019-20 / v1.8 / AM,MDP / 11 Mar 20 () = 1 10 ∑ () 10 =1 2 = 1 9 ∑[() − ()] 2 10 =1 Plot your results and comment on how they compare with your theoretical predictions. Comment on your results. (b) Now assume that you have 100 samples for both testing and training, such that + = 100. Assign = 3,5, 10, 50, 75,90 and respectively = 97,95, 90, 50, 25,10 to form training () and test () sets from the and samples to calculate the design (design() from Experiment 1 (a)) and test error (() from Experiment 2 (a)) estimates for each combination of train as test sets. Create a table as shown here depicting both design and test errors for each set of and samples. design() () 3 97 (results) … (etc) Comment on your results. Which combination of and is the best trade-off in terms of error estimates. 5.3 Experiment 3 In this experiment we shall investigate the dependence of the test() curves on the dimensionality of the pattern recognition problem. As the determination of the true error probability in high dimensional spaces is difficult, we shall take the estimated error test(), = 500) as the true error. You may choose the covariance matrix to be an identity matrix in this exercise. Choose the mean vectors so that the error probability is maintained in the range of 5-10%. Try to estimate test() for values = 3,5,10,20,50,100,200 for two class problems in = 5,10,15 dimensional spaces. The changes in the test error with the increase in training set size are different in different dimensions. If you are unable to design the classifier, consider what is the minimum number of training samples required as a function of dimensionality and why. How many training samples as a function of dimensionality do you need to achieve a reasonable performance (close to the true error rate)? 5.4 Experiment 4 The aim of this experiment is to explore the relationship between class separability and error probability. Choosing a suitable pattern space dimensionality, , generate a sequence of sets of normally distributed training data containing patterns from two classes. The Mahalanobis distance between the means of the two classes in each set should be monotonically increasing with the position of the set in the sequence. Estimate the error probability of the classifier in each case. Plot the error as a function of Mahalanobis distance. Comment on your findings.
