DATA7703 – Machine Learning Case Study 2020 (worth 10% of total course marks) Marcus Gallagher May 2020 1 Introduction In the course you have heard about the core trade-off that exists in machine learning between under/over fitting, model complexity and training set size. A recently-published paper[1] suggests that this is not an adequate explanation of over-parameterized models, including deep neural networks. Their idea is summarized in Figure 1. Figure 1: Figure 1 from the paper! Your task in this case study is to design and carry out a scientific experiment that tests the phenomena of double- descent curves in machine learning. This is an individual assessment task, so your submission must be your own individual work. 2 Procedure • Read the paper carefully. • Design an experiment/set of experiments. – You will need to decide what kind of model(s) and dataset(s) to use, as well as make decisions about the hyperparameters of the models and parameters of your experiments (e.g. number of trials to perform). – You are free to use any code you have written during the course, software libraries and software packages to implement your experiments, as long as you appropriately cite the work of others. You probably shouldn’t spend time implementing models/algorithms from scratch to do your experiments with – marks are based on the scientific experimental design, the results and your write-up of the experiment. However, you definitely should come up with your own experimental design from scratch. Taking (for example) a 1 blog post where someone has done something different and tweaking it to fit this task is unlikely to receive high marks. – Experiments can take a significant amount of computation time. You are not expected to do anything that requires more than (for example) 24hrs of compute time on a typical desktop or laptop computer. • Carry out your experiments, collect and analyse the results. 3 To Hand-in Write up a brief report (at most 5 pages of text, not including figures and tables) describing what you have done and the results. Note: this is a scientific experiment – I do not know if you are going to be able to successfully replicate the double-descent curve that the paper hypothesizes. It is likely to depend on many experimental factors. But you should not stress about this – achieving a high mark is not about whether your experiment is successful! Rather, it is about the quality, rigour and presentation of your work. I am interested in reading about what you did, why you did it, what can be concluded for your experiments and what you learned. 4 References [1] Belkin, M., Hsu, D., Ma, S., Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849-15854. 2
