CMPUT 328 Assignment 10 Classification and Detection on MNIST Double Digits Dataset Overview In this assignment, you are going to do classification on the MNIST Double Digits (MNISTDD) dataset. This dataset contains gray scale images of size 64 × 64. Each image has two MNIST digits (from 0 to 9) randomly placed inside it as in this visualization: Figure 1: Visualization of the first 64 images in the MNISTDD training set. Each image is 64 × 64. Please note that the digits in these images are not taken directly from MNIST, but rather generated by Generative Adversarial Networks (GANs) trained on it. Two digits in an image may partially or completely overlap each other as shown in Figure 1 (for example, see the last image in 3rd row) though complete overlap only happens in a small fraction of images. Most images do not have digits overlapping or only partially overlapping. Two digits in an image may also be of the same class. Your task in this assignment is to tell which digits are contained in each image and where are they located. Dataset The MNISTDD dataset is divided in to 3 subsets: train, validation and test containing 55K, 5K and 10K samples respectively. A sample consists of: • Image: A 64 × 64 image that has been vectorized to a 4096-dimensional vector. • Labels: A 2-dimensional vector that has two numbers in the range [0, 9] which are the two digits in the image. Note that these two numbers are always in ascending order. For example, if digits 7 and 5 are in a image, then this two-vector will be [5, 7] and not [7, 5]. • Bounding boxes: A 2 × 4 matrix which contains two bounding boxes that mark locations of the two digits in the image. The first row contain location for the first digit in labels and the second row for the second one. Each row of the matrix has 4 numbers which represent [row of the top left corner, column of the top left corner, row of the bottom right corner, column of the bottom right corner] in the exact order. Note: it is always the case that row of the bottom right corner – row of the top left corner = column of the bottom right corner – column of the top left corner = 28. This means that each bounding boxes has a size of 28 × 28 no matter how large or small the digit inside that box is. As an example, consider the following 64 × 64 image that contains the digits 5 and 2: • Image will be the above image but flattened to a 4096-dimensional vector. • Labels will be [2, 5] • Bounding boxes will be a matrix with the first row: [11, 27, 39, 55] and the second row: [6, 2, 34, 30]. This means that the digit 2 is located between the 11th and 39th rows and between 27th and 55th columns of the image. Same goes for digit 5 – it is located between the 6th and 34th rows and between 2nd and 30th columns of the image. Each set comprises 3 .npy files which can each be read using numpy.load()to obtain the corresponding matrix stored as a numpy.ndarray. Following are detailed descriptions of the 3 files where {SET_NAME} denotes the name of the subset (train, valid or test) and is the number of samples: • {SET_NAME}_X.npy: 2D matrix with dimension [, 4096] and containing the vectorized images Each row is a single vectorized image. • {SET_NAME}_Y.npy: 2D matrix with dimension [, 2] and containing the labels. Note that the labels are always in ascending order in each row. • {SET_NAME}_bboxes.npy: 3D matrix with dimension [, 2, 4] and containing the bounding boxes. For more information, see the description of bounding boxes above. For example, following are the dimensions of the numpy.ndarray in 3 files of the train set: • train_X.npy: [55000, 4096] • train_Y.npy: [55000, 2] • train_bboxes.npy: [55000, 2, 4] Task You are provided with the train and valid sets that can be downloaded from e-class in a zip archive named MNISTDD_train_valid.zip. This contains 6 files: train_X.npy, train_Y.npy, train_bboxes.npy, valid_X.npy, valid_Y.npy, valid_bboxes.npy. The test set is NOT released. You are also provided two python files: A10_eval.py and A10_submission.py. The latter has a single function classify_and_detect that you need to complete. It takes a single matrix containing all test (or validation) images as input and returns two NumPy arrays pred_class and pred_bboxes respectively containing the classification labels and detection bounding boxes in the same format as described above. To reiterate, the two digits in each row of pred_class must be in ascending order and the two rows in pred_bboxes corresponding to that image must match this ordering too. A10_eval.py can be run to evaluate the performance of your method as in terms of the classification accuracy (% of digits classified correctly) and Intersection over Union (IOU) of the detection boxes with the ground truth boxes. You are free to add any other functions or classes you need including those in other files imported from A10_submission.py. Just make sure to submit all files needed to run your code. You can use any machine learning method to solve this problem. There is no restriction or guideline as to what algorithm you can or should use. Marking This assignment uses relative marking where all qualifying submissions will be ranked by classification accuracy and detection IOU separately. A linear scaling from 50 – 100 will then be used for lowest to highest ranked submissions to generate a score for each metric. The overall score will be the mean of the two. There is no lab quiz for this assignment. What to Submit You need to submit a single zip file containing the modified A10_submission.py along with any other python files or checkpoints needed to run it on the test set. Testing will be done by running the main function in A10_eval.py and it is your responsibility to ensure that your code can be imported and run without errors. Apart from correct functioning, there is one more qualifying criterion for inclusion in the ranking process – testing time should not exceed 10 minutes on Colab CPU. Any submission not satisfying this will get no marks for this assignment. Submission deadline is November 29, 11:55 PM. This assignment is worth 8.7% of the overall grade.
