ECE 472 Robotics and Vision Prof. K. Dana Final Project: 3D Reconstruction, Deep Learning, Augmented Reality. Submission Instructions for Final Project Submit a 2 page progress report and a 4-5 page final report. Explain all algorithms in paragraph form demonstrating your technical knowledge of the code. Use equations (latex is highly recommended). Submit: 1) ProjectReport.pdf 2) Your Python Code 3) link to Images necessary to run the code in a compressed folder called Images/. (Do not use very high resolution images), 4) A list of dependencies required to run your code in a README file. Additional Instructions You may use opensource code except where noted. Give specific credit in your report about the source of this code. All additional code should be your own. For Part1, all images for reconstruction should be your own. This is an individual project. You are encouraged to discuss methods and issues with classmates, including discussion of opensource code and relevant tutorials; however, no copying of code, report text or images is allowed. Use Piazza to discuss what works and doesn’t work for you, especially in dealing with systems/software/library issues. 1. This semester you have learned algorithms for 3D reconstruction with two images (i.e. stereo reconstruction). However, 3D reconstruction with many images typically leads to much better results. Modern computer vision applications optimize 3D reconstruc- tion over many views (e.g. from multiple cameras or from the video feed of a single camera). This process is called structure from motion – SFM or multiview stereo and the refinement of the estimate is called bundle adjustment. (a) With your own set of images of an object (minimum 4 images) write code to reconstruct the scene using multiview stereo. Show the original images and the point cloud with sufficient detail to convey the 3D shape. Include these images in the final report. The 3D shape should not be trivial (e.g. not planes, spheres or cylinders). (b) In your report, describe the algorithm that is implemented in the components of your code. Use your own words. Use equations that you format (latex is highly recommended). Insert snippets of code in the report for clarification. (c) Some useful resources may include: • http://openmvg.readthedocs.io/en/latest/software/SfM/SfM/ • https://blog.mapillary.com/update/2014/12/15/sfm-preview.html • http://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html • https://github.com/snavely/bundler_sfm • https://bitbucket.org/devangel77b/python-sba • http://cdcseacave.github.io/openMVS/ Note: OpenMVS provides reconstructed surfaces, not just point clouds. Sur- faces are not required for this assignment. 2. (a) Fine tune a network for a 5 class classifier (not MNIST) Using Pytorch, select a pre-trained network to conduct a recognition experiment. Obtain your images from an existing database. 1 (b) In your report, describe the network you use (1-2 paragraphs). (c) In your report, report the accuracy, precision and recall for your experiment, for your each of the 5 classes. (d) Use drop-out regularization and batch-normalization. Explain how these work in your report. Compare the results with and without normalization. (e) In your report, show the confusion matrix for your experiments with the fine- tuned network. Describe what you see from the confusion matrix that you do not see with other metric. (f) Some useful resources may include: • http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html • https://github.com/Spandan-Madan/Pytorch_fine_tuning_Tutorial • http://cs231n.github.io/transfer-learning/ • https://gist.github.com/jcjohnson/6e41e8512c17eae5da50aebef3378a4c • https://flyyufelix.github.io/2016/10/03/fine-tuning-in-keras-part1.html 3. Augmented Reality Graduate Students (a) Do not use an augmented reality toolbox (b) Find a plane in the images (e.g. a checkerboard). For this part you may use the same sequence as for 3D reconstruction, or you may use a different sequence. (c) Associate a new coordinate frame with this plane (d) Build a wireframe model in this frame (e.g. cube) (e) Draw the wireframe model in the image (attached to the plane) (Remember you have the world coordinates and the Camera matrices, so you can render this synthetic object, in a similar way as you did in the hw assignments). Show three views of the attached cube. (f) Map an image (e.g. of yourself!) to one of the facets of this wireframe model Show three views. 4. Graduate Students Devise a prediction network or action network using one of the following two concepts learned in class. Describe your goal, your evaluation and the quality of the results. (a) LSTM or other recurrent network (b) Reinforcement Learning (c) Multimodal Deep Learning 2
