The Australian National University Research School of Computer Science, CECS COMP3430 – Data Wrangling – 2019 Record linkage project Due 11:55pm Sunday 20 October 2019 Worth 20% of the final grade for COMP3430 Draft – Last update September 11, 2019 Overview and Objectives For this project you will be having another look at the record linkage program that you developed in the lab sessions. Specifically, we provide you with two new data sets and ask you to work with the programs we have developed in the labs, and report on your findings. As with the previous assessments, the emphasis is on your understanding, descriptions, and justification as much as the raw (numerical) record linkage evaluation results that you are able to achieve. Important • Submit one zip archive file, named uNNNNNNN record linkage project.zip, where uNNNNNNN is your ANU ID. For example if your ANU ID is u1234567 you should submit the file u1234567 record linkage project.zip. Only use underscores and not spaces, and only lower-case letters in your file name (as this will greatly help our marking efforts – thanks). You receive a -1 mark penalty if you do not follow this naming convention. • Make sure that your student ID is included on the first page of your submitted report. You receive a -1 mark penalty if you do not include your student ID. • Do NOT include your name anywhere in your submission. All marking will be done anonymously. You receive a -1 mark penalty if you do include your name. • The zip file must contain: 1. Your report, a .pdf document named uNNNNNNN record linkage project report.pdf 2. Your output file for the best linkage results you were able to obtain (see task 3 below), a .csv file named uNNNNNNN record linkage project result.csv • The allowed total maximum length of your report is four (4) A4 pages (single pages, not 4 double pages!) and around 1,500 words. We expect you to use at least 12 point font size with a standard font (such as Times New Roman or Liberation Serif ) for all text in your submitted report. We encourage you to use larger font size or bold font for titles, section headers, etc. Include the total word-count of your report on the first page of your report. The 4 page maximum length does include any figures, tables, references and appendices. • Your submitted report does not need to have a cover page. • Word documents or any other formats besides PDF are not accepted and will not be marked. • Hand-written submissions are not accepted and will not be marked. • Make sure you submit the final version of your project before the submission deadline. Submission Submission will be done using Wattle. Click on the link COMP3430 record linkage project submission (to be made available) in week 11 to upload your ZIP file. You may submit as many draft versions of your project as you wish. However, you must make sure you submit a final version before the submission deadline. We will mark the final version present at the due date. Note that Wattle does not allow us to access earlier submitted versions of your project, therefore check carefully what you submit as the final version! We cannot accept submissions via email. Penalties The following will attract penalties: -1 mark if you do not follow the file naming convention discussed above. -1 mark if you do not include your student ID on the first page of your submitted report. -1 mark if you do include your name in your submitted report. -1 mark for every page over the maximum 4 page limit (so a 6 page report will attract a -2 penalty). -1 mark if you use a font size smaller than 12 points, or a difficult to read font type. Deadlines, Extensions and Late Submissions The record linkage project is due 11:55pm, Sunday 20 October 2019. Students will only be granted an extension on the submission deadline in extenuating circumstances, as de- fined by ANU policy (http://www.anu.edu.au/students/program-administration/assessments-exams/deferred-examinations). If you think you have grounds for an extension, you must notify the course convener as soon as possible and provide written evidence in support of your case (such as a medical certificate). The course convener will then decide whether to grant an extension and inform you as soon as practical. In accordance with the CECS and ANU late submission policy, no late submissions will be accepted, except where an extension has been approved by the course convener. Plagiarism No group work is permitted for this project. We do encourage you to discuss your work, but we expect you to do the project work by yourself. If you are unsure about what constitutes plagiarism, make sure you carefully read the ANU Academic Honesty Policy (http://academichonesty.anu.edu.au/). If you do include ideas or material from other sources, then you clearly have to make attribution by providing a reference to the material or source in your submitted project report. We do not require a specific referencing format, as long as you are consistent and your references allow us to find the source, should we need to while we are marking your report. Marking This project will be marked out of 20, and it will count for 20% of your final course mark. Note that not all project tasks are equally difficult. For some of the tasks there is no single right or wrong answer. Marks will be awarded based on your reasoning and the justification of your decisions and explanations, as well as clarity and correctness of writing. We will endeavour to release your marks and feedback within two teaching weeks after the submission deadline. If you feel we have made an error in marking, you have two weeks following the release of marks to raise any issues with the course convener, after which time your mark will be considered final. If you request that we re-mark your project, we will re-mark the entire project and your mark may go up or down as a result. Project Structure This project consists of four (4) tasks as described below which can be worth different numbers of marks. Make sure you answer all aspects of each task. If you have any questions on the project please post them on Wattle – however do not post any partial solutions, program codes, equations, calculations, URLs, etc. or any hints on how to solve any of the project tasks. Project Tasks For this project, we provide you with the following two new data sets, dataset A.csv and dataset B.csv, as well as a truth data set true matches.csv, available for download from Wattle in week 8. The tasks for this project are similar to what you had to do in lab 7 in week 9. You are required to run your record linkage program (including any modifications you have made to this program) on the two data sets provided, and write a report which addresses the following questions: 1. Blocking (6 marks): • How does blocking affect your results? Specifically, describe your choice of blocking method and choice of blocking keys. Discuss which attributes and/or attribute combination(s) in the given data sets were useful as blocking keys and which were not, and why. • If there is a trade-off between performance (reduction ratio, pairs completeness and pairs quality) and the quality of the final record linkage results, where do you think the optimal balance is, and why? • Do you think this trade-off would change on different data sets with different levels and characteristics of data quality? If so, how and why? 2. Comparison and Classification (6 marks): • How do different comparison techniques affect linkage results? Discuss and justify how you selected appropriate comparison functions for different attributes, and why these selected functions are suitable while others were not. • How do different classification techniques using different parameter settings affect linkage quality? Discuss and justify how you selected an appropriate classification function to obtain high linkage quality. • As discussed in the lectures in week 8, for suitable linkage quality measures, describe how the final record linkage quality changes with the choice of parameters and techniques? • Is the record linkage quality particul
arly sensitive to certain parameters or choice of comparison or classification techniques? If so, why is this the case? • Provide the numerical linkage evaluation results for other (not optimal, see below) parameter settings that you have used (you only have to provide the output file for your best obtained linkage results – see next task). Ideally you include tables or plots to show linkage quality results for different parameter settings. 3. Optimal Settings (4 marks): • What is the best linkage quality result you are able to achieve, both in the blocking and the classification steps? Why do you think this combination of parameters and techniques works well? • Are the results good for all evaluation measures discussed in the lectures in week 8, or only for some? If the results are good only for some measures, why do you think the results are not good for other measures? In addition to answering this task in your report, you must also submit the output file which contains the linked and classified matching record pairs (as a CSV file) for the best linkage result you were able to obtain. Use the Python program saveLinkResult.py which we use in lab 7 to write linkage output into a file. Your submitted output file must exactly follow this CSV file format! We will use a program to check linkage quality using this file to validate what you write in your report. If our program does not work with your submitted file because it does not follow the required file structure you will loose marks. 4. Data Quality (4 marks): • How dirty are these new data sets compared to all the data sets you have worked with in labs 3 to 7? Describe your impression after having conducted the linkage project. • How can you determine this? Describe the methodology you used to assess the data quality of the data sets we provided for this project (such as any calculations you used, or how you determined the data quality using data exploration and profiling). Visualisations: You should use appropriate data visualisations such as tables, plots, etc. for your descriptions to the above tasks. Marks will be awarded for good visualisations and appropriate tables. Assume you are presenting your record linkage project to an audience without a strong technical background, so make sure you adequately explain any visualisations you use (i.e. describe what tables and figures show and interpret the content of the obtained results).
