Page 1 OENG1116 – Summative Assessment 2 Individual Project Portfolio 1 Assignment context This project for OENG1116 is based on your work developing and selecting the most appropriate model for an engineering system, interpreting the simulation results of that model and comparing the results of different model architectures. This assessment task builds on project by using lecture, tutorial classes, activities undertaken during the semester till week 12 (Submission date is Friday, 5 June 2020, Time: 23.59 ). The following documents can be used for completion of this assessment • Lecture notes • Tutorial notes • Matlab toolbox (as indicated below) • Book chapters: (i)H. Khayyam, G. Golkarnarenji, R.N. Jazar, “Limited Data Modelling Approaches for Engineering Applications”, In Nonlinear Approaches in Engineering Applications; Jazar, R.N., Ed.; International Publication Springer: Cham, Switzerland, (2018) (ii) B Crawford, H Khayyam, AS Milani, RN Jazar, “Big Data Modeling Approaches for Engineering Applications” Nonlinear Approaches in Engineering Applications, 307-365 (2019) 2 Assignment overview. This individual assessment requires the student to present a project report, based on the activities proposed. The overall aim of this assignment report is to demonstrate the technical and non-technical learning outcomes of the unit by the student. This assignment has an overall weight of 35 % of the course. Page 2 3 Learning Outcomes A summary of the Course Learning Outcomes (CLOs) which will be assessed in this task are provided in table 1. Table 1: Summary of CLOs assessed in Assessment 2. This task assesses your Course Learning Outcomes (CLOs) CLOs 1- Analysis Ability to model non-deterministic (heuristic) systems and differentiate between nonlinear and linear models. Ability to numerically simulate linear and non-linear deterministic systems. Ability to estimate and validate a model based upon input and output data. Ability to create a model prediction based upon new input and validate the output data. Ability to understand and apply advanced theory of engineering fundamentals and specialist bodies of knowledge in the selected discipline area to predict the effect of engineering activities. Ability to apply underpinning natural, physical and engineering sciences, mathematics, statistics, computer and information sciences to engineering applications. 2- Research Ability to plan and execute a substantial research-based assessment tasks, with creativity and initiative in new situations in professional practice and with a high level of personal autonomy and accountability. Awareness of knowledge development and research directions within the engineering discipline. Ability to develop creative and innovative solutions to (heuristic) engineering challenges. Ability to assess, acquire and apply the competencies and resources appropriate to engineering activities. Ability to demonstrate professional use and management of information. Ability to clearly acknowledge your own contributions and the contributions from others and distinguish contributions you may have made as a result of discussions or collaboration with other people. Page 3 4 Assignment details and requirements Your report is related to the development of three different models for the given experimental data shown in Tables 2 and 3. The aim of the report is to justify all the decisions that you made to develop the different models, showing your skills to analyse non-deterministic (heuristic) systems. Table 2: Experimental data (Training) No. Inputs Outputs Input1 Input2 Input3 Output1 1 227 20 1 1.2446 2 227 20 4 1.2438 3 227 25 2 1.25 4 227 25 3 1.2417 5 227 30 3 1.2359 6 227 35 4 1.2244 7 230 20 2 1.2574 8 230 25 1 1.2417 9 230 25 3 1.2464 10 230 30 4 1.2341 11 230 35 1 1.2335 12 230 35 3 1.2317 13 233 20 3 1.257 14 233 25 1 1.2611 15 233 20 4 1.2601 16 233 25 2 1.2457 17 233 25 3 1.2465 18 233 25 4 1.2565 19 233 30 1 1.2429 20 233 30 3 1.2421 21 233 35 2 1.2363 22 236 20 1 1.2707 23 236 20 4 1.271 24 236 25 3 1.263 25 236 30 2 1.2547 26 236 35 1 1.2504 27 236 35 4 1.2474 Page 4 Table 3: Experimental data (Testing) No. Inputs Outputs Input1 Input2 Input3 Output1 28 227 35 1 1.2339 29 230 20 4 1.2588 30 233 35 3 1.2372 The output required for this assessment task is based on the 4 key areas as defined in Table 4, which provides full descriptions of the functionalities required for each model. In addition, the approximate length of the content has been specified (though not fixed) including the weight of each area (Considering the total 35 % of the assignment). Table 4: Description of key tasks required for report. Item Task Name (output) Description ULO(s) Approx. Length Weight 1 Modelling using Artificial Neural Network Read the collected data from Table 2. Perform data pre- processing if required. Develop a predictive model of input-output data sets based on Artificial Neural Networks (ANN) in MATLAB. Split the data into relevant ratios for training, validation and testing, providing justification on the ratios chosen. (i) Describe the network architecture, training procedure and every step carried out to improve the model. (ii) Define the fitting neural network through changing the number of hidden neurons (for example 5-25). (iii) If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different training algorithms (Levenberg–Marquardt, Conjugate Gradient, Quasi- Newton Algorithms, Bayesian Regularization, Gradient Decent). (iv) Evaluate the accuracy (error) of the developed model by using the data provided in Table 3. CLO1 ~2 to 3 page 10% 2 Modelling using Support Vector Machine Read the collected data from Table 2. Perform data pre-processing if required. Develop a predictive model of input-output data sets based on different Support Vector Machine (SVM) in MATLAB. (i) Describe the training procedure and every step carried out to improve the model. (ii) If applicable, find a solution to achieve the best fit in terms of model performance (use different SVM kernels Linear, Gaussian and Polynomial). (iii) Evaluate the accuracy (error) of the developed model by using the data provided in Table 3. CLO1 ~ 2-3 pages 10% 3 Modelling using Linear Non- Linear Regression Read the collected data from Table 2. Perform data pre-processing if required. Develop a predictive model of input-output data sets based on different Non-Linear Regression (NLR) in MATLAB. (i) Describe the training procedure and every step carried out to improve the model. (ii) If applicable, find a solution to achieve the best fit in terms of model performance (use different NLR models: Polynomial, Exponential, Power and combination). (iii) Evaluate the accuracy (error) of the model using the test dataset on Table 3. CLO1 ~ 2-3 pages 10% Page 5 4 Find RMSE, MSE and R Find MSE, RMSE and R for the models (1-3). Note: Use equations 1-3 to calculate RMSE, MSE and R: CLO1 ~ 1 page 2.5% 5 Compare Compare the three methods used. Discuss on the advantages/disadvantages of the different models in this application CLO1 ~ 1 page 2.5% Notes on structure and formatting: To make this task as simple as possible, the structure of the report should be based exactly on the tasks defined above. That is, you should have 7 sections (5 tasks and 2 Appendixes) in your report which contain the headings defined by the 5 Tasks Name in Table 4, Appendix A: Different ANN, SVM and NLR Methods Results and Appendix B: ANN, SVM and NLR Matlab Codes. There is no need for additional introduction and conclusion sections, or formatting such as Table of Contents, List of Figures, etc. However, you will still be assessed on the quality of the report and the clarity of the communication, via the assessment of CLO
1- 3 and throughout the report. 5 Marking criteria The assessment criteria are based on how well you have completed the 4 tasks defined in Table 4. • You will be scored for each of the key tasks defined in Table 4. The marks will then be weighted according to the marking rubric shown in Table 5. • To achieve the maximum score for each task, you will have clearly covered the information provided in the description, demonstrating that you have met the relevant Course Learning outcomes defined for each of the tasks in Table 4. Page 6 Table 5. Project Assessment Rubric Topic (Mark) Unsatisfactory (0<30%) Satisfactory (30%<70) Good (70%<80%) Outstanding (>80%) Task 1 Errors in reading the input/output data. No attempt on pre- processing data or no justification. Develop a predictive model of input- output data sets based on Artificial Neural Networks (ANN) in MATLAB. The ratios chosen for training, validation and testing are not justified. There is little or no justification on the parameters chosen to build the ANN (i)Brief descriptions of the network architecture, training procedure and every step carried out to improve the model. (ii) Lack of mathematical analysis of the results (iii) Code shows errors, does not work properly or there is no code shown for the ANN Successfully reading the collected data from Table 3, with proper justification. Perform data pre- processing if required. Develop a predictive model of input- output data sets based on Artificial Neural Networks (ANN) in MATLAB. Split the data into Different ratios for training, validation, and testing, discussing on the effects of different ratios on the ANN performance. (i)Describe the network architecture, training procedure and every step carried out to improve the model, showing a clear understanding of the effects of changing the different parameters considered (ii) Define the fitting neural network through changing the number of hidden neurons (for example 5-25). (iii)If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different training algorithms (more than one) (Levenberg–Marquardt, Conjugate Gradient, Quasi-Newton Algorithms, Bayesian Regularization, Gradient Decent). Successfully reading the collected data from Table 3, with proper justification. Perform data pre- processing if required. Develop a predictive model of input- output data sets based on Artificial Neural Networks (ANN) in MATLAB. Split the data into Different ratios for training, validation, and testing, discussing on the effects of different ratios on the ANN performance. (i)Describe the network architecture, training procedure and every step carried out to improve the model, showing a clear understanding of the effects of changing the different parameters considered (ii) Define the fitting neural network through changing the number of hidden neurons (for example 5-25). (iii)If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different training algorithms (more than one) (Levenberg–Marquardt, Conjugate Gradient, Quasi-Newton Algorithms, Bayesian Regularization, Gradient Decent). (iv) Mathematical analysis of the different ANNs performance provided to show less error. Successfully reading the collected data from Table 3, with proper justification. Perform data pre- processing if required. Develop a predictive model of input- output data sets based on Artificial Neural Networks (ANN) in MATLAB. Split the data into Different ratios for training, validation, and testing, discussing on the effects of different ratios on the ANN performance. (i)Describe the network architecture, training procedure and every step carried out to improve the model, showing a clear understanding of the effects of changing the different parameters considered (ii) Define the fitting neural network through changing the number of hidden neurons (for example 5-25). (iii)If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different training algorithms (more than one) (Levenberg–Marquardt, Conjugate Gradient, Quasi-Newton Algorithms, Bayesian Regularization, Gradient Decent). (iv) Mathematical analysis of the different ANNs performance provided to show less error. (v) Code compiles and work properly, built with a logical structure and including enough comments to help understanding the ANN developed. Task 2 Errors in reading the input/output data. No attempt on pre- processing data or no justification. Develop a predictive model of input- output data sets based on Support Vector Machine (SVM) in MATLAB. Input/output data is not used correctly for a SVM (i)Brief descriptions of the network architecture and every step carried out to improve the model. (ii) Lack of mathematical analysis of the results (iii) Code shows errors, does not work properly or there is no code shown for the SVM Successfully reading the input/output data. Perform data pre- processing if required. Develop a predictive model of input-output data sets based on Support Vector Machine (SVM) in MATLAB (i) Detailed description of the model architecture, and the steps carried out to improve the model, with proper justification of every parameter considered. (ii)If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different training algorithms (more than one) ((Linear, Gaussian and Polynomial). Successfully reading the input/output data. Perform data pre- processing if required. Develop a predictive model of input-output data sets based on Support Vector Machine (SVM) in MATLAB (i) Detailed description of the model architecture, and the steps carried out to improve the model, with proper justification of every parameter considered. (ii)If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different training algorithms (more than one) ((Linear, Gaussian and Polynomial). (iii) Evaluation of the accuracy of the different SVR models. Mathematical analysis of the SVRs performance provided to show less error. Successfully reading the input/output data. Perform data pre- processing if required. Develop a predictive model of input- output data sets based on Support Vector Machine (SVM) in MATLAB (i) Detailed description of the model architecture, and the steps carried out to improve the model, with proper justification of every parameter considered. (ii)If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different training algorithms (more than one) ((Linear, Gaussian and Polynomial). (iii) Evaluation of the accuracy of the different SVR models. Mathematical analysis of the SVRs performance provided to show less error. iv) Code compiles and work properly, built with a logical structure and including enough comments to help understanding the SVR developed Task3 Errors in reading the input/output data. No attempt on pre- processing data or no justification. Develop a Successfully reading the input/output data. Perform data pre- processing if required. Develop a predictive model of input-output Successfully reading the input/output data. Perform data pre- processing if required. Develop a predictive model of input-output data sets based on Successfully reading the input/output data. Perform data pre- processing if required. Develop a predictive model of input- output data sets based on Non-Linear Regression (NLR) in Page 7 predictive model of input-output data sets based on Non-Linear Regression (NLR) in MATLAB. Input/output data is not used correctly for a NLR (i)Brief descriptions of the implementation of the mathematical model and every step carried out to improve the model. (ii) Lack of mathematical analysis of the results (iii) Code shows errors, does not work properly or there
is no code shown for the NLR data sets based on Non-Linear Regression (NLR) in MATLAB. (i) Detailed description of the model architecture, and the steps carried out to improve the model, with proper justification of every parameter considered. (ii) If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different NLR approaches (more than one) (Polynomial, Exponential, Power and combination). Mathematical analysis of the different NLR performances provided (to show the error). Non-Linear Regression (NLR) in MATLAB. (i) Detailed description of the model architecture, and the steps carried out to improve the model, with proper justification of every parameter considered. (ii) If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different NLR approaches (more than one) (Polynomial, Exponential, Power and combination). Mathematical analysis of the different NLR performances provided (to show the error). (iii) Evaluation of the accuracy of the different NLR models. Mathematical analysis of the NLRs performance provided MATLAB. (i) Detailed description of the model architecture, and the steps carried out to improve the model, with proper justification of every parameter considered. (ii) If applicable, find a solution to achieve the best fit in terms of model performance by choosing and controlling different NLR approaches (more than one) (Polynomial, Exponential, Power and combination). Mathematical analysis of the different NLR performances provided (to show the error). (iii) Evaluation of the accuracy of the different NLR models. Mathematical analysis of the NLRs performance provided (iv) Code compiles and work properly, built with a logical structure and including enough comments to help understanding the NLRs developed. Task 4 No mathematical analysis, wrong RMSE, MSE and R values Only one of the parameters (RMSE, MSE or R) is correct Two of the parameters (RMSE, MSE or R) are correct Correct calculation of RMSE and MSE and R Task 5 No discussion of the findings or poor discussion with lots of fundamental mistakes. Description of the findings with no comparison/analysis between methods. The description does not contain fundamental mistakes. Correct description of the findings and comparison/analysis between the three different models. Minor mistakes in the analysis. Correct description of the findings and comparison/analysis between the three different models. Discussion clearly shows a strong understanding of the models built. Page 8 6 Submission requirements It is required to submit two files: a Report PDF file and a Matlab file on Canvas as follows: 1. Report PDF file: Prepare a report PDF file including the 5 tasks and Matlab code. PDF file are strongly recommended. Other file formats will not be accepted. (Use the name_surname_ID and extension for example: hamid_khayyam_s33784.PDF 2. Matlab file: Prepare a Matlab file including the 3 Matlab code with format m file. Other file formats will not be accepted. a. If you created a single Matlab file with different sections, name the file name_surname_ID.m b. If you created a Matlab file for every different section name them as follow i. name_surname_ID_ANN.m ii. name_surname_ID_SVM.m iii. name_surname_ID_NLR.m Then zip all the Matlab files of them in a unique file with the named name_surname_ID.zip 3. To access Assignment, please, LOG IN into your RMIT account and then access the Canvas: 4. Upload both the PDF file and the Matlab code. 7 Penalties and bonus marks Grace period for submission is 30 minutes. There are no bonus points associated with this assessment task
