School of Computer Science, University of St Andrews, CS5011, 2019-20 CS5011: A4 – Artificial Neural Networks – A ticketing routing agent Assignment: A4 – Assignment 4 Deadline: 20 December 2019 Weighting: 25% of module mark Please note that MMS is the definitive source for deadline and credit details. You are expected to have read and understood all the information in this specification and any accompanying documents at least a week before the deadline. You must contact the lecturer regarding any queries well in advance of the deadline. 1 Objective This practical aims to construct and use an artificial neural network applied in a help desk ticketing system. 2 Competencies • Design, implement, and document an artificial neural network system for multiclass clas- sification problems • Understand, use and adapt a neural network within an AI system. 3 Practical Requirements 3.1 Introduction IT Services intends to develop a new in-house Help Desk System1 to manage an increasingly large amount of requests logged through their online service desk. Requests from users are of a very diverse nature, and within IT Services responses are handled by a variety of teams (eg networking, data warehouse, web services, equipment, credentials, emergencies, etc… )2. As part of this system, you are tasked with developing a ticket routing agent in charge of the ticketing service; i.e., a service that routes requests (or tickets) to the most appropriate team for faster response. The agent should act as described below. Assume that the user has to tag the request by responding to a variety of yes/no questions about the nature of the request (e.g., is this about an incident? is this about a request? is this about logging-in? etc). Finally the user will enter the request. The agent has to log the answers to the questions and identify the most appropriate team to which it will then route the request. In addition, the agent should be able to improve its knowledge by interacting with those customers. 1https://en.wikipedia.org/wiki/Help_desk_software 2The data provided in this practical is entirely fictitious. Some ideas have been drawn from the CS ticketing ser- vice https://systems.wiki.cs.st-andrews.ac.uk/index.php/CS_ticketing_system and University IT services https://www.st-andrews.ac.uk/itsupport/ 1 3.2 The task For this practical, the ticket routing agent is to be developed using a neural system. A small set of team roles and ticket features is provided in the file tickets.csv as a starting point for this task. We assume that this dataset has been created by collecting previously routed tickets, where each line corresponds to a set of tags given by a user and its corresponding response team has been identified by IT Services. An example row in the dataset is shown below, where we notice that the request has been forwarded to the emergency team. Input – Tags Output Request Incident WebServices Login Wireless Printing IdCards Staff Students Response Team No Yes Yes Yes Yes No No No Yes Emergencies 3.3 Basic Agent The aim of this agent is to train a neural network to learn the classification patterns for the given dataset. The steps for this task are as follows: 1. Decide how to encode inputs and outputs for the given dataset. 2. Construct a training table with the chosen encoding for input/output of the dataset pro- vided. This step can be done as part of your code or with an external tool (e.g., a spread- sheet editor). 3. Construct a multilayer feedforward neural network, with one hidden layer, and appropriate activation function according to the encoding used. The network should have a fixed number of input units representing the coded tags. A fixed number of output units is to be designed to identify the response team. The single layer of hidden units is to provide hyperplanes that partition the input space into various regions for the identification of the response team. Decide how many hidden units should be included in the hidden layer. 4. Train the network using backpropagation, setting appropriate parameters such as learning rate, momentum, etc and evaluate its output. 5. Save the trained network on a file. 6. Now consider the current neural network, is the number of hidden units appropriate? Try with different numbers of hidden units, and analyse the results, then select one of the saved networks and add it in your submission. (If you attempt the intermediate agent, please use this chosen network for the next tasks). 3.4 Intermediate Agent In this part, you are required to develop a proof of concept system to show the use of the ticket routing agent. You are required to use the network constructed and saved in the previous step. Your system should be generic enough to be able to handle tags and related questions in a similar way. The agent should be able to identify the response team given the tags provided by the user, and in addition the user should be able to log new assigned tickets and learn from users preferences. The steps to be followed are described in order below. 1. Develop a basic text-based interface for the user to log a new ticket. 2. The agent should ask the user whether a tag applies to the new ticket, repeating the question for all the tags of the dataset (e.g. Incident? Yes/No, Request? Yes/No, etc) 2 3. The agent should accept yes/no answers to the questions from the user. 4. The agent should use the saved network from the basic agent to compute the output of the given input pattern. Note that the output from the network for a given input might need to be mapped to a response team. In particular, consider the case in which the user enters a pattern that does not exist in the training set. 5. The agent should communicate where this new ticket will be directed to. Please note that there is no need to ask or log the content of the ticket. 6. The system should then be ready to handle a new ticket. When all the above steps are completed, we focus on making this process more dynamic, such that the agent can identify early allocations, and learn new allocations from the users, as follows: 7. The user might get tired of having to give an answer for each of the tags. Design a method for the agent to make an early prediction. Discuss and implement this feature. 8. After a prediction has been made, the user might be still be unhappy with the choice of response team where their ticket will be directed to. If this is the case, the user may wish to suggest which response team (from a list of available ones) they think should handle the ticket. Modify the system to handle this option: the agent should log the new entry in the training table, retrain the network and be ready to accept new tickets. 3.5 Additional Advanced Learning Agent It is strongly recommended that you ensure you have completed the requirements for the ba- sic and intermediate agent before attempting any of these additional requirements. You may consider one or two (max) additional functionalities: 1. Use, compare and evaluate different training algorithms. Please ensure that a brief ex- planation of the algorithms used is given in the report. 2. The IT services might wish to introduce a new team that deals with a specific subset of tickets tagged in a certain way. They may also introduce new tags to better identify the new type of tickets. Modify the system with a new view for the IT admin to include additional response teams and tags on the fly. The network should be retrained to be ready for additional interactions. 3. Consider another agent that, given similar inputs, is able to identify the number of days needed for the ticket to be dealt with. Construct a new dataset for this scenario, train a new network, and extend your implementation such that the agent is able to inform the user of the waiting time, as well as the team likely to deal with the ticket. 4. Suggest and implement your own advanced requirements: for a significant advancement, you may apply the process developed in the basic/intermediate agent to a different sce- nario, or identify some additional components of the current problem to show how neural networks can be applied
in more complex settings of the given scenario. 4 Code and Report Specifications In this assignment, you have the choice of using Java or Python. You may only use Python if you are already familiar with this programming language. Your implementation must 3 run on the School lab Machines without the use of an IDE. For development in Java, you are required to use the library Encog3. For development in Python you are required to use the library scikit-learn4. Please do not use other libraries that are core to the assignment, but other libraries that are secondary to the objectives of the practical can be used. 4.1 Code Submission and Running Your source code should be placed in a directory called A4src/ and should include all non- standard external libraries. Your code should run using the following commands: java A4main [any param] Or python/python3 A4main [any param] where: Bas indicates the basic agent, Int indicates the intermediate agent, Adv indicates the advanced agent. If any additional parameters are to be used, ensure that these are specified in the report. Please include compiling instructions or specific environment settings in the report if required. Please note that code that does not adhere to these instructions will not be accepted. 4.2 Report You are required to submit a report describing your submission in PDF with the structure and requirements presented in the additional document CS5011_A_Reports. The core sections (Design, Implementation and Evaluation) have a limit of 1500 words in total. In the report, include a description of the structure of the network chosen with motivation on how the structure was selected and the encoding used, in addition to the general requirements. 5 Deliverables A single ZIP file must be submitted electronically via MMS by the deadline. Submissions in any other format will be rejected. Your ZIP file should contain: 1. A PDF report as discussed in Section 4.2 2. Your code as discussed in Section 4.1 3. A file containing your saved neural network from the basic agent in Section 3.3 6 Assessment Criteria Marking will follow the guidelines given in the school student handbook. The following issues will be considered: • Achieved requirements • Quality of the solution provided 3http://www.heatonresearch.com/encog/ 4http://scikit-learn.org/stable/ 4 • Examples and testing • Insights and analysis demonstrated in the report Some guideline descriptors for this assignment are given below: • For a mark of 8 or higher: the submission implements part of the basic agent, printing the current agent knowledge and is able to probe the cells in some order, adequately documented or reported. • For a mark of 11 or higher: the submission implements fully the basic agent. The code submitted is of an acceptable standard, and the report describes clearly what was done, with good style. • For a mark of 14 or higher: the submission implements fully the basic agent, and some good attempt to the intermediate agent. It contains clear and well-structured code and a clear report showing a good level of understanding of design and evaluation of learning agents. • For a mark of 17: the submission implements fully the basic and intermediate agent. It contains clear, well-designed code, together with a clear, insightful and well-written report, showing in-depth understanding of design and evaluation of logic agents. • Above 17: the submission implements fully the basic and intermediate agent, and one or two (max) advanced agent functionalities. The submission should demonstrate unusual clarity of implementation, together with an excellent and well-written report showing evi- dence of extensive understanding of design and evaluation of learning agents. 7 Policies and Guidelines Marking: See the standard mark descriptors in the School Student Handbook https://info.cs.st-andrews.ac.uk/student-handbook/learning-teaching/feedback.html# Mark_-Descriptors Lateness Penalty: The standard penalty for late submission applies (Scheme B: 1 mark per 8 hour period, or part thereof): https://info.cs.st-andrews.ac.uk/student-handbook/learning-teaching/assessment.html# latenesspenalties Good Academic Practice: The University policy on Good Academic Practice applies: https://www.st-andrews.ac.uk/students/rules/academicpractice/ Alice Toniolo ([email protected]) November 19, 2019 5
