CS6350.001 – Big data Management Analytics and Management Fall 2019 HOMEWORK 3: Window-based Stream Data Analytics with SPARK and Kafka (Submission Deadline: Nov 8th, 2019) The purpose of Homework 3 is to implement a system for real-time news stream classification, trend analysis and viasualization. For such implementation, you will work with Apache Spark and Apache Kafka, implement some of the state-of-the-art classifiers like SVM by using MLlib, and utilize Spark Streaming. You are asked to extract features and build a classifier over a stream of news articles. For the task of gathering real-time news articles, you will use stream tools provided by Guardian API. This part of the implementation is provided for you in the file stream_producer.py. Note that, for using this API, you are required to register/sign up for an API key in this link: https://open-platform.theguardian.com/access/ Stream_producer.py generates the Kafka streaming data from Guardian API every 1 second. The topic created has been named ‘guardian2’. When running this from your command prompt window, you should be able to see the news getting printed on the screen in the following format: Label index (news category) || headline + bodyText Each news article will be tagged with one category, such as Australia news, US news, Football, World news, Sport, Television & radio, Environment, Science, Media, News, Opinion, Politics, Business, UK news, Society, Life and style, Inequality, Art and design, Books, Stage, Film, Music, Global, Food, Culture, Community, Money, Technology, Travel, From the Observer, Fashion, Crosswords, Law, etc. Stream_producer.py script can capture a category for each news article and assign a label index for that category. Example of Stream_producer.py usage: $ python3 stream_producer.py API-key fromDate toDate For instance: python3 stream_producer.py API-key 2018-11-3 2018-12-24 To simulate real-time streaming and processing you will collect as much data as you can by streamming through kafka (by using ‘Stream_producer.py’). The classification model will be built offline with articles from such collected data. Following, in Spark context, you will need to create a Pipeline model with Tokenizer, Stopword remover, Labelizer, TF-IDF vectorizer, and your Classifier. You are allowed to test many different classification techniques and later present performance results (i.e., Accuracy, Recall, etc.) for the chosen classification technique. Finally, you will visualize the results over sliding Windows using Kibana. You need to use Spark Streaming for performing classification and visualize the output using ElasticSearch and Kibana. For example, a frequency count task can be performed using ElasticSearch whose results can be visualized in a tag cloud. As the stream goes on, the framework processes news articles continuously and periodically updates the tag cloud. The following figure depicts the framework structure. Note that, for this homework, you are required to set up standalone Spark and Kafka on your own system. Besides, you are allowed to use NLP processing tools, python Scikit-learn and MLlib in Spark packages. Additional Notes: (i) When applying/testing your classification model, you are required to use different window streamming (‘fromDate’ and ‘toDate’) than the one used for training the classification model. (ii) The label (categories of the news) must be used only for the training process (always as dependent variable). Furthermore, during testing, the labels can be utilized to report accuracy, recall, and precision for each batch. Submission: You have to upload your submission via e-learning before the due date. Please upload the following to eLearning: 1. Source files 2. Readme file 3. A short report showing the performance of the classifiers you tested and any further detail of your implementation.
