Kafka Athena Streaming Project

Project Description:

In this project I have implemented an End-To-End Data Engineering Project simulating real time data using stock market data using Kafka. The following technologies were used:

• Programming Language - Python

• Apache Kafka

• Cloud - AWS

1. EC2

2. S3

3. Glue Crawler

4. Glue Catalog

5. Athena

GitHub Link for the code

Architecture

EC2 and Kafka Setup:

• Choose launch instance from AWS EC2 page.

• Select a free tier eligible Amazon Machine Image. I have chosen Amazon Linux for this project.

• Choose a free tier Instance type from the instance type drop down.

• Select a existing key pair if available or create a new one for this EC2 instance.

• Allow SSH traffic from your IP or all IP(Not recommended).

• Choose the required storage. Upto 30 GB is free, however this much space is not required for this project.

• Launch the instance.

• We can connect to the instance using the key pair using putty or we can directly connect from the EC2 console.

• Next step is to setup Kafka in the EC2 instance.

• Please follow the steps highlighted below to set up kafka.

Python code to simulate streaming stock data:

• I have used a sample stock data and used the following code to simulate a streaming data.

• I have created a producer object and used a while to continuously send data to the consumer.

Python code get the data from consumer and upload to an S3 bucket:

• I have set up an s3 bucket.

• Then I created a consumer object and dumped the data as JSON to the previously created bucket.

• The data has started to be loaded to the s3 bucket.

Setting up a glue crawler to crawl the s3 bucket:

• Navigate to the AWS Glue Console.

• Navigate to the crawler page and click on create crawler.

• Go Next and check the Not Yet option in the is your data already mapped to glue tables question.

• Select Add data source and choose s3. Click on Browse S3 and choose the bucket where the data is streamed.

• In the next window choose an IAM role if you already have one or create a new IAM role for glue and give s3 access.

• In the next page create a new database or choose an existing database.

• Run the crawler.

Exploring the data in Athena:

• Navigate to the Athena console.

• In the query editor you would find the database you created in the crawler step.

• It will have a table in the name of the s3 bucket.

• The data will be constantly updated as more data is dumped in the s3 bucket.

• The data being constantly updated is showcased in the screenshots.