Date of Award

1-10-2024

Publication Type

Thesis

Degree Name

M.Sc.

Department

Computer Science

Keywords

Apache Kafka;Financial Market Prediction;Microservices Architecture;Real-Time Model Switching;Supervised Learning

Supervisor

Saeed Samet

Abstract

In today's globalized economy, financial markets are more interconnected than ever, generating vast amounts of data from thousands of sources every second. The need to accurately analyze and interpret this data is crucial for investors, analysts, and researchers alike. Traditional models for market prediction are limited by their ability to adapt to the real-time nature and 'big data' dimensions of these complex financial datasets. To address these challenges, this thesis proposes and implements a novel framework that combines Apache Kafka with a microservices framework. This framework offers a scalable, real-time solution for financial market prediction that effectively manages the 5Vs of big data: Volume, Velocity, Variety, Veracity, and Value. Apache Kafka's event-streaming capabilities serve as the backbone of the framework, enabling real-time data stream processing and distribution. The system captures data from multiple sources in real-time and feeds it to various sinks, thereby enhancing scalability and versatility. This real-time adaptation is optimized by an event-driven approach, ensuring immediate updates across all layers of the framework. One of the system's key features is real-time model switching, which dynamically selects the most appropriate machine learning model based on the market's current state, thereby maintaining prediction accuracy. Coupled with Change Data Capture (CDC) mechanisms, this ensures that the data fed into the model is always up to date. To enhance scalability while ensuring data quality, we employ a microservices architecture in which each service operates independently and can be updated without affecting other services. This provides high availability and fault tolerance, essential in a rapidly evolving financial environment. By integrating Apache Kafka and microservices into a unified framework that leverages real-time event streaming and dynamic model switching, this study presents an innovative approach to tackle the big data challenges in financial market prediction. The result is a system that not only demonstrates increased scalability but also successfully maintains prediction accuracy through its real-time model selection, making it an invaluable tool for financial market analysis.

Download

Share

COinS