RealtimeStreamingEngineering
RealtimeStreamingEngineering copied to clipboard
airscholar
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenAI LLM, Kafka and Elasticsearch. It covers each stage from data...
1.streaming-socket.py ```python import json import socket import time import pandas as pd def send_data_over_socket(file_path,host='127.0.0.1',port=9999,chunk_size=2): s = socket.socket(socket.AF_INET,socket.SOCK_STREAM) s.bind((host,port)) s.listen(1) print(f"Listing For connection {host}:{port}") conn,addr = s.accept() print(f"Connection From {addr}") last_sent_index...
Metadata
Owner
Metadata
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenAI LLM, Kafka and Elasticsearch. It covers each stage from data...