bug: spark_stream.py uses localhost instead of Docker service names for Kafka and Cassandra connections

[cschanhniem]

The spark_stream.py file has two issues that prevent it from working correctly inside the Docker Compose network:

Issue 1: Hardcoded localhost for service connections

• connect_to_kafka() uses kafka.bootstrap.servers, localhost:9092 — but the Kafka broker is a container with hostname broker. Inside Docker, localhost refers to the container itself, not the host. The correct connection string for inter-container communication would be broker:29092 (the internal listener port).
• create_cassandra_connection() uses Cluster([localhost]) — but Cassandra is a container with hostname cassandra. The correct value would be cassandra.
• create_spark_connection() uses spark.cassandra.connection.host, localhost — same issue.
• spark-streaming-kafka-integration also presumably needs an appropriate config for Docker networking.

Issue 2: Schema mismatch — dob field

The Cassandra table definition in create_table() includes a dob column:

dob TEXT,

And insert_data() accepts dob as a parameter and inserts it:

dob = kwargs.get(dob)

However, the Spark StructType schema in create_selection_df_from_kafka() does not include a dob field — only registered_date. This means any dob data sent through Kafka will silently be dropped and never written to Cassandra.

To reproduce:

1. Run docker compose up
2. The Spark streaming job connects to Kafka and Cassandra on localhost, which resolves to the Spark container itself rather than the broker/cassandra services
3. Even if connectivity is fixed, the dob field from Kafka messages will never be written to the database

Suggested fix:

• Replace localhost with Docker Compose service names (broker:29092 for Kafka, cassandra for Cassandra)
• Add StructField("dob", StringType(), False) to the schema definition in create_selection_df_from_kafka()