Kafka improvements

[de-sh]

1. Integration Tests:
   • [ ] Add tests using test containers to simulate Kafka.
2. Documentation:
   • [ ] Getting started with kafka ingestor on parseable
   • [ ] System design and decisions made
   • [ ] Configuration guide
3. Script Migration: Move the Kafka log stream generator script to the Quest repository for better maintainability and integration with existing test suites.
4. Test Cases: Add test cases to Quest to validate Kafka message ingestion and ensure the consistency of message counts etc.

Includes recommendations made by @hippalus in #1047

[hippalus]

@de-sh, items 3 and 4 have been completed. Additionally, @nitisht recommended moving the Python script located here to the Quest repository. This script can be enhanced to include test cases that verify the ingestion of Kafka messages and ensure that the number of events matches the expected count.

Completed Items:

3. Docker Improvements:

   • Added Docker Compose files for testing Kafka in distributed and standalone environments:
     • Distributed Test Environment
     • Standalone Test Environment
     • Dockerfile.kafka

4. Metrics Collector:

   • Completed the KafkaMetricsCollector implementation:
     • Metrics Implementation

IMHO: We can add the following items to this scope

• Script Migration: Move the Kafka log stream generator script to the Quest repository for better maintainability and integration with existing test suites.
• Test Cases: Add test cases to Quest to validate Kafka message ingestion and ensure the consistency of message counts etc.