feat: Feature/kafkaoptions kwargs

[zerafachris]

What this PR does / why we need it:

This is an attempt to introduce additional kafka settings to the Kafka stream as raised by https://github.com/feast-dev/feast/issues/4894

Which issue(s) this PR fixes:

Not applicable.

Misc

Unfortunately, this is a partial solution and am looking for support on this.

I have made most of the necessary changes but unit tests are failing due to:

===================================================================== short test summary info ======================================================================
FAILED sdk/python/tests/unit/test_stream_feature_view.py::test_stream_feature_view_serialization - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/test_stream_feature_view.py::test_stream_feature_view_udfs - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/test_stream_feature_view.py::test_stream_feature_view_initialization_with_optional_fields_omitted - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_feature_view[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_feature_view_udf[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_source[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/test_data_sources.py::test_proto_conversion - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_source_from_repo - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_feature_view_with_inline_stream_source[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_feature_view_with_inline_stream_source_from_repo - AssertionError: stdout: b'No project found in the repository. Using project name test5578hi0q50 defined in feature_store.yaml\nApplying changes for project tes...
=============================================== 10 failed, 519 passed, 2 skipped, 558 warnings in 234.66s (0:03:54) ================================================

Any support is greatly appreciated

[zerafachris]

With commit "1ef5d0afb35e3dd2f9f8aaee6da0a28f88982a82" I was able to add the proto definition for kafka_settings as a StreamFormat. Not sure if this is correct of not for dict[str, str]

[franciscojavierarceo]

I'll try to pull this down on Friday, sorry traveling for work.

[franciscojavierarceo]

Hey @zerafachris mind taking a look? did you still need this? I was planning on cutting a release for it and wanted to make sure we get this in if needed.

[zerafachris]

Hi @franciscojavierarceo, I resolved the conflicts. Ideally, yes, I would be keen to use this.

Unfortunately, I am still unable to fix the unit tests. The errors are:

===================================================================== short test summary info ======================================================================
FAILED sdk/python/tests/unit/test_stream_feature_view.py::test_stream_feature_view_serialization - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/test_stream_feature_view.py::test_stream_feature_view_udfs - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/test_stream_feature_view.py::test_stream_feature_view_initialization_with_optional_fields_omitted - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_feature_view[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_feature_view_udf[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_source[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_stream_source_from_repo - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/test_data_sources.py::test_proto_conversion - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/online_store/test_online_retrieval.py::test_get_online_features_milvus - feast.errors.FeastModuleImportError: Could not import module 'feast.infra.online_stores.milvus_online_store.milvus' while attempting to load class 'MilvusOnlin...
FAILED sdk/python/tests/unit/online_store/test_online_retrieval.py::test_milvus_lite_get_online_documents_v2 - feast.errors.FeastModuleImportError: Could not import module 'feast.infra.online_stores.milvus_online_store.milvus' while attempting to load class 'MilvusOnlin...
FAILED sdk/python/tests/unit/online_store/test_online_retrieval.py::test_milvus_native_from_feast_data - ModuleNotFoundError: No module named 'pymilvus'
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_feature_view_with_inline_stream_source[feature_store_with_local_registry] - typeguard.TypeCheckError: argument "data_source" (feast.core.DataFormat_pb2.StreamFormat) is not an instance of feast.core.DataSource_pb2.DataSource
FAILED sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py::test_apply_feature_view_with_inline_stream_source_from_repo - AssertionError: stdout: b'No project found in the repository. Using project name testht2m82xvdq defined in feature_store.yaml\nApplying changes for project tes...
=============================================== 13 failed, 520 passed, 3 skipped, 556 warnings in 205.98s (0:03:25) ================================================

[franciscojavierarceo]

I'll take a look at this today 👍

[ntkathole]

@zerafachris can you please sign the commit so that DCO check pass on PR and also rebase it ? I can help with the unit tests failures.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.