[Bug]: The pipeline fails when trying to write to kafka

[imadmoussa1]

What happened?

Hello,

I was trying to use the WriteToKafka from apache_beam.io.kafka. Evrything in pipeline is working but when I added the WriteToKafka I have this error: raise RuntimeError( RuntimeError: Service failed to start up with error 1 After checking the documentation I found that I need to use this because the Kafka in an extremal multi cross packages python apache_beam/runners/portability/expansion_service_main.py -p 18089 --fully_qualified_name_glob "*" while running this I have a different error: signal.pause() AttributeError: module 'signal' has no attribute 'pause' can you help with this problem ? I am using beam 2.4.0

Issue Priority

Priority: 0

Issue Component

Component: io-py-kafka

[TheNeuralBit]

Hi @imadmoussa1, according to the Python docs signal.pause is unix only. Are you running on a non-unix environment? (note we should support, so if that is the issue this is still a bug)

Also could you share the full stacktrace for both errors?

CC: @johnjcasey

[imadmoussa1]

Yes , I am trying to run the pipeline on windows machine

[TheNeuralBit]

It looks like that might be an issue in the Python expansion service, which we should track down (the full stacktrace for the signal.pause error would help here). But that shouldn't actually block your specific application. In your case you'll need a Java expansion service (since KafkaIO is a Java transform). If you can share the full stacktrace for the first RuntimeError you got we can help diagnose the issue there. One possible cause: do you have Java installed?

[imadmoussa1]

Yes I am running the java expansion service, and it seems I had this problem because I was using different java version.

[johnjcasey]

Has updating the expansion service resolved the issue?

[imadmoussa1]

yes the problem was resolved, Thank you