Env vars are not read anymore starting with 4.39.0

[venatir]

How to replicate:

Here's a sample file to test

input:
  type: stdin
pipeline:
  processors:
    - log:
        level: info
        message: "DB_CONNECTION_URL: ${DB_CONNECTION_URL}"
output:
  type: stdout

1. set the test env var and running this in 4.38.0. - works
2. set the test env var and running this in 4.39.0. - complains about missing env var.

This has been tested in the 4.38.0 and 4.39.0 tags for docker - redpandadata/connect in the dockerhub repository

[mihaitodor]

Hi @venatir 👋 I tried reproducing it on my end but no luck. Seems to work OK. How are you running that Docker container? Can you please share the full command line?

[venatir]

@mihaitodor Here's a way to replicate. 4.38.0 works. 4.39.0 doesn't

ver="4.39.0";
tmp_conf_file=`mktemp`
cat << EOF >$tmp_conf_file
input:
  type: stdin
pipeline:
  processors:
    - log:
        level: info
        message: "TEST: \${TEST}"
output:
  type: stdout
EOF

docker run -it --rm -e TEST="test" -v $tmp_conf_file:/my_conf.yaml redpandadata/connect:$ver -c /my_conf.yaml

[choult]

Confirmed recreated on my side when upgrading from benthos/4.6 to redpanda connect 4.39

Example codebase/docker-compose setup here:

https://github.com/choult/benthos-pres/blob/rp-connect-2992/docker-compose.yml

Switching to 4.38 for all containers works as expected

[mihaitodor]

Small update on this issue: The now deprecated flag -c is indeed ignoring environment variables and we'll have a fix out ASAP. In the meantime, if you wish to use v4.39.0, you should be able to do rpk connect run ... or redpanda-connect run ....

[peczenyj]

$ docker run -it --rm -e TEST="test" -v ./foo:/c redpandadata/connect:4.39.0 run /c/config.yaml
INFO Running main config from specified file       @service=redpanda-connect benthos_version=v4.39.0 path=/c/config.yaml
INFO Listening for HTTP requests at: http://0.0.0.0:4195  @service=redpanda-connect
INFO Input type stdin is now active                @service=redpanda-connect label="" path=root.input
INFO Launching a Redpanda Connect instance, use CTRL+C to close  @service=redpanda-connect
INFO Output type stdout is now active              @service=redpanda-connect label="" path=root.output
sqdsqd
INFO TEST: test                                    @service=redpanda-connect custom_source=true label="" path=root.pipeline.processors.0
sqdsqd
^CINFO Received SIGINT, the service is closing       @service=redpanda-connect

it seems a problem when you don't use the command run

BTW I can't get the -c flag to work on any version

[venatir]

Thanks @mihaitodor. I'll use run. It's my fault for not following the deprecation announcements, but I don't know anyone who does this for every product. I wonder if giving a warning that -c is deprecated can help? cc: @mihaitodor

@peczenyj , the -c flag, while deprecated still works. See my example above https://github.com/redpanda-data/connect/issues/2992#issuecomment-2466828353

[mihaitodor]

I wonder if giving a warning that -c is deprecated can help? cc: @mihaitodor

It really wasn't meant to break and the intent is to keep it around until at least another major version bump given how many people used it before run was introduced. Eventually, it will be covered in one of the migration guides.

I think @peczenyj got tripped up by the template docs which had a bug that was fixed here. If there's some other corner case, please let us know.

[harjis]

Hi!

I encountered a similar problem related to rpk connect lint and rpk connect test. With version 4.38.0 the following example works as expected. Starting from 4.39.0 the lint command complains about missing environment variables:

/config/connect.yaml(1,1) required environment variables were not set: [KAFKA_ADDRESSES]

printf "KAFKA_ADDRESSES=test" > .env
cat <<EOF > connect.yaml
input:
  label: "rpctest"
  kafka:
    addresses: ["\${KAFKA_ADDRESSES}"]
    topics: ["test"]
    target_version: 2.1.0
    consumer_group: "test"
    checkpoint_limit: 1024
    auto_replay_nacks: true
    start_from_oldest: true

pipeline:
  processors:
    - sleep:
        duration: 500ms
    - mapping: |
        root.name = this.test.uppercase()

output:
  stdout: {}
EOF

docker run \
--rm \
-it \
-v "$(pwd)/:/config/" \
docker.redpanda.com/redpandadata/connect:latest lint --env-file /config/.env /config/connect.yaml

[artemklevtsov]

Any updates? Still actual with rpk connect lint command.

[harjis]

We just updated to 4.57.1 and it looks like the issue no longer exists.