[BUG] Flytekit gRPC error: symbol not found in flat namespace '_CFRelease'

[cosmicBboy]

Describe the bug

There seems to be an issue with grpc when calling pyflyte run

equilibrium pyflyte run <aws:eq>
Traceback (most recent call last):
  File "/Users/bwilliams/miniforge3/envs/flyte/bin/pyflyte", line 5, in <module>
    from flytekit.clis.sdk_in_container.pyflyte import main
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/flytekit/__init__.py", line 163, in <module>
    from flytekit.core.base_sql_task import SQLTask
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/flytekit/core/base_sql_task.py", line 4, in <module>
    from flytekit.core.base_task import PythonTask, TaskMetadata
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/flytekit/core/base_task.py", line 28, in <module>
    from flytekit.core.context_manager import ExecutionParameters, FlyteContext, FlyteContextManager, FlyteEntities
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/flytekit/core/context_manager.py", line 30, in <module>
    from flytekit.clients import friendly as friendly_client  # noqa
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/flytekit/clients/friendly.py", line 18, in <module>
    from flytekit.clients.raw import RawSynchronousFlyteClient as _RawSynchronousFlyteClient
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/flytekit/clients/raw.py", line 9, in <module>
    import grpc
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: dlopen(/Users/bwilliams/miniforge3/envs/flyte/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_CFRelease'

Expected behavior

This should run the script being invoked

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[cosmicBboy]

@bmwilly if there's additional context/steps here to reproduce (e.g. environment setup steps, a toy example of the workflow you're trying to run) we'd appreciate it!

[bmwilly]

@cosmicBboy

• install miniforge3
• conda create -n flyte python=3.10
• conda activate flyte
• pip install flytekit
• pip install --no-binary :all: grpcio --ignore-installed
• pyflyte run

[eapolinario]

We are aware of an issue affecting python 3.10 on M1's, unfortunately the workaround doesn't seem to work on miniforge (or possibly conda?).

Good news is that the underlying issue is being fixed in the next release of the grpc libraries. Once those go out we're going to update flytekit to test on our side ASAP.

[bmwilly]

I'm still getting the same error, following the same steps as above (grpcio 1.47.0 is being installed).

[kumare3]

@eapolinario do you know if we have a fix?

[eapolinario]

~The attached fix didn't go out with grpc 1.47.0, instead, it went out only in 1.48.0 (as can be attested in https://github.com/grpc/grpc/releases/tag/v1.48.0). Working on a PR now.~

Edit: Not sure why, but https://github.com/grpc/grpc/pull/29857 is marked as being part of the release for both 1.47.0 and 1.48.0!

@bmwilly , can you confirm that both grpcio and grpcio-status are set to 1.47.0?

[eapolinario]

@bmwilly , can you also confirm what exact steps you're following to reproduce the error?

[pingsutw]

@bmwilly I found that this error only happened in miniforge3. To work around this error, you could use Anaconda.

[eapolinario]

@pingsutw , just for completion, can you share the repro steps you used? Also, do you understand why miniforge3 didn't benefit from the new grpcio version?

[bmwilly]

@eapolinario

@bmwilly , can you also confirm what exact steps you're following to reproduce the error?

Same steps that I wrote in this comment: https://github.com/flyteorg/flyte/issues/2619#issuecomment-1157693764

@bmwilly , can you confirm that both grpcio and grpcio-status are set to 1.47.0?

confirmed:

✗ mamba list grpcio                                                                                                                                                  <aws:eq>
# packages in environment at /Users/bwilliams/miniforge3/envs/flyte:
#
# Name                    Version                   Build  Channel
grpcio                    1.47.0                   pypi_0    pypi
grpcio-status             1.47.0                   pypi_0    pypi

[pingsutw]

I installed miniforge from this repo and the latest version of flytekit. Then ran this example by using pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2

[john-wallace-dev]

I resolved this by doing:

pip uninstall grpcio conda install grpcio conda install grpcio-tools

This correctly installed the M1 versions. For info, see: https://stackoverflow.com/questions/72620996/apple-m1-symbol-not-found-cfrelease-while-running-python-app

I was using Python 3.9.12 on a Mac Studio running macOS v12.5.1

[agilebean]

...do you understand why miniforge3 didn't benefit from the new grpcio version?

@eapolinario

can you please explain why? I'm seriously interested.

[eapolinario]

@agilebean , I couldn't get to the bottom of this.

Closing this ticket as we haven't seen this particular problem with grpcio versions.

[agilebean]

@eapolinario ok sure, but can you explain at least what you meant with "benefit" in:

...do you understand why miniforge3 didn't benefit from the new grpcio version?

In what sense should miniforge have benefited?