Upgrade internal azureml-dataprep azure-identity dependency to 1.8.0 or greater

[jackphillipsjmu]

Is your feature request related to a problem? Please describe. Using the Python azureml-dataprep library (versions >4.1) it has a dependency on azure-identity 1.7.0 which causes issues if you would like to use an upgraded version of the azure-identity library. This has caused us trouble with Azure Functions that need to use an updated identity package (related Stack Overflow issue) support for the updated identity library would be great. Note, this is a pipenv issue mostly but having the latest and greatest packages under the hood would be awesome!

Describe the solution you'd like Upgrade azureml-dataprep azure-identity to at a minimum version 1.8.0.

Describe alternatives you've considered Override dependency by using Pipfile with azure-identity in the packages and allocating azureml-dataprep in dev dependencies. Also, use a AzureCLI@2 task in pipelines over the Bash@3 task to supply proper information for testing with the DefaultAzureCredential() call.

Additional context N/A

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[xiangyan99]

@azureml-github

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github, @Azure/azure-ml-sdk.

Issue Details

---

Is your feature request related to a problem? Please describe. Using the Python azureml-dataprep library (versions >4.1) it has a dependency on azure-identity 1.7.0 which causes issues if you would like to use an upgraded version of the azure-identity library. This has caused us trouble with Azure Functions that need to use an updated identity package (related Stack Overflow issue) support for the updated identity library would be great. Note, this is a pipenv issue mostly but having the latest and greatest packages under the hood would be awesome!

Describe the solution you'd like Upgrade azureml-dataprep azure-identity to at a minimum version 1.8.0.

Describe alternatives you've considered Override dependency by using Pipfile with azure-identity in the packages and allocating azureml-dataprep in dev dependencies. Also, use a AzureCLI@2 task in pipelines over the Bash@3 task to supply proper information for testing with the DefaultAzureCredential() call.

Additional context N/A

Author:	jackphillipsjmu
Assignees:	bandsina
Labels:	question, Machine Learning, Service Attention, Client, customer-reported
Milestone:	-

[anliakho2]

We should be releasing a version with upgraded azure-identity dependency in December

[jackphillipsjmu]

Thank you for the update @anliakho2 glad to hear this is moving forward!

[tybyers]

Hi, I see a new version of azureml-dataprep (v 4.8.4) was released 4 days ago, but it still appears to have the dependency on azure-identity 1.7.0, according to the pip installer:

The conflict is caused by:
    The user requested azure-identity>=1.12.0
    azureml-dataprep 4.8.4 depends on azure-identity==1.7.0

This is causing a significant problem for us trying to build a suitable environment for our production system because azure-identity 1.7.0 has a bug with the ManagedIdentityCredential.

I tried a workaround in the conda yaml file by specifying:

- pip:
 - azure-ai-ml
 - azure-core
 - azureml-core
 - azureml-dataset-runtime
 - azureml-dataprep
 - azure-identity>=1.12.0 --install-option="--no-deps"

However, --install-option has been deprecated by pip as of October. I suppose I could specify an earlier version of pip, but otherwise I'm kind of stuck.

Will an azureml-dataprep version with upgraded azure-identity be released soon, as @anliakho2 mentioned above?

[tybyers]

Correcting the above, once I took away specifying the --no-deps clause, I also tried specifying azure-identity>=1.12.0:

- pip:
  - azure-identity>=1.12.0

and got the pip dependency hell failure.

When I just let it figure it out...

- pip:
  - azure-identity

It looks like it downloaded 1.12.0 like I was trying to force it to do all along..

Collecting azureml-dataprep
  Downloading https://<azure devops url>/azureml-dataprep/4.8.4/azureml_dataprep-4.8.4-py3-none-any.whl (43.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.5/43.5 MB 11.6 MB/s eta 0:00:00
Collecting azure-identity
  Downloading https://<azure devops url>/azure-identity/1.12/azure_identity-1.12.0-py3-none-any.whl (135 kB)

Thanks for updating azureml-dataprep -- just in time for us!

[tybyers]

Never mind on the above -- azureml_dataprep-4.8.4 still forces pip to uninstall azure-identity 1.12.0 and re-install 1.7.0, so I cannot create an Azure ML environment using a conda.yml specification that works. As a result, I can't use key vaults that aren't the workspace defaults, which creates technical debt for us because some of the secrets I need are in a vault controlled by another team, not our workspace default.

Will this team be releasing a version of azureml-dataprep that allows azure-identity >= 1.12.0 anytime soon?

[leonieroos]

Same issue for our team, we are blocked from fully using aml because of this inflexibility unfortunately. Will this be fixed soon or should we revert back to non-aml deployments for now?

[jackphillipsjmu]

For those fighting the good fight, most folks, in addition to my initial comments on the issue, might find it easier just to create a Docker-based environment that mimics what the conda/requirements.txt install performed in Azure ML.

You can use Azure's base images as well here. It just may take a little digging and testing. Also, if you're security allows it and a connection to the Azure ML Workspace is your end goal, then you can use Service Principal Authentication. This can be job or endpoint parameters set in the environment_variables for YAML definitions or provided as explicit environment variables in your Dockerfile. Not ideal, and doesn't encompass all use cases, but I hope it helps someone out there!

Previous Comment

Describe alternatives you've considered

Override dependency by using Pipfile with azure-identity in the packages and allocating azureml-dataprep in dev dependencies. 
Also, use a AzureCLI@2 task in pipelines over the Bash@3 task to supply proper information for testing with the DefaultAzureCredential() call.

[anliakho2]

@tybyers @leonieroos The change has barely missed last release but it is already in master. If it is blocking we can go hotfix route.

[shuyu42]

Before the release is ready, please consider using a the custom docker based environment as @jackphillipsjmu suggested above.

Here is the related document: Create an environment from a Docker build context. You can have something like this in the Dockerfile to force installing the desired version of azure-identity package:

RUN pip install azure-ai-ml azure-core azureml-core azureml-dataset-runtime azureml-dataprep
RUN pip install azure-identity>=1.12

[abhay8051]

I have been grappling with the issue of importing azure.ai.ml with error ""ImportError: cannot import name 'TokenCredential' from azure.core.credentials" when trying to create a pipeline, for last couple of days.

I am wondering why does it work on azure notebook with built-in "Python 3.10 - SDK V2" ?

[anliakho2]

the patch relaxing dependency requirements was released yesterday pip install azureml-dataprep==4.8.5

[ghost]

Hi @jackphillipsjmu. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @jackphillipsjmu, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.