Migration from Anaconda to conda-forge

[yoshipon]

What?

This PR adds a Python setup script for conda-forge. It updates CI, documentation, and tools/Makefile to use setup_miniforge.sh, while keeping the existing recipes to use Anaconda for reproducibility. The setup_miniforge.sh is written by copying setup_anaconda.sh.

Why?

While Anaconda is more reliable as a commercial product, conda-forge is more community-driven and has a permissive license.

See also

https://github.com/conda-forge/miniforge

[yoshipon]

I'll revert this PR to Draft because I found that I need to update Dockerfile.

[yoshipon]

I updated 1) to use miniforge in dockerfile and 2) to make the intel channel the default channel of miniforge. The intel channel enables us to use Intel MKL in NumPy/SciPy.

I'd like to confirm whether this script works well with CI before review.

[sw005320]

Any update?

[yoshipon]

I updated this PR to solve the MKL bug:

• https://github.com/pytorch/pytorch/issues/123097

The workaround is to install the version 2024.0 instead of 2024.1

I'd like to ask the following three questions:

1. I'm now writing conda install -y mkl=2024.0 on setup_miniforge.sh -> Is there any better place to write?
2. CI for Ubuntu looks not using conda -> Is it OK to change the CI to use conda?
3. Why unit tests are skiped might be because my PR is labeled as "Docker" -> Is it OK to remove this condition?

[sw005320]

I updated this PR to solve the MKL bug:

• ImportError undefined symbol: iJIT_NotifyEvent encountered when MKL 2024.1 is installed.  pytorch/pytorch#123097

The workaround is to install the version 2024.0 instead of 2024.1

I'd like to ask the following three questions:

1. I'm now writing conda install -y mkl=2024.0 on setup_miniforge.sh -> Is there any better place to write?

Is it only for the conda environment? If so, this is fine.

2. CI for Ubuntu looks not using conda -> Is it OK to change the CI to use conda?

Can you keep it false? We want to test it with various configurations (but minimizing the number of configurations to save the CI time). CI for Debian uses the conda environment, so it is good coverage.

3. Why unit tests are skiped might be because my PR is labeled as "Docker" -> Is it OK to remove this condition?

OK, done.

[sw005320]

There are several shell check errors. https://github.com/espnet/espnet/actions/runs/11606937418/job/32319685605?pr=5924#step:8:116 Can you fix them?

[sw005320]

I think https://github.com/espnet/espnet/blob/master/.github/workflows/ci_on_ubuntu.yml#L35 would not be very reasonable like your case. So, can you remove this condition?

[yoshipon]

Thank you for your prompt response.

Is it only for the conda environment? Yes, it is only for the conda (especially conda-forge) environment.

So, can you remove this condition? Yes, I removed the condition from the workflow now.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 53.82%. Comparing base (d38d76e) to head (7c061c3). Report is 16 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5924      +/-   ##
==========================================
+ Coverage   45.49%   53.82%   +8.32%     
==========================================
  Files         613      625      +12     
  Lines       54166    58429    +4263     
==========================================
+ Hits        24643    31447    +6804     
+ Misses      29523    26982    -2541     

Flag	Coverage Δ
test_integration_espnet1	62.56% <ø> (?)
test_integration_espnet2	48.21% <ø> (ø)
test_integration_espnetez	?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[sw005320]

Thanks, @yoshipon!

[sw005320]

@yoshipon, can you make a follow-up PR about the directory path?