data-setup icon indicating copy to clipboard operation
data-setup copied to clipboard

Installing requirements on windows/linux takes a huge time due to nvidia stuff

Open julesvanrie opened this issue 1 year ago • 3 comments

Since we included transformers, the python requirements install takes even longer than before.

Reasons (double check): transformers needs pytorch, which installs plenty of nvidia stuff too.

Couple of options:

  • Remove transformers stuff: we only use it one day, students could run the notebooks on Colab (like in the previous days of DL).
  • For pytorch to install a cpu-only version. Now we install nvidia cuda stuff, but most students don't have any benefit from it. (Pretty rare to have a (powerful) nvidia gpu in laptops.)

julesvanrie avatar Apr 05 '24 11:04 julesvanrie

On windows / linux 9.4 Gb of which:

  • nvidia 2.8G
  • tangled_up_in_unicode 1.8G
  • torch 1.4G
  • tensorflow 1.4G

On macos 4.3 Gb of which:

  • tangled_up_in_unicode 1.7G
  • torch 0.3G
  • tensorflow 0.9G

Maybe worth trying a cpu only of tensorflow too?

julesvanrie avatar Apr 05 '24 11:04 julesvanrie

tangled-up-in-unicode :

Dependency of pandas-profiling and visions, which in turn is a dependency of pandas-profiling.

So we can drop this if we drop pandas-profiling.

➜ pip install "pipdeptree<2.17.0"
➜ pipdeptree -r -p tangled-up-in-unicode
tangled-up-in-unicode==0.2.0
├── pandas-profiling==3.3.0 [requires: tangled-up-in-unicode==0.2.0]
└── visions==0.7.5 [requires: tangled-up-in-unicode>=0.0.4]
    └── pandas-profiling==3.3.0 [requires: visions==0.7.5]
➜  pipdeptree -r -p visions
visions==0.7.5
└── pandas-profiling==3.3.0 [requires: visions==0.7.5]

julesvanrie avatar Apr 06 '24 11:04 julesvanrie

ydata-profiling does not need tangled-up-in-unicode directly. visions doesn't need it anymore either in the latest version. But that requires pandas>=2.0.0 and we're still on 1.4.4).

julesvanrie avatar Apr 06 '24 11:04 julesvanrie