sdv-dev
Investigate why minimum unit tests take long for mac
Problem Description
As an engineer, I'd like to be able to see the results of my workflows more quickly so I can increase efficiency.
The unit test minimum builds seem to take ~20 minutes for macos. This is particularly troubling since we only get 5 concurrent mac machines.
Similarly the integration tests for mac sometimes take way longer.
Expected behavior
- Reduce the times for mac till it's similar to windows and linux (~3 min)
The issue seems to be that sometimes, certain integration tests can take significantly longer on macos as they do on linux. It seems to occur reliably on macos-latest when testing minimum requirements, and intermittently when using the latest. The worst offenders seem to be these three tests:
| TEST | TIME(s) macos-latest 3.10 minimum | TIME(s) macos-latest 3.8 latest | TIME(s) macos-13 (x86) 3.8 latest | TIME(s) macos-latest 3.10 latest | TIME(s) ubuntu-latest 3.10 latest |
|---|---|---|---|---|---|
| sequential/test_par.py::test_sythesize_sequences | 377.49 | 254.83 | 101.07 | 106.68 | 85.68 |
| single_table/test_ctgan.py::test_synthesize_table_ctgan | 117.62 | 52.69 | 32.87 | 23.45 | 15.31 |
| single_table/test_ctgan.py::test_ctgansynthesizer_with_constraints_generating_categorical_values | 95.13 | 44.62 | 23.6 | 15.05 | 10.94 |
I was not able to reproduce locally, even after limiting my system to match the specs of the github runners.
The full list of integration tests taking over 1s on various python versions and operating systems can be found here.
It seems that Github is still in the process of migrating all the macos-latest machines to macos-14 meaning that sometimes latest is still running on macos-12. I wonder how the times are if you explicitly set macos-14
Interesting, I tried forcing macos-14 and the minimum tests on 3.8 had an installation failure. I can reproduce this locally, it seems like our minimum requirements for 3.8 have issues on arm. The latest requirements both completed in a "normal" amount of time (~5 min).