FL4Health
FL4Health copied to clipboard
VectorInstitute
Smoke Tests Fail due to tolerances on my machine
For some reason smoke tests fail locally due to results being outside tolerances. Tried starting from a fresh environment and the issue persists. Not urgent but worth looking into eventually. Creating an issue so that I don't forget about it.
System Info
13th Gen Intel® Core™ i9-13900K × 32 Ubuntu 22.04.4 LTS NVIDIA GeForce RTX 4090 Driver Version: 555.42.06 Cuda Version: 12.1 Python Version: 3.9.19
Steps to recreate
python3.9 -m venv test_env
source test_env/bin/activate
cd ../../FL4Health
git checkout main
git pull
pip install --upgrade pip poetry
poetry install --with "codestyle, dev, dev-local, test, picai"
python -m tests.smoke_tests.run_smoke_test
Output
$ python -m tests.smoke_tests.run_smoke_test
2024-08-08 15:59:27 INFO Running smoke tests with parameters:
2024-08-08 15:59:27 INFO Server : examples.fedprox_example.server
2024-08-08 15:59:27 INFO Client : examples.fedprox_example.client
2024-08-08 15:59:27 INFO Config : tests/smoke_tests/fedprox_config.yaml
2024-08-08 15:59:27 INFO Dataset: examples/datasets/mnist_data/
2024-08-08 15:59:27 INFO Preloading MNIST dataset...
2024-08-08 15:59:27 INFO Preloading MNIST dataset...
INFO : Data directory: examples/datasets/mnist_data
2024-08-08 15:59:27 INFO Data directory: examples/datasets/mnist_data
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/train-images-idx3-ubyte.gz
100%|████████████████████████████████████████████████████| 9912422/9912422 [00:00<00:00, 22336309.08it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/train-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/train-labels-idx1-ubyte.gz
100%|█████████████████████████████████████████████████████████| 28881/28881 [00:00<00:00, 1481673.44it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/train-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/t10k-images-idx3-ubyte.gz
100%|████████████████████████████████████████████████████| 1648877/1648877 [00:00<00:00, 13375956.93it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/t10k-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/t10k-labels-idx1-ubyte.gz
100%|██████████████████████████████████████████████████████████| 4542/4542 [00:00<00:00, 12683441.26it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/t10k-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw
2024-08-08 15:59:33 INFO Finished preloading MNIST dataset
2024-08-08 15:59:33 INFO Starting server...
2024-08-08 15:59:36 INFO Server started
2024-08-08 15:59:36 INFO Starting client 0
2024-08-08 15:59:36 INFO Starting client 1
2024-08-08 15:59:36 INFO Collecting output for Client 0...
2024-08-08 15:59:46 INFO Output collected for Client 0
2024-08-08 15:59:46 INFO Collecting output for Client 1...
2024-08-08 15:59:46 INFO Output collected for Client 1
2024-08-08 15:59:46 INFO All clients finished execution
2024-08-08 15:59:46 INFO Collecting output for Server...
2024-08-08 15:59:49 INFO Output collected for Server
2024-08-08 15:59:49 INFO Server has finished execution
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/shawn/Code/FL4Health/tests/smoke_tests/run_smoke_test.py", line 462, in <module>
loop.run_until_complete(
File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home/shawn/Code/FL4Health/tests/smoke_tests/run_smoke_test.py", line 260, in run_smoke_test
assert len(server_errors) == 0, f"Server metrics check failed. Errors: {server_errors}"
AssertionError: Server metrics check failed. Errors: ["Saved value for metric 'val - prediction - accuracy' (0.4656666666666667) does not match the requested value (0.4627) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (2.022878646850586) does not match the requested value (2.0268) within requested tolerance (0.0005).", "Saved value for metric 'val - prediction - accuracy' (0.43377777777777776) does not match the requested value (0.4323) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (1.7308796644210815) does not match the requested value (1.7356) within requested tolerance (0.0005).", "Saved value for metric 'val - prediction - accuracy' (0.5417777777777778) does not match the requested value (0.5362) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (1.4460455179214478) does not match the requested value (1.4518) within requested tolerance (0.0005)."]
