clipper icon indicating copy to clipboard operation
clipper copied to clipboard

Some instructions that are required by Mxnet are not supported by some workers.

Open rkooo567 opened this issue 6 years ago • 0 comments

Following https://github.com/ucbrise/clipper/issues/706, https://github.com/ucbrise/clipper/issues/703, and https://github.com/ucbrise/clipper/pull/707.

We found out mxnet test is not even running at jenkins worker 7 and 8. It is because some pre-packages required for mxnet is not supported by cpus on worker 7 and 8. We temporarily fixed the problem by running our CI only at worker 2, but it will cause some problems in the future.

There are couple solutions we are thinking of. Please tell us about your opinion.

  1. Figure out build dependency and run CI at worker 2 if mxnet integration test should be run by a certain PR.
  2. Use other workers like 3, 4, 5. The problem is that it uses a different OS than worker 2, 7, and 8. We can use this solution by modifying our testing scripts more generic in any Linux based OS.
  3. Use 3-4 AWS EC2 instances and setup our own infrastructure that is separated from the lab. This has benefits that we can run a better Jenkins scripts and can be isolated from spark CI tests. But it will cost a lot.

rkooo567 avatar May 28 '19 05:05 rkooo567