[bug] Build fails due to Safety Report Creation error
Checklist
- [x] I've prepended issue tag with type of change: [bug]
- [X] (If applicable) I've attached the script to reproduce the bug
- [X] (If applicable) I've documented below the DLC image/dockerfile this relates to
- [ ] (If applicable) I've documented below the tests I've run on the DLC image
- [X] I'm using an existing DLC image listed here: https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html
- [ ] I've built my own container based off DLC (and I've attached the code used to build my own image)
Concise Description: Docker building fails due to a lack of proper Safety Report Creation.
DLC image/dockerfile: The default Pytorch 2.0 training image (using the provided buildspec)
Current behavior: The build fails due to a lacking safety report. Apparently due to an intentional fail as indicated in the comments here: https://github.com/aws/deep-learning-containers/blob/master/src/safety_report_generator.py#LL176C1-L177C86
Expected behavior: The safety report should be generated and read correctly by the JSON decoder.
Additional context:
I've created a t2.xlarge EC2 instance (Ubuntu 22.04) just for the build. The other steps related to ECR seems ok. I set the environment variables (REGION, ACCOUNT_ID and REPOSITORY_NAME) and I use the following command, as indicated in this repo's readme:
python src/main.py --buildspec pytorch/training/buildspec.yml --framework pytorch --image_types training --device_types gpu --py_versions py3
The error message:
Traceback (most recent call last):
File "/home/ubuntu/deep-learning-containers/src/main.py", line 183, in <module>
main()
File "/home/ubuntu/deep-learning-containers/src/main.py", line 179, in main
image_builder(buildspec_file, image_types, device_types)
File "/home/ubuntu/deep-learning-containers/src/image_builder.py", line 302, in image_builder
pushed_images += process_images(parent_images, "Parent/Independent")
File "/home/ubuntu/deep-learning-containers/src/image_builder.py", line 366, in process_images
build_images(common_stage_image_list, make_dummy_boto_client=True)
File "/home/ubuntu/deep-learning-containers/src/image_builder.py", line 508, in build_images
FORMATTER.progress(THREADS)
File "/home/ubuntu/deep-learning-containers/src/output.py", line 103, in progress
output[i] += "." * 10 + constants.STATUS_MESSAGE[futures[image].result()]
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/deep-learning-containers/src/image.py", line 164, in build
self.update_pre_build_configuration()
File "/home/ubuntu/deep-learning-containers/src/common_stage_image.py", line 54, in update_pre_build_configuration
generate_safety_report_for_image(
File "/home/ubuntu/deep-learning-containers/src/utils.py", line 266, in generate_safety_report_for_image
safety_scan_output = SafetyReportGenerator(container_id, ignore_dict=ignore_dict).generate()
File "/home/ubuntu/deep-learning-containers/src/safety_report_generator.py", line 178, in generate
scanned_vulnerabilities = json.loads(self.safety_check_output)
File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)