deep-learning-containers icon indicating copy to clipboard operation
deep-learning-containers copied to clipboard

[bug] Build fails due to Safety Report Creation error

Open emredog opened this issue 2 years ago • 0 comments

Checklist

  • [x] I've prepended issue tag with type of change: [bug]
  • [X] (If applicable) I've attached the script to reproduce the bug
  • [X] (If applicable) I've documented below the DLC image/dockerfile this relates to
  • [ ] (If applicable) I've documented below the tests I've run on the DLC image
  • [X] I'm using an existing DLC image listed here: https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html
  • [ ] I've built my own container based off DLC (and I've attached the code used to build my own image)

Concise Description: Docker building fails due to a lack of proper Safety Report Creation.

DLC image/dockerfile: The default Pytorch 2.0 training image (using the provided buildspec)

Current behavior: The build fails due to a lacking safety report. Apparently due to an intentional fail as indicated in the comments here: https://github.com/aws/deep-learning-containers/blob/master/src/safety_report_generator.py#LL176C1-L177C86

Expected behavior: The safety report should be generated and read correctly by the JSON decoder.

Additional context: I've created a t2.xlarge EC2 instance (Ubuntu 22.04) just for the build. The other steps related to ECR seems ok. I set the environment variables (REGION, ACCOUNT_ID and REPOSITORY_NAME) and I use the following command, as indicated in this repo's readme: python src/main.py --buildspec pytorch/training/buildspec.yml --framework pytorch --image_types training --device_types gpu --py_versions py3

The error message:

Traceback (most recent call last):
  File "/home/ubuntu/deep-learning-containers/src/main.py", line 183, in <module>
    main()
  File "/home/ubuntu/deep-learning-containers/src/main.py", line 179, in main
    image_builder(buildspec_file, image_types, device_types)
  File "/home/ubuntu/deep-learning-containers/src/image_builder.py", line 302, in image_builder
    pushed_images += process_images(parent_images, "Parent/Independent")
  File "/home/ubuntu/deep-learning-containers/src/image_builder.py", line 366, in process_images
    build_images(common_stage_image_list, make_dummy_boto_client=True)
  File "/home/ubuntu/deep-learning-containers/src/image_builder.py", line 508, in build_images
    FORMATTER.progress(THREADS)
  File "/home/ubuntu/deep-learning-containers/src/output.py", line 103, in progress
    output[i] += "." * 10 + constants.STATUS_MESSAGE[futures[image].result()]
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ubuntu/deep-learning-containers/src/image.py", line 164, in build
    self.update_pre_build_configuration()
  File "/home/ubuntu/deep-learning-containers/src/common_stage_image.py", line 54, in update_pre_build_configuration
    generate_safety_report_for_image(
  File "/home/ubuntu/deep-learning-containers/src/utils.py", line 266, in generate_safety_report_for_image
    safety_scan_output = SafetyReportGenerator(container_id, ignore_dict=ignore_dict).generate()
  File "/home/ubuntu/deep-learning-containers/src/safety_report_generator.py", line 178, in generate
    scanned_vulnerabilities = json.loads(self.safety_check_output)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

emredog avatar May 26 '23 08:05 emredog