Failure to upload summary to GCP

[nanlliu]

Describe the bug

Failure to upload summary to GCP.

Status code 412 --> should not happen if the request is HEAD OF GET.

However, I somehow had google.api_core.exceptions.PreconditionFailed: 412 PUT?

Screenshots

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2697, in _prep_and_do_upload
    created_json = self._do_upload(
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2520, in _do_upload
    response = self._do_resumable_upload(
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2356, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport, timeout=timeout)
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 515, in transmit_next_chunk
    return _request_helpers.wait_and_retry(
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py", line 155, in wait_and_retry
    response = func()
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 511, in retriable_request
    self._process_resumable_response(result, len(payload))
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/resumable_media/_upload.py", line 690, in _process_resumable_response
    status_code = _helpers.require_status_code(
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/resumable_media/_helpers.py", line 108, in require_status_code
    raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PERMANENT_REDIRECT: 308>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/filestore/users/nan/pyenv/versions/3.10.13/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/tensorboardX/event_file_writer.py", line 219, in run
    self._record_writer.flush()
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/tensorboardX/event_file_writer.py", line 69, in flush
    self._py_recordio_writer.flush()
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/tensorboardX/record_writer.py", line 193, in flush
    self._writer.flush()
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/tensorboardX/record_writer.py", line 154, in flush
    self.blob.upload_from_string(data=upload_buffer.getvalue(),
  File "/mnt/filestore/users/nan/pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 3125, in upload_from_string
    self.upload_from_file(
  File "/mnt/filestore/users/nan/pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2848, in upload_from_file
    self._prep_and_do_upload(
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2715, in _prep_and_do_upload
    _raise_from_invalid_response(exc)
  File "/mnt/filestore/users/nan/.venv/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 4905, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.PreconditionFailed: 412 PUT https://storage.googleapis.com/upload/storage/v1/b/o?uploadType=resumable&ifGenerationMatch=1733266651622079&upload_id=AFiumC4aOfReP-Day2traOzIWN3B-CJstuSVOdHMCGLhtm7GOtjddxvLRVTtA5fnyxTpB8TEAdt5ggaQyDDg5l9HUj2-8djf4IF1fQAJJKFbMatg: {
  "error": {
    "code": 412,
    "message": "At least one of the pre-conditions you specified did not hold.",
    "errors": [
      {
        "message": "At least one of the pre-conditions you specified did not hold.",
        "domain": "global",
        "reason": "conditionNotMet",
        "locationType": "header",
        "location": "If-Match"
      }
    ]
  }
}
: ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PERMANENT_REDIRECT: 308>)

Environment What is the result of pip list|grep -E "torch|proto|tensor" If the version is too old, please try to update first.

Python environment Virtualenv Which version of python are you using? Did you use Andconda or Virtualenv?

Additional context Add any other context about the problem here.

[lanpa]

Hi, is there a way to reproduce?