[Bug]: Failing to write to temporary file storage

[mehmetkoc182]

Related Template(s)

BigQuery to Bigtable & Parquet File Storage to Bigtable

What happened?

There seems to be some sort of mismatch around the dataflow worker permissions and cloud storage temp directory.

I've tried 2 different templates and the issue seems to persist on the "reading" step. However, the error log seems to be related to "write" permissions I believe happens to the temporary directory for the process.

I'm using uniform bucket level access and the worker account has full access to cloud storage and cloud storage objects.

Do I need different/extra permissions for these templates?

Beam Version

Newer than 2.43.0

Relevant log output

Error initializing dynamic plugin prober" err="error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system"




ima: Can not allocate sha384 (reason: -2)

[bvolpato]

Is there any other error? I think that the log mentioned is "normal" to happen on GCE instances.

[bvolpato]

@mehmetkoc182 Can you please elaborate / post any errors?

[bvolpato]

@mehmetkoc182 Let us know if you still want to troubleshoot this. We'll close this issue if there are no further interactions.

[Wuerike]

I think i have a similar error with the GCS to PubSub template, until the release 2023-03-21-00_RC00 it was working fine, but now at the 2023-03-28-00_RC00 it's failing on rename a temp file

Warning Log:

"Unable to delete temp files: "gs://my_bucket/tmp_dir/dax-tmp-2023-04-10_09_14_22-7957512643797177487-S01-0-adbd0cb3c13f0b01/[email protected]." Causes: Unable to delete file: gs://my_bucket/tmp_dir/dax-tmp-2023-04-10_09_14_22-7957512643797177487-S01-0-adbd0cb3c13f0b01/tmp-adbd0cb3c13f0932-shard--try-070f492ac9c1925b-endshard.sdfmeta."

Error Log:

Workflow failed. Causes: S01:Read Text Data/Read/Impulse+Read Text Data/Read/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+Read-Text-Data-Read-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/PairWithRestriction+Read-Text-Data-Read-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/SplitWithSizing failed., Unable to rename output files from gs://my_bucket/tmp_dir/dax-tmp-2023-04-10_09_14_22-7957512643797177487-S01-0-adbd0cb3c13f0b01/[email protected] to gs://my_bucket/tmp_dir/tmp-adbd0cb3c13f0932@*.sdfmeta., Unable to rename "gs://my_bucket/tmp_dir/dax-tmp-2023-04-10_09_14_22-7957512643797177487-S01-0-adbd0cb3c13f0b01/tmp-adbd0cb3c13f0932-shard--try-070f492ac9c1925b-endshard.sdfmeta" to "gs://my_bucket/tmp_dir/tmp-adbd0cb3c13f0932-00000-of-00001.sdfmeta".

[bvolpato]

@Wuerike thanks! Does the job fail? Or you see the operations being retried? Would you mind filing a support issue with the job id so the team can look at this internally?

[bvolpato]

@Wuerike seen a similar case, and it was related to missing permissions on the bucket. Unsure how the update might have affected it. Can you double check on that to be sure?

[Wuerike]

@bvolpato just to update, it was indeed a missing permission, I gave owner access to the service account and it worked. The problem is it was working before this is why I thought the reason could be any update.

[github-actions[bot]]

This issue has been marked as stale due to 180 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the issue at any time. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.