klio
klio copied to clipboard
[core] Add KlioWriteToText missing fields including file_name_suffix,…
Changes
Add KlioWriteToText missing fields including file_name_suffix, num_shards, append_trailing_newlines etc
A user reported his job erroring out when providing file_name_suffix when default event output was enabled in batch mode.
This PR enables missing fields for KlioWriteToText that are available in the beam conterpart.
Testing
Added a test use case for having more than the bare minimum configuration fields for the KlioWriteFileConfig. Tested this on a job using klio devtools for a test batch job. Tested to ensure that the job would continue to work with just:
events:
inputs:
<- snip ->
outputs:
- type: file
location: gs://shireen-batch-test/
as well as work with the config:
events:
inputs:
<- snip ->
outputs:
- type: file
location: gs://shireen-batch-test/
file_name_suffix: .txt
num_shards: 3
append_trailing_newlines: False
Checklist for PR author(s)
- [x] Format the pull request title like
[cli] Fixes bugs in 'klio job fake-cmd'. - [x] Changes are covered by unit tests (no major decrease in code coverage %) and/or integration tests.
- [x] Document any relevant additions/changes in the appropriate spot in
docs/src. - [x] For any change that affects users, update the package's changelog in
docs/src/reference/<package>/changelog.rst.