klio icon indicating copy to clipboard operation
klio copied to clipboard

[core] Add KlioWriteToText missing fields including file_name_suffix,…

Open shireen-bean opened this issue 4 years ago • 0 comments

Changes

Add KlioWriteToText missing fields including file_name_suffix, num_shards, append_trailing_newlines etc

A user reported his job erroring out when providing file_name_suffix when default event output was enabled in batch mode. This PR enables missing fields for KlioWriteToText that are available in the beam conterpart.

Testing

Added a test use case for having more than the bare minimum configuration fields for the KlioWriteFileConfig. Tested this on a job using klio devtools for a test batch job. Tested to ensure that the job would continue to work with just:

  events:
    inputs:
       <- snip ->
    outputs:
      - type: file
        location: gs://shireen-batch-test/

as well as work with the config:

  events:
    inputs:
       <- snip ->
    outputs:
      - type: file
        location: gs://shireen-batch-test/
        file_name_suffix: .txt
        num_shards: 3
        append_trailing_newlines: False

Checklist for PR author(s)

  • [x] Format the pull request title like [cli] Fixes bugs in 'klio job fake-cmd'.
  • [x] Changes are covered by unit tests (no major decrease in code coverage %) and/or integration tests.
  • [x] Document any relevant additions/changes in the appropriate spot in docs/src.
  • [x] For any change that affects users, update the package's changelog in docs/src/reference/<package>/changelog.rst.

shireen-bean avatar Jul 02 '21 17:07 shireen-bean