Tailing kubernetes logs doesn't put logs on new lines

[matthen]

🐛 Bug

When running torchx log --tail with a kubernetes job, the logs appear all on one line, with no new lines between them.

Module (check all that applies):

• [ ] torchx.spec
• [ ] torchx.component
• [ ] torchx.apps
• [ ] torchx.runtime
• [ ] torchx.cli
• [x] torchx.schedulers
• [ ] torchx.pipelines
• [ ] torchx.aws
• [ ] torchx.examples
• [ ] other

To Reproduce

Steps to reproduce the behavior:

torchx log --tail kubernetes:${JOB_NAME}

output on terminal looks like:

name/0 2024-12-06T09:57:30.580070417Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Using CUDA driver version 12060name/0 2024-12-06T09:57:30.580072867Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Configuring AWS-specific optionsname/0 2024-12-06T09:57:30.580075167Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Setting provider_filter to efaname/0 2024-12-06T09:57:30.580077497Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1name/0 2024-12-06T09:57:30.580079757Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Setting NCCL_NVLSTREE_MAX_CHUNKSIZE to 512KiB

(but running without --tail puts these all on new lines)

Environment

• torchx version (e.g. 0.1.0rc1): torchx-0.7.0
• Python version: 3.11.0rc1
• OS (e.g., Linux): ubuntu
• How you installed torchx (conda, pip, source, docker): pip
• Docker image and tag (if using docker): /
• Git commit (if installed from source): /
• Execution environment (on-prem, AWS, GCP, Azure etc): local macbook in Terminal app
• Any other relevant information:

[Sanjay-Ganeshan]

Can you try with a version built from source? I think this might already be addressed

[matthen]

Thanks for suggestion! I get the same issue after doing pip install -e git+https://github.com/pytorch/torchx.git#egg=torchx