dataform icon indicating copy to clipboard operation
dataform copied to clipboard

[Feature Request] Print BigQuery job costs and total time during CLI runs

Open kimujun opened this issue 8 months ago • 4 comments

Inspired by #1944


Thanks to #1944, jobIds are displayed during CLI runs.

These jobIds come from the response of getMetadata() and the response also includes totalBytesBilled and elapsedTime.

It would be helpful, for identifying heavy queries, to print the costs and total time using the data above.

kimujun avatar May 16 '25 09:05 kimujun

Total bytes billed added by @ashish10alex and will be available in 3.0.23. Elapsed time will be a little bit tricker to collect and show but feel free to contribute to Dataform CLI.

Ceridan avatar Jun 06 '25 10:06 Ceridan

@Ceridan Thank you for letting me know!

Elapsed time will be a little bit tricker to collect

Can you tell me what the problem is?

kimujun avatar Jun 17 '25 03:06 kimujun

There are a little bit more work for elapsed time since it requires proto changes as well.

Additionally, at this point I think we need to introduce a new option for run command, e.g. --format or --output to be able to specify what additional fields you want to see. At the moment we already have jobIDs and total bytes billed.

Ceridan avatar Jun 18 '25 09:06 Ceridan

@Ceridan That makes sense!

Format option seems great. I'll try it.

kimujun avatar Jun 20 '25 08:06 kimujun