sfdx force:data:soql:query --bulk only returns first batch of results when result set is very large

[codeulike]

Summary

If you use "sfdx force:data:soql:query --bulk" to fetch a large set of data (e.g. 1m rows) you only get the first batch of results, not the whole lot.

Steps To Reproduce:

Apologies, to reproduce this you'll need a repository with a lot of data in it.

1. Find a repository with 500,000 or more contacts (sorry!)
2. sfdx force:data:soql:query -u $username --query "select id, email from Contact" -r csv --bulk | Out-File "C:\temp\contact.csv"

Expected result

You should get a CSV file with all the contacts (500,000+)

Actual result

You get a CSV file with just the first batch of results, approx 250,000 of them ( Go to "Bulk Data Load Jobs" in Salesforce Setup, find the job, look at the batches of results, you will see that the number of lines in your CSV file matches just one of the batches)

The command

sfdx force:data:soql:bulk:report --bulkqueryid $id_of_bulk_load_job -u $username --resultformat csv | Out-File "C:\temp\contact2.csv"

... suffers from the same problem

I think "sfdx force:data:soql:bulk:report" needs a 'batch id' parameter in the same way that "force:data:bulk:status" does.

System Information

• Powershell 7
• sfdx-cli/7.172.0

[github-actions[bot]]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[codeulike]

This seems to be fixed in v1.70.0, possibly before that (what happened above with jsforce and how that went through the pipeline isnt very clear to me)

i.e. sf data query --query "select id, email from Contact" --target-org $username -r csv --bulk

... will now work correctly with large datasets that span several batches

edit: And thanks for fixing! being able to hit the bulk api from the command line is sooo useful

[cristiand391]

@codeulike That's right, I can't recall in which version of jsforce the fix was included but it should definitely fixed in latest sf/sfdx. Thanks for checking!