Python bigquery export script does not export with correct schema when compared with gcloud export

[sureshs123]

I have tried to export the asset details to biquery for writing queries to fetch certain data, Initially i tried with gcloud sdk and it exported the required data in the perfect schema . when i try to do the same with the python code to export assets to biquery using this script https://github.com/googleapis/python-asset/blob/main/samples/snippets/quickstart_exportassets.py it seems to be exporting the data in incorrect schema which i am not able to query

Steps to reproduce

1. Run https://github.com/googleapis/python-asset/blob/main/samples/snippets/quickstart_exportassets.py script to export serviceaccountKey asset type to bigquery.
2. Perform the same export using gcloud asset export command and compare with that of the python script .
3. Please find the attached screenshots of the schema

When exported using gcloud sdk this is the schema obtained which is the expected schema.

When exported using client library this is the schema obtained which is the incorrect schema.

Am I missing any additional config in the python script? I am trying to export serviceAccountKey resource to bigquery. Do let me know if any additional details are required.

Thanks!

[parthea]

@lvvvvvf, as author of the sample samples/snippets/quickstart_exportassets.py, please could you take a look?

[sureshs123]

@parthea Any updates on this?? Its a blocker for the setup that we are trying

[parthea]

I'm going to transfer this issue to the google-cloud-python repository as we are preparing to move the code for google-cloud-asset to that repository in the next 1-2 weeks.

[lvvvvvf]

I think you need to add separate_tables_per_asset_type=true to bigquery_destination, to expand the metadata schema.

Thanks

---

From: sureshs123 @.> Sent: Tuesday, April 18, 2023 7:03 AM To: googleapis/google-cloud-python @.> Cc: lvvvvvf @.>; Mention @.> Subject: [googleapis/google-cloud-python] Python bigquery export script does not export with correct schema when compared with gcloud export (Issue #11092)

I have tried to export the asset details to biquery for writing queries to fetch certain data, Initially i tried with gcloud sdk and it exported the required data in the perfect schema . when i try to do the same with the python code to export assets to biquery using this script https://github.com/googleapis/python-asset/blob/main/samples/snippets/quickstart_exportassets.py it seems to be exporting the data in incorrect schema which i am not able to query

Steps to reproduce

1. Run https://github.com/googleapis/python-asset/blob/main/samples/snippets/quickstart_exportassets.py script to export serviceaccountKey asset type to bigquery.
2. Perform the same export using gcloud asset export command and compare with that of the python script .
3. Please find the attached screenshots of the schema

When exported using gcloud sdk this is the schema obtained which is the expected schema.

[image]https://user-images.githubusercontent.com/92539532/164981335-3dfa665c-649b-4d37-bdda-c49866df3043.png

When exported using client library this is the schema obtained which is the incorrect schema.

[image]https://user-images.githubusercontent.com/92539532/164981398-7581676c-da96-4383-9f8b-118c4c509745.png

Am I missing any additional config in the python script? I am trying to export serviceAccountKey resource to bigquery. Do let me know if any additional details are required.

Thanks!

— Reply to this email directly, view it on GitHubhttps://github.com/googleapis/google-cloud-python/issues/11092, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM3DCLKZK7SUCPELO6GM3O3XB2NMPANCNFSM6AAAAAAXCWBUEU. You are receiving this because you were mentioned.Message ID: @.***>