google-cloud-python

google-cloud-python copied to clipboard

Support JsonStreamWriter Interface like java client.

1

Hi, i recently started using storage write api and python client only supports protobuf methods to ingest the data into bigquery.it is kind of frustrating to use it lack of...

slice-amandata

add test case for append_rows_stream.close() 499 error

1

Add a test case that covers the issue in googleapis/python-bigquery-storage#638. Thanks for stopping by to let us know something could be better! **PLEASE READ**: If you have a support contract...

Linchin

Deadlock possible when calling bigquery_storage.BigQueryReadClient() on multiple threads

1

I'm using `bigquery_storage.BigQueryReadClient()` inside a [Prefect](https://www.prefect.io/) task, and if I run five or so instances of that task in parallel it seems to deadlock about 50% of the time. Wrapping...

d1manson

Google Big Query Write append_rows does not respect retry configuration

15

Default configuration for append_rows() call [here](https://github.com/googleapis/python-bigquery-storage/blob/v2.19.1/google/cloud/bigquery_storage_v1/services/big_query_write/transports/base.py#L154) and [here](https://github.com/googleapis/python-bigquery-storage/blob/v2.19.1/google/cloud/bigquery_storage_v1/services/big_query_write/async_client.py#L468) sets that by default request would be retired in case of `google.api_core.exceptions.ServiceUnavailable` exception with a timeout of 1 day. However I observed...

kamilglod

collate is not supported by Storage API

2

I'm using Python BigQuery API to perform queries and create some extracts. Everything went fine, except when I made a change on the schema to add collate (case ignore) to...

nuevoleonkx

Conversion from bq schema to proto

2

**Is your feature request related to a problem? Please describe.** The java implementation can convert bigquery's schema to proto and write json without preparing proto files. I think it would...

richwomanbtc

Increase coverage in handwritten code

1

See build log [here](https://github.com/googleapis/python-bigquery-storage/actions/runs/3585098065/jobs/6032647828) ``` google/cloud/bigquery_storage_v1/types/annotations.py 2 2 0 0 0% 18-24 google/cloud/bigquery_storage_v1/types/arrow.py 15 0 0 0 100% google/cloud/bigquery_storage_v1/types/avro.py 11 0 0 0 100% google/cloud/bigquery_storage_v1/types/protobuf.py 9 0 0 0 100%...

parthea

feat!: convert arrow data types to Python types in `rows()`

3

I notice we just loop through all the rows in a record batch here: https://github.com/googleapis/python-bigquery-storage/blob/f7fc4ec45e62739139bf865497ce2e5fbd948d31/google/cloud/bigquery_storage_v1/reader.py#L695 This might result in some odd types, such as [Int64Scalar](https://arrow.apache.org/docs/python/generated/pyarrow.Int64Scalar.html?highlight=scalar#pyarrow.Int64Scalar) and [TimestampScalar](https://arrow.apache.org/docs/python/generated/pyarrow.TimestampScalar.html?highlight=scalar#pyarrow.TimestampScalar) sneaking in. I...

tswast

deprecate `ReadRowsStream.to_dataframe`, `ReadRowsIterable.to_dataframe`

4

These higher-level functions seemed useful at the time of implementation, but they were not needed by `google-cloud-bigquery`, `pandas-gbq`, or even experimental Dask (https://github.com/dask/dask/issues/3121#issuecomment-578328715). Instead, only the lower-level `ReadRowsPage.to_arrow` and `ReadRowsPage.to_dataframe`...

tswast

show deprecation warning when `read_session` passed to `ReadRowsStream.rows`, `ReadRowsStream.to_arrow`, `ReadRowsStream.to_dataframe`

3

Follow-up to https://github.com/googleapis/python-bigquery-storage/pull/228, which quietly deprecates `read_session`. I want to wait until `google-cloud-bigquery` doesn't rely on this behavior before we more loudly deprecate it.

tswast