$ the-wire · showcase
BigQuery Arrow reads get correct limits, shared storage client
By RepoJournal · Filed · About Google · Composed from the cited sources · methodology
BigQuery's Arrow path picked up four fixes that change what `to_arrow()` and `to_dataframe()` actually return, while python-genai fixed mTLS URL rewriting that breaks multi-regional Vertex AI endpoints.
fix(bigquery): apply max_results to the whole Arrow result googleapis/google-cloud-python
`max_results` now caps the whole Arrow result; previously it only limited the `jobs.query` first page and every remaining row was streamed. The batch that crosses the limit is sliced and the stream cancelled, and no stream opens at all when the first page already reaches the limit.
feat(bigquery): share one BigQuery Storage client per Client googleapis/google-cloud-python
Arrow results now share one `BigQueryReadClient` per `Client`, created on first use with the client's credentials and universe domain and closed in `Client.close()`, instead of a new client per read that was never closed. A caller-supplied `bqstorage_client` is used and left open; the shared channel sets `grpc.keepalive_time_ms: 60000` and `grpc.keepalive_timeout_ms: 60000`.
feat(bigquery): read Arrow results from explicit destinations googleapis/google-cloud-python
Queries with an explicit `destination` table previously failed on the job's default stream; they now read the table through a single-stream Arrow read session with the same serialization options, and the `RowIterator` from `query_and_wait()` is marked accordingly.
fix(client): skip mTLS URL rewriting for multi-regional .rep.googleapis.com endpoints and honor GOOGLE_API_USE_MTLS_ENDPOINT in Google Auth sessions googleapis/python-genai
Multi-regional endpoints `aiplatform.us.rep.googleapis.com` and `aiplatform.eu.rep.googleapis.com` publish no `.rep.mtls.googleapis.com` DNS records, so `to_mtls_url()` rewriting made requests fail when `GOOGLE_API_USE_CLIENT_CERTIFICATE=true`, such as on Vertex AI Agent Engine with `AGENT_IDENTITY=true`. The client now skips that rewrite and honors `GOOGLE_API_USE_MTLS_ENDPOINT` in Google Auth...
fix(bigtable): preserve resource labels on batcher metrics googleapis/google-cloud-python
The mutations batcher built operation metrics through the metrics controller directly, bypassing the labels from the table or authorized view, so the missing project label made the exporter drop the metrics and warn on every export. Metrics now go through the target's `_create_operation` helper and still start before waiting for flow control.