[Feature Request]: BigQuery To Vertex AI Matching Engine Index (via Avros)

[shadiramadan]

Related Template(s)

New template

What feature(s) are you requesting?

https://cloud.google.com/vertex-ai/docs/matching-engine/using-matching-engine

Vertex AI Matching Engine is a newer product offering from Google. It's great- but the data import process in terms of integrating with other Google offerings for data sources could have been better thought out.

We use BigQuery to store all of our vectors/ other data.

Vertex AI Matching Engine supports creating an index from Avros. The one caveat being it only supports FLOAT32 vector data. Which is fine and desired- we don't need FLOAT64 precision. But! BigQuery exports only in FLOAT64- Vertex AI Matching Engine errors if the numbers are FLOAT64.

Our current process is BigQuery -> Avros -> Custom code to rebuild Avro to use FLOAT32 -> Vertex AI Index

It would be nice if there was a DataFlow template to handle this data conversion automatically so I can delete the Avro export and the workaround code I have just for this FLOAT32 vs 64 issue. I opened a feature request to Google for them to just cast the FLOAT64 values to FLOAT32 and not error but it's been 2 months and I'm not sure they have even triaged it.