mercari
The writing is slow
Can you please tell why the library for writing from BigqueryToSpanner is very slow? About 20 records per second. Should I generate template with additional options to improve performance of library/
20 cases per second is certainly slow. In my case, I insert 1 million records in about 10 minutes (total Dataflow job). Do you know where the bottleneck is in the Dataflow step? As a cause of slowness, the write processing on the Spanner side may be a bottleneck. Whether Spanner is a bottleneck can be confirmed by the CPU usage rate of the Spanner console. If the rate is higher than 80% you need to increase the number of instances of Spanner.
Solved! The problem was with incorrect data i was trying to write into spanner (in my case it was a duplicate combined primary key). Now, with correct primary keys, it is loading into spanner about 3 milliions per 8 minutes.