io
io copied to clipboard
When avrorecorddataset is used to read HDFS data, the speed will be very slow when about 2% of the data remains in each round of training
I use tensorflow IO version 0.19.1, the batch size of samples is set to 1024, the read parallelism is set to 36, and the total number of steps 147000 and 6000 are both tested. When about 2% data is left, the training speed becomes very slow. At the beginning, it takes about 45ms for a step. At the end, it takes more than 400 ms for a step. What is the mechanism of this problem? Is there any version that solves this problem? Or how to optimize this problem for the time being? Thanks!