Andy S. Lou comments

Repositories
Issues
Comments

Results 2 comments of


                                            Andy S. Lou

TFDV on Dataflow getting OOMs frequently

Hi Chris! Do you have the beam stage names where the OOM is occuring? During the combine phase (when calculating statistics), we have a limit on the size of the...

TFDV on Dataflow getting OOMs frequently

> Could you clarify what this does exactly? When we load the recordbatch from the dataset it comes in a specific batch size (typically tfdv.constants.DEFAULT_DESIRED_INPUT_BATCH_SIZE or some batch size specified...