Purvang

Results 39 comments of Purvang

Image semantic segmentation example with tf.data object on multi gpu training

Hi @banasraf Thank you for your reply. I have modified function as below to apply cropping on images and labels. ``` @pipeline_def(batch_size=args.batch_size) def data_pipeline(shard_id): pngs = fn.readers.caffe2( path=os.path.join(args.dataset, "train", "images"),...

Image semantic segmentation example with tf.data object on multi gpu training

@banasraf Is there any additional information that I need to provide? please let me know. Thank you

Image semantic segmentation example with tf.data object on multi gpu training

@banasraf Thank you for your answer. This is the simplest example I am trying to work with. After adding your suggestions, I am still have issue running it without any...

Image semantic segmentation example with tf.data object on multi gpu training

Thank you @JanuszL . I was able to start training with the changes. But there are two scenarios that I am having difficulties with. 1) Training Time: Now after adding...

Image semantic segmentation example with tf.data object on multi gpu training

@JanuszL Thanks for your reply. Using DALI, there are 59 iterations to complete each epoch, when batch size is 50 (assuming per Gpu based on last message) Therefore Global batch...

Image semantic segmentation example with tf.data object on multi gpu training

Sure @JanuszL ``` https://drive.google.com/drive/folders/1iYk1wIRupy34z6bixHdErMX8859lF-UD?usp=sharing ``` This link has both script and as well dataset. Please let me know if there is anything missing. Thank you

Image semantic segmentation example with tf.data object on multi gpu training

Thank you for your reply @JanuszL . With this change, I was able to get correct iterations per batch. I am using 2 x A100 40Gb Nvidia Gpus and with...

Image semantic segmentation example with tf.data object on multi gpu training

@JanuszL Thank you for your reply. After trial and error, I have limited each Gpu memory to 4GB for tensorflow out of 40GB gpu memory, when I use DALI data...

Image semantic segmentation example with tf.data object on multi gpu training

Thank you @JanuszL . First of all, based on your previous reply about tensorflow competing gpu memory, ``` tf.config.experimental.set_memory_growth(gpu, True) ``` was already present and I changed it to limit...

Image semantic segmentation example with tf.data object on multi gpu training

@JanuszL . Thank you for the answer. point noted regarding the image format. ``` https://drive.google.com/drive/folders/1iYk1wIRupy34z6bixHdErMX8859lF-UD?usp=sharing ``` This gdrive location contains both scripts, one with DALI and other one with tf.data....