How to run microbatches on different gpus?

Open adis98 opened this issue 4 years ago • 3 comments

Hi,

I've been experimenting with GPipe and was wondering if it is possible to run different micro-batches on different GPUS? For example if there are 16 micro-batches, is it possible to run 8 micro-batches on GPU 0 and the other 8 on GPU 1? It would be faster than pipelining all the micro-batches on the same machine right?

Mar 18 '21 04:03 adis98

@jonathanasdf, could you help me with this?

Mar 24 '21 12:03 adis98

Sorry I'm not familiar with GPipe. @bignamehyp ?

Mar 24 '21 21:03 jonathanasdf

@bignamehyp , if the above mentioned scenario is not possible....does lingvo support data-parallelism? i.e, running multiple minibatches in parallel and then aggregating their results? If yes, what are the command line arguments for it?

Mar 24 '21 23:03 adis98