seirasto
seirasto
@albertvillanova We were finally able to process the full NQ dataset on our machines using 600 gb with 5 workers. Maybe these numbers will work for you as well.
I asked my colleague who ran the code and he said apache beam.
@albertvillanova Since we have already processed the NQ dataset on our machines can we upload it to datasets so the NQ PR can be merged?
> I asked my colleague who ran the code and he said apache beam. He looked into it further and he just used DirectRunner. @albertvillanova