A mpi script for calibration

[zrxdaly]

Is your feature request related to a problem? Please describe. I guess some may also need this to calibrate tons of files, I tested a script for mpi calibration.

1. install mpi4py: https://mpi4py.readthedocs.io/en/latest/install.html
2. code example mainly from here https://gist.github.com/joezuntz/7c590a8652a1da6dc4c9:

import dtscalibration
from mpi4py import MPI
...
comm = MPI.COMM_WORLD
rank = MPI.COMM_WORLD.Get_rank()
size = MPI.COMM_WORLD.Get_size()

def cali_fun(file):
    # calibration function

comm.Barrier()

for i, file in enumerate(grouped_files):
    if i%size!=rank: continue
    print("Task number %d being done by processor %d of %d" % (i, rank, size))
    cali_fun(file)

[BSchilperoort]

Hi Dai, is this script just an example for how other people can do this?

Python also has the built-in multiprocessing module available, in case you just want to split up the jobs on your own machine.

[zrxdaly]

Hi Dai, is this script just an example for how other people can do this?

Yes, just thought it would be nice to share it here.

Python also has the built-in multiprocessing module available, in case you just want to split up the jobs on your own machine.

Oh nice, just tested it, multiprocessing module is also very handy to use for local machine. Since the data is quite a lot, I use mpi4py in Defltblue.