Max Imakaev

Cooler zoomify issues summary thread

1

On Open2C there have been several reports about cooler zoomify misbehaving in a non-reproducible fashion. Using this as a meta-thread summarizing our observations. @sergpolly and @Phlya reported that cooler zoomify...

default value instead of np.nans for cooler matrix

2

Many algorithms need a matrix that has simply zeroes for missing bins, not NANs. It would be nice to have a default value for c.matrix(balance=True)

Some multithreading happening inside multiprocessing on Anaconda's python

15

Cooler zoomify was observed to use more than 100% CPU per cooler, suggesting that perhaps MKL multithreading is happening inside cooler. Suggested solution: run this at cooler startup? try: import...

@gspracklin was also going to submit a pull request with trimming.

merge_zoom_library_group_coolers and merge_stats... restart again when distiller is restarted (without --resume at least).

2

I would expect them to be marked as completed.

Download step fails sometimes because ftp server on GEO is very screwed up, and some folders can just disappear. It would be nice to have an option to simply bypass...

raise error if important parameters not provided

1

For example, if assembly_name is missing (e.g. if one tried to re-use the old cooler with assembly parameter), genome name just gets replaced with .null., but distiller finishes.

check inputs: invalid characters in group names, missing libraries in library groups, etc...

3

Distiller does not check (before running) for consistency of library groups or experiments. As such, a misprint in library groups, in a complex experiment, would result in error.

Numpy and Pandas supplied by Anaconda may parallelized by default (MKL, numexpr, etc.). We don't need it because we are running different processes in parallel. It is hard to do...

Will make several things somewhat easier.