google-research
sofima.warp.render.tiles raise cv2.error: OpenCV(4.7.0) (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'
Following the stitching example for Sofima, the stitching is fine for small number of tiles, however I have 1631 EM tiles for stitching using the following tile_id_map
id_map = np.arange(1632).reshape(34, -1)
tile_id_map = id_map.copy()
for i in range(id_map.shape[0]):
if i % 2 == 1:
tile_id_map[i, :] = tile_id_map[i, :][::-1]
Each tile is down sampled to about 5 MB
def read_image(image_path, x, y):
f = open(image_path, 'rb')
image_data = f.read()
image = PIL.Image.open(io.BytesIO(image_data)).convert("L")
image_np = np.array(image)
image.close()
print(f"Read image {os.path.basename(image_path)} \r", end=" ")
return x, y, image_np
with futures.ThreadPoolExecutor(max_workers=20) as executor:
for y in range(tile_id_map.shape[0]):
for x in range(tile_id_map.shape[1]):
tile_id = tile_id_map[y, x]
image_path = os.path.join(args.folder, f"down_sampled_Stitched_{tile_id:0>4}.tiff")
f = executor.submit(read_image, image_path, x, y)
future_to_tile_id.update({f: tile_id})
for f in futures.as_completed(future_to_tile_id):
try:
x, y, image_np = f.result()
tile_map[(x, y)] = image_np
except Exception as exc:
print(f"Generated Execption: {exc}")
The rest of the stitching code is the same as Sofima stitching example using tile_space = (tile_id_map.shape[0],tile_id_map.shape[1]) and cx, cy = stitch_rigid.compute_coarse_offsets(tile_space, tile_map, ((1500, 500),(1500, 500)))
However, warp_subvolume raise error when trying to warp the tile using sofima.warp.render.tiles .
opencv version is 4.7
File "/home/youssef/workspace/sofima_trials/./stitchsupertiles.py", line 289, in <module>
# Warp the tiles into a single image.
File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 517, in render_tiles
_render_tile(tile_x=x, tile_y=y, coord_map=coord_map)
File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 465, in _render_tile
warped_img, warped_mask = warp_subvolume(
File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 182, in warp_subvolume
f.result()
File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 169, in _warp_section
warped[c, z, ...] = cvx2.remap(
cv2.error: OpenCV(4.7.0) /home/conda/feedstock_root/build_artifacts/libopencv_1678824110780/work/modules/imgproc/src/imgwarp.cpp:1724: error: (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'
There are several issues concerning opencv remap() dealing with large images. I am not sure if further down sampling of tiles would solve the issue and whether it is possible to pickle the result tile meshes for down sampled tiles and re-apply for large images.
Thanks in advance.
OpenCV version 4.9.0.80
I'm also experiencing this issue, with 3D alignment of a single large volume of 54 sections, i.e.
data_2_warp.shape
(34039, 28719, 54, 1)
I'm warping section for section:
warped = [np.transpose(data_2_warp[:, :, 0:1, 0], [2, 1, 0])]
for z in tqdm(range(1, data_2_warp.shape[2])):
data_box = bounding_box.BoundingBox(start=(0, 0, 0), size=(data_2_warp.shape[0], data_2_warp.shape[1], 1))
out_box = bounding_box.BoundingBox(start=(0, 0, 0), size=(data_2_warp.shape[0], data_2_warp.shape[1], 1))
data = np.transpose(data_2_warp[data_box.start[0]:data_box.end[0],
data_box.start[1]:data_box.end[1],
z:z+1, 0:1], [3, 2, 1, 0])
warped.append(
warp.warp_subvolume(data, data_box, inv_map_full[:, z:z+1, ...], box_high, stride, out_box, 'lanczos', parallelism=1)[0, ...])
Warping a subvolume (e.g. 2000x2000x54) of this array works fine and the result is as expected. But applying the warping to the full data_2_warp array gives the exact same error after about 5 minutes, for the first iteration of the loop:
---------------------------------------------------------------------------
error Traceback (most recent call last)
Cell In[155], line 11
5 out_box = bounding_box.BoundingBox(start=(0, 0, 0), size=(data_2_warp.shape[0], data_2_warp.shape[1], 1))
7 data = np.transpose(data_2_warp[data_box.start[0]:data_box.end[0],
8 data_box.start[1]:data_box.end[1],
9 z:z+1, 0:1], [3, 2, 1, 0])
10 warped.append(
---> 11 warp.warp_subvolume(data, data_box, inv_map_full[:, z:z+1, ...], box_high, stride, out_box, 'lanczos', parallelism=1)[0, ...])
File ~/miniconda3/envs/sofima/lib/python3.9/site-packages/sofima/warp.py:176, in warp_subvolume(image, image_box, coord_map, map_box, stride, out_box, interpolation, offset, parallelism)
173 fs.add(exc.submit(_warp_section, z=z))
175 for f in futures.as_completed(fs):
--> 176 f.result()
178 # Map IDs back to the original space, which might be beyond the range of
179 # int32.
180 if orig_to_low is not None:
File ~/miniconda3/envs/sofima/lib/python3.9/concurrent/futures/_base.py:439, in Future.result(self, timeout)
437 raise CancelledError()
438 elif self._state == FINISHED:
--> 439 return self.__get_result()
441 self._condition.wait(timeout)
443 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
File ~/miniconda3/envs/sofima/lib/python3.9/concurrent/futures/_base.py:391, in Future.__get_result(self)
389 if self._exception:
390 try:
--> 391 raise self._exception
392 finally:
393 # Break a reference cycle with the exception in self._exception
394 self = None
File ~/miniconda3/envs/sofima/lib/python3.9/concurrent/futures/thread.py:58, in _WorkItem.run(self)
55 return
57 try:
---> 58 result = self.fn(*self.args, **self.kwargs)
59 except BaseException as exc:
60 self.future.set_exception(exc)
File ~/miniconda3/envs/sofima/lib/python3.9/site-packages/sofima/warp.py:163, in warp_subvolume.<locals>._warp_section(z)
155 dx, dy = cv.convertMaps(
156 dx,
157 dy,
158 dstmap1type=maptype,
159 nninterpolation=(interpolation == cv.INTER_NEAREST),
160 )
162 for c in range(0, image.shape[0]):
--> 163 warped[c, z, ...] = cv.remap(
164 image[c, z, ...], dx, dy, interpolation=interpolation
165 )
error: OpenCV(4.9.0) /io/opencv/modules/imgproc/src/imgwarp.cpp:1744: error: (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'
Would a workaround be to use the distributed mesh optimization? Unfortunately I am running into some other error there.
After doing some Googling, I found the following on the openCV forum: https://forum.opencv.org/t/about-shrt-max-in-function-cv-remap/2884
It appears that imwarp.cpp uses (short) integer arithmetics for performance reasons, so the image cannot be larger than 0x7fff (32767 pixels). It seems that this is an unchangeable parameter.
I tried cropping the x dimension of my data array to 32767-1 before warping (setting it to 32767 still made it fail), since all other dimensions are within the limit.
When doing this, it doesn't throw the error on my side. So it seems that there is a hard limit of 32766 pixels on each dimension of your image array.
The best workaround here is likely to simply work with smaller image fragments, even if the sections themselves are very large. Mesh and flow processing can still be done as usual, but the rendering of the warped data should be done tile-wise (with user-defined tiles -- something like 1k^2 or 2k^2 pixels would be typical). In the em_alignment notebook, rendering is done section-wise, but warp.warp_subvolume can process arbitrary bounding boxes.
