NVlabs
Input point cloud has no data!
尊敬的wen 我进行自建数据纹理生成出现点云数据加载错误，具体问题如下： (BundleSDF) a@robot:~/project/BundleSDF$ python run_custom.py --mode run_video --video_dir /data/dataset/output --out_folder /data/dataset/bundlesdf_output --use_segmenter 1 --use_gui 1 --debug_level 2 Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=19, releaselevel='final', serial=0) Warp 1.3.3 initialized: CUDA Toolkit 12.5, Driver 12.4 Devices: "cpu" : "x86_64" "cuda:0" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) "cuda:1" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) CUDA peer access: Supported fully (all-directional) Kernel cache: /home/a/.cache/warp/1.3.3 Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=19, releaselevel='final', serial=0) Warp 1.3.3 initialized: CUDA Toolkit 12.5, Driver 12.4 Devices: "cpu" : "x86_64" "cuda:0" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) "cuda:1" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) CUDA peer access: Supported fully (all-directional) Kernel cache: /home/a/.cache/warp/1.3.3 [2024-09-26 14:45:26.256] [warning] [Bundler.cpp:49] Connected to nerf_port 9999 [2024-09-26 14:45:26.257] [warning] [FeatureManager.cpp:2084] Connected to port 5555 default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}} Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=19, releaselevel='final', serial=0) Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=19, releaselevel='final', serial=0) Warp 1.3.3 initialized: CUDA Toolkit 12.5, Driver 12.4 Devices: "cpu" : "x86_64" "cuda:0" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) "cuda:1" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) CUDA peer access: Supported fully (all-directional) Kernel cache: /home/a/.cache/warp/1.3.3 Warp 1.3.3 initialized: CUDA Toolkit 12.5, Driver 12.4 Devices: "cpu" : "x86_64" "cuda:0" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) "cuda:1" : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled) CUDA peer access: Supported fully (all-directional) Kernel cache: /home/a/.cache/warp/1.3.3 GUI started [bundlesdf.py] percentile denoise start [bundlesdf.py] percentile denoise done [bundlesdf.py] processNewFrame start 000000 [bundlesdf.py] process frame 000000 [pcl::PLYWriter::writeASCII] Input point cloud has no data! [pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud! [pcl::PLYWriter::writeASCII] Input point cloud has no data! 段错误 (核心已转储)
请问这是由什么原因造成的呢？ Best wishes!
Hi, @wenbowen123, a great and useful work. I am also getting the same error. Please help. I am sharing a few details to provide more info about the situation.
- Here are my unique values in the depth array w.r.t. your previous question.
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826, 1.836, 1.846, 1.856, 1.866, 1.876, 1.886, 1.897, 1.907,
1.918, 1.929, 1.939, 1.95 , 1.961, 1.973, 1.984, 1.996, 2.007,
2.019, 2.03 , 2.043, 2.055, 2.068, 2.08 , 2.093, 2.105, 2.118,
2.132, 2.145, 2.158, 2.172, 2.186, 2.2 , 2.214, 2.228, 2.243])
- I am capturing RGB, depth from a Fetch Robot. Below is the logic for depth
if depth.encoding == "32FC1":
depth_cv = self.cv_bridge.imgmsg_to_cv2(depth)
depth_cv = np.array(depth_cv)
depth_cv[np.isnan(depth_cv)] = 0
depth_cv = depth_cv * 1000
depth_cv = depth_cv.astype(np.uint16)
- This is how I run it:
python run_custom.py --mode run_video --video_dir ../data/test_obj_pose --out_folder ../data/test_obj_pose/out/bundlesdf --use_segmenter 0 --use_gui 1 --debug_level 2
-
I create masks using SAMv2
-
❌ Here's a video of the run as well.
https://github.com/user-attachments/assets/75468ae1-79b7-4555-b926-ecedcdce23f5
-
✅ The same setup runs with the example milk bottle data.
- Here's the proof.
- First run uses
ipdbto see if everything runs. - The second run is without
ipdband runs smoothly with GUI.
- First run uses
- Here's the proof.
https://github.com/user-attachments/assets/3f6b2040-b51c-42c7-b9ee-aaa8834bfd15
@jishnujayakumar at which step did you print the depth? It'd be great to keep track of the depth from here , and see at which step it become all zero. Then I can advise more.
Thanks for the reply, @wenbowen123. Following is the depth trace after line#531. Seems it's not zero. A simple difference between milk example and my data is that the depth values in the former is between 0-1 while mine goes beyond 1. Not sure if this should be an issue as I have raw depth values and as I understand BundleSDF should work with raw depth values. Please correct if I am wrong.
> /home/jishnu/Projects/mm-demo/vie/BundleSDF/bundlesdf.py(539)run()
538
--> 539 thres = np.percentile(depth[valid], percentile)
540 depth[depth>=thres] = 0
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826, 1.836, 1.846, 1.856, 1.866, 1.876, 1.886, 1.897, 1.907,
1.918, 1.929, 1.939, 1.95 , 1.961, 1.973, 1.984, 1.996, 2.007,
2.019, 2.03 , 2.043, 2.055, 2.068, 2.08 , 2.093, 2.105, 2.118,
2.132, 2.145, 2.158, 2.172, 2.186, 2.2 , 2.214, 2.228, 2.243])
> /home/jishnu/Projects/mm-demo/vie/BundleSDF/bundlesdf.py(540)run()
539 thres = np.percentile(depth[valid], percentile)
--> 540 depth[depth>=thres] = 0
541 logging.info("percentile denoise done")
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826, 1.836, 1.846, 1.856, 1.866, 1.876, 1.886, 1.897, 1.907,
1.918, 1.929, 1.939, 1.95 , 1.961, 1.973, 1.984, 1.996, 2.007,
2.019, 2.03 , 2.043, 2.055, 2.068, 2.08 , 2.093, 2.105, 2.118,
2.132, 2.145, 2.158, 2.172, 2.186, 2.2 , 2.214, 2.228, 2.243])
> /home/jishnu/Projects/mm-demo/vie/BundleSDF/bundlesdf.py(541)run()
540 depth[depth>=thres] = 0
--> 541 logging.info("percentile denoise done")
542
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826])
[bundlesdf.py] percentile denoise done
> /home/jishnu/Projects/mm-demo/vie/BundleSDF/bundlesdf.py(543)run()
542
--> 543 frame = self.make_frame(color, depth, K, id_str, mask, occ_mask, pose_in_model)
544 os.makedirs(f"{self.debug_dir}/{frame._id_str}", exist_ok=True)
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826])
> /home/jishnu/Projects/mm-demo/vie/BundleSDF/bundlesdf.py(544)run()
543 frame = self.make_frame(color, depth, K, id_str, mask, occ_mask, pose_in_model)
--> 544 os.makedirs(f"{self.debug_dir}/{frame._id_str}", exist_ok=True)
545
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826])
ipdb> frame
<my_cpp.Frame object at 0x7fef343bbdf0>
ipdb> n
> /home/jishnu/Projects/mm-demo/vie/BundleSDF/bundlesdf.py(546)run()
545
--> 546 logging.info(f"processNewFrame start {frame._id_str}")
547 # self.bundler.processNewFrame(frame)
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826])
[bundlesdf.py] processNewFrame start 000000
> /home/jishnu/Projects/mm-demo/vie/BundleSDF/bundlesdf.py(548)run()
547 # self.bundler.processNewFrame(frame)
--> 548 self.process_new_frame(frame)
549 logging.info(f"processNewFrame done {frame._id_str}")
ipdb> np.unique(depth)
array([0. , 1.425, 1.431, 1.437, 1.443, 1.449, 1.456, 1.462, 1.468,
1.474, 1.481, 1.487, 1.493, 1.5 , 1.507, 1.513, 1.52 , 1.527,
1.533, 1.54 , 1.547, 1.554, 1.561, 1.568, 1.575, 1.583, 1.59 ,
1.597, 1.605, 1.612, 1.62 , 1.628, 1.635, 1.643, 1.651, 1.659,
1.667, 1.675, 1.683, 1.692, 1.7 , 1.708, 1.717, 1.726, 1.734,
1.743, 1.752, 1.761, 1.77 , 1.779, 1.788, 1.798, 1.807, 1.817,
1.826])
ipdb> n
[bundlesdf.py] process frame 000000
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
[pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud!
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
Segmentation fault (core dumped)
Seems the problem was not normalizing depth. After normalizing as follows the pipeline works. @wenbowen123, please let me know if this is the correct way.
percentile = self.cfg_track['depth_processing']["percentile"]
if percentile<100: # Denoise
logging.info("percentile denoise start")
valid = (depth>=0.1) & (mask>0)
thres = np.percentile(depth[valid], percentile)
depth[depth>=thres] = 0
# normalize the depth values between 0 and 1
min_depth, max_depth = np.min(depth), np.max(depth)
depth = (depth - min_depth) / (max_depth - min_depth)
logging.info("percentile denoise done")
ipdb> np.unique(depth)
array([0. , 0.7803943 , 0.78368018, 0.78696605, 0.79025192,
0.79353779, 0.7973713 , 0.80065717, 0.80394304, 0.80722892,
0.81106243, 0.8143483 , 0.81763417, 0.82146769, 0.8253012 ,
0.82858708, 0.83242059, 0.83625411, 0.83953998, 0.84337349,
0.84720701, 0.85104053, 0.85487404, 0.85870756, 0.86254107,
0.86692223, 0.87075575, 0.87458927, 0.87897043, 0.88280394,
0.8871851 , 0.89156627, 0.89539978, 0.89978094, 0.9041621 ,
0.90854326, 0.91292442, 0.91730559, 0.92168675, 0.92661555,
0.93099671, 0.93537788, 0.94030668, 0.94523549, 0.94961665,
0.95454545, 0.95947426, 0.96440307, 0.96933187, 0.97426068,
0.97918949, 0.98466594, 0.98959474, 0.99507119, 1. ])
Here's a demo run:
https://github.com/user-attachments/assets/cb1d4be1-d798-41bc-b622-950f826569cc
oh, the default bundlesdf sets a clip value for depth at 1.0. I just updated the code. Please pull again and set the zfar here depending on your max possible depth. Normalizing depth is not the right way.
Thank you for the clarification. It works at my end now.
Here's an example of using my masks with milk data cam_K and my.cam_K.
W.r.t. the output, I have one question. The output poses seem to be visually similar using the two diff cam_K. Ideally, I should be using mine. but here using any works roughly. Have you seen such a behavior? (Do the intrinsic parameters matter?)
https://github.com/user-attachments/assets/d68658cc-a809-45ab-a1cb-c0314f3a733f