Thanks a lot for sharing the code! I want to know how to use your code to realize dynamic video training and rendering just like the video below.
Now I can only render the free view video with the object still, just like the first video on your project page.
When I try to execute python train.py --base_folder $EXPERIMENT_PATH --gin_bindings="data_dir='$DATASET_PATH'" --gin_configs configs/hypernerf_interp_ds_2d.gin
I0925 16:29:32.415290 140117065011584 train.py:135] *** Starting experiment
I0925 16:29:32.415382 140117065011584 train.py:139] *** Loading Gin configs from: ['configs/hypernerf_interp_ds_2d.gin']
I0925 16:29:32.430606 140117065011584 xla_bridge.py:236] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0925 16:29:32.494084 140117065011584 xla_bridge.py:236] Unable to initialize backend 'tpu': INVALID_ARGUMENT: TpuPlatform is not available.
I0925 16:29:32.494200 140117065011584 train.py:159] exp_dir = /data/TinyMe/chickchickenout
I0925 16:29:32.494389 140117065011584 train.py:163] summary_dir = /data/TinyMe/chickchickenout/summaries/train
I0925 16:29:32.494438 140117065011584 train.py:167] checkpoint_dir = /data/TinyMe/chickchickenout/checkpoints
I0925 16:29:32.494497 140117065011584 train.py:171] Starting process 0. There are 1 processes.
I0925 16:29:32.494542 140117065011584 train.py:173] Found 1 accelerator devices: [GpuDevice(id=0, process_index=0)].
I0925 16:29:32.494578 140117065011584 train.py:175] Found 1 total devices: [GpuDevice(id=0, process_index=0)].
I0925 16:29:32.698378 140117065011584 train.py:187] Creating datasource
I0925 16:29:32.698573 140117065011584 interp.py:71] *** Loading dataset IDs from /data/TinyMe/chickchicken/dataset.json
I0925 16:29:32.699014 140117065011584 core.py:237] Creating datasource of type InterpDataSource with use_appearance_id=True, use_camera_id=False, use_warp_id=True, use_depth=False, use_time=False, train_stride=1, val_stride=1
I0925 16:29:32.699630 140117065011584 train.py:200] Initializing models.
2022-09-25 16:29:45.751242: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2085] Execution of replica 0 failed: INTERNAL: Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
File "train.py", line 370, in
app.run(main)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 203, in main
model, params['model'] = models.construct_nerf(
File "/data/TinyMe/hypernerf/hypernerf/models.py", line 701, in construct_nerf
params = model.init({
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 998, in init
_, v_out = self.init_with_output(
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 968, in init_with_output
return self.apply(
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 936, in apply
return apply(
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/core/scope.py", line 687, in wrapper
y = fn(root, *args, **kwargs)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 1178, in scope_fn
return fn(module.clone(parent=scope), *args, **kwargs)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 275, in wrapped_module_method
y = fun(self, *args, **kwargs)
File "/data/TinyMe/hypernerf/hypernerf/models.py", line 631, in call
z_vals, points = model_utils.sample_pdf(
File "/data/TinyMe/hypernerf/hypernerf/model_utils.py", line 216, in sample_pdf
z_samples = piecewise_constant_pdf(key, bins, weights, num_coarse_samples,
File "/data/TinyMe/hypernerf/hypernerf/model_utils.py", line 184, in piecewise_constant_pdf
bins_g0, bins_g1 = minmax(bins)
File "/data/TinyMe/hypernerf/hypernerf/model_utils.py", line 179, in minmax
x1 = jnp.min(jnp.where(~mask, x[..., None], x[..., -1:, None]), -2)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
return fun(*args, **kwargs)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/_src/api.py", line 412, in cache_miss
out_flat = xla.xla_call(
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/core.py", line 1616, in bind
return call_bind(self, fun, *args, **params)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/core.py", line 1607, in call_bind
outs = primitive.process(top_trace, fun, tracers, params)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/core.py", line 1619, in process
return trace.process_call(self, fun, tracers, params)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/core.py", line 615, in process_call
return primitive.impl(f, *tracers, **params)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/interpreters/xla.py", line 623, in _xla_call_impl
out = compiled_fun(*args)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/interpreters/xla.py", line 913, in _execute_compiled
out_bufs = compiled.execute(input_bufs)
jax._src.traceback_util.UnfilteredStackTrace: RuntimeError: INTERNAL: Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "train.py", line 370, in
app.run(main)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 203, in main
model, params['model'] = models.construct_nerf(
File "/data/TinyMe/hypernerf/hypernerf/models.py", line 701, in construct_nerf
params = model.init({
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 998, in init
_, v_out = self.init_with_output(
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 968, in init_with_output
return self.apply(
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 936, in apply
return apply(
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/core/scope.py", line 687, in wrapper
y = fn(root, *args, **kwargs)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 1178, in scope_fn
return fn(module.clone(parent=scope), *args, **kwargs)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/flax/linen/module.py", line 275, in wrapped_module_method
y = fun(self, *args, **kwargs)
File "/data/TinyMe/hypernerf/hypernerf/models.py", line 631, in call
z_vals, points = model_utils.sample_pdf(
File "/data/TinyMe/hypernerf/hypernerf/model_utils.py", line 216, in sample_pdf
z_samples = piecewise_constant_pdf(key, bins, weights, num_coarse_samples,
File "/data/TinyMe/hypernerf/hypernerf/model_utils.py", line 184, in piecewise_constant_pdf
bins_g0, bins_g1 = minmax(bins)
File "/data/TinyMe/hypernerf/hypernerf/model_utils.py", line 179, in minmax
x1 = jnp.min(jnp.where(~mask, x[..., None], x[..., -1:, None]), -2)
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/site-packages/jax/interpreters/xla.py", line 913, in _execute_compiled
out_bufs = compiled.execute(input_bufs)
RuntimeError: INTERNAL: Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2022-09-25 16:29:45.849669: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:1047] could not synchronize on CUDA context: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered :: *** Begin stack trace ***
_PyDict_DelItem_KnownHash
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
PyObject_CallFunctionObjArgs
PyObject_ClearWeakRefs
_PyModule_ClearDict
PyImport_Cleanup
Py_FinalizeEx
Py_RunMain
Py_BytesMain
__libc_start_main
*** End stack trace ***
2022-09-25 16:29:45.849777: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_executable.cc:124] Check failed: pair.first->SynchronizeAllActivity()
Fatal Python error: Aborted
Current thread 0x00007f6f8be31180 (most recent call first):
File "/home/hy/miniconda3/envs/hypernerf/lib/python3.8/weakref.py", line 351 in remove
Aborted (core dumped)
And in sdf_2d.ipynb, it seems that the definition of the hyper_encoder module is missing.
thanks for your helping!
https://user-images.githubusercontent.com/54303737/192090214-c22ff504-50c5-491c-8c7c-62436219591b.mp4
google
