TensorRT icon indicating copy to clipboard operation
TensorRT copied to clipboard
verified publisher icon pytorch

❓ [Question] Jetson AGX Orin build and install torch_tensorrt wheel file Failed

Open breknddone opened this issue 1 year ago • 5 comments

❓ Question

I follow this tutorial to install Torch-TensorRT, but in the last step:

cuda_version=$(nvcc --version | grep Cuda | grep release | cut -d ',' -f 2 | sed -e 's/ release //g')
export TORCH_INSTALL_PATH=$(python -c "import torch, os; print(os.path.dirname(torch.__file__))")
export SITE_PACKAGE_PATH=${TORCH_INSTALL_PATH::-6}
export CUDA_HOME=/usr/local/cuda-${cuda_version}/
# replace the MODULE.bazel with the jetpack one
cat toolchains/jp_workspaces/MODULE.bazel.tmpl | envsubst > MODULE.bazel
# build and install torch_tensorrt wheel file
python setup.py --use-cxx11-abi install --user

some errors happened:

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
    LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
    PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/debug.out -- /bin/sh -i)
ERROR: /home/lab223/TensorRT/core/conversion/var/BUILD:20:11: Compiling core/conversion/var/Var.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/58/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/core/DeprecatedTypePropertiesRegistry.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:59:11: Compiling core/conversion/converters/NodeConverterRegistry.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/57/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/ops/cudnn_batch_norm_ops.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:39:11: Compiling core/conversion/converters/converter_util.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/56/execroot/_main/external/_main~_repo_rules~libtorch/include/ATen/ops/native_dropout_backward_cpu_dispatch.h (???????)
Target //:libtorchtrt failed to build
INFO: Elapsed time: 1000.299s, Critical Path: 574.06s
INFO: 7984 processes: 7938 internal, 46 linux-sandbox.
ERROR: Build did NOT complete successfully

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • PyTorch Version (e.g., 1.0): 2.5.0
  • CPU Architecture: arm64(Jetson AGX Orin)
  • OS (e.g., Linux): Linux
  • How you installed PyTorch: pip
  • Build command you used (if compiling from source): python setup.py --use-cxx11-abi install --user
  • Are you using local sources or building from archives: building from archives
  • Python version: 3.10.15
  • CUDA version: 12.6
  • GPU models and configuration: -
  • Any other relevant information: Install torch_tensorrt in the model's anaconda virtual environment

Additional context

It seems a I/O exception.But Jetson still has 11GB of space.please help me!thanks!!!!

breknddone avatar Dec 18 '24 18:12 breknddone

Can you change the bazel build command in setup.py to add --sandbox-debug and --verbose_failures and share the output? This message does not convey that much information

narendasan avatar Dec 18 '24 19:12 narendasan

Can you change the bazel build command in setup.py to add --sandbox-debug and --verbose_failures and share the output? This message does not convey that much information

sure.

DEBUG: Sandbox debug output for CppCompile 
//core/conversion/tensorcontainer:tensorcontainer:
1734547440.188853798: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1734547440.188938086: src/main/tools/linux-sandbox.cc:165: Netns is 0
1734547440.188945350: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1734547440.189515846: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 13275
1734547440.190429253: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1734547440.190705893: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1734547440.193842786: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/_hermetic_tmp -> /tmp
1734547440.199941085: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1734547440.199974141: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.199992253: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1734547440.200001981: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.200251197: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1734547440.200269181: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1734547440.200277373: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547440.200297469: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1734547440.200308765: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547440.200315773: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1734547440.200322333: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1734547440.200336125: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1734547440.200346397: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1734547440.200355933: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1734547440.200364477: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1734547440.200379421: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1734547440.200387037: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1734547440.200395357: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1734547440.200404892: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1734547440.200475804: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1734547440.200488412: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1734547440.200494492: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547440.200501532: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1734547440.200507708: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1734547440.200513532: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1734547440.200520348: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1734547440.200526172: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1734547440.200532700: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1734547440.200540124: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1734547440.200549500: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1734547440.200556764: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1734547440.200563388: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1734547440.200615580: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1734547440.200658108: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1734547440.200669852: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1734547440.200697052: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1734547440.200705468: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1734547440.200711548: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1734547440.200719836: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1734547440.200728252: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1734547440.200735516: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1734547440.200742172: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1734547440.200752508: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/bszk
1734547440.200776476: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1734547440.200784572: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547440.200791356: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547440.200798428: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.200808636: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547440.200815292: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.200945436: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1734547440.201218492: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1734547597.851306145: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x00
1734547597.852080283: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 0
1734547597.853295897: src/main/tools/linux-sandbox.cc:243: child exited normally with code 0

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/debug.out -- /bin/sh -i)
DEBUG: Sandbox debug output for CppCompile //core/ir:ir:
1734547666.568098834: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1734547666.568173129: src/main/tools/linux-sandbox.cc:165: Netns is 0
1734547666.568179816: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1734547666.568852633: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 13473
1734547666.572746858: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1734547666.572907127: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1734547666.573341860: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/_hermetic_tmp -> /tmp
1734547666.573399581: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1734547666.573421658: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.573440472: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1734547666.573451735: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.573617731: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1734547666.573633249: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1734547666.573641088: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547666.573661630: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1734547666.573672317: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547666.573680380: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1734547666.573686715: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1734547666.573699865: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1734547666.573710360: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1734547666.573720823: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1734547666.573729430: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1734547666.573743604: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1734547666.573751603: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1734547666.573759186: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1734547666.573767537: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1734547666.573834313: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1734547666.573846952: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1734547666.573852807: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547666.573861798: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1734547666.573869829: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1734547666.573877348: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1734547666.573884803: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1734547666.573891043: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1734547666.573900098: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1734547666.573908577: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1734547666.573918015: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1734547666.573927998: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1734547666.573938717: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1734547666.573989751: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1734547666.574002453: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1734547666.574012948: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1734547666.574039409: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1734547666.574047952: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1734547666.574054671: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1734547666.574063694: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1734547666.574073197: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1734547666.574080940: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1734547666.574089643: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1734547666.574101226: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/bszk
1734547666.574123239: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1734547666.574131366: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547666.574138469: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547666.574144613: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.574155491: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547666.574162786: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.574342157: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1734547666.574685476: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1734547783.023817798: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x00
1734547783.023855490: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 0
1734547783.024807844: src/main/tools/linux-sandbox.cc:243: child exited normally with code 0

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/debug.out -- /bin/sh -i)
DEBUG: Sandbox debug output for CppCompile //core/ir:ir:
1734547645.822595679: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1734547645.822668214: src/main/tools/linux-sandbox.cc:165: Netns is 0
1734547645.822675189: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1734547645.823208308: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 13464
1734547645.823293929: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1734547645.823395485: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1734547645.823736787: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/_hermetic_tmp -> /tmp
1734547645.823784685: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1734547645.823803563: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.823821352: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1734547645.823832647: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.823980661: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1734547645.823995507: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1734547645.824004946: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547645.824025039: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1734547645.824034862: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547645.824042445: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1734547645.824049004: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1734547645.824061579: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1734547645.824074857: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1734547645.824083368: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1734547645.824091783: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1734547645.824106341: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1734547645.824114308: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1734547645.824121955: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1734547645.824130594: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1734547645.824195066: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1734547645.824206905: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1734547645.824213208: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547645.824221271: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1734547645.824228150: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1734547645.824236245: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1734547645.824242933: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1734547645.824249364: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1734547645.824257395: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1734547645.824265874: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1734547645.824275441: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1734547645.824283664: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1734547645.824292655: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1734547645.824338345: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1734547645.824350407: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1734547645.824359846: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1734547645.824384707: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1734547645.824393186: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1734547645.824399777: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1734547645.824406497: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1734547645.824414048: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1734547645.824421855: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1734547645.824428958: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1734547645.824440700: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/bszk
1734547645.824459642: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1734547645.824468121: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547645.824474456: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547645.824482199: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.824493078: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547645.824499701: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.824679135: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1734547645.824927585: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1734547798.097325437: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x00
1734547798.097386295: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 0
1734547798.098490380: src/main/tools/linux-sandbox.cc:243: child exited normally with code 0

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/debug.out -- /bin/sh -i)
ERROR: /home/lab223/TensorRT/core/conversion/var/BUILD:20:11: Compiling core/conversion/var/Var.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/58/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/core/DeprecatedTypePropertiesRegistry.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:59:11: Compiling core/conversion/converters/NodeConverterRegistry.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/57/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/ops/cudnn_batch_norm_ops.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:39:11: Compiling core/conversion/converters/converter_util.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/56/execroot/_main/external/_main~_repo_rules~libtorch/include/ATen/ops/native_dropout_backward_cpu_dispatch.h (???????)
Target //:libtorchtrt failed to build
INFO: Elapsed time: 1000.299s, Critical Path: 574.06s
INFO: 7984 processes: 7938 internal, 46 linux-sandbox.
ERROR: Build did NOT complete successfully

breknddone avatar Dec 18 '24 19:12 breknddone

Hmm, how did you install bazel and can you share bazel/bazelisk version numbers?

narendasan avatar Dec 18 '24 20:12 narendasan

I use this to install bazel：

wget -v https://github.com/bazelbuild/bazelisk/releases/download/v1.20.0/bazelisk-linux-arm64
sudo mv bazelisk-linux-arm64 /usr/bin/bazel
chmod +x /usr/bin/bazel

Hmm, how did you install bazel and can you share bazel/bazelisk version numbers?

breknddone avatar Dec 18 '24 20:12 breknddone

Can you quickly try bazelisk v1.25.0, we saw some issues with earlier bazelisk versions https://github.com/pytorch/TensorRT/pull/3328

narendasan avatar Dec 18 '24 20:12 narendasan