TensorRT icon indicating copy to clipboard operation
TensorRT copied to clipboard
verified publisher icon pytorch

❓ [Question] Jetson AGX Orin Install in Jetpack 6.1 Build did NOT complete successfully

Open breknddone opened this issue 1 year ago • 5 comments

❓ Question

I follow this tutorial to install Torch-TensorRT, but in the last step:

# build and install torch_tensorrt wheel file
python setup.py --use-cxx11-abi install --user

some errors happened:

using CXX11 ABI build
Jetpack version: 6.1
building libtorchtrt cmd=['/usr/bin/bazel', 'build', '//:libtorchtrt', '--compilation_mode=opt', '--distdir=third_party/dist_dir/x86_64-linux-gnu', '--config=linux', '--platforms=//toolchains:jetpack_6.1']
DEBUG: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/external/rules_python~/python/private/python.bzl:46:10: WARNING: Ignoring toolchain 'python_3_11' from module 'rules_pkg': Toolchain 'python_3_11' from module 'torch_tensorrt' already registered Python version 3.11 and has precedence
INFO: Analyzed target //:libtorchtrt (127 packages loaded, 13849 targets configured).
ERROR: /home/lab223/TensorRT/core/util/BUILD:60:11: Compiling core/util/Exception.cpp failed: (Exit 1): gcc failed: error executing CppCompile command (from target //core/util:exception) /home/lab223/anaconda3/envs/rnw/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 25 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
gcc: fatal error: cannot execute 'cc1plus': execvp: No such file or directory
compilation terminated.
Target //:libtorchtrt failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 8.444s, Critical Path: 4.05s
INFO: 329 processes: 329 internal.
ERROR: Build did NOT complete successfully

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • PyTorch Version (e.g., 1.0): 2.5.0
  • CPU Architecture: arm64(Jetson AGX Orin)
  • OS (e.g., Linux): Linux
  • How you installed PyTorch: pip
  • Build command you used (if compiling from source): python setup.py --use-cxx11-abi install --user
  • Are you using local sources or building from archives: building from archives
  • Python version: 3.10.15
  • CUDA version: 12.6
  • GPU models and configuration: -
  • Any other relevant information: Install torch_tensorrt in the model's anaconda virtual environment

Additional context

please help me!thanks!!!!

breknddone avatar Dec 11 '24 09:12 breknddone

Can you provide the results of adding --sandbox_debug and --verbose_failures to the build command in setup.py? Seems like from what is here there is a toolchain issue with g++.

narendasan avatar Dec 11 '24 20:12 narendasan

Can you provide the results of adding --sandbox_debug and --verbose_failures to the build command in setup.py? Seems like from what is here there is a toolchain issue with g++.

I add --sandbox_debug and --verbose_failures to setup.py, and the debug info shows:

using CXX11 ABI build
Jetpack version: 6.1
building libtorchtrt cmd=['/usr/bin/bazel', 'build', '//:libtorchtrt', '--compilation_mode=opt', '--distdir=third_party/dist_dir/x86_64-linux-gnu', '--config=linux', '--platforms=//toolchains:jetpack_6.1', '--sandbox_debug', '--verbose_failures']
INFO: Analyzed target //:libtorchtrt (0 packages loaded, 0 targets configured).
DEBUG: Sandbox debug output for CppCompile //core/util:exception:
1733989836.324773440: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1733989836.324872248: src/main/tools/linux-sandbox.cc:165: Netns is 0
1733989836.324877687: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1733989836.325472613: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 52445
1733989836.325506626: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1733989836.325653557: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1733989836.325868995: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/_hermetic_tmp -> /tmp
1733989836.325927262: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1733989836.325944541: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/execroot/_main
1733989836.325960347: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1733989836.325968923: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/execroot/_main
1733989836.326134893: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1733989836.326149963: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1733989836.326158059: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1733989836.326176361: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1733989836.326184968: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1733989836.326191816: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1733989836.326197703: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1733989836.326210598: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1733989836.326220869: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1733989836.326227653: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1733989836.326236900: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1733989836.326250755: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1733989836.326257762: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1733989836.326264962: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1733989836.326272545: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1733989836.326339195: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1733989836.326350362: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1733989836.326355770: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1733989836.326362713: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1733989836.326368921: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1733989836.326375480: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1733989836.326381368: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1733989836.326386967: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1733989836.326393399: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1733989836.326458641: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1733989836.326478543: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1733989836.326492142: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1733989836.326502797: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1733989836.326508973: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /run/snapd/ns, nullptr, 4135, nullptr) failure (Permission denied) ignored
1733989836.326560169: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1733989836.326601637: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1733989836.326610404: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1733989836.326618244: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1733989836.326624707: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1733989836.326631746: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1733989836.326639298: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1733989836.326646337: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1733989836.326652193: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1733989836.326660000: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1733989836.326666656: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1733989836.326676863: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1733989836.326682750: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1733989836.326693149: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/execroot/_main
1733989836.326707772: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1733989836.326712764: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/execroot/_main
1733989836.326861551: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1733989836.327124665: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1733989836.343405748: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x100
1733989836.343437649: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 1
1733989836.344321766: src/main/tools/linux-sandbox.cc:243: child exited normally with code 1

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/debug.out -- /bin/sh -i)
ERROR: /home/lab223/TensorRT/core/util/BUILD:60:11: Compiling core/util/Exception.cpp failed: (Exit 1): linux-sandbox failed: error executing CppCompile command
  (cd /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/execroot/_main && \
  exec env - \
	LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/4/debug.out -- /home/lab223/anaconda3/envs/rnw/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++14' -MD -MF bazel-out/aarch64-opt/bin/core/util/_objs/exception/Exception.d '-frandom-seed=bazel-out/aarch64-opt/bin/core/util/_objs/exception/Exception.o' -iquote . -iquote bazel-out/aarch64-opt/bin '-std=c++17' '-fdiagnostics-color=always' '-std=c++17' '-fdiagnostics-color=always' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c core/util/Exception.cpp -o bazel-out/aarch64-opt/bin/core/util/_objs/exception/Exception.o)
gcc: fatal error: cannot execute 'cc1plus': execvp: No such file or directory
compilation terminated.
Target //:libtorchtrt failed to build
INFO: Elapsed time: 0.358s, Critical Path: 0.06s
INFO: 4 processes: 4 internal.
ERROR: Build did NOT complete successfully

And my gcc/g++ location and version：

(rnw) lab223@ubuntu:~/TensorRT$ which gcc
/home/lab223/anaconda3/envs/rnw/bin/gcc
(rnw) lab223@ubuntu:~/TensorRT$ which g++
/usr/bin/g++
(rnw) lab223@ubuntu:~/TensorRT$ gcc --version
gcc (conda-forge gcc 12.1.0-17) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(rnw) lab223@ubuntu:~/TensorRT$ g++ --version
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

how can I solve this ?

breknddone avatar Dec 12 '24 07:12 breknddone

this seems relevant since you are using conda https://github.com/pytorch/vision/issues/8540

narendasan avatar Dec 12 '24 21:12 narendasan

this seems relevant since you are using conda pytorch/vision#8540

I follow this method, it seems worked.

conda install -c conda-forge gxx_linux-64
cd /home/ms/anaconda3/envs/NAME/bin
ln -s /home/ms/anaconda3/envs/NAME/bin/aarch64-conda_cos7-linux-gnu-g++ g++

But some new errors happend:

using CXX11 ABI build
Jetpack version: 6.1
building libtorchtrt cmd=['/usr/bin/bazel', 'build', '//:libtorchtrt', '--compilation_mode=opt', '--distdir=third_party/dist_dir/x86_64-linux-gnu', '--config=linux', '--platforms=//toolchains:jetpack_6.1', '--sandbox_debug', '--verbose_failures']
INFO: Analyzed target //:libtorchtrt (0 packages loaded, 0 targets configured).
DEBUG: Sandbox debug output for CppLink //cpp/lib:libtorchtrt_plugins.so:
1734072959.687516534: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1734072959.687671851: src/main/tools/linux-sandbox.cc:165: Netns is 0
1734072959.687678509: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1734072959.688578559: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 77219
1734072959.689758736: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1734072959.689900160: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1734072959.690371328: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/_hermetic_tmp -> /tmp
1734072959.690450395: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1734072959.690468641: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/execroot/_main
1734072959.690484710: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1734072959.690493769: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/execroot/_main
1734072959.690668293: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1734072959.690681193: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1734072959.690688395: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734072959.690709074: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1734072959.690718582: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734072959.690724824: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1734072959.690731034: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1734072959.690744510: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1734072959.690755426: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1734072959.690762501: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1734072959.690770855: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1734072959.690783340: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1734072959.690790574: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1734072959.690798641: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1734072959.690806067: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1734072959.690871882: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1734072959.690882157: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1734072959.690887343: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734072959.690893777: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1734072959.690900916: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1734072959.690906709: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1734072959.690912247: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1734072959.690917785: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1734072959.690923835: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1734072959.690931646: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1734072959.690941089: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1734072959.690948452: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1734072959.690954630: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1734072959.690959880: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /run/snapd/ns, nullptr, 4135, nullptr) failure (Permission denied) ignored
1734072959.691003574: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1734072959.691048037: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1734072959.691056328: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1734072959.691063915: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1734072959.691071149: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1734072959.691077744: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1734072959.691084466: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1734072959.691091604: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1734072959.691097462: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1734072959.691105593: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1734072959.691111483: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1734072959.691120990: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734072959.691126016: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734072959.691134819: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/execroot/_main
1734072959.691146823: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734072959.691153001: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/execroot/_main
1734072959.691290776: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1734072959.691626410: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1734072959.804221858: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x100
1734072959.804259631: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 1
1734072959.805278921: src/main/tools/linux-sandbox.cc:243: child exited normally with code 1

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
	ZERO_AR_DATE=1 \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/debug.out -- /bin/sh -i)
ERROR: /home/lab223/TensorRT/cpp/lib/BUILD:5:10: Linking cpp/lib/libtorchtrt_plugins.so failed: (Exit 1): linux-sandbox failed: error executing CppLink command
  (cd /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/execroot/_main && \
  exec env - \
	LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
	ZERO_AR_DATE=1 \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/153/debug.out -- /home/lab223/anaconda3/envs/rnw/bin/x86_64-conda-linux-gnu-cc @bazel-out/aarch64-opt/bin/cpp/lib/libtorchtrt_plugins.so-2.params)
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~tensorrt_S_S_Cnvinferplugin___Ulib_Saarch64-linux-gnu/libnvinfer_plugin.so while searching for nvinfer_plugin
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -lnvinfer_plugin
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~tensorrt_S_S_Cnvinfer_Ulib___Ulib_Saarch64-linux-gnu/libnvinfer.so while searching for nvinfer
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -lnvinfer
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~cuda_S_S_Ccudart___Ulib64/libcudart.so while searching for cudart
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -lcudart
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~libtorch_S_S_Ctorch___Ulib/libtorch.so while searching for torch
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -ltorch
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~libtorch_S_S_Ctorch___Ulib/libtorch_cpu.so while searching for torch_cpu
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -ltorch_cpu
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~libtorch_S_S_Ctorch___Ulib/libtorch_cuda.so while searching for torch_cuda
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -ltorch_cuda
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~libtorch_S_S_Ctorch___Ulib/libtorch_global_deps.so while searching for torch_global_deps
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -ltorch_global_deps
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~libtorch_S_S_Cc10_Ucuda___Ulib/libc10_cuda.so while searching for c10_cuda
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -lc10_cuda
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: warning: skipping incompatible bazel-out/aarch64-opt/bin/_solib_aarch64/_U@@_Umain~_Urepo_Urules~libtorch_S_S_Cc10___Ulib/libc10.so while searching for c10
/home/lab223/anaconda3/envs/rnw/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld.gold: error: cannot find -lc10
collect2: error: ld returned 1 exit status
Target //:libtorchtrt failed to build
INFO: Elapsed time: 0.546s, Critical Path: 0.15s
INFO: 13 processes: 13 internal.
ERROR: Build did NOT complete successfully

breknddone avatar Dec 13 '24 06:12 breknddone

hmm, can you share the module.bzl file you are using?

narendasan avatar Dec 18 '24 19:12 narendasan