ck icon indicating copy to clipboard operation
ck copied to clipboard
Published 1 month ago • verified publisher icon mlcommons

Problem running dlrm-v2, Can not perform a '--user' install in virtualenv

Open WarrenSchultz opened this issue 2 years ago • 2 comments

Ubuntu 22.04

cm test core CM version: 1.5.1

Python executable used by CK: /usr/bin/python3

Path to CM package: /home/ptuser/.local/lib/python3.10/site-packages/cmind Path to CM core module: /home/ptuser/.local/lib/python3.10/site-packages/cmind/core.py Path to CM internal repo: /home/ptuser/.local/lib/python3.10/site-packages/cmind/repo

Path to CM repositories: /home/ptuser/CM

cmr "generate-run-cmds inference _find-performance _r4.0" --model=dlrm-v2-99 --implementation=nvidia-original --device=cuda --backend=tensorrt --category=edge --division=open --execution-mode=valid --results_dir=$HOME/MLPerf_OOB --quiet --clean

`Using MLCommons Inference source from /home/ptuser/CM/repos/local/cache/8e28206686e24735/inference

Cleaning results in /home/ptuser/MLPerf_OOB/test_results

Running loadgen scenario: Offline and mode: performance

  • cm run script "app mlperf inference generic _nvidia-original _dlrm-v2-99 _tensorrt _cuda _valid _r4.0_default _offline"
    • cm run script "detect os"
    • cm run script "get sys-utils-cm"
    • cm run script "get python"
    • cm run script "get mlcommons inference src"
    • cm run script "get cuda-devices"
      • cm run script "get cuda _toolkit" rm: cannot remove 'a.out': No such file or directory

Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Tue_Jul_11_02:20:44_PDT_2023 Cuda compilation tools, release 12.2, V12.2.128 Build cuda_12.2.r12.2/compiler.33053471_0

Compiling program ...

Running program ...

GPU Device ID: 0 GPU Name: Tesla V100-PCIE-16GB GPU compute capability: 7.0 CUDA driver version: 12.2 CUDA runtime version: 12.2 Global memory: 16935682048 Max clock rate: 1380.000000 MHz Total amount of shared memory per block: 49152 Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 2048 Maximum number of threads per block: 1024 Max dimension size of a thread block X: 1024 Max dimension size of a thread block Y: 1024 Max dimension size of a thread block Z: 64 Max dimension size of a grid size X: 2147483647 Max dimension size of a grid size Y: 65535 Max dimension size of a grid size Z: 65535

  • cm run script "reproduce mlperf nvidia inference _dlrm-v2-99 _offline _tensorrt _cuda _gpu_memory.16"
    • cm run script "detect os"
    • cm run script "detect cpu"
      • cm run script "detect os"
    • cm run script "get sys-utils-cm"
    • cm run script "get cuda _cudnn"
    • cm run script "get tensorrt"
    • cm run script "build nvidia inference server _ctuning"
      • cm run script "detect os"
      • cm run script "detect cpu"
        • cm run script "detect os"
      • cm run script "get sys-utils-cm"
      • cm run script "get python3"
      • cm run script "get cuda _cudnn"
      • cm run script "get tensorrt _dev"
      • cm run script "get gcc"
      • cm run script "get cmake"
      • cm run script "get generic sys-util _glog-dev"
      • cm run script "get generic sys-util _gflags-dev"
      • cm run script "get generic sys-util _libgmock-dev"
      • cm run script "get generic sys-util _libre2-dev"
      • cm run script "get generic sys-util _libnuma-dev"
      • cm run script "get generic sys-util _libboost-all-dev"
      • cm run script "get generic sys-util _rapidjson-dev"
      • cm run script "get nvidia mlperf inference common-code _ctuning"
      • cm run script "get generic-python-lib _pycuda"
      • cm run script "get generic-python-lib _opencv-python"
      • cm run script "get generic-python-lib _nvidia-dali"
      • cm run script "get mlperf inference nvidia scratch space" Updating Loadgen HEAD is now at 1bddedf Fix generate_final_report.py (#1485) Updating Power-Dev repo HEAD is now at e9e16b1 Merge pull request #317 from dmiskovic-NV/dejan-undo-range-extension make[1]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA' mkdir -p build/plugins/NMSOptPlugin cd build/plugins/NMSOptPlugin
        && cmake -DCMAKE_BUILD_TYPE=Release /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/code/plugin/NMSOptPlugin
        && make -j CMake Warning (dev) at CMakeLists.txt:19 (find_package): Policy CMP0146 is not set: The FindCUDA module is removed. Run "cmake --help-policy CMP0146" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

The following variables are derived from the values of the previous variables unless provided explicitly:

-- Configuring done (0.0s) CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "nmsoptplugin". This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/NMSOptPlugin make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/NMSOptPlugin' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/NMSOptPlugin' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/NMSOptPlugin' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/NMSOptPlugin' [100%] Built target nmsoptplugin make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/NMSOptPlugin' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/NMSOptPlugin' mkdir -p build/plugins/RNNTOptPlugin cd build/plugins/RNNTOptPlugin
&& cmake -DCMAKE_BUILD_TYPE=Release /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/code/plugin/RNNTOptPlugin
&& make -j CMake Warning (dev) at CMakeLists.txt:19 (find_package): Policy CMP0146 is not set: The FindCUDA module is removed. Run "cmake --help-policy CMP0146" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

-- Configuring done (0.0s) CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "rnntoptplugin". This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "rnntoptplugin". This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "rnntoptplugin". This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/RNNTOptPlugin make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/RNNTOptPlugin' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/RNNTOptPlugin' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/RNNTOptPlugin' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/RNNTOptPlugin' [100%] Built target rnntoptplugin make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/RNNTOptPlugin' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/RNNTOptPlugin' mkdir -p build/plugins/pixelShuffle3DPlugin cd build/plugins/pixelShuffle3DPlugin
&& cmake -DCMAKE_BUILD_TYPE=Release /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/code/plugin/pixelShuffle3DPlugin
&& make -j CMake Warning (dev) at CMakeLists.txt:19 (find_package): Policy CMP0146 is not set: The FindCUDA module is removed. Run "cmake --help-policy CMP0146" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

The following variables are derived from the values of the previous variables unless provided explicitly:

-- Configuring done (0.0s) CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "pixelshuffle3dplugin". This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "pixelshuffle3dplugin". This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/pixelShuffle3DPlugin make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/pixelShuffle3DPlugin' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/pixelShuffle3DPlugin' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/pixelShuffle3DPlugin' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/pixelShuffle3DPlugin' [100%] Built target pixelshuffle3dplugin make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/pixelShuffle3DPlugin' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/pixelShuffle3DPlugin' mkdir -p build/plugins/conv3D1X1X1K4Plugin cd build/plugins/conv3D1X1X1K4Plugin
&& cmake -DCMAKE_BUILD_TYPE=Release /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/code/plugin/conv3D1X1X1K4Plugin
&& make -j CMake Warning (dev) at CMakeLists.txt:19 (find_package): Policy CMP0146 is not set: The FindCUDA module is removed. Run "cmake --help-policy CMP0146" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

The following variables are derived from the values of the previous variables unless provided explicitly:

-- Configuring done (0.0s) CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "conv3D1X1X1K4Plugin". This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "conv3D1X1X1K4Plugin". This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D1X1X1K4Plugin make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D1X1X1K4Plugin' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D1X1X1K4Plugin' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D1X1X1K4Plugin' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D1X1X1K4Plugin' [100%] Built target conv3D1X1X1K4Plugin make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D1X1X1K4Plugin' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D1X1X1K4Plugin' mkdir -p build/plugins/conv3D3X3X3C1K32Plugin cd build/plugins/conv3D3X3X3C1K32Plugin
&& cmake -DCMAKE_BUILD_TYPE=Release /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/code/plugin/conv3D3X3X3C1K32Plugin
&& make -j CMake Warning (dev) at CMakeLists.txt:19 (find_package): Policy CMP0146 is not set: The FindCUDA module is removed. Run "cmake --help-policy CMP0146" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

The following variables are derived from the values of the previous variables unless provided explicitly:

-- Configuring done (0.0s) CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "conv3D3X3X3C1K32Plugin". This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D3X3X3C1K32Plugin make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D3X3X3C1K32Plugin' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D3X3X3C1K32Plugin' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D3X3X3C1K32Plugin' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D3X3X3C1K32Plugin' [100%] Built target conv3D3X3X3C1K32Plugin make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D3X3X3C1K32Plugin' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/conv3D3X3X3C1K32Plugin' mkdir -p build/plugins/retinanetConcatPlugin cd build/plugins/retinanetConcatPlugin
&& cmake -DCMAKE_BUILD_TYPE=Release /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/code/plugin/retinanetConcatPlugin
&& make -j CMake Warning (dev) at CMakeLists.txt:18 (find_package): Policy CMP0146 is not set: The FindCUDA module is removed. Run "cmake --help-policy CMP0146" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

The following variables are derived from the values of the previous variables unless provided explicitly:

-- Configuring done (0.0s) CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "retinanetconcatplugin". This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "retinanetconcatplugin". This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/retinanetConcatPlugin make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/retinanetConcatPlugin' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/retinanetConcatPlugin' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/retinanetConcatPlugin' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/retinanetConcatPlugin' [100%] Built target retinanetconcatplugin make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/retinanetConcatPlugin' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/retinanetConcatPlugin' mkdir -p build/plugins/DLRMv2EmbeddingLookupPlugin cd build/plugins/DLRMv2EmbeddingLookupPlugin
&& cmake -DCMAKE_BUILD_TYPE=Release /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/code/plugin/DLRMv2EmbeddingLookupPlugin
&& make -j CMake Warning (dev) at CMakeLists.txt:18 (find_package): Policy CMP0146 is not set: The FindCUDA module is removed. Run "cmake --help-policy CMP0146" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

The following variables are derived from the values of the previous variables unless provided explicitly:

-- Configuring done (0.0s) CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

CUDA_ARCHITECTURES is empty for target "dlrmv2embeddinglookupplugin". This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/DLRMv2EmbeddingLookupPlugin make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/DLRMv2EmbeddingLookupPlugin' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/DLRMv2EmbeddingLookupPlugin' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/DLRMv2EmbeddingLookupPlugin' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/DLRMv2EmbeddingLookupPlugin' [100%] Built target dlrmv2embeddinglookupplugin make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/DLRMv2EmbeddingLookupPlugin' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/plugins/DLRMv2EmbeddingLookupPlugin' make[1]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA' make[1]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA' Building loadgen... CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required): Compatibility with CMake < 3.5 will be removed from a future version of CMake.

Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions.

mlperf_loadgen v3.1 -- Using C++ compiler flags: -O3 -Wno-error=switch -DDALI_1_15=1 -Wno-error=maybe-uninitialized -Wno-error=range-loop-construct -O3 -W -Wall -- Using C++ standard: 14 -- Using static linker flags: -- Using shared linker flags: -L/usr/local/cuda/lib64 -L/home/ptuser/CM/repos/local/cache/6d4edc1374724424/TensorRT-8.6.1.6/lib -O3 -- Using output path: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build CMake Warning (dev) at CMakeLists.txt:31 (find_package): Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules are removed. Run "cmake --help-policy CMP0148" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers. Use -Wno-dev to suppress it.

-- Using Python interpreter: /home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/bin/python -- Configuring done (0.1s) -- Generating done (0.0s) -- Build files have been written to: /home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build make[2]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' make[3]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' [ 7%] Building CXX object CMakeFiles/mlperf_loadgen.dir/version_generated.cc.o [ 15%] Linking CXX static library libmlperf_loadgen.a make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' [ 84%] Built target mlperf_loadgen make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' make[4]: Entering directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' [ 92%] Linking CXX executable benchmark make[4]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' [100%] Built target benchmark make[3]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' make[2]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/build/inference/loadgen/build' running bdist_wheel running build running build_ext building 'mlperf_loadgen' extension x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c bindings/python_api.cc -o build/temp.linux-x86_64-3.10/bindings/python_api.o In file included from ../third_party/pybind/include/pybind11/cast.h:16, from ../third_party/pybind/include/pybind11/attr.h:13, from ../third_party/pybind/include/pybind11/pybind11.h:44, from ../third_party/pybind/include/pybind11/functional.h:12, from bindings/python_api.cc:27: ../third_party/pybind/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’: ../third_party/pybind/include/pybind11/detail/internals.h:200:27: warning: ‘void PyEval_InitThreads()’ is deprecated [-Wdeprecated-declarations] 200 | PyEval_InitThreads(); | ~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/python3.10/Python.h:130, from ../third_party/pybind/include/pybind11/detail/common.h:112, from ../third_party/pybind/include/pybind11/pytypes.h:12, from ../third_party/pybind/include/pybind11/cast.h:13, from ../third_party/pybind/include/pybind11/attr.h:13, from ../third_party/pybind/include/pybind11/pybind11.h:44, from ../third_party/pybind/include/pybind11/functional.h:12, from bindings/python_api.cc:27: /usr/include/python3.10/ceval.h:122:37: note: declared here 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void); | ^~~~~~~~~~~~~~~~~~ x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c early_stopping.cc -o build/temp.linux-x86_64-3.10/early_stopping.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c generated/version_generated.cc -o build/temp.linux-x86_64-3.10/generated/version_generated.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c issue_query_controller.cc -o build/temp.linux-x86_64-3.10/issue_query_controller.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c loadgen.cc -o build/temp.linux-x86_64-3.10/loadgen.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c logging.cc -o build/temp.linux-x86_64-3.10/logging.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c results.cc -o build/temp.linux-x86_64-3.10/results.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c test_settings_internal.cc -o build/temp.linux-x86_64-3.10/test_settings_internal.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c utils.cc -o build/temp.linux-x86_64-3.10/utils.o x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=3 -DMINOR_VERSION=1 -I. -I../third_party/pybind/include -I/home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/include -I/usr/include/python3.10 -c version.cc -o build/temp.linux-x86_64-3.10/version.o x86_64-linux-gnu-g++ -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -L/usr/local/cuda/lib64 -L/home/ptuser/CM/repos/local/cache/6d4edc1374724424/TensorRT-8.6.1.6/lib -O3 -std=c++14 -O3 -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.10/bindings/python_api.o build/temp.linux-x86_64-3.10/early_stopping.o build/temp.linux-x86_64-3.10/generated/version_generated.o build/temp.linux-x86_64-3.10/issue_query_controller.o build/temp.linux-x86_64-3.10/loadgen.o build/temp.linux-x86_64-3.10/logging.o build/temp.linux-x86_64-3.10/results.o build/temp.linux-x86_64-3.10/test_settings_internal.o build/temp.linux-x86_64-3.10/utils.o build/temp.linux-x86_64-3.10/version.o -o build/lib.linux-x86_64-3.10/mlperf_loadgen.cpython-310-x86_64-linux-gnu.so /home/ptuser/CM/repos/local/cache/1215b7169adb405c/mlperf-tests/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64/wheel copying build/lib.linux-x86_64-3.10/mlperf_loadgen.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel running install_egg_info running egg_info writing mlperf_loadgen.egg-info/PKG-INFO writing dependency_links to mlperf_loadgen.egg-info/dependency_links.txt writing top-level names to mlperf_loadgen.egg-info/top_level.txt reading manifest file 'mlperf_loadgen.egg-info/SOURCES.txt' writing manifest file 'mlperf_loadgen.egg-info/SOURCES.txt' Copying mlperf_loadgen.egg-info to build/bdist.linux-x86_64/wheel/mlperf_loadgen-3.1.egg-info running install_scripts creating build/bdist.linux-x86_64/wheel/mlperf_loadgen-3.1.dist-info/WHEEL creating 'dist/mlperf_loadgen-3.1-cp310-cp310-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it adding 'mlperf_loadgen.cpython-310-x86_64-linux-gnu.so' adding 'mlperf_loadgen-3.1.dist-info/METADATA' adding 'mlperf_loadgen-3.1.dist-info/WHEEL' adding 'mlperf_loadgen-3.1.dist-info/top_level.txt' adding 'mlperf_loadgen-3.1.dist-info/RECORD' removing build/bdist.linux-x86_64/wheel ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv. make[1]: *** [Makefile.build:374: build_loadgen] Error 1 make[1]: Leaving directory '/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA' make: *** [/home/ptuser/CM/repos/local/cache/b45fc52df28b4924/repo/closed/NVIDIA/Makefile.build:210: build] Error 2

CM error: Portable CM script failed (name = build-mlperf-inference-server-nvidia, return code = 256) `

WarrenSchultz avatar Sep 23 '23 15:09 WarrenSchultz

@arjunsuresh @WarrenSchultz - there were many fixes to CM-MLPerf automations in the past few months. Did we fix this issue? Should I close this ticket?

gfursin avatar Jan 16 '24 08:01 gfursin

@arjunsuresh @WarrenSchultz - there were many fixes to CM-MLPerf automations in the past few months. Did we fix this issue? Should I close this ticket?

I don't have access to the quad GPU system for testing anymore. Let me see if I can put something together and do some checking.

WarrenSchultz avatar Jan 16 '24 14:01 WarrenSchultz

@arjunsuresh - I believe we can now run DLRMv2 via CM and we will have a result in the upcoming submission, right? I will close this issue and we can reopen it if the issue reappears ...

gfursin avatar Feb 27 '24 15:02 gfursin