Fix CMake errors/unsupported configs

[matthiasdiener]

Followup from #2625.

• [x] gni-*
• [x] *-crayx?
• [x] pami(lrts)-* (#2921)
• [x] Windows
• [x] -D options
• [x] gcc-10 etc. (#2841)
• [x] ./build charm++ netlrts-linux-x86_64.cmake gcc-10 -std=c++20 smp -g --enable-lbuserdata (#2884)
• [ ] mpi-win-x86_64
• [ ] OpenMP omp-smp (#3396, #3436)
• [ ] ParFUM etc.
• [x] ~~Tau~~
• [ ] MSYS2 shells
• [ ] options such as tsan, nolb, ooc (see src/arch/common/conv-mach-*.sh)

---

Directory in src/libs/ck-libs/	Built with ./buildold?	Built with CMake?	Same?
MeshStreamer/	:white_check_mark:	:white_check_mark:	:white_check_mark:
NDMeshStreamer/	:white_check_mark:	:white_check_mark:	:white_check_mark:
ParFUM-Iterators/	:x:	:x:	:white_check_mark:
ParFUM-Tops-Dev/	:x:	:x:	:white_check_mark:
ParFUM-Tops/	:white_check_mark:	:x:	:x:
ParFUM/	:white_check_mark:	:x:	:x:
TMRC2D/	:x:	:x:	:white_check_mark:
ampi/	:white_check_mark:	:white_check_mark:	:white_check_mark:
amr/	:x:	:x:	✅
armci/	:white_check_mark:	:white_check_mark:	:white_check_mark:
barrier/	:white_check_mark:	:white_check_mark:	:white_check_mark:
boomarray/	:x:	:x:	:white_check_mark:
cache/	:white_check_mark:	:white_check_mark:	:white_check_mark:
ckloop/	:white_check_mark:	:white_check_mark:	:white_check_mark:
collide/	:white_check_mark:	:white_check_mark:	:white_check_mark:
completion/	:white_check_mark:	:white_check_mark:	:white_check_mark:
datatransfer/	:white_check_mark:	:x:	:x:
dummy/	:x:	:x:	:white_check_mark:
fem/	:white_check_mark:	:x:	:x:
fftlib/	:x:	:x:	:white_check_mark:
idxl/	:white_check_mark:	:white_check_mark:	:white_check_mark:
ifem/	:white_check_mark:	:x:	:x:
io/	:white_check_mark:	:white_check_mark:	:white_check_mark:
irecv/	:white_check_mark:	:white_check_mark:	:white_check_mark:
liveViz/	:white_check_mark:	:white_check_mark:	:white_check_mark:
liveViz3d/	:x:	:x:	:white_check_mark:
mblock/	:white_check_mark:	:white_check_mark:	:white_check_mark:
metis/	:x:	:x:	:white_check_mark:
msa2/	:x:	:x:	:white_check_mark:
multiphaseSharedArrays/	:white_check_mark:	:white_check_mark:	:white_check_mark:
netfem/	:white_check_mark:	:x:	:x:
pmaf/	:x:	:x:	:white_check_mark:
pose/	:white_check_mark:	:white_check_mark:	:white_check_mark:
pythonCCS/	:white_check_mark:	:white_check_mark:	:white_check_mark:
search/	:white_check_mark:	:white_check_mark:	:white_check_mark:
sparseContiguousReducer/	:white_check_mark:	:white_check_mark:	:white_check_mark:
sparseReducer/	:x:	:x:	:white_check_mark:
state_space_searchengine/	:white_check_mark:	:white_check_mark:	:white_check_mark:
taskGraph/	:white_check_mark:	:white_check_mark:	:white_check_mark:
tcharm/	:white_check_mark:	:white_check_mark:	:white_check_mark:
tmr/	:x:	:x:	:white_check_mark:

---

Please leave comments here with things that fail.

Please include

• the ./build command
• the machine you build on
• the CMake summary after the config step

[evan-charmworks]

Passing smp as part of the build tuplet instead of as a separate word is silently accepted but does not actually build in SMP mode. For example, ./build AMPI mpi-linux-x86_64-smp -j4 -g3 and ./build AMPI netlrts-linux-x86_64-smp -j4 -g3.

[matthiasdiener]

Passing smp as part of the build tuplet instead of as a separate word is silently accepted but does not actually build in SMP mode. For example, ./build AMPI mpi-linux-x86_64-smp -j4 -g3 and ./build AMPI netlrts-linux-x86_64-smp -j4 -g3.

This seems to be an undocumented feature. Are there any users/use cases for this?

[epmikida]

The -no-use-new-std (and probably the -use-new-std arguments to build seem to be ignored. I'm working on a project that uses a std later than c++11 but when I build charm with the cmake script, it results in charmc always using the --std=c++11 flag which crashes my build.

Here's the build command I used: ./build charm++ mpi-linux-x86_64 --build-shared -no-use-new-std -j8

Whereas if I switch to ./buildold with the same command I get a charmc that works.

[matthiasdiener]

The -no-use-new-std (and probably the -use-new-std arguments to build seem to be ignored. I'm working on a project that uses a std later than c++11 but when I build charm with the cmake script, it results in charmc always using the --std=c++11 flag which crashes my build.

Here's the build command I used: ./build charm++ mpi-linux-x86_64 --build-shared -no-use-new-std -j8

Whereas if I switch to ./buildold with the same command I get a charmc that works.

I can't reproduce the issue. When using the old build system with ./buildold charm++ netlrts-linux-x86_64 -j8 -g --with-production -no-use-new-std, I get the following charmc behavior (gcc 5.4.0):

$ charmc -c hello.C -verbose
charmc: Executing g++ -DCMK_GFORTRAN -I../../../bin/../include -D__CHARMC__=1
-U_FORTIFY_SOURCE -fno-stack-protector -std=c++11 -fno-lifetime-dse 
-c hello.C -o hello.o

$ charmc -c hello.C -verbose -no-use-new-std
charmc: Executing g++ -DCMK_GFORTRAN -I../../../bin/../include -D__CHARMC__=1
-U_FORTIFY_SOURCE -fno-stack-protector -std=c++11 -fno-lifetime-dse 
-c hello.C -o hello.o

$ charmc -c hello.C -verbose -no-use-new-std -std=c++14
charmc: Executing g++ -DCMK_GFORTRAN -I../../../bin/../include -D__CHARMC__=1
-std=c++14 -U_FORTIFY_SOURCE -fno-stack-protector -std=c++11 -fno-lifetime-dse 
-c hello.C -o hello.o

Ie, in all cases -std=c++11 is added to the compile command, and the C++14 argument in the last case is essentially ignored. I could not see any substantial differences with the new build system. Looking at the code, -no-use-new-std/-use-new-std seems to be mostly irrelevant for C++, with only small functional changes for C. The only possible reason i can think of is that in the old build system, -std=c++11 is not added to the build command if the compiler enables C++11 by default, while the new system always adds -std=c++11.

What is your compiler version? What is the error message you get?

[epmikida]

For this particular build I'm using mpicc, with gcc version 8.4.0 (Ubuntu 8.4.0-1ubuntu1~18.04) under the hood.

I tried to reproduce what you did above but with the MPI builds, and I had the opposite results:

./buildold charm++ mpi-linux-x86_64 --build-shared -no-use-new-std -j8 and ./buildold charm++ mpi-linux-x86_64 --build-shared -j8 both produce the following from compiling with -verbose: charmc -c hello.C -verbose charmc: Executing /home/epmikida/Programming/flecsi_devel/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-8.3.0/openmpi-3.1.6-uvzp5gldea6cyniij3747hhzujzpvrcs/bin/mpic++ -DCMK_GFORTRAN -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -I../../../bin/../include -D__CHARMC__=1 -fno-stack-protector -fno-lifetime-dse -c hello.C -o hello.o And it produced this regardless of whether or not I passed -no-use-new-std, -use-new-std, or nothing to the charmc command. But when doing the same exact builds with CMAKE, the -std=c++11 showed up every time no matter what.

I'll check to see if the patch you posted fixes the issue for this config. Maybe the -use-new-std is behaving differently for MPI builds? Or like you said, maybe it's based on compiler version? Either way I get different behavior when using ./build vs ./buildold

[epmikida]

Oh, and forgot to mention, even when using ./buildold to build charm with -use-new-std, the -std=c++11 still doesn't show up. So in all cases I've tried the -use-new-std/-no-use-new-std has no effect on the build at all.

[matthiasdiener]

For this particular build I'm using mpicc, with gcc version 8.4.0 (Ubuntu 8.4.0-1ubuntu1~18.04) under the hood.

I tried to reproduce what you did above but with the MPI builds, and I had the opposite results:

./buildold charm++ mpi-linux-x86_64 --build-shared -no-use-new-std -j8 and ./buildold charm++ mpi-linux-x86_64 --build-shared -j8 both produce the following from compiling with -verbose: charmc -c hello.C -verbose charmc: Executing /home/epmikida/Programming/flecsi_devel/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-8.3.0/openmpi-3.1.6-uvzp5gldea6cyniij3747hhzujzpvrcs/bin/mpic++ -DCMK_GFORTRAN -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -I../../../bin/../include -D__CHARMC__=1 -fno-stack-protector -fno-lifetime-dse -c hello.C -o hello.o And it produced this regardless of whether or not I passed -no-use-new-std, -use-new-std, or nothing to the charmc command. But when doing the same exact builds with CMAKE, the -std=c++11 showed up every time no matter what.

I'll check to see if the patch you posted fixes the issue for this config. Maybe the -use-new-std is behaving differently for MPI builds? Or like you said, maybe it's based on compiler version? Either way I get different behavior when using ./build vs ./buildold

Thanks for looking at this. I think the issue is what I had mentioned above: Since gcc-8 uses C++14 by default (ie, without any -std=c++ option), the old build system never adds std=c++11 to the command line, while the new one does. The (-no)-use-new-std option seems to make no difference. Reproducing the behavior of the old build system in cmake will be difficult... The proposed fix in #2909 will maybe fix your issue, but it is not a general solution, since for many compilers, we need to add the -std=c++11 option to all commands.

As a workaround, can you try building with ./build .... -std=c++14 ?

---

Side note: This behavior changed with gcc-6. Versions less than 6 use C++98 by default, versions after that C++14.

[epmikida]

Tried ./build charm++ mpi-linux-x86_64 --build-shared -j8 -std=gnu++17

But the -verbose output is: charmc: Executing mpicxx -fPIC -DCMK_GFORTRAN -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -I../../../bin/../include -D__CHARMC__=1 -U_FORTIFY_SOURCE -fno-stack-protector -fno-lifetime-dse -std=c++11 -c hello.C -o hello.o

For now, the ./buildold is a suitable workaround.

[epmikida]

On a side note, maybe (-no)-use-new-std should be removed if they don't have any function. Or maybe they should be converted to an option that just allows you to explicitly set the standard that charmc should use.

[matthiasdiener]

./build charm++ mpi-linux-x86_64 --build-shared -j8 -std=gnu++17

Ok, I'll check what is going wrong with this.

On a side note, maybe (-no)-use-new-std should be removed if they don't have any function. Or maybe they should be converted to an option that just allows you to explicitly set the standard that charmc should use.

I think that works, but only for C.

[matthiasdiener]

@epmikida: I looked a bit more at charmc, and I think the right way to enable this functionality is to use its -c++-option argument, which adds the argument only for C++ source files, after the arguments that charmc adds automatically:

$ ./charmc -c hello.C -c++-option -std=c++14 -verbose
# [...]
charmc: Executing clang++ -fPIC -dynamic -fno-common -stdlib=libc++ -DCMK_GFORTRAN
-mmacosx-version-min=10.7 -D_DARWIN_C_SOURCE -I../../../bin/../include -D__CHARMC__=1 
-U_FORTIFY_SOURCE -fno-stack-protector -std=c++11 -std=c++14 -c hello.C -o hello.o

I.e., -std=c++14 comes after -std=c++11 in the compiler command line when using this charmc option and therefore should determine the language level used. Can you check if this is an adequate workaraound for you?

[epmikida]

../../../bin/charmc -c hello.C -c++-option -std=gnu++17 -verbose

charmc: Executing /home/epmikida/Programming/flecsi_devel/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-8.3.0/openmpi-3.1.6-uvzp5gldea6cyniij3747hhzujzpvrcs/bin/mpic++ -DCMK_GFORTRAN -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -I../../../bin/../include -D__CHARMC__=1 -U_FORTIFY_SOURCE -fno-stack-protector -fno-lifetime-dse -std=c++11 -c hello.C -o hello.o

Does not seem to work. I don't know if it is because it's an MPI build or what, but I can't seem to get the -std=X option to show up at all.

[epmikida]

And it's certainly recognizing the option. If I have a typo in -c++-option it throws an error for an unrecognized option.

[evan-charmworks]

Does reverting or cherry-picking ca4221e4a5c5f23ba820a90b23352cc85f854e5c affect -c++-option?

[matthiasdiener]

Does reverting or cherry-picking ca4221e affect -c++-option?

Interesting. @epmikida Can you clarify which Charm++ version you are using? As far as I can see, ca4221e should only affect the option ordering. It is not clear to me (and I can't reproduce it) why -std=gnu++17 doesn't appear at all.

[epmikida]

Ah I see, for my previous comment I was on 6498e1752. I didn't realize the -c++-option had been updated in a commit since then.

I updated master to the most recent commit (e4a6e7b3f) and now compiling hello.C (in examples/charm++/hello) seems to work. It also works with the larger project that originally uncovered this issue so yes, this workaround should work. Thanks.

[evan-charmworks]

It looks like the CMake build system is creating metis objects directly in src/libs/ck-libs/metis/ instead of the tuplet subfolder.

[evan-charmworks]

A remaining to-do item is Microsoft MPI support on Windows.

[evan-charmworks]

If I build with -fsanitize=address, linking the hello example fails because it lacks this argument (and presumably anything else on the build line).

[evan-charmworks]

I believe the i386 autobuilds are actually testing x86_64 because the platform parts of the build tuplet passed to buildcmake are not used when invoking cmake.

[evan-charmworks]

Some CIs print a warning that we set CMP0075 to OLD.

 CMake Deprecation Warning at CMakeLists.txt:25 (cmake_policy):
  The OLD behavior for policy CMP0075 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

https://cmake.org/cmake/help/latest/policy/CMP0075.html

[jbakosi]

Attempting to use ninja as cmake's generator as

cmake -GNinja -DNETWORK=mpi <charm-root>

I get

...
-- Configuring done
-- Generating done
CMake Error:
  Running
   '/usr/bin/ninja' '-C' '/home/jbakosi/code/charm/b' '-t' 'cleandead'
  failed with:
   ninja: error: build.ninja:1437: bad $-escape (literal $ must be written as $$)
CMake Error:
  Running
   '/usr/bin/ninja' '-C' '/home/jbakosi/code/charm/b' '-t' 'recompact'
  failed with:
   ninja: error: build.ninja:1437: bad $-escape (literal $ must be written as $$)
CMake Generate step failed.  Build files cannot be regenerated correctly.

The problem is the dollar sign in hwloc's build command at https://github.com/UIUC-PPL/charm/blob/main/cmake/hwloc.cmake#L46, which ninja does not like. I suspect the solution is to escape it so it results in double-dollar-signs, however, I cannot seem to be able to trick cmake into doing that any way I have tried. Just wanted to note it here, maybe someone has a solution eventually.

Background: I'd like to build charm as a cmake externalproject using ninja. In the meantime, a workaround is:

ExternalProject_Add(charm ... BUILD_COMMAND make -j) 

[mszpindler]

• [ ] ofi-crayshasta at release 7.0.0 needs this fix https://github.com/spack/spack/issues/40129#issuecomment-1757848414

[jbakosi]

Does the cmake build support tracemode projections?

[evan-charmworks]

Does the cmake build support tracemode projections?

Yes, just make sure you've built Charm with --enable-tracing.

[jbakosi]

Thanks, @evan-charmworks. I needed to pass -DTRACING=true -DTRACING_COMMTHREAD=true to the charm cmake and use the link argument -tracemode projections.