How to add seqan3 as external project in cmake?

[JensUweUlrich]

Platform

• SeqAn version: 3.0.0
• Operating system: Ubuntu LTS 18.04
• Compiler: GCC-7.4.0

Question

I would like to add Seqan3 as an external project to the cmake configuration in order to simplify installation/update of my application. I could not find any HowTo, so how would you suggest to do it?

Thanks Jens

[eseiler]

Hey Jens,

there's some information in our CMake file.

You can do something like this:

# Find SeqAn3
find_package (SeqAn3 REQUIRED
              HINTS ${SEQAN3_DIR}/build_system)

# Update the list of file names below if you add source files to your project.
add_executable (my_program
                my_program.cpp)

# Link the SeqAn3 library
target_link_libraries (my_program seqan3::seqan3)

HINTS ${SEQAN3_DIR}/build_system tells CMake where to look for seqan3. In this example we expect a call like cmake [...] -DSEQAN3_DIR=path. If this is relative to the CMakeLists.txt file, you could also use something like HINTS ${CMAKE_SOURCE_DIR}/seqan3/build_system.

Edit: Removed obsolete second version.

[marehr]

You don't need the second version at all. (at least possible since cmake 3.4, might also work with prior versions, but we require that as minimum for different reasons)

[eseiler]

Good to know, we should add a HowTo.

[marehr]

I once proposed in a strategy meeting to remove those variables, because this way of using CMake is out-dated.

[h-2]

@JensUweUlrich I thought that the setup guide covers this: http://docs.seqan.de/seqan/3-master-user/setup.html#autotoc_md72

Is this something you read and found that it didn't help you? How would you propose that we change the documentation to make finding this easier? (honest question)

[JensUweUlrich]

Well... that's not exactly what I meant. This assumes, that you already downloaded seqan to some place in your filesystem. There is a possibility to define external projects in CMake like described here. I already did this for google test & google mock. This is really comfortable because cmake then automatically downloads and builds the latest version of the external project.

[rrahn]

If I am not mistaken you should be able to also just pull SeqAn to some directory. You need to disable the build step though since you don’t need it. Only thing that should be left is to tell Cmake where to find the SeqAn cmake config file and you should be good to go by including the SeqAn project as usual.

_{Sent with GitHawk}

[JensUweUlrich]

Ok. That was my idea as well. I will try and tell you how it could work.

[h-2]

Well... that's not exactly what I meant. This assumes, that you already downloaded seqan to some place in your filesystem. There is a possibility to define external projects in CMake like described here

Ah, now I know what you mean. Yes, I think we should definitely add instructions for that! @joshuak94 could you look into that and maybe add a Tutorial to the setup section once we have agreed on a procedure?

[JensUweUlrich]

Okay. I figured out how to do it. The directory structure of my app is like the one described in the tutorial with a build and src directory. Within src is the CMakeLists.txt that looks like this:

cmake_minimum_required (VERSION 3.4)

SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread")

project (AppName CXX)

# add all subdirectories
add_subdirectory(seqan)
add_subdirectory(main)
add_subdirectory(test)

Within the seqan directory is another CMakeLists.txt file with the following content:

# Download and unpack seqan at configure time
configure_file(CMakeLists.txt.in ${CMAKE_CURRENT_BINARY_DIR}/seqan3-download/CMakeLists.txt)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" .
  RESULT_VARIABLE result
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/seqan3-download)
if(result)
  message(FATAL_ERROR "CMake step for seqan3 failed: ${result}")
endif()

execute_process(COMMAND ${CMAKE_COMMAND} --build .
  RESULT_VARIABLE result
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/seqan3-download )
if(result)
  message(FATAL_ERROR "Build step for seqan3 failed: ${result}")
endif()

set(SeqAn3_DIR ${CMAKE_CURRENT_BINARY_DIR}/seqan3/seqan3-src/build_system)
find_package (SeqAn3 REQUIRED)

And additionally there is a CMakeLists.txt.in file in the same directory:

cmake_minimum_required(VERSION 3.4)

project(seqan3-download)

include(ExternalProject)
ExternalProject_Add(seqan3
  GIT_REPOSITORY    https://github.com/seqan/seqan3.git
  GIT_TAG           master
  GIT_SHALLOW		1
  SOURCE_DIR        "${CMAKE_CURRENT_BINARY_DIR}/seqan3/seqan3-src"
  CMAKE_ARGS		"-DCMAKE_PREFIX_PATH=<SOURCE_DIR>/build_system"
  CONFIGURE_COMMAND	""
  BUILD_COMMAND		""
  INSTALL_COMMAND	""
)

Then you don't need to specify -DCMAKE_PREFIX_PATH when calling cmake from the build directory. This works for me now. The only I thing I could not find out is how to tell cmake to clone seqan recursively ("GIT_CONFIG "--recursive" did not work). Maybe this is a little help for the tutorial.

[eseiler]

If you just want to clone SeqAn3, you could also use git submodules. Then you just need find_package (SeqAn3 REQUIRED HINTS ${CMAKE_SOURCE_DIR}/submodules/seqan3/build-system) or similar. If you then clone your project with --recurse-submodules, the submodules of SeqAn3 are also cloned.

[smehringer]

@JensUweUlrich @eseiler Is there something left to do here or is this resolved?

[JensUweUlrich]

Can be closed from my side.

[smehringer]

@eseiler is there something we should update in the tutorials?

[h-2]

The only I thing I could not find out is how to tell cmake to clone seqan recursively ("GIT_CONFIG "--recursive" did not work).

I still think we should figure this out and add things to the tutorials.

[mr-c]

@marehr Maybe this issue should be re-opened & retitled ?

[rlorigro]

Hi, This may not be ideal for everyone, but this solution is working for me using CMake ExternalProject:

# Need to explicitly enable ExternalProject functionality
include(ExternalProject)

# Download or update library as an external project
ExternalProject_Add(project_seqan3
        GIT_REPOSITORY https://github.com/seqan/seqan3.git
        PREFIX ${CMAKE_SOURCE_DIR}/external/seqan3/
        CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${CMAKE_SOURCE_DIR}/external/seqan3/ -DCMAKE_INSTALL_LIBDIR=${CMAKE_SOURCE_DIR}/external/seqan3/lib
        BUILD_IN_SOURCE True
        INSTALL_DIR ${CMAKE_SOURCE_DIR}/external/seqan3/
        INSTALL_COMMAND make install
        )

# Define main library as dependent on the downloaded project (transitively)
add_dependencies(GetBlunted project_seqan3)

# Ensure that main library has access to primary dependencies' and secondary dependencies' headers
include_directories(
        ${CMAKE_SOURCE_DIR}/external/seqan3/include/
        ${CMAKE_SOURCE_DIR}/external/seqan3/include/seqan3/submodules/cereal/include/
        ${CMAKE_SOURCE_DIR}/external/seqan3/include/seqan3/submodules/range-v3/include/
        ${CMAKE_SOURCE_DIR}/external/seqan3/include/seqan3/submodules/sdsl-lite/include/
        )

I am running this on:

 CMake 3.19.3 
-- CMAKE_CXX_COMPILER_ID: GNU
-- CMAKE_CXX_COMPILER_VERSION: 7.5.0
-- CMAKE_SYSTEM: Linux-4.15.0-132-generic

I also needed to specify -fconcepts for this build: set(CMAKE_CXX_FLAGS "-fconcepts")

I have not had any issue with submodules for this project, but I have in the past for other projects, and the way I worked around that issue is by using the following syntax (e.g. for the SPOA repo):

# Download or update library as an external project
ExternalProject_Add(project_spoa
        GIT_REPOSITORY https://github.com/rvaser/spoa.git
        GIT_SUBMODULES vendor/bioparser vendor/cereal vendor/cpu_features vendor/simde
        PREFIX ${CMAKE_SOURCE_DIR}/external/
        CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${CMAKE_SOURCE_DIR}/external/spoa/ -DCMAKE_INSTALL_LIBDIR=${CMAKE_SOURCE_DIR}/external/spoa/lib
        BUILD_IN_SOURCE True
        INSTALL_DIR ${CMAKE_SOURCE_DIR}/external/spoa/
        INSTALL_COMMAND make install
        )

I hope this is helpful for your own documentation or for any users trying to build this way. I have built from scratch and found that no additional target_include_... or target_link_... commands are needed, because it is header-only. Creating the ExternalProject and the dependency of the main project on that project is enough to ensure that the files exist before using include_directories() to include the headers.

However, I did find that the error messages here: https://github.com/seqan/seqan3/blob/3e881d0d4881b076887ebfb66b6a23c2317b75f3/include/seqan3/core/platform.hpp#L95

were a bit unclear or misleading because I needed to explicitly include all 3 submodules' include directories.

[marehr]

@rlorigro Thank you for your insights.

I would generally suggest to use FetchContent that is the modern way to include external projects via cmake (fully available since 3.14):

cmake_minimum_required (VERSION 3.14)

project (my_app LANGUAGES CXX VERSION 1.0.0)

message (STATUS "Fetching external project seqan3")
include (FetchContent)
FetchContent_Declare (
    seqan3_fetch_content
    GIT_REPOSITORY "https://github.com/seqan/seqan3.git"
    GIT_TAG "master"
)

# this will do the download, find_package(seqan3) and exposes seqan3::seqan3 at cmake configure time
FetchContent_MakeAvailable (seqan3_fetch_content)

add_executable (my_app my_app.cpp)

# this will set-up all includes and other dependencies needed by seqan3
target_link_libraries (my_app PUBLIC seqan3::seqan3)

This has the advantage that everything will be available at cmake configure time and thus behaves similar to an explicit git submodule. And you don't need to set include paths and other flags needed for seqan3 yourself, because you can directly "link" against seqan3 (in cmake terminology this not only means linking shared libraries, but setting the correct include paths for header-only libraries, too).

[rlorigro]

This is nice, thanks. I've never seen this recommended in all my googling.

[rlorigro]

Unfortunately for other projects (spoa) FetchContent it doesn't seem to work. Tells me spoa::spoa target is not found

[marehr]

Unfortunately for other projects (spoa) FetchContent it doesn't seem to work. Tells me spoa::spoa target is not found

Yes, this because spoa does not follow "modern" cmake paradigms.

If you look at the code, they named their library ${PROJECT_NAME} which is spoa

so doing

target_link_libraries (my_app PUBLIC spoa)

should work.

[eseiler]

Core Meeting 27-1-2021

Resolution:

• We will add @marehr's solution to the documentation (as a new section "Including SeqAn3 as external project" under the "Setup" tab)

[rlorigro]

@marehr I see. What is happening is that my system installation of spoa (required for other software) was being located first, and it seems that FetchContent has a mechanism that does not allow duplication of libraries. However the system version of spoa appears to be an older one that is not compatible. I think in many cases you would want to specify which version is used, so there must be some way to specify this? But I'm not finding an obvious override in the docs.

(sorry for using this git issue for discussion, I'm just very interested in switching over to FetchContent)

[marehr]

@rlorigro No, everything fine, since if I understood you correctly, we might face the same issue and it seems worthwhile to think about this.

From what I understood, you

• have a system-wide install of "spoa" (I guess you can't remove it, since it would be the easiest solution)
• an extra external dependency on "spoa" that conflicts with your system-wide install
• you want to prioritise your local checkout over the system-wide install

Can you share your output make VERBOSE=1, I'm not sure whether target_link_libraries (my_app PUBLIC spoa) prioritises /usr/lib/spoa.so over a cmake library target or whether the linker finds /usr/lib/spoa.so as well as your local build of spoa.so.

[rlorigro]

Yes that's exactly right. To make this debugging simpler, I created a mock project that looks like this:

######################################
# ---------------------------------- #
# -------- COMPILER VERSION -------- #
# ---------------------------------- #
######################################

cmake_minimum_required(VERSION 3.10)
project(main_project VERSION 0.0.0)

# Compiler options.
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_definitions(-ggdb3 -O0 -Wall)       # Debugging + No optimization


#########################################
# ------------------------------------- #
# -------- SOURCES AND HEADERS -------- #
# ------------------------------------- #
#########################################

# Include header files
include_directories(
        "inc"
)

# Define our shared library sources. NOT test/executables.
set(SOURCES
        src/a.cpp
        )

project(main_project)
add_library(main_project STATIC ${SOURCES})

##############################################
# ------------------------------------------ #
# -------- LINKING EXTERNAL LIBRARY -------- #
# ------------------ spoa ------------------ #
# ------------------------------------------ #
##############################################


message (STATUS "Fetching external project spoa")
include (FetchContent)
FetchContent_Declare (
        spoa_fetch_content
        GIT_REPOSITORY "https://github.com/rvaser/spoa.git"
        GIT_TAG "master"
)


# this will do the download, find_package(spoa) and exposes 'spoa' at cmake configure time
FetchContent_MakeAvailable (spoa_fetch_content)


############################################
# ---------------------------------------- #
# -------- Generating executables -------- #
# ---------------------------------------- #
############################################

# -------- EXECUTABLES --------

set(EXECUTABLES
        test_spoa
        )

foreach(FILENAME_PREFIX ${EXECUTABLES})
    add_executable(${FILENAME_PREFIX} src/executable/${FILENAME_PREFIX}.cpp)
    target_link_libraries(${FILENAME_PREFIX}
            main_project
            spoa
            )
endforeach()

# -------- final steps --------

# Where to install
set(BINARY_INSTALL_DIR ${CMAKE_CURRENT_SOURCE_DIR}/bin)

install(TARGETS
        main_project
        DESTINATION ${BINARY_INSTALL_DIR})

And actually, this does not reproduce the problem... The problem happens here when I switch to FetchContent: https://github.com/vgteam/GetBlunted/blob/a09f9338c8d9a017d982ad5787765a56204efb70/CMakeLists.txt

From the original CMakeLists.txt that used ExternalProject: https://github.com/vgteam/GetBlunted/blob/9022fb4babf28b1f0574feadef18b55ecd6cc9a6/CMakeLists.txt#L262

VERBOSE=1 output implies that the downloaded project is being linked, but the code that calls spoa is completely broken when I use FetchContent, because of missing references to new methods/classes.

[ 92%] Building CXX object CMakeFiles/test_spoa.dir/src/test/test_spoa.cpp.o
/usr/bin/c++ -DSEQAN3_HAS_BZIP2=1 -DSEQAN3_HAS_ZLIB=1 -I/home/ryan/code/GetBlunted-vg/GetBlunted/inc -I/home/ryan/code/GetBlunted-vg/GetBlunted/external/gfak/include -I/home/ryan/code/GetBlunted-vg/GetBlunted/external/bdsg/include -I/home/ryan/code/GetBlunted-vg/GetBlunted/external/bdsg/include/handlegraph -I/home/ryan/code/GetBlunted-vg/GetBlunted/external/bdsg/include/bdsg -I/home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/spoa_fetch_content-src/include -I/home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/spoa_fetch_content-src/vendor/simde -I/home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/spoa_fetch_content-src/vendor/cpu_features/include -I/home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/spoa_fetch_content-src/vendor/cereal/include -I/home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/seqan3_fetch_content-src/include -isystem /home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/seqan3_fetch_content-src/submodules/sdsl-lite/include -isystem /home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/seqan3_fetch_content-src/submodules/range-v3/include -isystem /home/ryan/code/GetBlunted-vg/GetBlunted/build/_deps/seqan3_fetch_content-src/submodules/cereal/include -fexceptions -fconcepts   -O3 -Wall -pthread -fopenmp -std=gnu++1z -o CMakeFiles/test_spoa.dir/src/test/test_spoa.cpp.o -c /home/ryan/code/GetBlunted-vg/GetBlunted/src/test/test_spoa.cpp

Example error:

error: ‘class spoa::Graph’ has no member named ‘AddAlignment’; did you mean ‘add_alignment’?

So in the end I have no explanation for what is causing this. Obviously this is getting a bit in the weeds, so no offense taken if it's no longer a good investment of your time. The only difference that I can think of between FetchContent_makeAvailable and ExternalProject is that ExternalProject lets me specify libspoa.a for the imported project.

[marehr]

@rlorigro I just wanted to give you a heads-up that I read your post, but I haven't had time to experiment with your use case. I try to do it in the next few days, you can ping me, if I forget to follow-up, because I'm terrible at remembering things 😆