Ultimaker
Build instructions are outdated and don't work
Apparently some new "Conan" dependency management system has been introduced, but there is no documentation in the readme for how to build with it, and following the instructions no longer works. Running cmake errors out with
CMake Error at CMakeLists.txt:199 (find_package):
By not providing "Findclipper.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "clipper", but
CMake did not find one.
Could not find a package configuration file provided by "clipper"
(requested version 6.4.2) with any of the following names:
clipperConfig.cmake
clipper-config.cmake
Add the installation prefix of "clipper" to CMAKE_PREFIX_PATH or set
"clipper_DIR" to a directory containing one of the above files. If
"clipper" provides a separate development package or SDK, be sure it has
been installed.
Did you look at these instructions:
https://github.com/Ultimaker/Cura/wiki/Running-Cura-from-Source
And ofcourse the readme.md in CuraEngine self?
https://github.com/Ultimaker/CuraEngine#readme
Let me know if this works for you.
Also be aware that we are still working on optimizing things for: "setting up the development environment" for the next two weeks.
But don't hesitate to create a ticket if things don't work out. If you could do that in the Cura repo it will be noticed quicker by us.
I will transfer this ticket there now
I wasn't aware of the additional instructions in the Cura wiki. I'll see if I can extract the information I need from there. I'm not trying to install the Cura frontend only build CuraEngine, and it's not clear right away how to do that. The instructions in the CuraEngine README no longer work, as clipper and other dependency code has been removed and no instructions on how to get it have been added.
OK, trying to follow those directions has Conan downloading hundreds of MBs of packages that duplicate basic system functionality/libraries I already have and want to use rather than just getting the missing clipper, stb, etc...
You will only have to build those packages such as: boost, clipper, Arcus etc once. This will also be done automatically. Once build, they will be stored in the local Conan cache Only updated or rebuild if they're changed, in version or no longer ABI compatible.
Yes it is a bit of a change but compared to the old workflow we have shaved down build time by a lot. It is possible to setup a working development environment within 2 minutes instead of a day (no exaggeration btw)
We now also have reproducable builds, no longer a dependency hell, which could cause bugs and eat away time and resources from developers trying to figure out what the difference is between systems. While finding out it was a different version of an indirect dependencies.
Not to mention that we have a cost reduction in the ball park of 1 senior developer on a yearly basis.
I know it is a change, but we honestly think this is already a huge improvement on what we had and I hope you guys can give use feedback what works for you and what can be improved. Because one of reasons we switched is that it should greatly reduce the complexity for developers to set up the environment.
Please try out the work flow for a bit, see if it is an improvement and give us feedback or PR's if you think that somethings need to be changed.
Hundreds of mb sounds like it is indeed installing the frontend with cpython etc. For Curaengine you would only have to install Conan and the Conan configuration.
conan config install https://github.com/ultimaker/conan-config.git
conan profile new default --detect --force
Once Conan is installed the only instructions for building CuraEngine are
conan install . curaengine/5.2.0-alpha@richfelker/testing --build=missing --update
cmake . -DCMAKE_TOOLCHAIN_FILE=cmake-build-release/conan/conan_toolchain.cmake
cmake --build .
@rburema can you update the README.md for Curaengine as part of ticket CURA-8830
It is indeed outdated
conan install . curaengine/5.2.0-alpha@richfelker/testing --build=missing --update
~~This fails for me with ERROR: Package recipe with name curaengine!=cura-build-environment~~
~~Also conan install . cura-build-environment/5.2.0-alpha@piezo/testing --build=missing --update fails with:~~
ERROR: Failed requirement 'arcus/5.0.0-a+1@ultimaker/testing' from 'conanfile.py (cura-build-environment/5.2.0-alpha@piezo/testing)'
ERROR: Unable to find 'arcus/5.0.0-a+1@ultimaker/testing' in remotes
Edit: Ok I missed the renaming of the master branch and was using stale git object, it now works as advertised. But this stills stand
And what if I want to use a locally installed Arcus, since I also have changes in local source? How can I compile using system zlib, protobuf and boost packages?
It used to be possible to build cura and its dependencies in one command using scripts from Archlinux User Repository. It seems that the maintainers of said scripts have now given up.
$ conan --version
Conan version 1.51.0
$ conan profile show default
Configuration for profile default:
[settings]
os=Linux
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=clang
compiler.version=14
compiler.libcxx=libstdc++
build_type=Release
[options]
[conf]
[build_requires]
[env]
$ find /home/piezo/.conan/
/home/piezo/.conan/
/home/piezo/.conan/.gitignore
/home/piezo/.conan/remotes.json
/home/piezo/.conan/global.conf
/home/piezo/.conan/.conan.db
/home/piezo/.conan/artifacts.properties
/home/piezo/.conan/profiles
/home/piezo/.conan/profiles/default
/home/piezo/.conan/profiles/cura_build.jinja
/home/piezo/.conan/profiles/cura.jinja
/home/piezo/.conan/version.txt
/home/piezo/.conan/generators
/home/piezo/.conan/generators/VirtualPythonEnv.py
/home/piezo/.conan/generators/PyCharmRunEnv.py
/home/piezo/.conan/generators/GitHubActionsBuildEnv.py
/home/piezo/.conan/generators/GitHubActionsRunEnv.py
/home/piezo/.conan/cacert.pem
/home/piezo/.conan/settings.yml
/home/piezo/.conan/conan.conf
/home/piezo/.conan/config_install.json
@Piezoid it looks like that you're running that from the cura-build-environment repo, which is deprecated.
The Conan install command for Curaengine should be run in the root of Curaengine.
The biggest change in our workflow is that we no longer rely on long complex interdependent build scripts, but that each repo can now be seen as a single modular project. I hope this documentation explain the philosophy best https://github.com/Ultimaker/Cura/wiki/Running-Cura-from-Source#process-description-and-workflow
@Piezoid as described earlier in this thread, our old build process wasn't maintainable anymore and very very costly both money wise and on development hours.
It used to be possible to build cura and its dependencies in one command
It is now possible the install the latest Cura (main) in one line (once Conan etc is setup)
conan install cura/latest@ultimaker/testings --build=missing --update -g VirtualPythonEnv -o de tools=True
The aim is to make this workflow easy for all OSes, build systems and development machines. I can currently setup a whole development environment within minutes, and we can build an AppImage within 6 minutes.
Yes at the moment you still can encounter some issues, because we are still working on setting things up. But we are confident that we will get there within the next two weeks.
@Piezoid
And what if I want to use a locally installed Arcus, since I also have changes in local source?
If you want to develop multiple projects like for instance arcus and Curaengine. Clone them both and then put Arcus into editable mode https://docs.conan.io/en/latest/developing_packages/editable_packages.html
cd libArcus
conan editable . arcus/5.2.0-alpha@piezod/Ultimaker
cmake --preset Release
cmake --build --preset Release
cd ../Curraengine
Conan install . curaengine/5.2.0-alpha@ultimaker/testing --require-override=arcus/5.2.0-alpha@piezod/Ultimaker --build=missing --update
cmake --preset Release
The present commands are used for Cmake version 3.23+ otherwise the regular Cmake configure/ generate commands give earlier
What that basically does is to override the arcus package specified in conandata.yml it will then use that previously created editable package which will be used to generate the CMake find modules using the paths pointing to your local Arcus repo. Just make sure that you build the actually binary of Arcus for the release build.
More documentation to come... As I said still work in progress a.t.m.
@Piezoid it really looks like you're are using the cura-build-environment repo. That one is no longer used.
The aim is to make this workflow easy for all OSes, build systems and development machines. I can currently setup a whole development environment within minutes, and we can build an AppImage within 6 minutes.
I get that it greatly simplifies the process on your end for CI and releasing artifacts. However, maintainers of linux packages will need to somehow integrate this in their workflow and build against their own packages. Shipping a separate build of openssl, Qt, etc is not acceptable on these channels. Maybe there is way around this, but the fact that conan-io/conan#1330 is still open is not encouraging.
I will take a closer look at the instructions, the editable mode seems to cover my needs for development.
@Piezoid it really looks like you're are using the
cura-build-environmentrepo. That one is no longer used.
As I corrected in my edit, I was using a stale master branch (instead of the new main branch) of curaengine that had cura-build-environment as a project name.
Hundreds of mb sounds like it is indeed installing the frontend with cpython etc. For Curaengine you would only have to install Conan and the Conan configuration.
conan config install https://github.com/ultimaker/conan-config.git conan profile new default --detect --force
Can you clarify what these commands are supposed to be doing and where they're supposed to be run from? Are they setting up some user-account-wide conan configuration imported from your github, or are they project-local?
Once Conan is installed the only instructions for building CuraEngine are
conan install . curaengine/5.2.0-alpha@richfelker/testing --build=missing --update cmake . -DCMAKE_TOOLCHAIN_FILE=cmake-build-release/conan/conan_toolchain.cmake cmake --build .
What is "curaengine/5.2.0-alpha@richfelker/testing"? I am not trying to pull CuraEngine source from my github. I'm trying to use my local repository with the commits I want in it. When I tried to do something like this before, it started downloading all sorts of unrelated stuff under ~/.conan.
How can I compile using system zlib, protobuf and boost packages?
Why would you want to do that?
If you would really want to do that you would have to remove those requirements out of the conandata.yml and make sure that CMake module path has find modules with the same namings scheme and which declare the same cmake target names.
Which is a pain and this is mainly one of the reasons why we moved to Conan. Maintaining Cura with packaged installed via multiple different system package manager (or OSes without a package manager Windows) is a pain, error prone and filled up the swear-jar very quickly.
The benefit of using Conan is that binaries are reused for every project managed by Conan. Depending on the user options, compiler, architecture and os. See also https://docs.conan.io/en/latest/getting_started.html#installing-dependencies
Why would you want to do that?
Because I don't want N copies of these things installed, and this is the way we do things when we're compiling a package to ship in a distro or to run on our own systems: we use the distro-provided libraries we already have rather than every program having its own pet way to get yet another copy of the same thing.
If you would really want to do that you would have to remove those requirements out of the
conandata.ymland make sure that CMake module path has find modules with the same namings scheme and which declare the same cmake target names.
Being unfamiliar with Conan, I'm not sure what I need to do here. There are a bunch of different headings "None", "5.1.0", etc. all with the same deps under them. I am not trying to make an artifact for distribution so I do not care about naming versions; I will be building from "git mainline" all the time, and just want to be able to build current source without downloading any library code except what's specific to CuraEngine. (I don't even need Arcus, I disable it at build time).
Can you clarify what these commands are supposed to be doing and where they're supposed to be run from?
These need to be run once and can run from everywhere. The will pull in the Conan configuration to make sure that all projects are build with the same flags, environment settings etc. You will also add a link to the cura-ce Artifactory server. Which is a server containing the conanfile.py for dependencies, also known as recipes. These contain the information to build the dependencies and how to consume it.
All that information is used to generate, in our case CMake find configuration files` but you can also output files for xcode, msbuild, auto tools etc.
I forgot that you guys need to remove the cura Artifactory remote which requires credentials and is used by us internally. This contains the recipes and pre built binaries to speed up the build steps. Unfortunately it is to costly to also allow an arbitrary number of community developers to download those.
conan remote remove cura
What is "curaengine/5.2.0-alpha@richfelker/testing"? I am not trying to pull CuraEngine source from my github. Notice the
.in theconan installcommand which specifieke that we're using the conanfile.py in the root of CuraEngine. So you're building from the local repo.
When I tried to do something like this before, it started downloading all sorts of unrelated stuff under ~/.conan
It should only download, dependencies and dependencies from dependencies. That is the whole idea about Conan
You can use the following commands to get some extra insights
conan inspect .
conan info .
Sorry accidentally closed it, typing it all on a phone. Also apologizing for the "autocorrect" words and capitalization you might encounter in my comments
OK, from what I can tell of the docs:
The config install is intended to share the Conan client configuration. For example, in a company or organization, is important to have common settings.yml, profiles, etc.
So on the one hand folks are telling me that Conan is supposed to be a system for building "every project managed by Conan", but on the other hand you're giving me instructions that, as far as I can tell, are basically "clobber any Conan configuration I had and clone Ultimaker's organizational configuration onto my system". This really does not make sense, and reinforces my frustration that Conan seems to be designed around organizational/enterprise use, not as a system for consumers of FOSS to build it on their own systems with their own user accounts.
Because I don't want N copies of these things installed, and this is the way we do things when we're compiling a package to ship in a distro or to run on our own systems:
This is what Conan tries to solve, while taking into account ABI and API compatibility.
If you want to deploy stuff to the userspace you can add -g deploy https://docs.conan.io/en/latest/reference/generators/deploy.html?highlight=deploy
If you want to run it directly you can use the -g VirtualRunEnv this will create a shell script which will set all the environment variables like LD_LIBRARY_PATH, PATH etc. Such that your project can find those installed in the local Conan cache.
I will be building from "git mainline" all the time, and just want to be able to build current source without downloading any library code except what's specific to CuraEngine
git clone https://github.com/ultimaker/curaengine
cd curaengine
conan install . curraengine/5.2.0-alpha@ultimaker/testings --build=missing --update
cmake --preset Release
cmake --build --preset Release
Should be enough
Why would you want to do that?
Mainly, I'm anticipating the requirements that will come from linux distribution packagers. They will either come ask these question or give up.
But also because I'm still struggling to have conan build all the dependencies. Running this in Cura source directory:
CONAN_SYSREQUIRES_MODE=enabled conan install . cura/5.2.0-alpha@piezo/stable --build=missing --update -o cura:devtools=True -g VirtualPythonEn
fails while building tcl/8.6.10, while I have a perfectly good version on my system.
I had to add to specify CONAN_SYSREQUIRES_MODE because the verify default mode can't find the X11 libraries that are nevertheless present (with system includes).
Then, Tcl build fails to link math function as if tclsh was not linked with libm.
I'll try again tomorrow with gcc/binutils instead of llvm/llvm-ld.
on the other hand you're giving me instructions that, as far as I can tell, are basically "clobber any Conan configuration I had and clone Ultimaker's organizational configuration onto my system".
You can install the Ultimaker configuration to an other directory and set an environment variable to use that. They took that into account. We simply didn't get to writing the complete documentation yet for Cura and Curraengine. Since we're still in the process of switching to this workflow
After removing the gratutious system lib deps from conandata.yml, it was still trying to download a huge pile of duplicate system libs as indirect deps of Arcus. After adding -o enable_arcus=False to the conan install command, it actually succeeded, but now I don't know how to get it to build. CMake is complaining that
Target "_CuraEngine" links to:
boost::boost
so I think there is inconsistent naming going on in how the package is referred to in CMake, or something was broken that prevents the system boost from being found like it was before the Conan changes..
You can install the Ultimaker configuration to an other directory and set an environment variable to use that.
OK, great, but the docs should be warning people that they need to do this, and telling them how! Since, as written, they instruct the user to clobber their own Conan configuration if they already have/use Conan.
@Piezoid can you share your CMD line output maybe increase the log level to debug first:
export CONAN_LOGGING_LEVEL=debug
@richfelker agreed and we will add those instructions but I hope you also understand why that documentation is currently not there yet. We're still busy working on this, while also trying to maintain our regular release schedule.
It will be added, just know that we are pressed for resources atm so any help in this regard is very much appreciated and please know that we didn't switch to a new system to frustrate you guys (whom we really appreciate). The old system just wasn't maintainable anymore.
Full recipe that works for me (if I didn't miss anything):
cd /path/to/CuraEngine
conan remote add cura-ce https://ultimaker.jfrog.io/artifactory/api/conan/cura-community True
CONAN_REVISIONS_ENABLED=1 conan install -o enable_arcus=False . --build=missing --update
sed -ie 's/boost::boost/Boost::boost/g' CMakeLists.txt
sed -i -e /boost/d -e /protobuf/d -e /zlib/d -e /gtest/d conandata.yml
mkdir build
cd build
cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake-build-releas
e/conan/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_ARCUS=OFF -DCMAKE_CXX_FLAGS="-static -O2"
make
The CMAKE_CXX_FLAGS are my own preference and non-essential.
Should probably be edited with an environment variable to use a project-directory-local Conan configuration root, but I don't know how to do that yet.
(Edited to add command to strip deps from conandata.yml.)
boost::boost so I think there is inconsistent naming going on in how the package is referred to in CMake
The CMake target namings follow the naming scheme of Conan (which is lowercase by default)
We had to make a decision, and our old CMake scripts were even more inconsistent in their naming scheme. Keep in mind that Linux is just one of the three operating system on which we need to build stuff