RIOT-OS
Bug: make compile-commands missing source code
Description
The command make compile-commands does not integrate source code from external repositories pulled in via the pkg mechanism.
Steps to reproduce the issue
cd tests/pkg_u8g2/
make # to get it to pull the package source code
make compile-commands
Expected results
The compile-commands.json would include source from build/pkg/u8g2 that was used in the build
Actual results
The compile-commands.json does NOT include source from build/pkg/u8g2 that was used in the build. you can verify this here (or with ANY app) with the following sequence of commands:
CC_OUTPUT=$(jq -r '[.[].output| sub("^/YOUR_WORKING_DIRECTORY/"; "") ] | unique[] ' ../../compile_commands.json | sort)
FIND_OUTPUT=$(find bin/ -iname "*.o" | sort)
so that will give you 2 variables. The first contains all the unique .o files covered in teh compile-commands json. The second contains all the .o files in the bin directory. Ideally, they match. However, if you diff them, or compare line count, you will see differences.
For example, from the tests/pkg_u8g2, I get the following:
echo "${CC_OUTPUT}" | wc -l
46
echo "${FIND_OUTPUT}" | wc -l
144
looking into some of the differences, diff <(echo "${CC_OUTPUT}") <(echo "${FIND_OUTPUT}") | less shows stuff like this:
bin/native/u8g2_csrc/u8g2_bitmap.o
> bin/native/u8g2_csrc/u8g2_box.o
> bin/native/u8g2_csrc/u8g2_buffer.o
> bin/native/u8g2_csrc/u8g2_circle.o
> bin/native/u8g2_csrc/u8g2_cleardisplay.o
> bin/native/u8g2_csrc/u8g2_d_memory.o
> bin/native/u8g2_csrc/u8g2_d_setup.o
> bin/native/u8g2_csrc/u8g2_font.o
> bin/native/u8g2_csrc/u8g2_fonts.o
> bin/native/u8g2_csrc/u8g2_hvline.o
> bin/native/u8g2_csrc/u8g2_input_value.o
> bin/native/u8g2_csrc/u8g2_intersection.o
> bin/native/u8g2_csrc/u8g2_kerning.o
> bin/native/u8g2_csrc/u8g2_line.o
> bin/native/u8g2_csrc/u8g2_ll_hvline.o
> bin/native/u8g2_csrc/u8g2_message.o
> bin/native/u8g2_csrc/u8g2_polygon.o
> bin/native/u8g2_csrc/u8g2_selection_list.o
> bin/native/u8g2_csrc/u8g2_setup.o
> bin/native/u8g2_csrc/u8log.o
> bin/native/u8g2_csrc/u8log_u8g2.o
> bin/native/u8g2_csrc/u8log_u8x8.o
> bin/native/u8g2_csrc/u8x8_8x8.o
...
which is all source from the package itself.
and even from examples/hello-world they differ:
diff <(echo "${CC_OUTPUT}") <(echo "${FIND_OUTPUT}")
24a25
> bin/native/cpu/tramp.o
Versions
Operating System Environment
----------------------------
Operating System: "Ubuntu" "20.04.3 LTS (Focal Fossa)"
Kernel: Linux 5.4.0-92-generic x86_64 x86_64
System shell: /bin/dash (probably dash)
make's shell: /bin/dash (probably dash)
Installed compiler toolchains
-----------------------------
native gcc: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
arm-none-eabi-gcc: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 9-2019-q4-major) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]
avr-gcc: avr-gcc (GCC) 5.4.0
mips-mti-elf-gcc: missing
msp430-elf-gcc: missing
riscv-none-elf-gcc: missing
riscv64-unknown-elf-gcc: missing
riscv-none-embed-gcc: missing
xtensa-esp32-elf-gcc: xtensa-esp32-elf-gcc (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a5) 5.2.0
xtensa-esp8266-elf-gcc: xtensa-esp8266-elf-gcc (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a5) 5.2.0
clang: clang version 10.0.0-4ubuntu1
Installed compiler libs
-----------------------
arm-none-eabi-newlib: "3.1.0"
mips-mti-elf-newlib: missing
msp430-elf-newlib: missing
riscv-none-elf-newlib: missing
riscv64-unknown-elf-newlib: missing
riscv-none-embed-newlib: missing
xtensa-esp32-elf-newlib: "2.2.0"
xtensa-esp8266-elf-newlib: "2.2.0"
avr-libc: "2.0.0" ("20150208")
Installed development tools
---------------------------
ccache: ccache version 3.7.7
cmake: cmake: /usr/local/lib/libcurl.so.4: no version information available (required by cmake)
cppcheck: Cppcheck 1.90
doxygen: 1.8.17
git: git version 2.25.1
make: GNU Make 4.2.1
openocd: Open On-Chip Debugger 0.10.0+dev-01337-g07df04b3-dirty (2020-07-21-14:39)
python: Python 2.7.18
python2: Python 2.7.18
python3: Python 3.8.10
flake8: 3.9.2 (mccabe: 0.6.1, pycodestyle: 2.7.0, pyflakes: 2.3.1) CPython 3.8.10 on
coccinelle: spatch version 1.0.8 compiled with OCaml version 4.08.1
Note that you can also use the output of bear as a comparison, as follows:
bear make
# build output...
❯ jq -r '[.[]] | length' compile_commands.json
144
❯ find bin/ -iname "*.o" | sort | wc -l
144
its not a file-by-file comparison, but the matching number leads me to believe that bear correctly caught all source used in the build.
@maribu - you did some of the work on the compile-commands target. Any thoughts here? We're currently wrapping our build with bear but would prefer to be able to omit that as its "yet another tool" we need to manage. Thanks!
The issue that external packages are build with whatever build system they are usually built. RIOT just configures the build system to produce compatible binaries for linking it into the RIOT firmware.
The best would be to check if u8g2 has support for generation of a compile_commanda.json upstream and use that. If not, it is better to request that feature upstream.
I don't really think that we at a RIOT side can provide patches to enable compile commands for packages (there are many...). But e.g. for packages using modern build systems such cmake or meson that do support generation or compile_commands.json natively, it should be relatively straight forward to add. Maybe for Makefile using packages one could just hook up bear as a maintainable way to generate compile_commands.json.
I think we can close this. The reported issue is absolutely correct in the assessment that no compile commands are generated for packages. But since they packages consists of external code build with an external build systems, there is no universal solution on how to generate them.
IMO generation of a compile_commands.json for external code is out of scope.
If anyone disagrees, please reopen (or even better, PR a solution).
Thinking out loud - could one solution be the option for a package to "register" or provide some hooks into the RIOT build system? It wouldn't cover all packages, of course, but opt-in is better than nothing?
Certainly :) In fact, when cmake is the build system the generation of compile-commands.json is relatively straight forward. And u8g2 seems to use cmake :)