Compilation which requires intermediate (build system) compilation of tools doesn't work.

[sleemanj]

Some things, for example, binutils and gcc, have compilation processes that involve compiling temporary stuff that are run on the build machine to do tasks during the compilation process.

This fails even when instructing the build processes to use the correct "native" gcc (CC_FOR_BUILD environment variable, --build and --host configure arguments), because the crossbuild setup does not seem to have any way to allow one to compile within the container something that must execute within the container if the container is started as a cross compiler.

Note that it is not possible to pre-compile this stuff outside of the container as it's done as part of the make (or in some cases, configure) process itself.

Eh, this is easier to demonstrate than describe.

ubuntu@ip-172-30-1-27:~/test$ sudo docker run -it --rm -v /home:/home -w $(pwd) -e CROSS_TRIPLE=x86_64-w64-mingw32  multiarch/crossbuild:dev bash
root@aa68f36ac0c1:/home/ubuntu/test# cat hello.c
#include <stdio.h>

int main(void)
{
  printf("Hello World\n");
  exit(0);
}
root@aa68f36ac0c1:/home/ubuntu/test# /usr/bin/gcc hello.c
hello.c: In function 'main':
hello.c:6:3: warning: incompatible implicit declaration of built-in function 'exit'
  exit(0);
  ^
/tmp/ccJpOPiQ.s: Assembler messages:
/tmp/ccJpOPiQ.s:7: Warning: .type pseudo-op used outside of .def/.endef ignored.
/tmp/ccJpOPiQ.s:7: Error: junk at end of line, first unrecognized character is `m'
/tmp/ccJpOPiQ.s:22: Warning: .size pseudo-op used outside of .def/.endef ignored.
/tmp/ccJpOPiQ.s:22: Error: junk at end of line, first unrecognized character is `m'
/tmp/ccJpOPiQ.s:24: Error: junk at end of line, first unrecognized character is `-'

so the assembler fails, almost certainly gcc has picked up the wrong assembler. We also (I expect the root of the problem) have this interesting thing going on:

root@aa68f36ac0c1:/home/ubuntu/test# /usr/bin/gcc-ar-4.9 
/usr/x86_64-w64-mingw32/bin/ar: no operation specified

so looks like these gcc components in /usr/bin (eg, /usr/bin/gcc-ar-4.9) are getting replaced with the cross compile gcc components (but they are not actually physically replaced, it's like the executable itself is doing the substitution).

If we start with a null cross compile environment (that is, not actually a cross compiler), we can compile stuff for the container from within the container as expected.

ubuntu@ip-172-30-1-27:~/test$ sudo docker run -it --rm -v /home:/home -w $(pwd) -e CROSS_TRIPLE=x86_64-linux-gnu  multiarch/crossbuild:dev bash
root@4cdc067643db:/home/ubuntu/test# gcc hello.c
hello.c: In function 'main':
hello.c:6:3: warning: incompatible implicit declaration of built-in function 'exit'
  exit(0);
  ^
root@4cdc067643db:/home/ubuntu/test# ./a.out
Hello World    

and the gcc components are correct...

root@4cdc067643db:/home/ubuntu/test# /usr/bin/gcc-ar-4.9                                                                                                                                                    
/usr/bin/ar: no operation specified     

[sleemanj]

The answer is something like this I think (executed as root inside the container of course)...

$ rm /usr/bin/gcc
$ cat >/usr/bin/gcc <<'EOF'
#!/bin/bash
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export LD_LIBRARY_PATH=/usr/lib:/lib
/usr/bin/gcc-4.9 "$@"
EOF
$ chmod +x /usr/bin/gcc

with this wrapper we can call /usr/bin/gcc and it will compile suitable for execution in the container, while calling gcc will of course use the cross-compiler.

I don't know if others will also need to be wrapped, probably at least g++.

Using the GCC build process as a guide, I have this. Note that only ones which are symlinks are wrapped, and in effect that means, gcc, g++ and ld, the others are left as-is for now, so far the build is progressing so we shall see if this is enough or if the ar, as, nm, and ranlib need to also be wrapped.

for native in gcc g++ ar as  ld nm ranlib
do
  if [ -L /usr/bin/$native ]
  then
    NATIVE_BIN="$(realpath /usr/bin/$native)"
    rm /usr/bin/$native
    cat >/usr/bin/$native <<EOF
#!/bin/bash
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export LD_LIBRARY_PATH=/usr/lib:/lib
${NATIVE_BIN} "\$@"
EOF
    chmod +x /usr/bin/$native    
  fi
done

export CC_FOR_BUILD=/usr/bin/gcc
export CXX_FOR_BUILD=/usr/bin/g++
export AR_FOR_BUILD=/usr/bin/ar
export AS_FOR_BUILD=/usr/bin/as
export LD_FOR_BUILD=/usr/bin/ld
export NM_FOR_BUILD=/usr/bin/nm
export RANLIB_FOR_BUILD=/usr/bin/ranlib

[nlm]

I did have this kind of problem with busybox compilation. I think #27 might solve yours too.