OpenBLAS on dSPACE realtime hardware

[mohseninima]

Hello,

I am currently trying to update some code to use OpenBLAS and implement it on a dSPACE 1103 PowerPC board but I am having some issues. The build steps are a little confusing and I will try my best to explain. There are 3 devices in this setup

1. A laptop where the OpenBLAS is compiled -Haswell i7-4700HQ -Ubuntu 18.04 WSL 64 bit

2. The host for the dSPACE system -Sandy Bridge-E Xeon E5-1620 -Windows 7 64 bit -MATLAB 32 bit

3. Real-time board -dSPACE 1103 -Power PC 750 GX -Receives final compiled code from the host

I first compile OpenBLAS on the laptop using make DYNAMIC_ARCH=1 BINARY=32 HOSTCC=gcc CC=i686-w64-mingw32-gcc FC=i686-w64-mingw32-gfortran CFLAGS='-static-libgcc -static-libstdc++ -static -ggdb' FFLAGS='-static' && mv -f libopenblas.dll.a libopenblas.lib

I then copy over the lib/dll.a/include files to the host PC. In my existing code MYCODE.c I include cblas.h and update my functions to use cblas. I then use MATLAB to compile a mex file by using the following command mex -v MYCODE.c libopenblas.lib -g -I'openblas/include' -lmwlapack

This compiles successfully and I am able to run my mex file and obtain correct results.

Now to upload the code to the real-time board I first create a model in Simulink that uses the mex file and call that using rtwbuild('MYMODEL') which will take my C files and compile them using the PPCTools37 compiler for the real-time board PPC architecture. I then receive the error

COMPILING "..\MYCODE.c"
(F) C0005; "C:\Users\XXXX\Desktop\openblasproject\openblas\include\common.h", line 87 pos 20; could not open source file "unistd.h"

Any idea why I would be receiving this error?

Thank You

[martin-frbg]

This file is included from common.h when _MSC_VER is not defined (that is, when the compiler is not recognized as MSVC, some kind of Unix-like environment like mingw/msys is assumed). You can try to #define _MSC_VER in your code before it includes any of the OpenBLAS headers, or find out how the PPCTools37 compiler introduces itself in the default preprocessor defines that it adds.

[brada4]

Matlab includes MKL BLAS. You do not need OpenBLAS at that point. Why would yourcode.c incluse internal OpenBLAS header at all?

[mohseninima]

Answering @martin-frbg

This file is included from common.h when _MSC_VER is not defined (that is, when the compiler is not recognized as MSVC, some kind of Unix-like environment like mingw/msys is assumed). You can try to #define _MSC_VER in your code before it includes any of the OpenBLAS headers, or find out how the PPCTools37 compiler introduces itself in the default preprocessor defines that it adds.

When using #define _MSC_VER I now get the following when compiling the code for PPC.

(F) C0005; "C:\Users\XXXX\Desktop\openblasproject\openblas\include\common.h", line 117 pos 21; could not open source file "windows.h"

I did some more searching and the full compiler name is "Microtec PowerPC C/C++ Compiler 3.7" The following compiler mirrors are defined

_MRI Defined as 1. _MICROTEC Defined as 1. _TARGET_PPC Defined as 1 if the compiler is being used for a PPC target; otherwise undefined. _VERSION The version of the compiler in literal string format. _UNIX Defined as 1 if the compilation host is any UNIX variant. _SOLARIS Defined as 1 if the compilation host is Solaris. _LINUX Defined as 1 if the compilation host is Linux. _WINDOWS Defined as 1 if the compilation host is any version of Windows.

For @brada4's question

Matlab includes MKL BLAS. You do not need OpenBLAS at that point. Why would yourcode.c incluse internal OpenBLAS header at all?

From what I know MKL is really only optimized for x86 systems and I did not get much if any performance benefit between reference BLAS and MKL on this PPC system. I am trying to push the limits of this system and I am hoping OpenBLAS will help me do so.

[brada4]

You build (cross-build linux to win32) on your laptop for use with 5 years old last 32bit matlab release - what does it have to do with some BSP compiler failing in a completely independent build? Can you get errors and show command lines from that BSP compiler? That build has absolutely nothing to do with matlab or mingw gcc you used before.

[mohseninima]

It does have something to do with the previous build. The mex file is used to construct a simulink model that the PPC compiler uses in combination with the original source code to create the final compiled software that is sent to the real-time system. This is standard for the system, the dSPACE software uses MATLAB to interface with the dSPACE hardware. The only thing I am doing differently than normal is trying to incorporate OpenBLAS. The 5 year old 32 bit matlab release is required (I would love to upgrade if I could).

I have included the output and the makefile that is called when running rtwbuild('MYMODEL') output.txt MYMODEL.mk.txt

[brada4]

I see no openblas built in your log nor the error you encounter....

[mohseninima]

I see no openblas built in your log nor the error you encounter....

Line 73 and 77 in output.txt

[martin-frbg]

Does it compile when you simply change the #if !defined(_MSC_VER) to #if 0 in line 86 of common.h (and remove the #define _MSC_VER again, as it seems to cause other problems) ? If that does the trick, we could probably change that line to something like #if !defined(_MSC_VER) && !defined(_TARGET_PPC)

[brada4]

USER_LIBS = "C:\Users\XXXX\Desktop\openblasproject\libopenblas.lib" "C:\Program Files (x86)\MATLAB\R2013a\extern\lib\win32\microsoft\libmwlapack I dont think ppc blas ever link with win32 lapack. C0005 code is from visual studio compiler, nothing to do with BSP compiler.

[mohseninima]

@martin-frbg Setting #if 0 ends up causing the same error as defining _MSC_VER

(F) C0005; "C:\Users\XXXX\Desktop\openblasproject\openblas\include\common.h", line 117 pos 21; could not open source file "windows.h" #include <windows.h>

@brada4 This is an error message from the PPC compiler

C0005 could not open source file “filename”

Also linking with the MATLAB libmwlapack and libmwblas works fine on their own (this is the built in MATLAB MKL). I am going step by step when implementing OpenBLAS by first replacing the BLAS before trying LAPACK. The error messages I am receiving are before anything from libmwlapack is even loaded.

[brada4]

Could you try cross-building PPC component outside matlab? One single line error for unknown command is not really helpful. Are you certain microtec compiler emulates MSVC error messages? Is it documented anywhere in the world?

[martin-frbg]

I see now that the #include <windows.h> in line 117 is independent of _MSC_VER, it depends on #ifdef OS_WINDOWS. Probably best to add an #ifndef _TARGET_PPC ... #endif around that line (or simply comment it out for now) and see where that gets you. (Cannot negate the entire ifdef OS_WINDOWS as its #else branch would again try to include unistd.h)

[mohseninima]

@martin-frbg Doing that in combination with setting if !defined(_MSC_VER) to #if 0 I now receive the following error-5-1.txt

@brada4 I can try but it might take time, it is not documented (and I am not sure if possible) to compile outside of matlab. I am certain about the error

[brada4]

It should be possible to compile using common make , e.g make CC=ppc-cross.exe HOSTCC=cl.exe TARGET=PPC440 FC=NONE

[martin-frbg]

The "new" errors from common.h appear to be over use of the phrase __inline void somefunction rather than void __inline somefunction (and same with "int"). What is worrying is that it still appears to compile some x86 code (else it should not need to include common_x86.h, and the errors from param.h appear to be in settings for Intel CPUs as well). Wouldn't it need to create PPC code (and access an OpenBLAS compiled for PPC440 target to run on the dSPACE board rather than the x86 cpu of its Windows host ?

[brada4]

The errors say you are building x86 assembly code for supposedly powerpc with a powerpc compiler. Maybe you start taking things seriously and at least get a compiler matching your target?

Sorry if this sounds overreacting.

[martin-frbg]

@brada4 mind your words please...

[martin-frbg]

Maybe I am confused about what you are trying to compile here with the PPC compiler - is it OpenBLAS itself, or is it your own code ? (I expect you will need to compile both, as you will probably need a PPC version of the library if you want the BLAS calls to be executed on the PPC440 board - the mingw compile on your laptop created the windows-style library for the Xeon cpu only). In your own code, you would probably include openblas_config.h and cblas.h, but not OpenBLAS' internal headers like common.h

[mohseninima]

I have tried making some diagrams to show what is going on So this is what works normally (for the past 6+ years)

This is what I am currently trying to do

and if I am understanding correctly @martin-frbg , you are proposing this? I would update the library for the new target?

[martin-frbg]

I am not sure if I understand the mex workflow - where you currently have "Reference BLAS", is this the source code or a binary ? (I expect this would have to be either sourcecode or a precompiled ppc binary if you want the BLAS calls to be executed on the dSPACE board)

[mohseninima]

I am not sure if I understand the mex workflow - where you currently have "Reference BLAS", is this the source code or a binary ? (I expect this would have to be either sourcecode or a precompiled ppc binary if you want the BLAS calls to be executed on the dSPACE board)

It is the source

[martin-frbg]

OK, then you would probably start from the OpenBLAS source as well, and build it for TARGET=PPC440 rather than relying on the autodetection that would only see the Xeon and Windows. But whatever you do you should not include common.h in your own MYCODE.c

[brada4]

In principle at the point OpenBLAS is mixed in in the picture it could be blob produced outside mex just like the "other code" in same static library format. Almost same as with win32 dll you produced earlier. So mymodel.c would be same as mycode.c (with added matlab function wrappers) then compiled into native PPC executable.

[mohseninima]

Ok, so I am now trying to make a static library for the powerpc portion of the compilation. Would this be the correct command?

make TARGET=PPC440 HOSTCC=gcc CC=powerpc-linux-gnu-gcc FC=powerpc-linux-gnu-gfortran CFLAGS='-static-libgcc -static-libstdc++ -static -ggdb' FFLAGS='-static'

I receive the following output. Compilation fails at line 1784 complaining about junk at the end of the line failedcompile.txt

[brada4]

CFLAGS should be replaced with CCOMMON_OPT and FFLAGS alike, the "standard" names are changed internally e.g. different set for building LAPACK etc.

I think you need to use your dspace compiler (gcc will work if target is really kind of Linux)

OpenBLAS does not use libstdc++ , and it actually builds static library, see Makefile.rule for all possible options (technically "ar" archive on Linux, but check with "file" command so it is same as dspace static components)

[martin-frbg]

The "junk at end of line" messages are from the assembler apparently not understanding register names like r14 (or more likely, misinterpreting the DCBT macro that is defined (as a L1_PREFETCH) in common_power.h. Not sure if this is due to "wrong" compiler (or rather, assembler) or some other problem.

[brada4]

Probably it hits C comments that normally are stripped out by C compiler, but maybe not so everywhere in long dormant codes...

like: xor r1,r1 // we did something

[brada4]

@mohseninima could you try building with make -i and all your parameters (after make clean) It will run build through showing all errors, no working library produced, just that it will not stop on first encountered error.

[martin-frbg]

@brada4 no, it is stumbling over instructions like DCBT(A01,PREA) (where "A01" is #defined to be r14 , "PREA" is r24) that expand to dcbt r24, r14 according to common_power.h. No comments anywhere near these. As far as I can tell, PPC440 support in OpenBLAS is inherited from the original GotoBLAS.

[mepholic]

I'm hitting this issue trying to build with target PPC970. After looking at this carefully, and running the affected files through the preprocessor (ex: cpp -I src/OpenBLAS-0.3.7 src/OpenBLAS-0.3.7/kernel/power/gemm_kernel_altivec.S), you can see that the DCBT macro is interpreted as something like: dcbt 8, 29, 7

The IBM Documentation claims:

The dcbt instruction serves as both a basic and extended mnemonic. The dcbt mnemonic with three operands is the basic form, and the dcbt with two operands is the extended form. In the extended form, the TH field is omitted and assumed to be 0b0000.

The example assembly code on that page goes on to show them using the extended mnemonic, but does not make use of the basic mnemonic. My assembler (GNU assembler version 2.32 (powerpc64-foxkit-linux-musl) using BFD version (GNU Binutils) 2.32) does not seem to like the "basic mnemonic" syntax with 3 arguments to the dcbt instruction. When I attempt to build OpenBLAS, I get failures in the same files as @mohseninima did above. For instance:

../kernel/power/gemm_kernel_altivec.S: Assembler messages:
../kernel/power/gemm_kernel_altivec.S:348: Error: junk at end of line: `7'
../kernel/power/gemm_kernel_altivec.S:355: Error: junk at end of line: `8'
../kernel/power/gemm_kernel_altivec.S:409: Error: junk at end of line: `7'
make[1]: *** [Makefile.L3:671: strmm_kernel_LN.o] Error 1

I'm not sure which assembler this code was written for or tested on, but it appears that GNU assembler is not one of them.

Additionally, I question the macro definition of DCBT() in this code, which is the chosen form for my hardware based on the defines in the immediately preceding code (I'm building on Linux, not Darwin or FreeBSD). Regardless, it appears that the DCBT() macro reverses the order of the arguments passed to dcbt, which seems to counter the IBM documentation linked above. I'm not sure if it's intentional or not, but it certainly doesn't look right.

I have hardware that I can readily test fixes on. Any help that can be provided in fixing this issue would be greatly appreciated! For reference, my host machine is an IBM Power9, which should be backwards compatible with the PPC970.