oracle
install unknown packages
I'd like to install tensorflow, keras and some other packages:
bash-4.2# graalpython -m ginstall install keras
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
Unknown package: 'keras'
bash-4.2# graalpython -m ginstall install tensorflow
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
Unknown package: 'tensorflow'
Is there any way to make these packages available in graalpython? Is there a general approach for any package?
I'm highly interested in making these packages available in graalpython. What's the current state for tensorflow? Is somebody working on it? Can you give me some hints on how to add these packages to graalpython, by myself? I took a glance at ginstall and see a lot of custom patches. Before i go into a lot of trial and error, I'd like to know if there's a procedure/walkthrough to create these patches and contribute to ginstall.
@nodueck there's no fixed "procedure" to add something to ginstall. If a package can be added, we just add it, if it needs patches, we put them there. Tensorflow, however, will not work. Someone from our team spent 3 months documenting the issues arising around Tensorflow, and they are many, including no Fortran compiler being available that can generate LLVM bitcode for Tensorflow, the complex build system not working on Graal Python, missing threading and multiprocessing support on Graal Python, ... the list goes on. It will take quite a while to fix them, it's unlikely to work in the next 6 months.
There's no current progress on these. We are working on getting basic compatibility with things like the core types and modules working first (things like import, traceback, the fraction and complex types etc - these all currently are incomplete). We will get back to these C extension modules in the latter half of 2021
including no Fortran compiler being available that can generate LLVM bitcode for Tensorflow
As of LLVM 11 (released a couple weeks ago), there is a project called flang that compiles Fortran to LLVM bitcode. https://github.com/llvm/llvm-project/tree/llvmorg-11.0.0/flang We should be able to include that in the llvm toolchain distribution in Sulong and be able to compile Fortan programs in the same way as clang - embedding LLVM bitcode into the object file.
@rahulmutt thanks, we have been expecting the availability of flang hoping for exactly what you describe :) Unfortunately, the state of flang (née F18) is that it currently does not fully integrate with LLVM. Most importantly, it does not produce LLVM bitcode, yet. We'll have to wait a bit longer for it to mature and fully integrate with the LLVM project before we can use it. (c.f: https://releases.llvm.org/11.0.0/tools/flang/docs/ReleaseNotes.html
Flang is able to parse a comprehensive subset of the Fortran language and check it for correctness. Flang is not yet able to generate LLVM IR for the source code and thus is unable to compile a running binary.
Flang is able to unparse the input source code into a canonical form and emit it to allow testing. Flang can also invoke an external Fortran compiler on this canonical input.
)
I am trying to summarize unknown packages situation in #228
Niko @nodueck can you point to specific issues with tensorflow ?
@paulvi installation of multiple dependencies of tensorflow doesn't even work, leave alone tensorflow itself. It's a huge amount of code and we have some work ahead of us to get it to build, install, load, and then run.
tensorflow 2.15.0 can now be installed with pip, torch 2.2.1 as well. Unfortunately, they still have to be built from source during the installation, which is slow and requires you to have the right compiler and libraries in the system. Hopefully, in the future we will have prebuilt binary wheels for them.
