cppflow2: Discussions of SavedModel API in TF2

[ljn917]

This issue serves as a shared note to track the status of SavedModel C API in TF 2.0, and also for discussions on possible implementations in cppflow2.

The original RFC proposal is here. The loading functions are defined here

However, as of this writing, the new SavedModel C API in TF 2.0 is still work-in-progress. It is not enabled in bazel. To enable the API, the following patch is needed. In addition, headers need to be copied manually. Although this patch enables the experimental APIs in the libtensorflow, calling these APIs will throw unimplemented errors. (at least in v2.3) I saw commits were submitted to support the new SavedModel C API in v2.4, so some of them may work now.

diff --git a/tensorflow/BUILD b/tensorflow/BUILD
index d00608ccc9..8bab3a6e3e 100644
--- a/tensorflow/BUILD
+++ b/tensorflow/BUILD
@@ -746,6 +746,7 @@ tf_cc_shared_object(
         "//tensorflow/c:version_script.lds",
         "//tensorflow/c/eager:c_api",
         "//tensorflow/c/eager:c_api_experimental",
+        "//tensorflow/c/experimental/saved_model/public:c_saved_model_api",
         "//tensorflow/core:distributed_tensorflow_dependencies",
         "//tensorflow/core:tensorflow",
     ],

There seems to be two ways to call the model, ConcreteFunction (TF_GetSavedModelConcreteFunction) and SignatureDefFunction (TF_GetSavedModelSignatureDefFunction) defined in this header, but both of them return TFE_Op* (TF_SignatureDefFunctionMakeCallOp and TF_ConcreteFunctionMakeCallOp). They should be compatible with eager execution in cppflow2.

Additional notes

• The ConcreteFunction signature may involve high-level python class, e.g. RaggedTensorSpec for RaggedTensor. They may require significant work to be implemented in C++. The corresponding SignatureDefFunction is rather low-level, where all the Tensors are flattened, e.g. the prevous RaggedTensor is represented by two flat Tensors. From this perspective, starting with SignatureDefFunction seems to be a good idea.

Edit:

• ConcreteFunction (TF_GetSavedModelConcreteFunction) implementation seems to be available on the master branch.
• SignatureDefFunction (TF_GetSavedModelSignatureDefFunction) is not implemented yet.
• The tests provide some examples.

Edit2: A WIP for the TF v2 SavedModel API in cppflow: https://github.com/ljn917/cppflow/tree/cppflow2-savedmodel

Known limitations:

• Concrete function can only have one input signature. It means it cannot load keras models. You need to annotate the functions with tf.function.
• SignatureDef function is not supported in libtensorflow yet.

[bnascimento]

Thanks @ljn917 @serizba I stumbled into this thread, after failing alot to use c/cc tensorflow 2.x APIs. Your work here gave me lots of good insights. Please keep it up