TensorFlow.NET icon indicating copy to clipboard operation
TensorFlow.NET copied to clipboard

[BUG Report]: Keras model.save() throws exception when saving to network path on Windows

Open abrguyt opened this issue 1 year ago • 0 comments

Description

Using the custom packaging of the latest Tensorflow.NET version from the Github Repo (to overcome the bug in the official NuGet package - see https://github.com/SciSharp/TensorFlow.NET/issues/1246#issuecomment-2630024514 I got Tensorflow.NET / Keras model fitting, saving and loading working well.

Just when saving a Keras model to a Windows network path like '\FS-01\COM-Data$\ML Models' an message-less exception is thrown in

at Tensorflow.Eager.EagerRunner.SetOpAttrList(Context ctx, SafeEagerOpHandle op, String key, Object values, TF_AttrType type, Dictionary`2 attr_list_sizes, Status status) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Eager\EagerRunner.TFE_FastPathExecute.cs:line 326

Looking at the source it seems line 323 in the file 'TensorFlowNET.Core\Eager\EagerRunner.TFE_FastPathExecute.cs' is causing this with

throw new NotImplementedException("");

Saving the model to a local folder on the file system works perfectly well - I suspect somehow the unusual characters in the network file path cause this exception; but I can't dive deep in the Tensorflow.NET source to figure out why this is happening - any suggestions?

Full stack trace:

Source: Tensorflow.Binding
 Strack Trace:    at Tensorflow.Eager.EagerRunner.SetOpAttrList(Context ctx, SafeEagerOpHandle op, String key, Object values, TF_AttrType type, Dictionary`2 attr_list_sizes, Status status) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Eager\EagerRunner.TFE_FastPathExecute.cs:line 326
   at Tensorflow.Eager.EagerRunner.SetOpAttrs(SafeEagerOpHandle op, Object[] attrs) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Eager\EagerRunner.TFE_FastPathExecute.cs:line 241
   at Tensorflow.Eager.EagerRunner.TFE_ExecuteCancelable(Context ctx, String device_name, String op_name, Tensor[] inputs, Object[] attrs, Int32 num_outputs) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Eager\EagerRunner.TFE_Execute.cs:line 63
   at Tensorflow.Eager.EagerRunner.TFE_Execute(Context ctx, String device_name, String op_name, Tensor[] inputs, Object[] attrs, Int32 num_outputs) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Eager\EagerRunner.TFE_Execute.cs:line 36
   at Tensorflow.Eager._execute.quick_execute(String op_name, Int32 num_outputs, Tensor[] inputs, Object[] attrs, Context ctx, String name) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Eager\execute.cs:line 31
   at Tensorflow.io_ops.save_v2_eager_fallback(Tensor prefix, String[] tensor_names, String[] shape_and_slices, Tensor[] tensors, String name, Context ctx) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Operations\io_ops.cs:line 58
   at Tensorflow.io_ops.save_v2(Tensor prefix, String[] tensor_names, String[] shape_and_slices, Tensor[] tensors, String name) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Operations\io_ops.cs:line 40
   at Tensorflow.tensorflow.IoApi.save_v2(Tensor prefix, String[] tensor_names, String[] shape_and_slices, Tensor[] tensors, String name) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\APIs\tf.io.cs:line 45
   at Tensorflow.Checkpoint.SingleDeviceSaver.save(Tensor file_prefix, CheckpointOptions options) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\functional_saver.cs:line 79
   at Tensorflow.Checkpoint.MultiDeviceSaver.<>c__DisplayClass5_2.<save>b__6(ITensorFlowObject _) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\functional_saver.cs:line 285
   at Tensorflow.Binding.tf_with[T](T py, Action`1 action) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Binding.Util.cs:line 199
   at Tensorflow.Checkpoint.MultiDeviceSaver.<>c__DisplayClass5_0.<save>g__save_fn|1() in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\functional_saver.cs:line 283
   at Tensorflow.Checkpoint.MultiDeviceSaver.save(Tensor file_prefix, CheckpointOptions options) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\functional_saver.cs:line 305
   at Tensorflow.Checkpoint.MultiDeviceSaver.save(String file_prefix, CheckpointOptions options) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\functional_saver.cs:line 309
   at Tensorflow.Checkpoint.TrackableSaver.<>c__DisplayClass14_0.<save_cached_when_graph_building>b__0() in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\checkpoint.cs:line 119
   at Tensorflow.Checkpoint.TrackableSaver.save_cached_when_graph_building(String file_prefix, Tensor object_graph_tensor, CheckpointOptions options) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\checkpoint.cs:line 136
   at Tensorflow.Checkpoint.TrackableSaver.save(String file_prefix, Nullable`1 checkpoint_number, Session session, CheckpointOptions options) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Checkpoint\checkpoint.cs:line 180
   at Tensorflow.SavedModelUtils.save_and_return_nodes(Trackable obj, String export_dir, ConcreteFunction signatures, SaveOptions options, Boolean experimental_skip_checkpoint) in D:\temp\TensorFlow.NET-master\src\TensorFlowNET.Core\Training\Saving\SavedModel\save.cs:line 46
   at Tensorflow.Keras.Saving.SavedModel.KerasSavedModelUtils.save_model(Model model, String filepath, Boolean overwrite, Boolean include_optimizer, ConcreteFunction signatures, SaveOptions options, Boolean save_traces)
   at Tensorflow.Keras.Engine.Model.save(String filepath, Boolean overwrite, Boolean include_optimizer, String save_format, SaveOptions options, ConcreteFunction signatures, Boolean save_traces)

Reproduction Steps

No response

Known Workarounds

No response

Configuration and Other Information

Windows Server 2022, Windows 10 Tensorflow.NET v 0.150 Tensorflow.Keras v 0.15

abrguyt avatar Feb 10 '25 05:02 abrguyt