chychkan
error in traning model with metal
Hi,guys with edit ".dfl/DeepFaceLab/core/lears/nn.py" line 201 "suggest_all_gpu=True" we can choose metal when traning model but it still report an error : Choose one or several GPU idxs (separated by comma).
[CPU] : CPU [0] : METAL
[0] Which GPU indexes to choose? : 0
Metal device set to: Apple M1
systemMemory: 16.00 GB maxCacheSize: 5.33 GB
Initializing models: 0%| | 0/7 [00:00<?, ?it/s] Error: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) placed on device Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0 Assign_210 (Assign) /device:GPU:0
[[node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform
(defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) ]]
Errors may have originated from an input operation. Input Source operations connected to node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: In[0] encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape:
Operation defined at: (most recent call last)
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self.build_sub(sublayer, f"{name}{i}")
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Additional information about colocations:No node-device colocations were active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation. Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277>
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1579, in get_variable
return get_variable_scope().get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1322, in get_variable
return var_store.get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 578, in get_variable
return _true_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 531, in _true_getter
return self._get_single_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 952, in _get_single_variable
v = variables.VariableV1(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 268, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 213, in _variable_v1_call
return previous_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 206, in
Traceback (most recent call last): File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1380, in do_call return fn(*args) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1362, in run_fn self.extend_graph() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1403, in extend_graph tf_session.ExtendSession(self.session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0 Assign_210 (Assign) /device:GPU:0
[[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 657, in on_initialize model.init_weights() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Saveable.py", line 106, in init_weights nn.init_weights(self.get_weights()) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/ops/init.py", line 48, in init_weights nn.tf_sess.run (ops) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 970, in run result = self._run(None, fetches, feed_dict, options_ptr, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1193, in run results = self.do_run(handle, final_targets, final_fetches, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1373, in do_run return self.do_call(run_fn, feeds, fetches, targets, options, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1399, in do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) placed on device Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0 Assign_210 (Assign) /device:GPU:0
[[node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform
(defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) ]]
Errors may have originated from an input operation. Input Source operations connected to node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: In[0] encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape:
Operation defined at: (most recent call last)
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self.build_sub(sublayer, f"{name}{i}")
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Additional information about colocations:No node-device colocations were active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation. Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277>
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1579, in get_variable
return get_variable_scope().get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1322, in get_variable
return var_store.get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 578, in get_variable
return _true_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 531, in _true_getter
return self._get_single_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 952, in _get_single_variable
v = variables.VariableV1(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 268, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 213, in _variable_v1_call
return previous_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 206, in
getting this error
`Initializing models: 0%| | 0/5 [00:00<?, ?it/s] Error: Graph execution error:
Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self.build_sub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self.build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in build_sub self.build_sub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self.build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0
[[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1617, in get_variable
return get_variable_scope().get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1333, in get_variable
return var_store.get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 589, in get_variable
return _true_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 542, in _true_getter
return self._get_single_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 963, in _get_single_variable
v = variables.VariableV1(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 265, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 210, in _variable_v1_call
return previous_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 203, in
Traceback (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1377, in do_call return fn(*args) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in run_fn self.extend_graph() File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1400, in extend_graph tf_session.ExtendSession(self.session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0
[[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 657, in on_initialize model.init_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Saveable.py", line 106, in init_weights nn.init_weights(self.get_weights()) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/ops/init.py", line 48, in init_weights nn.tf_sess.run (ops) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 967, in run result = self._run(None, fetches, feed_dict, options_ptr, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1190, in _run results = self._do_run(handle, final_targets, final_fetches, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1370, in _do_run return self._do_call(_run_fn, feeds, fetches, targets, options, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1396, in _do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self.build_sub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self.build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in build_sub self.build_sub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self.build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0
[[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1617, in get_variable
return get_variable_scope().get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1333, in get_variable
return var_store.get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 589, in get_variable
return _true_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 542, in _true_getter
return self._get_single_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 963, in _get_single_variable
v = variables.VariableV1(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 265, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 210, in _variable_v1_call
return previous_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 203, in
`
Wondering if this is our answer https://developer.apple.com/forums/thread/700169?answerId=704851022#704851022
@Smiril I am using your fork but cant train my model on GPU . It says tf.reduce_mean has some problem. Please help me run it on the GPU. Thanks
@Smiril I am using M2 macbook pro. Are there any settings I need to change?