invalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [37007] rhs shape= [12003]

[SeekPoint]

(venv) xxx590cc37a5:conv_seq2seq xxx.xxx$ (venv) xxx590cc37a5:conv_seq2seq xxx.xxx$ python -m bin.train --config_paths="example_configs/conv_seq2seq.yml,example_configs/train_seq2seq.yml,example_configs/text_metrics_bpe.yml" \

--model_params " vocab_source: vocab.bpe.32000 vocab_target: vocab.bpe.32000"
--input_pipeline_train " class: ParallelTextInputPipelineFairseq params: source_files: train.tok.clean.bpe.32000.en target_files: train.tok.clean.bpe.32000.de"
--input_pipeline_dev " class: ParallelTextInputPipelineFairseq params: source_files: newstest2013.tok.bpe.32000.en target_files: newstest2013.tok.bpe.32000.de"
--batch_size 32
--eval_every_n_steps 5000
--train_steps 1000000
--output_dir models \

INFO:tensorflow:Loading config from /Users/xxx.xxx/xxx_prj/conv_seq2seq/example_configs/conv_seq2seq.yml INFO:tensorflow:Loading config from /Users/xxx.xxx/xxx_prj/conv_seq2seq/example_configs/train_seq2seq.yml INFO:tensorflow:Loading config from /Users/xxx.xxx/xxx_prj/conv_seq2seq/example_configs/text_metrics_bpe.yml INFO:tensorflow:Final Config: bucxxxts: 10,20,30,40 default_params:

• {separator: ' '}
• {postproc_fn: seq2seq.data.postproc.strip_bpe} hooks:
• {class: PrintModelAnalysisHook}
• {class: MetadataCaptureHook}
• {class: SyncReplicasOptimizerHook}
• class: TrainSampleHook params: {every_n_steps: 1000} metrics:
• {class: LogPerplexityMetricSpec} model: ConvSeq2Seq model_params: decoder.class: seq2seq.decoders.ConvDecoderFairseq decoder.params: {cnn.kwidths: '3,3,3', cnn.layers: 3, cnn.nhids: '256,256,256'} embedding.dim: 256 encoder.class: seq2seq.encoders.ConvEncoderFairseq encoder.params: {cnn.kwidths: '3,3,3,3', cnn.layers: 4, cnn.nhids: '256,256,256,256'} optimizer.clip_gradients: 0.1 optimizer.learning_rate: 0.25 optimizer.name: Momentum optimizer.params: {momentum: 0.99, use_nesterov: true} source.max_seq_len: 50 source.reverse: false target.max_seq_len: 50

WARNING:tensorflow:Ignoring config flag: default_params INFO:tensorflow:Setting save_checkpoints_secs to 600 INFO:tensorflow:Creating ParallelTextInputPipelineFairseq in mode=train INFO:tensorflow: ParallelTextInputPipelineFairseq: !!python/unicode 'num_epochs': null !!python/unicode 'shuffle': true !!python/unicode 'source_delimiter': !!python/unicode ' ' !!python/unicode 'source_files': [t, r, a, i, n, ., t, o, k, ., c, l, e, a, n, ., b, p, e, ., '3', '2', '0', '0', '0', ., e, n] !!python/unicode 'target_delimiter': !!python/unicode ' ' !!python/unicode 'target_files': [t, r, a, i, n, ., t, o, k, ., c, l, e, a, n, ., b, p, e, ., '3', '2', '0', '0', '0', ., d, e]

INFO:tensorflow:Creating ParallelTextInputPipelineFairseq in mode=eval INFO:tensorflow: ParallelTextInputPipelineFairseq: !!python/unicode 'num_epochs': 1 !!python/unicode 'shuffle': false !!python/unicode 'source_delimiter': !!python/unicode ' ' !!python/unicode 'source_files': [n, e, w, s, t, e, s, t, '2', '0', '1', '3', ., t, o, k, ., b, p, e, ., '3', '2', '0', '0', '0', ., e, n] !!python/unicode 'target_delimiter': !!python/unicode ' ' !!python/unicode 'target_files': [n, e, w, s, t, e, s, t, '2', '0', '1', '3', ., t, o, k, ., b, p, e, ., '3', '2', '0', '0', '0', ., d, e]

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_xxxep_checkpoint_max': 5, '_tf_random_seed': None, '_task_type': None, '_environment': 'local', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11b951e90>, '_tf_config': gpu_options { per_process_gpu_memory_fraction: 1.0 } , '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_evaluation_master': '', '_xxxep_checkpoint_every_n_hours': 4, '_master': ''} INFO:tensorflow:Creating PrintModelAnalysisHook in mode=train INFO:tensorflow: PrintModelAnalysisHook: {}

INFO:tensorflow:Creating MetadataCaptureHook in mode=train INFO:tensorflow: MetadataCaptureHook: {!!python/unicode 'step': 10}

INFO:tensorflow:Creating SyncReplicasOptimizerHook in mode=train INFO:tensorflow: SyncReplicasOptimizerHook: {}

INFO:tensorflow:Creating TrainSampleHook in mode=train INFO:tensorflow: TrainSampleHook: {!!python/unicode 'every_n_secs': null, !!python/unicode 'every_n_steps': 1000, !!python/unicode 'source_delimiter': !!python/unicode ' ', !!python/unicode 'target_delimiter': !!python/unicode ' '}

INFO:tensorflow:Creating LogPerplexityMetricSpec in mode=eval INFO:tensorflow: LogPerplexityMetricSpec: {}

INFO:tensorflow:Training model for 5000 steps INFO:tensorflow:Creating ConvSeq2Seq in mode=train INFO:tensorflow: ConvSeq2Seq: !!python/unicode 'decoder.class': !!python/unicode 'seq2seq.decoders.ConvDecoderFairseq' !!python/unicode 'decoder.params': {cnn.kwidths: '3,3,3', cnn.layers: 3, cnn.nhids: '256,256,256'} !!python/unicode 'embedding.dim': 256 !!python/unicode 'embedding.init_scale': 0.04 !!python/unicode 'embedding.share': false !!python/unicode 'encoder.class': !!python/unicode 'seq2seq.encoders.ConvEncoderFairseq' !!python/unicode 'encoder.params': {cnn.kwidths: '3,3,3,3', cnn.layers: 4, cnn.nhids: '256,256,256,256'} !!python/unicode 'inference.beam_search.beam_width': 0 !!python/unicode 'inference.beam_search.choose_successors_fn': !!python/unicode 'choose_top_k' !!python/unicode 'inference.beam_search.length_penalty_weight': 1.0 !!python/unicode 'optimizer.clip_embed_gradients': 5 !!python/unicode 'optimizer.clip_gradients': 0.1 !!python/unicode 'optimizer.learning_rate': 0.25 !!python/unicode 'optimizer.lr_decay_rate': 0.9 !!python/unicode 'optimizer.lr_decay_steps': 5000 !!python/unicode 'optimizer.lr_decay_type': !!python/unicode 'exponential_decay' !!python/unicode 'optimizer.lr_min_learning_rate': 1.0e-05 !!python/unicode 'optimizer.lr_staircase': true !!python/unicode 'optimizer.lr_start_decay_at': 0 !!python/unicode 'optimizer.lr_stop_decay_at': 2147483647 !!python/unicode 'optimizer.name': !!python/unicode 'Momentum' !!python/unicode 'optimizer.params': {!!python/unicode 'momentum': 0.99, !!python/unicode 'use_nesterov': true} !!python/unicode 'optimizer.sync_replicas': 0 !!python/unicode 'optimizer.sync_replicas_to_aggregate': 0 !!python/unicode 'position_embeddings.num_positions': 100 !!python/unicode 'source.max_seq_len': 50 !!python/unicode 'source.reverse': false !!python/unicode 'target.max_seq_len': 50 !!python/unicode 'vocab_source': !!python/unicode 'vocab.bpe.32000' !!python/unicode 'vocab_target': !!python/unicode 'vocab.bpe.32000'

INFO:tensorflow:Creating vocabulary lookup table of size 37007 INFO:tensorflow:Creating vocabulary lookup table of size 37007 INFO:tensorflow:Creating ConvEncoderFairseq in mode=train INFO:tensorflow: ConvEncoderFairseq: {cnn.kwidth_default: 3, cnn.kwidths: '3,3,3,3', cnn.layers: 4, cnn.nhid_default: 256, cnn.nhids: '256,256,256,256', embedding_dropout_xxxep_prob: 0.9, nhid_dropout_xxxep_prob: 0.9, position_embeddings.combiner_fn: tensorflow.add, position_embeddings.enable: true}

INFO:tensorflow:Creating ConvDecoderFairseq in mode=train INFO:tensorflow: ConvDecoderFairseq: {!!python/unicode 'cnn.kwidth_default': 3, !!python/unicode 'cnn.kwidths': !!python/unicode '3,3,3', !!python/unicode 'cnn.layers': 3, !!python/unicode 'cnn.nhid_default': 256, !!python/unicode 'cnn.nhids': !!python/unicode '256,256,256', !!python/unicode 'embedding_dropout_xxxep_prob': 0.9, !!python/unicode 'max_decode_length': 49, !!python/unicode 'nhid_dropout_xxxep_prob': 0.9, !!python/unicode 'nout_embed': 256, !!python/unicode 'out_dropout_xxxep_prob': 0.9, !!python/unicode 'position_embeddings.combiner_fn': !!python/unicode 'tensorflow.add', !!python/unicode 'position_embeddings.enable': true}

INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/V/read:0", shape=(256, 256), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/V:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/g/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/g:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/b/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/b:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/V:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/g:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/b:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/V:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/g:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/b:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/V:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/g:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/b:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/V:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/g:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/b:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/V/read:0", shape=(256, 256), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/V:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/g/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/g:0 INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/b/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/b:0 INFO:tensorflow:Create CheckpointSaverHook. 47 ops no flops stats due to incomplete shapes. Consider passing run_meta to use run_time shapes. Parsing GraphDef... Parsing RunMetadata... Parsing OpLog... Preparing Views... INFO:tensorflow:_TFProfRoot (--/31.97m params) model/conv_seq2seq/Variable (0/0 params) model/conv_seq2seq/decode/W (37007x256, 9.47m/9.47m params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_out/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_out/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_out/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_query/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_query/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_query/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_out/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_out/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_out/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_query/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_query/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_query/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_out/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_out/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_out/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_query/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_query/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_query/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_0/V (3x256x512, 393.22k/393.22k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_0/b (512, 512/512 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_0/g (512, 512/512 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_1/V (3x256x512, 393.22k/393.22k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_1/b (512, 512/512 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_1/g (512, 512/512 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_2/V (3x256x512, 393.22k/393.22k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_2/b (512, 512/512 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_2/g (512, 512/512 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/linear_mapping_before_cnn/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/linear_mapping_before_cnn/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/linear_mapping_before_cnn/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/linear_mapping_after_cnn/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/linear_mapping_after_cnn/b (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/linear_mapping_after_cnn/g (256, 256/256 params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/V (256x37007, 9.47m/9.47m params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b (37007, 37.01k/37.01k params) model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/g (37007, 37.01k/37.01k params) model/conv_seq2seq/decode/pos (100x256, 25.60k/25.60k params) model/conv_seq2seq/encode/W (37007x256, 9.47m/9.47m params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/V (3x256x512, 393.22k/393.22k params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/b (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/g (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/V (3x256x512, 393.22k/393.22k params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/b (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/g (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/V (3x256x512, 393.22k/393.22k params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/b (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/g (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/V (3x256x512, 393.22k/393.22k params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/b (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/g (512, 512/512 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/b (256, 256/256 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/g (256, 256/256 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/V (256x256, 65.54k/65.54k params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/b (256, 256/256 params) model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/g (256, 256/256 params) model/conv_seq2seq/encode/pos (100x256, 25.60k/25.60k params)

W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main "main", fname, loader, pkg_name) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 277, in tf.app.run() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 272, in main schedule=FLAGS.schedule) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 106, in run return task() File "seq2seq/contrib/experiment.py", line 104, in continuous_train_and_eval monitors=self._train_monitors) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 280, in new_func return func(*args, **kwargs) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 426, in fit loss = self._train_model(input_fn=input_fn, hooks=hooks) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 981, in _train_model config=self.config.tf_config) as mon_sess: File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 315, in MonitoredTrainingSession return MonitoredSession(session_creator=session_creator, hooks=all_hooks) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 601, in init session_creator, hooks, should_recover=True) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 434, in init self._sess = _RecoverableSession(self._coordinated_creator) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 767, in init _WrappedSession.init(self, self._create_session()) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 772, in _create_session return self._sess_creator.create_session() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 494, in create_session self.tf_sess = self._session_creator.create_session() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 375, in create_session init_fn=self._scaffold.init_fn) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 256, in prepare_session config=config) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 188, in _restore_checkpoint saver.restore(sess, ckpt.model_checkpoint_path) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1428, in restore {self.saver_def.filename_tensor_name: save_path}) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run run_metadata_ptr) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run feed_dict_string, options, run_metadata) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run target_list, options, run_metadata) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [37007] rhs shape= [12003] [[Node: save/Assign_96 = Assign[T=DT_FLOAT, _class=["loc:@model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b, save/RestoreV2_96)]]

Caused by op u'save/Assign_96', defined at: File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main "main", fname, loader, pkg_name) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 277, in tf.app.run() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 272, in main schedule=FLAGS.schedule) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 106, in run return task() File "seq2seq/contrib/experiment.py", line 104, in continuous_train_and_eval monitors=self._train_monitors) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 280, in new_func return func(*args, **kwargs) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 426, in fit loss = self._train_model(input_fn=input_fn, hooks=hooks) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 981, in _train_model config=self.config.tf_config) as mon_sess: File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 315, in MonitoredTrainingSession return MonitoredSession(session_creator=session_creator, hooks=all_hooks) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 601, in init session_creator, hooks, should_recover=True) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 434, in init self._sess = _RecoverableSession(self._coordinated_creator) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 767, in init _WrappedSession.init(self, self._create_session()) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 772, in _create_session return self._sess_creator.create_session() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 494, in create_session self.tf_sess = self._session_creator.create_session() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 366, in create_session self._scaffold.finalize() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 183, in finalize self._saver.build() File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1070, in build restore_sequentially=self._restore_sequentially) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 671, in build restore_sequentially, reshape) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 445, in _AddShardedRestoreOps name="restore_shard")) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 414, in _AddRestoreOps assign_ops.append(saveable.restore(tensors, shapes)) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 155, in restore self.op.get_shape().is_fully_defined()) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign use_locking=use_locking, name=name) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op op_def=op_def) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op original_op=self._default_original_op, op_def=op_def) File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [37007] rhs shape= [12003] [[Node: save/Assign_96 = Assign[T=DT_FLOAT, _class=["loc:@model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b, save/RestoreV2_96)]]

[tobyyouup]

@loveJasmine Have you load the correct checkpoint? Or can you try on a new folder without loading an existing checkpoint?