Reproduce Issue

[ghost]

Hi,

I am trying to reproduce the results reported in the paper, but I encounter some problems. Let me use vanillaLSTM.py to illustrate. I test VanillaLSTM on JS_non_terminal.pickle and JS_terminal_1k_whole.pickle, and the following is the configuration according to the paper:

class TestConfig(object):
  """Tiny config, for testing."""
  init_scale = 0.05
  learning_rate = 0.001
  max_grad_norm = 5
  num_layers = 1
  num_steps = 50
  attn_size = 50
  hidden_sizeN = 300
  hidden_sizeT = 1200
  sizeH = 1500
  max_epoch = 8
  max_max_epoch = 8
  keep_prob = 1.0
  lr_decay = 0.6
  batch_size = 128
  vocab_size = 95, 50001

However, the train accuracy and valid accuracy I obtain is around 0.48-0.49. If I test on the test set (uncomment lines 360-364), the test accuracy is 0.46. The paper reports a test accuracy of 0.699. Could you provide some instructions on how to reproduce the results?

Moreover, I find you have modified the batch_size and num_steps for test set in lines 343-344. What is the reason for that?

Thanks!

[jack57lee]

Hi,

Sorry for the late response! And thanks for your interest in our paper!

Seems that max_epoch should be set to 1. If you set it to 8, you will not have learning rate decay.

As for setting num_steps to 1, we follow the example in official Tensorflow tutorial. I do not quite remember the reason. It might be we are generating in an auto-regressive manner (one-by-one) during testing.

Sorry again for the unclear documentation. I will find some time to refine it.