sigmoid and tanh on the same tensor?

[keunwoochoi]

Hi, thanks for the codes!

Although Figure 4 in the original paper is described as in your code, Eq. 2 (which I assume to be more correct) says there are two different convolutions, i.e., I think it should be

(ResidualBlock)

    def forward(self, x, skip_size)
        input_tanh = self.dilated(x)
        input_sigmoid = self.dilated(x)

        # pixelCNN
        gated_tanh = self.gated_tanh(input_tanh)  # [-1, 1]
        gated_sigmoid = self.gate_sigmoid(input_sigmoid)  # [0, 1]
        gated = gated_tanh * gated_sigmoid  # [0, 1]

instead of

    def forward(self, x, skip_size)
        output = self.dilated(x)

        # pixelCNN
        gated_tanh = self.gated_tanh(output)  # [-1, 1]
        gated_sigmoid = self.gate_sigmoid(output)  # [0, 1]
        gated = gated_tanh * gated_sigmoid  # [0, 1]

But I'm also guessing, have you possibly looked into it before?

[golbin]

@keunwoochoi You're right.

And more correctly, tanh receives filter, sigmoid receives gate and z is multiplied them.

    def forward(self, x, skip_size)
        filter = self.dilated(x)
        gate = self.dilated(x)

        # pixelCNN
        filtered = self.gated_tanh(filter)
        gated = self.gate_sigmoid(gate)
        z = filtered * gated

Thanks your comments.