Rasa App not sending images to livechat

[thecion]

Hello ppl, I'm having a problem trying to work with images in livechat through RasaApp.

This appears to be happening only when using a Rasa Bot with RasaApp, since I can send images to the user by myself and the user can reply with another image during a "normal" conversation.

Testing:

• I have an utter for the images setup like this in my domain.yml:

utter_explica_numprocesso:
  - image: https://s.conjur.com.br/img/b/tabela-numeracao.jpeg
    text: É um número grande, conforme esta imagem.
  - image: https://s.conjur.com.br/img/b/tabela-numeracao.jpeg
    text: É um número grande, conforme essa imagem. Mas ele é fundamental. Vamos lá?

• Images are working fine if I use the Integration menu (Administration -> Integration), in which I have set up an outgoing webhook for communicating with the Rasa Server. Below a screenshot of a Rasa Bot sending an Image to the user inside RocketChat, via integration webhook:

• However the same utter does not work with livechat and RasaApp, as shown in the screenshot below

• The image is not shown as well in the omnichannel chat log:

My questions are:

1. Should RasaApp work with linked images?
2. If not, how should I send files to the App API POST incoming?

Best regards.

[murtaza98]

Hey @thecion Unfortunately the app isn't handling media properly :disappointed: and there is definitely room for improvement there.

Could you help me by confirming the payload which the Rasa rest api returns back to the app if it has a image response?

[
  {"text": "Hey Rasa!"}, {"image": "http://example.com/image.jpg"}
]

Also any idea what other types of files can be shared by the Rasa bot so I can add support to that as well?

[thecion]

Hello @murtaza98 , thanks for your quick reply and I appreciate your effort in making this App even better.

Could you help me by confirming the payload which the Rasa rest api returns back to the app if it has a image response?

First, I have some logs from our Rasa server:

• From the Rasa Log:

[
   BotUttered("É um número grande, conforme esta imagem.",
   {
      "elements":null,
      "quick_replies":null,
      "buttons":null,
      "attachment":"https://s.conjur.com.br/img/b/tabela-numeracao.jpeg",
      "image":"https://s.conjur.com.br/img/b/tabela-numeracao.jpeg",
      "custom":null
   },
   {
      "channel":"rocketchat",
      "utter_action":"utter_explica_numprocesso"
   },
   1625241198.3047285)
]"'"

• From our tracker store

{
   "event":"bot",
   "timestamp":1625238699.9047103,
   "metadata":{
      "channel":"rocketchat",
      "utter_action":"utter_explica_numprocesso"
   },
   "text":"\u00c9 um n\u00famero grande, conforme esta imagem.",
   "data":{
      "elements":null,
      "quick_replies":null,
      "buttons":null,
      "attachment":"https://s.conjur.com.br/img/b/tabela-numeracao.jpeg",
      "image":"https://s.conjur.com.br/img/b/tabela-numeracao.jpeg",
      "custom":null
   }
} 

From what I understand Rasa sends an URL of the image, not the image itself. I could be wrong here (if anyone wanna correct me it would be very helpful =) ), but it seems RocketChat is the responsible for showing the image to the user.

Rasa is using the chat.postMessage endpoint from RocketChat API, as show in its DEBUG log:

2021-07-02 13:50:56 DEBUG    urllib3.connectionpool  - Starting new HTTP connection (1): 192.168.100.54:3000
2021-07-02 13:50:56 DEBUG    urllib3.connectionpool  - http://192.168.100.54:3000 "POST /api/v1/chat.postMessage HTTP/1.1" 200 None

I'm able to replicate the message Rasa sends by sending this to the RocketChat API.

{ 
    "channel": "#channelname or string id",
    "attachments": [
        {
        "image_url": "https://images-na.ssl-images-amazon.com/images/I/71dhm1tcD%2BL._AC_SL1500_.jpg"
        }
    ]
}

Also any idea what other types of files can be shared by the Rasa bot so I can add support to that as well?

In our case I believe image files like PNG, JPG, GIF are the most common. Since these bots usually work in small chat windows I don't think people would send anything too big (well I wouldn't anyway :P ).

[murtaza98]

Please check if this comment is helpful here -> https://github.com/RocketChat/Apps.Rasa/issues/37#issuecomment-1117441330