Parts of message are lost on IRC if split into multiple messages

[Phoenix616]

When the message is too long to fit into a single IRC message and it has to split it into multiple it will loose parts of the message at the split location (looks like 13 chars in one case, not sure if that's all the time or maybe depending on the user name length)

[zachbr]

I have confirmed this in a few messages. Oddly it seems like it's happening lower than Dis4IRC. More investigation needed on my end.

Thankfully, this is not the default case. The bot will prefer to send out to the PasteService by default. This was only revealed because the PasteService returned an error so it fell back to a multimessage. Would still be nice to get it fixed.

[zachbr]

Finally circled back to this and spent some more time looking into it. Last time I got to nearly the same place and then just sort of stopped. This looks like one of those fun "wow IRC is old"-type issues.

Per the IRC RFC:

   IRC messages are always lines of characters terminated with a CR-LF
   (Carriage Return - Line Feed) pair, and these messages SHALL NOT
   exceed 512 characters in length, counting all characters including
   the trailing CR-LF. Thus, there are 510 characters maximum allowed
   for the command and its parameters.  There is no provision for
   continuation of message lines.  See section 6 for more details about
   current implementations.

This the good ol' days of plain C, so 512 characters really means 512 bytes. Despite that, most IRC networks now use the UTF-8 encoding, and many languages (like Java!) no longer encode strings as ASCII. So if the IRC library is splitting messages into words and doing some fancy work to cut those words into useful boundaries so as to limit their size to a maximum of 512 characters (on Java's definition of a character), it will fail to handle cases where those characters actually end up encoded as multiple bytes. Now if you have some bridge software that is adding Unicode zero-width spaces into nicknames to prevent people from pinging themselves when they speak, well you just broke the function of the message splitter.

Sure enough:

List element at index 0 is of length 448 encoded as 452 bytes. Max bytes remaining is 449.
List element at index 1 is of length 442 encoded as 442 bytes. Max bytes remaining is 449.
List element at index 2 is of length 186 encoded as 186 bytes. Max bytes remaining is 449.

Some IRC networks (like EsperNet) silently truncate messages that are too long. Other networks may kick back an error message but I am not familiar with which those would be. Ergo (formerly Oragono) in it's default configuration actually does kick back an error message, which helped confirm this.

I will mess with this some more soon hopefully, running out of time today. This is the code that is currently doing the "cutting" to split long messages into multiple, each to be sent separately. I have pinged the library developer as well but we all live busy lives. One of us will hopefully get to this before too long. Famous last words given the date on this issue.