File upload with over 65kb cannot send to the packet

[captainjaypee01]

I'm trying to upload over 65kb and sending this data to the broker, When I tried to send it I received an error saying "Sending data over the socket to the broker failed."

So, I tried to track down which part of the code I'm getting the error and I found this function

https://github.com/php-mqtt/client/blob/master/src/MqttClient.php

/**
     * Writes some data to the socket. If a {@see $length} is given, and it is shorter
     * than the data, only {@see $length} amount of bytes will be sent.
     *
     * @param string   $data
     * @param int|null $length
     * @return void
     * @throws DataTransferException
     */
    protected function writeToSocket(string $data, int $length = null): void
    {
        $calculatedLength = strlen($data);
        $length           = min($length ?? $calculatedLength, $calculatedLength);

        $result = @fwrite($this->socket, $data, $length);

        if ($result === false || $result !== $length) {
            $this->logger->error('Sending data over the socket to the broker failed.');
            throw new DataTransferException(
                DataTransferException::EXCEPTION_TX_DATA,
                'Sending data over the socket failed. Has it been closed?'
            );
        }

        $this->bytesSent += $length;

        $this->logger->debug('Sent data over the socket: {data}', ['data' => $data]);

        // After writing successfully to the socket, the broker should have received a new message from us.
        // Because we only need to send a ping if no other messages are delivered, we can safely reset the ping timer.
        $this->lastPingAt = microtime(true);
    }

Based on this one, the result is having a different value from the $length or $calculatedLength. So, I'm not sure if this is a problem with the PHP. I'm also using the wirepas protobuffer file to serialize structured data.

I hope anyone can help me regarding this, because I was also thinking to use a python script just to do the upload. Our main backend is written in PHP Laravel that's why we're still sticking on this PHP-MQTT Library.

[Namoshek]

Which broker are you using and have you verified it accepts messages larger than 64kb?

[captainjaypee01]

I'm using mosquitto as the broker. If I upload the file lower 65kb it can send to the broker, but if its greater than 65kb the error comes in. I have resolution if the data is greater than 65kb I'm splitting the data so it will not check the whole data. And I'm getting the same size as it needs to compared with the given $length

[Namoshek]

This is strange, because there is actually a test for messages with a size up to 2MB (see here).

[captainjaypee01]

As you can see here in this code, I tried to splitting the data. The @fwrite can only accept 65kb which is weird for me, thats why I have to split the whole data and concatenate it so the @fwrite function can handle the whole data of 4096 size per loop.

if(strlen($data) > 60000){
            $pieces = str_split($data, 4096);
            foreach($pieces as $piece){
                $result += @fwrite($this->socket, $piece, $length);
                Log::info("PIECES $result");
            }
        }
        else{
            $result = @fwrite($this->socket, $data, $length);
        }

[Namoshek]

What kind of hardware (pc, rpi, ...), operating system and PHP version are you using? I cannot reproduce this behavior.

[captainjaypee01]

Local - Laptop Dell Processor: i7-10510U Memory: 16GB Ram OS: Windows 10 PHP: PHP 7.4.24 (cli) (built: Sep 21 2021 13:38:25) ( ZTS Visual C++ 2017 x64 ) Copyright (c) The PHP Group Zend Engine v3.4.0, Copyright (c) Zend Technologies

Testing environment OS: Linux PHP: PHP 7.4.30 (cli) (built: Jun 27 2022 08:20:11) ( NTS ) Copyright (c) The PHP Group Zend Engine v3.4.0, Copyright (c) Zend Technologies with Zend OPcache v7.4.30, Copyright (c), by Zend Technologies

[captainjaypee01]

These are the files and filetypes that I'm trying to upload into the socket

Also below here are the error result:

[2022-10-06 07:15:28] local.INFO: MQTT CLIENT LENGTH 107395  
[2022-10-06 07:15:28] local.INFO: MQTT CLIENT RESULT 65536  
[2022-10-06 07:15:28] local.INFO: MQTT CLIENT CALCULATED LENGTH 107395  
[2022-10-06 07:15:28] local.ERROR: MQTT [{host}:{port}] [{clientId}] Sending data over the socket to the broker failed. {"host":"www.lingjacksmart.com","port":8883,"clientId":"abc45de1be194bf03cf6"} 
[2022-10-06 07:15:28] local.ERROR: [65] Transferring data over socket failed: Sending data over the socket failed. Has it been closed? {"userId":282,"exception":"[object] (PhpMqtt\\Client\\Exceptions\\DataTransferException(code: 65): [65] Transferring data over socket failed: Sending data over the socket failed. Has it been closed? at C:\\Users\\John Paul D Dala\\Documents\\projects\\smartiot\\mobileapi-azure\\vendor\\php-mqtt\\client\\src\\MqttClient.php:1155)
[stacktrace]
#0 C:\\Users\\John Paul D Dala\\Documents\\projects\\smartiot\\mobileapi-azure\\vendor\\php-mqtt\\client\\src\\MqttClient.php(1105): PhpMqtt\\Client\\MqttClient->writeToSocket('2\\xFF\\xC6\\x06\\x00.gw-reques...', 107395)

[Namoshek]

I'm not familiar with OTAP files, but if they are in binary format and they contain specific byte sequences (e.g. a BOM), they could in theory interfere with the protocol. Can you test a different file (e.g. a text file with the same character 100.000 times) and see if the same error occurs?

And is your broker located behind a proxy by any chance?