OCI Object Store Premature end of Content-Length delimited message body

[akshaysu]

Hi OCI team, I'm facing an issue when getting Objects from OCI OS. The issue is persistent when I try to download large files in parallel. Error message is "Premature end of Content-Length delimited message body (expected: 33,554,160; received: 1,474,560)" Same message is occurring for all files(InputStream).

OCI SDK version : 2.41.1

org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 33,554,160; received: 9,240,576)
        at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
        at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)
        at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:79)
        at com.oracle.bmc.io.internal.WrappedResponseInputStream.read(WrappedResponseInputStream.java:49)
        at com.oracle.bmc.io.internal.ContentLengthVerifyingInputStream.read(ContentLengthVerifyingInputStream.java:44)
        at com.oracle.bmc.io.internal.AutoCloseableContentLengthVerifyingInputStream.read(AutoCloseableContentLengthVerifyingInputStream.java:52)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:257)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2444)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2527)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2507)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:334)
        at oracle.nosql.driver.values.JsonUtils.createValueFromJson(JsonUtils.java:191)
        at oracle.nosql.driver.values.JsonUtils.parseObject(JsonUtils.java:298)
        at oracle.nosql.driver.values.JsonUtils.createValueFromJson(JsonUtils.java:257)
        at oracle.nosql.driver.values.JsonReader$MapValueJsonIterator.makeValue(JsonReader.java:174)
        at oracle.nosql.driver.values.JsonReader$MapValueJsonIterator.next(JsonReader.java:156)
        at oracle.nosql.driver.values.JsonReader$MapValueJsonIterator.next(JsonReader.java:106)
        at com.oracle.nosql.tools.migrator.objectstorage.ObjectStorageJsonSource.getNext(ObjectStorageJsonSource.java:115)
        at com.oracle.nosql.tools.migrator.MigratorSink.run(MigratorSink.java:55)
        at com.oracle.nosql.tools.migrator.MigratorPipeline.run(MigratorPipeline.java:107)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

This is an IOException thrown by InputStream return by GetObjectResponse.

OCI_OS_Get_error.log

[KartikShrikantHegde]

Thanks for reporting the issue. Our object storage team will take a look and get back to you.

[y-chandra]

@akshaysu - Can you please try this workaround to see if this resolves the issue?

[y-chandra]

Since we haven't heard from you in a while, we'll be closing this issue for now. Please feel free to reopen if the issue hasn't been fixed.

[akshaysu]

@y-chandra I reproduced the issue with below minimalistic code

public class App {
    public static void main(String[] args) throws IOException,
            InterruptedException {

        final ConfigFileReader.ConfigFile configFile =
                ConfigFileReader.parseDefault();

        final AuthenticationDetailsProvider provider =
                new ConfigFileAuthenticationDetailsProvider(configFile);
        ObjectStorageClient objectStorageClient = null;
        InputStream inputStream = null;
        try {
            //Do not auto close stream 
            shouldAutoCloseResponseInputStream(false);
            //Get OCI OS handle
            objectStorageClient =
                    ObjectStorageClient.builder().region(Region.AP_MUMBAI_1).build(provider);

            //Get object
            GetObjectRequest request = GetObjectRequest.builder()
                    .bucketName("bucket-migrator")
                    .namespaceName("mynamespace")
                    .objectName("zips.json")
                    .build();

            //get input stream
            inputStream = objectStorageClient.getObject(request)
                    .getInputStream();


            //read from inputStream and writ to out
            byte[] buf = new byte[128];
            while (true) {
                int r = inputStream.read(buf);
                if (r == -1) break;
                System.out.write(buf, 0, r);
                //sleep for 100ms This is important to reproduce
                Thread.sleep(100);
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            inputStream.close();
            objectStorageClient.close();
        }
    }
}

org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 3,182,409; received: 753,664)
	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
	at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
	at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:74)
	at com.oracle.bmc.io.internal.WrappedResponseInputStream.read(WrappedResponseInputStream.java:44)
	at com.oracle.bmc.io.internal.ContentLengthVerifyingInputStream.read(ContentLengthVerifyingInputStream.java:37)
	at com.example.App.main(App.java:64)

[JAL9302]

@akshaysu - did you try this workaround that @y-chandra had suggested above, to see if this resolves the issue?

[akshaysu]

@jodoglevy Yes. You can see above in the code auto close is set to false. Still I'm facing the same issue. shouldAutoCloseResponseInputStream(false);

[akshaysu]

@jodoglevy @y-chandra Any update on this?

[y-chandra]

@akshaysu - We are looking into the issue and will get back to you once we have an update.

[y-chandra]

I tried to replicate the issue, but instead of the above mentioned error, I got a connection reset error, which makes me think that this issue is caused by Object Storage service closing the connection before all the data is downloaded. Can you please open a customer support ticket with OCI mentioning the above issue?

[y-chandra]

@akshaysu - Did you get a chance to open a customer support ticket?

[akshaysu]

@y-chandra Yes. I reported the issue to Object storage internal team and they're looking into it. Thanks

[y-chandra]

Closing this issue for now. Please feel free to reopen if you need to.