Paper icon indicating copy to clipboard operation
Paper copied to clipboard
verified publisher icon pilgr

KryoException: Buffer underflow

Open pilgr opened this issue 10 years ago • 19 comments

Very rarely crash happening in AppDialer app using Paper, affects only 0,05% sessions.

Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
a (io.paperdb.PaperTable)
       at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
       at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:373)
       at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:127)
       at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:667)
       at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
       at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:557)
       at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
       at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:204)
       at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:128)
       at io.paperdb.Paper.get(Paper.java:52)
       at name.pilgr.appdialer.Storage.getHistory(Storage.java:54)
       at name.pilgr.appdialer.search.StatsManager.init(StatsManager.java:18)
       at name.pilgr.appdialer.Engine.setFire(Engine.java:197)
       at name.pilgr.appdialer.Engine.doStartUp(Engine.java:235)
       at name.pilgr.appdialer.Engine$1.run(Engine.java:70)
       at name.pilgr.appdialer.util.CrashableTask.run(CrashableTask.java:18)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
       at java.util.concurrent.FutureTask.run(FutureTask.java:237)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:841)

pilgr avatar Jul 04 '15 11:07 pilgr

Looks like happens only once per affected user. All further data read attempts are successful.

pilgr avatar Jul 04 '15 11:07 pilgr

@pilgr Hi, I was using this library as an experiment in one of my projects. But I am having this issue every time I tried to retrieve an object for the first time. It is a specific object only.

raviteja06 avatar Dec 22 '15 05:12 raviteja06

@raviteja06 Is it 100% reproducible in your case? Could you write a unit test to reproduce it?

pilgr avatar Dec 22 '15 08:12 pilgr

@pilgr I guess it's mostly because of different things I added in the development process. I cleared the cache and tried it, Never got that again.

raviteja06 avatar Dec 22 '15 08:12 raviteja06

I got this too on 0.1% of the sessions. I have never been able to reproduce on one of my devices so far but it is a significant cause of crashes. Also most of these crashes happen at startup which is not good.

martinbonnin avatar Sep 19 '16 06:09 martinbonnin

Could it be linked to https://github.com/EsotericSoftware/kryo/issues/128 ? In all cases, can you put a section about Thread Safety in the README.md ? I understand it's ok to call Paper.book().read/write() from multiple threads, am I correct ?

martinbonnin avatar Sep 19 '16 06:09 martinbonnin

@martinbonnin yes, you can call read/write from different threads.

pilgr avatar Sep 19 '16 13:09 pilgr

@pilgr I'm also seeing this crash on a client's app, which uses paperdb v1.1:

Fatal Exception: io.paperdb.PaperDbException: Couldn't read/deserialize file /data/user/0/com.justeat.app.uk/files/log/counter.pt for table counter
       at io.paperdb.DbStoragePlainFile.readTableFile(SourceFile:241)
       at io.paperdb.DbStoragePlainFile.select(SourceFile:130)
       at io.paperdb.Book.read(SourceFile:65)
...
Caused by com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
mContent (io.paperdb.PaperTable)
       at com.esotericsoftware.kryo.io.Input.require(SourceFile:199)
       at com.esotericsoftware.kryo.io.Input.readVarInt(SourceFile:373)
       at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(SourceFile:127)
       at com.esotericsoftware.kryo.Kryo.readClass(SourceFile:667)
       at com.esotericsoftware.kryo.serializers.ObjectField.read(SourceFile:118)
       at com.esotericsoftware.kryo.serializers.FieldSerializer.read(SourceFile:551)
       at com.esotericsoftware.kryo.Kryo.readObject(SourceFile:686)
       at io.paperdb.DbStoragePlainFile.readTableFile(SourceFile:222)
       at io.paperdb.DbStoragePlainFile.select(SourceFile:130)
       at io.paperdb.Book.read(SourceFile:65)
...

We're also seeing it in a very small number of crashes per session, but according to Crashlytics the number of crashes is greater than the number of users impacted (each affected user has seen the crash 1.3x on average).

Crashlytics device insights indicate that the crash has only been seen on HTC and OnePlus devices running Android 6.0 or 6.0.1. Does that correspond with what you're seeing with AppDialer?

fingertricks avatar Oct 03 '16 13:10 fingertricks

I face this error too:

device: Mi 2
Android version: 5.0.2

+--- io.paperdb:paperdb:2.6
|    +--- com.esotericsoftware:kryo:4.0.1
|    |    +--- com.esotericsoftware:reflectasm:1.11.3
|    |    |    \--- org.ow2.asm:asm:5.0.4
|    |    +--- com.esotericsoftware:minlog:1.3.0
|    |    \--- org.objenesis:objenesis:2.5.1
|    \--- de.javakaffee:kryo-serializers:0.40
|         \--- com.esotericsoftware:kryo:4.0.0 -> 4.0.1 (*)

Process: com.angelia.wancai, PID: 26251

io.paperdb.PaperDbException: Couldn't read/deserialize file /data/data/com.angelia.wancai/files/io.paperdb/key_app_configuration.pt for table key_app_configuration

at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:298)

at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:158)

at io.paperdb.Book.read(Book.java:73)

at io.paperdb.Book.read(Book.java:58)

at com.angelia.wancai.MainApplication.config(MainApplication.java:98)

at com.angelia.wancai.utils.AppConfigurationUtils$2.onResponse(AppConfigurationUtils.java:74)

at okhttp3.RealCall$AsyncCall.execute(RealCall.java:135)

at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)

at java.lang.Thread.run(Thread.java:818)

Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow.

Serialization trace:

mContent (io.paperdb.PaperTable)

at com.esotericsoftware.kryo.io.Input.require(Input.java:199)

at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:373)

at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:127)

at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)

at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:712)

at io.paperdb.DbStoragePlainFile.readContent(DbStoragePlainFile.java:307)

at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:291)

... 10 more

hwangjr avatar Jan 04 '18 01:01 hwangjr

I am facing this error when .pt file show 0 bytes

steven274 avatar Jul 09 '18 10:07 steven274

I have updated the latest version of 2.6. While crash occurring on the latest version also. Please check the below track.

Caused by io.paperdb.PaperDbException: Couldn't read/deserialize file /data/user/0/xxx.xx.xxxxxxxxx/files/mallow/TableName.pt for table TableName.pt at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:298) at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:158) at io.paperdb.Book.read(Book.java:73) at xxx.xx.xxxxxxxxx(PaperDB.java:110) at xxx.xx.xxxxxxxxx(MainActivity.java:382) at xxx.xx.xxxxxxxxx(MainActivity.java:347) at android.app.Activity.performCreate(Activity.java:6701) at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1118) at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2677) at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2789) at android.app.ActivityThread.-wrap12(ActivityThread.java) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1527) at android.os.Handler.dispatchMessage(Handler.java:110) at android.os.Looper.loop(Looper.java:208) at android.app.ActivityThread.main(ActivityThread.java:6267) at java.lang.reflect.Method.invoke(Method.java) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1063) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:924)

Caused by com.esotericsoftware.kryo.KryoException: Buffer underflow. Serialization trace: journeyState (com.fleetrover.datamodel.TrackCurrentLocation) mContent (io.paperdb.PaperTable) at com.esotericsoftware.kryo.io.Input.require(Input.java:199) at com.esotericsoftware.kryo.io.Input.readAscii_slow(Input.java:616) at com.esotericsoftware.kryo.io.Input.readAscii(Input.java:594) at com.esotericsoftware.kryo.io.Input.readString(Input.java:472) at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:199) at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:188) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132) at com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:145) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:734) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:712) at io.paperdb.DbStoragePlainFile.readContent(DbStoragePlainFile.java:307) at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:291) at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:158) at io.paperdb.Book.read(Book.java:73) at xxx.xx.xxxxxxxxx(PaperDB.java:110) at xxx.xx.xxxxxxxxx(MainActivity.java:382) at xxx.xx.xxxxxxxxx(MainActivity.java:347) at android.app.Activity.performCreate(Activity.java:6701) at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1118) at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2677) at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2789) at android.app.ActivityThread.-wrap12(ActivityThread.java) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1527) at android.os.Handler.dispatchMessage(Handler.java:110) at android.os.Looper.loop(Looper.java:208) at android.app.ActivityThread.main(ActivityThread.java:6267) at java.lang.reflect.Method.invoke(Method.java) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1063) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:924)

Manikandan92 avatar Dec 12 '18 10:12 Manikandan92

same here as @Manikandan92

vzts avatar Jan 10 '19 15:01 vzts

I can reproduce the problem 100% of the times. For me, it's happening because I moved the class I was saving to a different package.

condesales avatar Mar 27 '20 17:03 condesales

We went in production recently (3 days ago) and we just registered two episodes: both for the same user.

Still no idea about what can cause it?

Joseph82 avatar Jun 24 '20 17:06 Joseph82

This issue is still present in version 2.7.1. It is rare and not immediately reproducible like 2.6 but it still happens. Please catch the exception within Paper.

The crash occurs within our SDK at this read line.

com.esotericsoftware.kryo.KryoException: Buffer underflow. Serialization trace: mContent (io.paperdb.PaperTable)

com.esotericsoftware.kryo.io.Input.readVarInt Input.java:373
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass DefaultClassResolver.java:127
com.esotericsoftware.kryo.Kryo.readClass Kryo.java:693
com.esotericsoftware.kryo.serializers.ObjectField.read ObjectField.java:118
com.esotericsoftware.kryo.serializers.FieldSerializer.read FieldSerializer.java:543
com.esotericsoftware.kryo.Kryo.readObject Kryo.java:712
io.paperdb.DbStoragePlainFile.readContent DbStoragePlainFile.java:332
io.paperdb.DbStoragePlainFile.readTableFile DbStoragePlainFile.java:316
io.paperdb.DbStoragePlainFile.select DbStoragePlainFile.java:166
io.paperdb.Book.read Book.java:73
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchEvent EventStoreManager.java:123
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchAllEvents EventStoreManager.java:147
com.salesforce.androidsdk.analytics.SalesforceAnalyticsManager.publishAllEvents SalesforceAnalyticsManager.java:279
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:605
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:591
android.os.AsyncTask$2.call AsyncTask.java:333
java.util.concurrent.FutureTask.run FutureTask.java:266
android.os.AsyncTask$SerialExecutor$1.run AsyncTask.java:245
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java:1167
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java:641
java.lang.Thread.run Thread.java:764

io.paperdb.DbStoragePlainFile.readTableFile DbStoragePlainFile.java:323
io.paperdb.DbStoragePlainFile.select DbStoragePlainFile.java:166
io.paperdb.Book.read Book.java:73
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchEvent EventStoreManager.java:123
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchAllEvents EventStoreManager.java:147
com.salesforce.androidsdk.analytics.SalesforceAnalyticsManager.publishAllEvents SalesforceAnalyticsManager.java:279
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:605
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:591
android.os.AsyncTask$2.call AsyncTask.java:333
java.util.concurrent.FutureTask.run FutureTask.java:266
android.os.AsyncTask$SerialExecutor$1.run AsyncTask.java:245
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java:1167
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java:641
java.lang.Thread.run Thread.java:764

brandonpage avatar Nov 17 '20 23:11 brandonpage

Hello @pilgr, do you have any intention of fixing or catching this issue within Paper?

brandonpage avatar Dec 07 '20 21:12 brandonpage

@brandonpage since this is incredibly rare and hardly to be reproduced, I'm not sure how to fix it. We may add some workarounds like second read attempts though. I'm wondering how often does it happen comparing to the total number of the user sessions?

pilgr avatar Dec 07 '20 21:12 pilgr

@pilgr This buffer underflow crash is also happening in our app as we are using SalesforceMobileSDK, which use version 2.7.1 of paper db. This issue only affected a few of our users but once it happens it will crash consistently (crash 200+ times for some of our users). Also because this crash is happening to those users consistently, the buffer underflow crash has pushed our crash per session rate to almost 4%

rufuszhu avatar Dec 07 '20 21:12 rufuszhu

@rufuszhu my understanding is if issue happens all the time for particular user, that might be caused by app update which used new package name for the class holding data. Please check out this message mentioned above.

I can reproduce the problem 100% of the times. For me, it's happening because I moved the class I was saving to a different package.

If the class has been stored with one package name and then on read the different class is expected (even if only package name has been changed) that may cause the issue.

For the same exception which happens very rarely I assume some other root cause takes place, that is still unknown.

pilgr avatar Dec 07 '20 22:12 pilgr