ParquetWriter is incompatible with JDK25

[hugoncosta]

Describe the bug, including details regarding any error messages, version, and platform.

Hello, I'm trying to upgrade my code base to JDK25 and I'm having this issue on my tests

     java.lang.UnsupportedOperationException: getSubject is not supported
        at java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3852)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3842)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3630)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:541)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
        at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58)
        at com.amazon.networkvalidator.parquet.ParquetWriter.write(ParquetWriter.kt:75)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest$test writing to parquet$1.invokeSuspend(ParquetWriterTest.kt:88)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest$test writing to parquet$1.invoke(ParquetWriterTest.kt)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest$test writing to parquet$1.invoke(ParquetWriterTest.kt)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$1.invokeSuspend(TestBuilders.kt:318)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
        at kotlinx.coroutines.test.TestDispatcher.processEvent$kotlinx_coroutines_test(TestDispatcher.kt:24)
        at kotlinx.coroutines.test.TestCoroutineScheduler.tryRunNextTaskUnless$kotlinx_coroutines_test(TestCoroutineScheduler.kt:99)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$workRunner$1.invokeSuspend(TestBuilders.kt:327)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:47)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at kotlinx.coroutines.test.TestBuildersJvmKt.createTestResult(TestBuildersJvm.kt:10)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:310)
        at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:168)
        at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0$default(TestBuilders.kt:160)
        at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0$default(Unknown Source)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest.test writing to parquet(ParquetWriterTest.kt:76) 

It appears this is an issue within the Hadoop libraries upon which Parquet Java depends on

Component(s)

No response

[hugoncosta]

Raised a Jira in Apache itself - [​HADOOP-19703] UserGroupInformation.java is using a non-support operation in JDK25 - ASF JIRA

[hugoncosta]

Work is ongoing in https://github.com/apache/hadoop/pull/7892

[ChristopherWirt]

I managed to get around this by - disabling the filesystem cache HADOOP_CONF.set("fs.file.impl.disable.cache", "true");