jackrabbit-oak icon indicating copy to clipboard operation
jackrabbit-oak copied to clipboard
verified publisher icon apache

OAK-8413 Use the new Azure SDK in the Azure Segment Store

Open ierandra opened this issue 1 year ago • 5 comments

ierandra avatar Sep 27 '24 12:09 ierandra

@ierandra, the build fails with the following error in Oak Segment Azure: [ERROR] org.apache.jackrabbit.oak.segment.azure: Version increase required; detected 3.0.0, suggested 4.0.0 [ERROR] org.apache.jackrabbit.oak.segment.azure.util: Version increase required; detected 1.71.0, suggested 2.0.0

dulceanu avatar Sep 30 '24 10:09 dulceanu

@dulceanu I was using -Dbaseline.skip in the build command. Please let me know if I should update the versions so that we don't get these messages.

ierandra avatar Sep 30 '24 12:09 ierandra

@ierandra, yes, the versions need to be updated so that the build passes without skipping the baseline check.

dulceanu avatar Sep 30 '24 12:09 dulceanu

Now oak-segment-azure build fails when issuing mvn clean install with lots of test related setup preconditions not met. Could you see what's this about, please?

dulceanu avatar Oct 01 '24 09:10 dulceanu

@dulceanu I fixed all the tests. build should be ok now.

ierandra avatar Oct 02 '24 10:10 ierandra

This doesn't seem to compile (after applying the diff to oak trunk):

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project oak-segment-azure: Compilation failure: Compilation failure:
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[43,47] cannot find symbol
[ERROR]   symbol:   class AzureStorageCredentialManager
[ERROR]   location: package org.apache.jackrabbit.oak.segment.azure
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[134,88] cannot find symbol
[ERROR]   symbol:   class AzureStorageCredentialManagerV8
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[163,76] cannot find symbol
[ERROR]   symbol:   class AzureStorageCredentialManagerV8
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[169,63] cannot find symbol
[ERROR]   symbol:   class AzureStorageCredentialManagerV8
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[54,19] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[56,19] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[64,22] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[64,67] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[72,29] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[72,74] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[105,18] cannot find symbol
[ERROR]   symbol:   class BlobItem
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[121,23] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile.AzureJournalReader
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[123,23] cannot find symbol
[ERROR]   symbol:   class BlobItem
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile.AzureJournalReader
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[131,36] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile.AzureJournalReader
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[131,77] cannot find symbol
[ERROR]   symbol:   class BlobItem
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile.AzureJournalReader
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[166,17] cannot find symbol
[ERROR]   symbol:   class AppendBlobClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile.AzureJournalWriter
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[287,32] cannot find symbol
[ERROR]   symbol:   class BlobContainerClient
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile.CombinedReader
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[287,78] cannot find symbol
[ERROR]   symbol:   class BlobItem
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile.CombinedReader
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureArchiveManager.java:[62,21] cannot find symbol
[ERROR]   symbol:   class CloudBlobDirectory
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureArchiveManager
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureArchiveManager.java:[69,32] cannot find symbol
[ERROR]   symbol:   class CloudBlobDirectory
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureArchiveManager
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[141,35] cannot find symbol
[ERROR]   symbol:   class AzurePersistenceV8
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[185,20] cannot find symbol
[ERROR]   symbol:   variable AzureUtilitiesV8
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[193,14] cannot find symbol
[ERROR]   symbol:   class AzureStorageCredentialManagerV8
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/tool/ToolUtils.java:[193,84] cannot find symbol
[ERROR]   symbol:   class AzureStorageCredentialManagerV8
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzurePersistence.java:[68,16] constructor AzureArchiveManager in class org.apache.jackrabbit.oak.segment.azure.AzureArchiveManager cannot be applied to given types;
[ERROR]   required: CloudBlobDirectory,org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor,org.apache.jackrabbit.oak.segment.spi.monitor.FileStoreMonitor,org.apache.jackrabbit.oak.segment.remote.WriteAccessController
[ERROR]   found: com.azure.storage.blob.BlobContainerClient,com.azure.storage.blob.BlobContainerClient,java.lang.String,org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor,org.apache.jackrabbit.oak.segment.spi.monitor.FileStoreMonitor,org.apache.jackrabbit.oak.segment.remote.WriteAccessController
[ERROR]   reason: actual and formal argument lists differ in length
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[107,13] cannot find symbol
[ERROR]   symbol:   class ListBlobsOptions
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[107,53] cannot find symbol
[ERROR]   symbol:   class ListBlobsOptions
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[109,18] cannot find symbol
[ERROR]   symbol:   class BlobItem
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile
[ERROR] /C:/projects/apache/oak/trunk/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureJournalFile.java:[110,87] cannot find symbol
[ERROR]   symbol:   variable BlobType
[ERROR]   location: class org.apache.jackrabbit.oak.segment.azure.AzureJournalFile

...etc...

reschke avatar Oct 29 '24 14:10 reschke

@reschke I updated the PR with changes from apache/jackrabbit-oak trunk and should be fine now

ierandra avatar Oct 29 '24 15:10 ierandra

It does indeed; thanks.

reschke avatar Oct 29 '24 18:10 reschke

This looks a bit weird:

[WARNING] com.azure.core.credential: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] com.azure.core.credential: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] com.azure.identity: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] com.azure.identity: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] com.microsoft.azure.storage: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] com.microsoft.azure.storage: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] com.microsoft.azure.storage.blob: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] com.microsoft.azure.storage.blob: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] com.microsoft.azure.storage.core: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] com.microsoft.azure.storage.core: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.counters: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.counters: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.maps: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.maps: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues.atomic: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues.atomic: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues.atomic.unpadded: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues.atomic.unpadded: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues.unpadded: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.queues.unpadded: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.util: Excessive version increase; detected 1.71.0, suggested 1.70.0
[WARNING] io.netty.util.internal.shaded.org.jctools.util: Version has been increased but analysis detected no changes; detected 1.71.0, suggested 1.70.0

are we exporting these packages?

This looks wrong...:

                        <Export-Package>
                            org.apache.jackrabbit.oak.segment.azure,
                            org.apache.jackrabbit.oak.segment.azure.queue,
                            org.apache.jackrabbit.oak.segment.azure.util,
                            com.fasterxml.jackson.dataformat.xml,
                            com.fasterxml.jackson.dataformat.xml.deser,
                            com.microsoft.azure.storage,
                            com.microsoft.azure.storage.core,
                            com.microsoft.azure.storage.blob,
                            com.azure.core.credential,
                            com.azure.identity,
                            com.azure.blob,
                            com.azure.storage.common,
                            com.azure.storage.internal.avro.implementation
                        </Export-Package>

reschke avatar Oct 29 '24 18:10 reschke

Opened https://issues.apache.org/jira/browse/OAK-11236 - the embedding/exporting issues started earlier.

reschke avatar Oct 29 '24 19:10 reschke

The build on my machine fails because of baseline plugin

$ mvn clean install -DskipTests
....
[ERROR] org.apache.jackrabbit.oak.segment.azure.util: Version increase required; detected 1.0.0, suggested 2.0.0

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  39.965 s
[INFO] Finished at: 2024-11-12T10:02:29+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.felix:maven-bundle-plugin:5.1.9:baseline (baseline) on project oak-segment-azure: Baseline failed, see generated report -> [Help 1]
[

smiroslav avatar Nov 12 '24 09:11 smiroslav

@smiroslav I configure it back to 2.0.0. should work now

ierandra avatar Nov 12 '24 10:11 ierandra

@ierandra, I would prefer if you let me resolve the PR comments that I have started. This post captures well the resoninig I have in mind https://www.danclarke.com/resolving-pr-comments

smiroslav avatar Nov 18 '24 12:11 smiroslav

@reschke

I'm only looking into POMs here. It is unclear why apparently apparently all new dependencies get embedded. What's the reason for that?

As I mention in my previous comment if I don't do it if fails at build phase for CQ: https://jenkins-evergreen.ci.corp.adobe.com/job/Sprouts/job/aemcs-developer-releases/5335/artifact/1-BuildAEM.log

ierandra avatar Nov 26 '24 10:11 ierandra

@smiroslav I fixed all the comments. could you please take a look?

ierandra avatar Nov 27 '24 10:11 ierandra

Thanks @ierandra I am having a look.

In the meantime, I started the build, and it failed.
A bunch of test are failing on my local machine as well when I run

mvn test -pl oak-segment-azure

[ERROR] testInitialSize(org.apache.jackrabbit.oak.segment.azure.AzureTarFilesTest)  Time elapsed: 0 s  <<< ERROR!
java.lang.IllegalArgumentException: The value of the parameter 'maxRetries' should be between 1 and 2147483647.
        at com.azure.storage.common.implementation.StorageImplUtils.assertInBounds(StorageImplUtils.java:192)
        at com.azure.storage.common.policy.RequestRetryOptions.<init>(RequestRetryOptions.java:98)
        at com.azure.storage.common.policy.RequestRetryOptions.<init>(RequestRetryOptions.java:64)

smiroslav avatar Dec 06 '24 16:12 smiroslav

@smiroslav I just run the command you mentioned (mvn test -pl oak-segment-azure) and the build was with success. Please make sure you git-fetch-pull the latest changes.

ierandra avatar Dec 16 '24 13:12 ierandra

@ierandra the build first failed in jenkins https://github.com/apache/jackrabbit-oak/actions/runs/12028412044/job/34038518960?pr=1748

smiroslav avatar Dec 17 '24 10:12 smiroslav

https://github.com/apache/jackrabbit-oak/actions/runs/12028412044/job/34597299155?pr=1748

Error:  deactivate(org.apache.jackrabbit.oak.segment.azure.AzureSegmentStoreServiceTest)  Time elapsed: 0.006 s  <<< ERROR!
java.lang.IllegalArgumentException: The value of the parameter 'maxRetries' should be between 1 and 2147483647.
	at com.azure.storage.common.implementation.StorageImplUtils.assertInBounds(StorageImplUtils.java:192)
	at com.azure.storage.common.policy.RequestRetryOptions.<init>(RequestRetryOptions.java:98)
	at com.azure.storage.common.policy.RequestRetryOptions.<init>(RequestRetryOptions.java:64)
	at org.apache.jackrabbit.oak.segment.azure.util.AzureRequestOptions.getRetryOptionsDefault(AzureRequestOptions.java:56)
	at org.apache.jackrabbit.oak.segment.azure.util.AzureRequestOptions.getRetryOptionsDefault(AzureRequestOptions.java:43)
	at org.apache.jackrabbit.oak.segment.azure.AzuriteDockerRule.getReadBlobContainerClient(AzuriteDockerRule.java:104)
	at org.apache.jackrabbit.oak.segment.azure.AzureSegmentStoreServiceTest.setup(AzureSegmentStoreServiceTest.java:85)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

smiroslav avatar Dec 18 '24 15:12 smiroslav