Warn about deprecated and ignored index.mapper.dynamic index setting for better backwards compatibility

[peternied]

Description

When migrating from older versions of OpenSearch indices with a setting of index.mapper.dynamic - even if the value is null or the default of true prevents migration. With this change these indices can be successfully migrated.

Related Issues

• Resolves https://github.com/opensearch-project/OpenSearch/issues/3879

Check List

• [X] New functionality includes testing.
  • [X] All tests pass
• [ ] ~New functionality has been documented.~
  • [ ] ~New functionality has javadoc added~
• [X] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [X] Commits are signed per the DCO using --signoff
• [X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [ ] ~Public documentation issue/PR created~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peternied]

@shwetathareja Could you take a look into this issue? I'd would also like to backport this change to 2.12 and 1.3

[github-actions[bot]]

:x: Gradle check result for d44473559c56d7cee8a041c63858f94d414b5f18: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 57b8d91d6571e04f3e890e9434bf9bad900d8a41: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Compatibility status:

Checks if related components are compatible with change 7e28535

Incompatible components

Incompatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/alerting.git]

[github-actions[bot]]

:x: Gradle check result for 2a9f7594cfadfea086faf530bd5dc63c45ed7ce7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peternied]

Note; failures in gradle check are legit BWC tests that are impacted by this change.

[github-actions[bot]]

:x: Gradle check result for cc53284bca26ee6d3adadf7d4530c858f0b57f81: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 800950533621455f0a8044bda47cd1815adedc82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peternied]

@derek-ho What do you think of this change?

[peternied]

@nknize Mind taking a look at this PR - seems like you might have some thought on how we handle cases like this from the work you did to remove the default mapping [1]?

• [1] https://github.com/opensearch-project/OpenSearch/pull/2151

[peternied]

@reta mind taking a look, you might have some context from when you reviewed the removal of the default mapping [1]?

• [1] https://github.com/opensearch-project/OpenSearch/pull/2151

[github-actions[bot]]

:grey_exclamation: Gradle check result for 800950533621455f0a8044bda47cd1815adedc82: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (e97bee8) 71.47% compared to head (7e28535) 71.38%.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11193      +/-   ##
============================================
- Coverage     71.47%   71.38%   -0.09%     
+ Complexity    59938    59880      -58     
============================================
  Files          4972     4972              
  Lines        281770   281773       +3     
  Branches      40910    40910              
============================================
- Hits         201385   201153     -232     
- Misses        63710    63883     +173     
- Partials      16675    16737      +62     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[opensearch-trigger-bot[bot]]

This PR is stalled because it has been open for 30 days with no activity.

[opensearch-trigger-bot[bot]]

This PR is stalled because it has been open for 30 days with no activity.

[github-actions[bot]]

:grey_exclamation: Gradle check result for 747f7c5a519f8fda08b70de0a9708652a0eefc04: UNSTABLE

• TEST FAILURES:

      3 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

:x: Gradle check result for 8dd856987a9368bed52f8edb042ed0673789f639: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

@peternied the DCO check is failing ... :(

[github-actions[bot]]

:x: Gradle check result for 549e7894aeca4d347f768d25c9aa9e3323abf8f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peternied]

This because of the version bump is midflight?

=== Standard output of node `node{:qa:mixed-cluster:v2.13.0-0}` ===

»    ↓ errors and warnings from /var/jenkins/workspace/gradle-check/search/qa/mixed-cluster/build/testclusters/v2.13.0-0/logs/opensearch.stdout.log ↓
» WARN ][o.o.g.DanglingIndicesState] [v2.13.0-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
» WARN ][o.o.d.FileBasedSeedHostsProvider] [v2.13.0-0] expected, but did not find, a dynamic hosts list at [/var/jenkins/workspace/gradle-check/search/qa/mixed-cluster/build/testclusters/v2.13.0-0/config/unicast_hosts.txt]
»   ↑ repeated 4 times ↑
» ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [v2.13.0-0] uncaught exception in thread [main]
»  org.opensearch.bootstrap.StartupException: OpenSearchException[failed to bind service]; nested: IndexFormatTooNewException[Format version is not supported (resource BufferedChecksumIndexInput(NIOFSIndexInput(path="/var/jenkins/workspace/gradle-check/search/qa/mixed-cluster/build/testclusters/v2.13.0-0/data/nodes/0/_state/_5.cfs") [slice=_5.fnm])): 1 (needs to be between 0 and 0)];
»  	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  Caused by: org.opensearch.OpenSearchException: failed to bind service
»  	at org.opensearch.node.Node.<init>(Node.java:1331) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.node.Node.<init>(Node.java:413) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	... 6 more

[reta]

This because of the version bump is midflight?

I think should be fixed, we've just merged https://github.com/opensearch-project/OpenSearch/pull/12394 to main

[github-actions[bot]]

:x: Gradle check result for 549e7894aeca4d347f768d25c9aa9e3323abf8f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 549e7894aeca4d347f768d25c9aa9e3323abf8f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:grey_exclamation: Gradle check result for 7e2853538b0ccbe6d3d34a8b6443ab27cd38fa2b: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot[bot]]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11193-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f34ca960149c4f37f18bddfa4a0b0d7d2dcee9e4
# Push it to GitHub
git push --set-upstream origin backport/backport-11193-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11193-to-2.x.

[reta]

@peternied it seems like this fix is incomplete, nonetheless the value of INDEX_MAPPER_DYNAMIC_SETTING should be ignored, it won't be, org.opensearch.action.support.AutoCreateIndex:

    public AutoCreateIndex(
        Settings settings,
        ClusterSettings clusterSettings,
        IndexNameExpressionResolver resolver,
        SystemIndices systemIndices
    ) {
        this.resolver = resolver;
        dynamicMappingDisabled = !MapperService.INDEX_MAPPER_DYNAMIC_SETTING.get(settings);
        this.systemIndices = systemIndices;
        this.autoCreate = AUTO_CREATE_INDEX_SETTING.get(settings);
        clusterSettings.addSettingsUpdateConsumer(AUTO_CREATE_INDEX_SETTING, this::setAutoCreate);
    }