[BUG]Filtering by exclusion

[cauneacm]

Is there a way to exclude templates and components so that we only create a filter for indexes during metadata migration? It appears that the current mechanism only supports inclusion-based filters, which can be error-prone. In our case, we don’t need to migrate any templates or components to the target cluster, and excluding them would help avoid the “illegal_argument_exception” errors we’re seeing. This exclusion is crucial for achieving a successful metadata migration.

[peternied]

@cauneacm thanks for filing this issue, could you post a call stack of that failure case?

I suspect this could be worked around by adding a name in the allow list that won't possibly appear, excluding all index templates index components.

However, there might be some code paths that a triggered before the allow list filtering is applied - the call stack would help us know where.

I'd be happy to review a pull request if that would be faster to get this fixed for you.

[AndreKurait]

As an example, the args are defined in DataFilterArgs so you can provide --index-template-allowlist none --component-template-allowlist none

[cauneacm]

@AndreKurait If I add a dummy value or none I will get the following error:
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.lang.Exception.getMessage()" because the return value of "org.opensearch.migrations.metadata.CreationResult.getException()" is null
at org.opensearch.migrations.cli.Items.failureMessage(Items.java:97)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.opensearch.migrations.cli.Items.appendFailures(Items.java:80)
at org.opensearch.migrations.cli.Items.appendSection(Items.java:57)
at org.opensearch.migrations.cli.Items.asCliOutput(Items.java:40)
at org.opensearch.migrations.commands.MigrationItemResult.asCliOutput(MigrationItemResult.java:20)
at org.apache.logging.slf4j.Log4jEventBuilder.addArgument(Log4jEventBuilder.java:74)
at org.opensearch.migrations.MetadataMigration.main(MetadataMigration.java:80)

From my understanding, if the template or component is not found in the snapshot, the migration will fail.

@peternied
I'll try to find another way and if it works I'll create a PR.

[cauneacm]

After I ran the console migrate without any flags, just pointing to the snapshot we want to use, and tested the cat-indices command, I realized that the metadata did get migrated despite all the error messages. I guess this is just confusing since the console shows no indication of successful actions. I will close this for now since it's not an issue anymore.