NIFI-9892: Updated Azure storage related processors to adhere to NiFi…

[markap14]

… best practices and cleaned up code a bit. Fixed several integration tests.

Thank you for submitting a contribution to Apache NiFi.

Please provide a short description of the PR here:

Description of PR

Enables X functionality; fixes bug NIFI-YYYY.

In order to streamline the review of the contribution we ask you to ensure the following steps have been taken:

For all changes:

• [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

• [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

• [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

• [ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

• [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
• [ ] Have you written or updated unit tests to verify your changes?
• [ ] Have you verified that the full build is successful on JDK 8?
• [ ] Have you verified that the full build is successful on JDK 11?
• [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
• [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
• [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

• [ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

[jfrazee]

@markap14 I looked these through and they all look like good changes (which you probably don't need to be told). There's one thing I wanted to call out that I think might be relevant with changing the default to ${filename}. I think it will break a quick creation of List -> Fetch without a user having to edit Fetch. The gist is that azure.blobname and filename aren't the same. The former includes the full path and the latter is just the basename (or it was that way).

See https://github.com/apache/nifi/pull/3906#issuecomment-623016616

[jfrazee]

Overall, great updates, @markap14! Using the processors felt much more internally consistent now.

I noted that while ListAzureBlobStorage does set the filename attribute, it doesn't set it to a value that can be directly used by FetchAzureBlobStorage. Instead, the azure.blobname contains what would be the usable value. For example, filename contained test.txt and azure.blobname contained resources/test.txt in my test, and it's the latter that is needed by FetchAzureBlobStorage. Additionally, path was set to ./, which I think is not consistent. What do you think about setting path to the value before the filename, and defaulting the FetchAzureBlobStorage Blob Name property to ${path}/${filename}? Not sure how this would work at the root level, however.

Same issue for the *_12 processors. I have not exercised the ADLS processors yet.

Oops. Overlapping replies.

${path}/${filename} is not inconsistent with some of the other Fetch processors so I think it's a reasonable default. The behavior of ${path} is also weird enough to call broken so it'd seem ok to change.

[markap14]

Great catches @jfrazee and @gresockj . You are both indeed right - I did overlook the necessity of including the path attribute. Will update the Fetch processors to use ${path}/${filename} and ensure that the List processors are properly populating the path attribute.

[github-actions[bot]]

We're marking this PR as stale due to lack of updates in the past few months. If after another couple of weeks the stale label has not been removed this PR will be closed. This stale marker and eventual auto close does not indicate a judgement of the PR just lack of reviewer bandwidth and helps us keep the PR queue more manageable. If you would like this PR re-opened you can do so and a committer can remove the stale tag. Or you can open a new PR. Try to help review other PRs to increase PR review bandwidth which in turn helps yours.