NIFI-13344 Implement backend for uploading and managing custom NARs

[bbende]

Summary

This PR includes the work for NIFI-13344 which introduces an API and framework components for uploading custom NARs and then managing the NARs that have been uploaded. This covers the "push model" described in the Feature Proposal for Custom NAR Improvements.

The REST API introduced includes the following end-points:

Upload a NAR:
POST - /controller/nar-manager/upload (application/octet-stream)

List uploaded NARs:
GET -  /controller/nar-manager/nars

Get summary for a single NAR:
GET -  /controller/nar-manager/nars/{id}

Get more details for a single NAR like the extensions contained in it:
GET -  /controller/nar-manager/nars/{id}/detalls

Download a NAR:
GET -  /controller/nar-manager/nars/{id}/download

Delete a previously uploaded NAR:
DELETE -  /controller/nar-manager/nars/{id}

Some rules and restrictions of the current functionality...

If you upload a NAR with the same coordinate, it will replace the existing NAR, which includes:

• Stopping and ghosting any components instantiated from the NAR
• Unloading the previous NAR
• Loading the new NAR
• Unghosting the previously ghosted components
• Restoring the components state if needed

You can upload NARs in any order and if a NAR is missing it's dependency it will remain unloaded in a "Missing Dependency" state until the dependency is uploaded, at which it automatically becomes loaded/installed. Similar to how dropping NARs into ./extensions works.

You can't upload a NAR with the same coordinate as a NAR that is not part of the "NAR Manager", meaning one of the NARs that was provided from a lib directory or a NarProvider.

On delete, the force argument can be passed which will stop and ghost any components to allow delete to proceed.

One other restriction is not being allowed to delete or replace a NAR that another NAR has a dependency on.

In a cluster, when a node joins the cluster it attempts to synchronize its NARs with the coordinator. This requires access to the REST APIs described above, which means the node identities will need READ on /controller.

CLI commands are implemented for all of the above end-points:

#> nifi upload-nar -nar nifi-example-processors-nar-1.0.nar
#> nifi upload-nar -nar nifi-example-service-api-nar-1.0.nar
#> nifi upload-nar -nar nifi-example-service-nar-1.1.nar

#> nifi list-nars

#   Id                                     Group             Artifact                       Version   State
-   ------------------------------------   ---------------   ----------------------------   -------   ---------
1   1c6575c8-e488-33d3-a226-7803bfc30004   org.apache.nifi   nifi-example-processors-nar    1.0       Installed
2   afd8c799-de79-33ad-9ad0-413bd5beff3b   org.apache.nifi   nifi-example-service-api-nar   1.0       Installed
3   5f63686a-b59f-30f6-ad2b-9b372678398e   org.apache.nifi   nifi-example-service-nar       1.1       Installed

#> nifi list-nar-component-types -nid 1c6575c8-e488-33d3-a226-7803bfc30004

#   Type                                             Kind
-   ----------------------------------------------   ---------
1   org.apache.nifi.processors.example.MyProcessor   PROCESSOR

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• [X] Apache NiFi Jira issue created

Pull Request Tracking

• [X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• [X] Pull Request based on current revision of the main branch
• [XX] Pull Request refers to a feature branch with one commit containing changes

Verification

There are two system tests that exercise the APIs, one in a standalone instance, and and one with a two node cluster to exercise the replication of uploads.

In addition, have been running a two node cluster locally and testing various API calls and scenarios.

Build

• [X] Build completed using mvn clean install -P contrib-check
  • [X] JDK 21

UI Contributions

• [ ] NiFi is modernizing its UI. Any contributions that update the current UI also need to be implemented in the new UI.

Licensing

• [ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
• [ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• [ ] Documentation formatting appears as expected in rendered files

[bbende]

@gresockj thanks for the thorough review! I pushed up a commit to address some changes, a few things to still address:

• user guide
• some stuff around the MaxLengthInputStream
• the afterProperties method question

Regarding the comment about needing to refresh the UI... Currently the UI retrieves the lists of processor and controller service types one time when the canvas is loaded. We can file something to try and improve it, but I consider it a separate effort since it already impacts stuff like dropping a NAR into ./extenions, or a NAR being pulled from a NarProvider and download into ./extensions.

[bbende]

@exceptionfactory thanks for the review! I pushed a couple of commits that should address most of your comments, let me know of anything else.

[bbende]

@exceptionfactory @gresockj I pushed up another commit that adds support for handling of NARs containing Python processors. This required some changes on the Python side to track the coordinate of the NAR that the processor came from because this wasn't available on the Java side before, and all Python components on the Java side are represented as coming from org.apache.nifi:python-extensions with the version coming from the Processor's version.