1978 add kafka plumbing

[rorymckinley]

Validation Steps

How can a reviewer validate your work?

Notes for the reviewer

Related issue

Fixes #

Review checklist

• [ ] I have performed a self-review of my code
• [ ] I have verified that all appropriate authorization policies (:owner, :admin, :editor, :viewer) have been implemented and tested
• [ ] If needed, I have updated the changelog
• [ ] Product has QA'd this feature

[rorymckinley]

@stuartc @jyeshe Opening this PR to get your views on the GenServer setup that is intended to ensure that we process messages with the same trigger-topic-key combination in the order they arrive. All messages with the same combination are considered to form part of a MessageCandidateSet and the earliest of these messages (based on the timestamp received from the KafkaCluster) is a MessageCandidate.

There are two moving parts:

The MessageCandidateSetServer . The server is responsible for building sets from TriggerKafkaMessage records and handing out a set upon request.

The MessageCandidateSetWorker requests a set from the server and then identifies the candidate for the set. If the candidate is not linked to a WorkOrder, the worker creates a new WorkOrder and then links it to the candidate. If the candidate is linked to a WorkOrder with a state of :success, the worker deletes this candidate. After either operation, or a no-op, the worker will request a new candidate set.

There is obviously a host of supporting functionality still needed (e.g. what do we do if a candidate set gets stuck?) as well as optimisations (e.g. candidate sets with nil keys do not need to retain sequence). But my goal is to get a happy-day implementation deployed to prod lightning as soon as I have made it play nicely with the rest of Lightning, so that Jembi can start testing in that environment.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 89.34708% with 31 lines in your changes missing coverage. Please review.

Project coverage is 89.20%. Comparing base (1bdc5fc) to head (71501db).

Files	Patch %	Lines
...ghtning_web/live/job_live/kafka_setup_component.ex	0.00%	18 Missing :warning:
...kafka_triggers/message_candidate_set_supervisor.ex	0.00%	3 Missing :warning:
lib/lightning_web/live/workflow_live/components.ex	0.00%	3 Missing :warning:
lib/lightning/vault.ex	60.00%	2 Missing :warning:
lib/lightning_web/live/components/icon.ex	0.00%	2 Missing :warning:
...ing/kafka_triggers/message_candidate_set_worker.ex	83.33%	1 Missing :warning:
...ightning/workflows/triggers/kafka_configuration.ex	98.63%	1 Missing :warning:
lib/lightning_web/live/workflow_live/edit.ex	0.00%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2136      +/-   ##
==========================================
- Coverage   89.20%   89.20%   -0.01%     
==========================================
  Files         279      294      +15     
  Lines        9406     9689     +283     
==========================================
+ Hits         8391     8643     +252     
- Misses       1015     1046      +31     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[stuartc]

@rorymckinley is this PR now defunct, since Kafka support was merged in last week?