Daniel K
Daniel K
In the perspective to support streaming (Atum currently supports batch processing only), we need to work ideas on how to support streaming, too. There is a `Dataset.observe`-based PoC on this...
Currently, the code in `examples` submodule already serves as integration test (sort of), but pursuing other features (separate module/runnable aspirations/... see #99). But it would be nice to have actual...
The current example* submodules have a couple of shortcomings that would be nice to address: - they are part of the project build - so they are released (unnecessarily) -...
While #37 solves funcioning of Atum on the S3 persistence layer, it would be quite beneficial in terms of project quality to cover these routines via integration tests as well....
When Atum's _control measure tracking_ is enabled, the measure tracking fails in case one attempts to write the dataframe directly to Kafka, e.g. ```scala spark.enableControlMeasuresTracking(somePath).setControlMeasuresWorkflow(someName) df.selectExpr("topic", "CAST(key AS STRING)", "CAST(value...
## Background In Enceladus API V2 (and further on in V3), there are multiple serialization techniques used to get JSON representation of objects, and currently, they behave differently in terms...
## Background & Feature Originating in https://github.com/AbsaOSS/enceladus/pull/2055#discussion_r870068092, the idea is that when enabling an entity, all its dependencies should be checked to be enabled as well. ## Feature Since V3...
This epic group scripts and changes with relation to Menas DB migration including checks, initialization, and related functionality.
## Background In #1772 ( PR #1779), it was suggested to use Hermes to compare actual v. expected dataframes instead of MrPower's fast dataset compare. At the of writing this...