Maciej Obuchowski

Option to dump OpenLineage events that correspond to dataset/namespace from web

3

Having this feature would make debugging, and replicating errors much faster.

delete: add possibility to soft-delete datasets

4

First part of #1736 - ability to "soft delete" datasets - API to "hide" them which can be used from UI. This PR does not include the UI part. It...

support for OpenLineage's RUNNING eventType

2

`OpenLineage` introduces `RUNNING` event type which models continuous streaming job that it currently running - to differentiate it from generic `OTHER` event type. Related issues are https://github.com/OpenLineage/OpenLineage/issues/946 and discussion here:...

[marquez.service.models.Source](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/service/models/Source.java#L32) requires that `connectionUrl` was not-null. However, if `DataSourceDatasetFacet` does not exist in the LineageEvent, [the field can be set to null](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/db/OpenLineageDao.java#L451).

https://marquezproject.github.io/marquez/ is the proper website - the top level site should redirect there.

performance tests: generate and execute synthetic events

1

Write script that generates OpenLineage events reasonably similar to real-world scenario. Proposed scenario: ``` Suppose there are 5000 datasets (dataset_1..5000), with 50 columns each (dataset_column_X_1..50). Generate 1000 different jobs such...

performance test: create script that execute load testing of GET endpoints

1

Given database with (large) amount of lineage events and Marquez API, design script based on https://k6.io/ that executes GET requests on the endpoints, particularly those that are used by the...

Building on top of possibility added in https://github.com/MarquezProject/marquez/pull/2032 - add endpoint to soft delete everything in particular dataset - both jobs and datasets.

insert: do not always detect parens after table name as projection

1

Before this PR, query ``` INSERT INTO s.table_a (SELECT col1, col2 FROM s_two.table_x WHERE condition IS TRUE AND condition_2 IS FALSE GROUP BY col2, col1) ``` fails because parser always...

ci: synchronize OpenAPI docs

1

https://marquezproject.ai/openapi.html has a doc version that does not correspond with latest release or main branch.