Moritz Mack

Support VR test including TestStream for Spark runner in streaming mode

6

Run VR tests for Spark streaming runner rather than custom tests (test are already run as part of the "normal" unit test run). If `forceStreaming` is set to `true`, the...

Improved pipeline translation in SparkStructuredStreamingRunner

28

Improved pipeline translation in `SparkStructuredStreamingRunner` (closes #22445, #22382): - Make use of Spark `Encoder`s to leverage structural information in translation (and potentially benefit from Catalyst optimizer). Though note, the possible...

[Bug]: Issues with Watermark propagation in Spark runner (streaming)

2

### What happened? I was running into [this issue](https://github.com/apache/beam/pull/22620#issuecomment-1208303417) when adding support for VR tests for Spark in streaming mode (https://github.com/apache/beam/pull/22620). It looks like the runner only supports a global...

### What needs to happen? https://github.com/apache/beam/pull/16947 added Schema support for AWS models. With that `Coder`s for AWS IOs became (mostly) deprecated (see related changes). Respective code should be removed after...

[Task]: Isolate usage of Scala APIs in ScalaInterop utility where unreadable in Java

1

### What needs to happen? Isolate usage of Scala APIs in ScalaInterop utility where unreadable in Java ### Issue Priority Priority: 2 ### Issue Component Component: runner-spark

Introduce constants for special ReservedClusterStateMetadata versions

1

Introducing constants for special ReservedClusterStateMetadata versions to make it easier to trace their usage.

Extended feature validations in REST tests and improved wording.

2

Extended feature validations in REST tests and improved wording for better guidance on mistakes / misunderstandings.

Migrate remaining usage of skip version in YAML specs to cluster_features

3

Migrate remaining usage of skip version in YAML specs to cluster features, using the new `known_issues` and `awaits_fix`, e.g.: ```yaml - skip: # replaces version: "8.3.0 - 8.3.2" known_issues: -...

Require 'capabilities' test runner feature if using capabilities check

1

Require 'capabilities' test runner feature if using capabilities check introduced in #108425. cc @elastic/clients-team

Better isolate failure cause in DeprecationHttpIT.testDeprecatedSettingsReturnWarnings

1

Use separate xOpaqueId for cleanup in `DeprecationHttpIT.testDeprecatedSettingsReturnWarnings` to better isolate the cause of #108628 In rare cases we receive deprecation headers in`DeprecationHttpIT.testDeprecatedSettingsReturnWarnings`, but the warning seems to be missing in...