[SPARK-48726] Create the StateSchemaV3 file format, and write this out for the TransformWithStateExec operator

[ericm-db]

What changes were proposed in this pull request?

In this PR, we introduce the StateSchemaV3 file that is used to keep track of a list of ColumnFamilySchema which we write from the TransformWithState operator. We collect the Column Family schemas from the driver, and write them out as a part of a planning rule.

Why are the changes needed?

These changes are needed to enable schema evolution for this operator in the future.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit tests and ran existing unit tests

[info] Run completed in 11 seconds, 673 milliseconds.
[info] Total number of tests run: 4
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 43 s, completed Jun 26, 2024, 10:38:35 AM

Was this patch authored or co-authored using generative AI tooling?

No