[SPARK-39838][SQL] Preserve explicit empty column metadata

[kupferk]

What changes were proposed in this pull request?

This PR fixes the issue SPARK-39838 where an explicitly set empty Column Metadata object would be optimized away, making it impossible to reset column metadata.

Why are the changes needed?

The behaviour changed from Spark 3.2.x to Spark 3.3. While Spark 3.2.x would respect explicitly resetting a columns metadata to empty, Spark 3.3 inherits the metadata from child nodes. From my point of view, Spark 3.2.x behaviour is correct while Spark 3.3 is unexpected.

Does this PR introduce any user-facing change?

Not really, except that it restores the behaviour of Spark 3.2.x

How was this patch tested?

A unittest was added

[AmplabJenkins]

Can one of the admins verify this patch?

[github-actions[bot]]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[JordanMLee]

is it possible to get this one reopened?

[beliefer]

cc @MaxGekk