[Feature] It may be a memory bottleneck when read all manifests.

[5herhom]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

Snapshot.dataManifests() will read all the ManifestFileMeta in memory . If there are too many files in a table ,it may cause oom. And when ManifestFileMeta.merge() to merge the manifests , the merge map is also a memory container , which will case the same problem. I think we can use a spillable container to save manifests .

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• [X] I'm willing to submit a PR!