parseable
parseable copied to clipboard
Remove redundant ListObjectsV2 call when fetching from S3
In S3 query flow, datafusion ListingOptions is configured with extension to look for, this internally makes datafusion list files using ListObjectsV2 before actually fetching the file. This call maybe removed if filename is known beforehand.
Todo:
- [ ] What to do in case of multiple parseable instances with different hostnames.
- [ ] Figure out how to provide table without doing list object calls.