Core: Use Bulk Delete when dropping table data and metadata

[amogh-jahagirdar]

This change updates the implementation of dropping table data and metadata to use bulk delete if applicable.

[jackye1995]

Currently it does not have any testing, I know we have some issue using s3mock, can we add an integ test in aws module for delete all files in a table?

[amogh-jahagirdar]

Currently it does not have any testing, I know we have some issue using s3mock, can we add an integ test in aws module for delete all files in a table?

Yeah, I was just looking into it. If needed, I can add integration tests to the TestGlueCatalogTable. TestGlueCatalogTable already tests purging and the underlying file IO is S3, so if those succeed we should be good. I will run those and validate as well as add any other cases.

[jackye1995]

TestGlueCatalogTable already tests purging and the underlying file IO is S3, so if those succeed we should be good. I will run those and validate as well as add any other cases.

I don't think we have enough files generated to trigger the full code path, let's make sure we do that

[amogh-jahagirdar]

@szehon-ho @danielcweeks @aokolnychyi @rdblue Would be happy to get your feedback as well!

[aokolnychyi]

I'd love to take a look later today.

[aokolnychyi]

@amogh-jahagirdar, let me know what you think about the suggestions above. If you want to, we can do them in a follow-up PR as well to avoid blocking this feature. Looking forward to consuming this in actions!

[amogh-jahagirdar]

Thanks @aokolnychyi for the reviews, and sorry for the delay, I got busy with internal work. I've updated the PR with the suggested refactoring.

[aokolnychyi]

Let me take a look. Sorry for the delay.

[amogh-jahagirdar]

Thanks @aokolnychyi @jackye1995 @singhpk234 for the reviews, I've addressed all the nits!

[jackye1995]

Thanks for the contribution @amogh-jahagirdar , and thanks for reviewing @singhpk234 @aokolnychyi !