compass icon indicating copy to clipboard operation
compass copied to clipboard

ability to overwrite asset instead of merge

Open bsushmith opened this issue 3 years ago • 0 comments

Is your feature request related to a problem? Please describe. As part of UpdatePatchAsset, we patch and merge the asset before writing into DB.

For the asset field data which contains the crux of asset information, it is impossible to overwrite in the current code. Only if the existing asset data is nil, it is possible to overwrite it. Otherwise, the existing asset.data is merged with incoming changes before writing to DB.

Would like to have a feature flag, where the user can decide if they would like to always take it in the latest changes or merge it with existing values.

Describe the solution you'd like A feature flag to decide whether to merge or take it latest changes without merging.

Describe alternatives you've considered I wanted to remove preview rows for bigquery tables in a project, and configured meteor recipe in such a way by having max_preview_rows value set as 0. But since compass does only merge rather than overwrite, it didn't happen. I've tried to delete the asset and then insert the asset again. but since the DeleteAsset takes in asset id, it became cumbersome to delete all the assets using the id.

Additional context Am not sure about the process of finding asset diff here - https://github.com/odpf/compass/blob/main/internal/store/postgres/asset_repository.go#L301. By the time, the code flow reaches here, the asset is already merged with incoming changes. This diff is used to track asset versions in asset_versions table. Please correct me If I am wrong here, but shouldn't the diff be calculated with incoming changes rather than merged changes?

bsushmith avatar Jul 27 '22 18:07 bsushmith