Patrick Duin

Prometheus loves tags, we should tag our metrics with the function call and the metastore. We currently put them all in the name this isn't great and hard to create...

I'd like WaggleDance to support Presto views. Currently Presto views aren't parsed and database names aren't transformed in the Presto View this is because Presto stores the view as: Presto...

Improve validation of ACCESS_CONTROL_TYPE settings for the FederatedMetastore

3

From the docs: > Primary metastores can configure access-control-type to have any of the described access-control-types whereas federated metastores may only be configured to READ_ONLY and READ_AND_WRITE_ON_DATABASE_WHITELIST. The validation for...

Depending on the batch size given we're doing potentially large listPartition calls here: https://github.com/HotelsDotCom/circus-train/blob/main/circus-train-core/src/main/java/com/hotels/bdp/circustrain/core/HiveEndpoint.java#L152 and https://github.com/HotelsDotCom/circus-train/blob/main/circus-train-core/src/main/java/com/hotels/bdp/circustrain/core/HiveEndpoint.java#L154 We should use a partitionIterator: https://github.com/HotelsDotCom/hcommon-hive-metastore/blob/master/src/main/java/com/hotels/hcommon/hive/metastore/iterator/PartitionIterator.java if possible. Benefit is that we'll be nicer...

Update of metadata during a replication fails when base path is not matched in all partitions

1

Sometimes replication fails with an error like: ``` Caused by: com.hotels.bdp.circustrain.api.CircusTrainException: Partition path 's3://bucket/db/table/1539359466542/acquisition_instant=20181012T155106Z' does not seem to belong to data source path 's3://bucket/table' ``` Or something similar. This is...

_As a user of CircusTrain_ _I'd like to know if I can change struct columns without breaking replication_ Acceptance Criteria: * Write an integration test for the following scenario. -...

`ThriftHiveMetaStore` code starts up a HiveMetastore which starts a Hive Thrift server. Hive doesn't provide any hooks to shut it down. So in our tests the started threads keep running...