Vitali Makarevich comments

Results 27 comments of


                                            Vitali Makarevich

Cannot connect using telnet in Ch02 listing_211

u can manually put context into channel.on('join', func, channel) and change this in function to channel like this.clients should be channel.clients

[MINOR] Add parallel listing of existing partitions

@hudi-bot run azure

[MINOR] Add parallel listing of existing partitions

@yihua will you review this if I fix conflicts?

[MINOR] Add parallel listing of existing partitions

@yihua @nsivabalan is there any chance you'll be able to take a look on it? It's a significant improvement and makes sync much faster... We've been running it in production...

[MINOR] Add parallel listing of existing partitions

@yihua @nsivabalan

[MINOR] Add parallel listing of existing partitions

Thanks for the review, will take a look this week.

[MINOR] Add parallel listing of existing partitions

@yihua I found that functionality I'm trying to bring heavily collides with https://github.com/apache/hudi/pull/10572. So for the case of loading a list of partitions, there are now the following mechanisms: 1....

[MINOR] Add parallel listing of existing partitions

@parisni can you take a look at this PR if it addresses your concerns? We are operating 2M+ partition syncs with this code within seconds.

[MINOR] Add parallel listing of existing partitions

@parisni I will add improvements in the following PR. The list I plan to add: 1. IT as you mentioned 2. `.excludeColumnSchema(true)` - although as I remember seeing AWS Glue...

[MINOR] Add parallel listing of existing partitions

@danny0405 small follow-up just to confirm - config changes will be picked up automatically right? I opened `asf-site` branch and it looks like if I just add a new parameter...