Mukund Thakur
Mukund Thakur
@HappyRay Is there any complete design document of azkaban. It would be really helpful for us to build a good design for making azkaban HA.
I have two ideas to solve this. **IDEA 1** We put all the in memory state of azkaban web server info( like runnableFlows etc ) in some data store(DS) ,...
This sounds great.
I tried setting fs.gs.outputstream.type to SYNCABLE_COMPOSITE but I don't see any parallelism neither any improvement in runtimes for a 5GB file upload.
Okay. Thanks for the information. I used hadoop fs client to upload the files. Command: hadoop fs -put
We have multipart upload interface in hadoop and we have already implemented it for S3A. https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/multipartuploader.md https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/MultipartUploader.java https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AMultipartUploader.java CC @steveloughran
pushed to branch-3.3 as well.
merged to branch-3.3 as well.
`he class holds strong references so finalize() won't be called, ever.` Are you referring to DELETE_ON_EXIT map? Changes looks good. there is just one checkstyle reported.
> you know, the other thing to think of here is have finalize() methods in s3a/abfs, though they aren't ever encouraged You mean in the S3AFileSystem and ABFS right? Yes,...