Fabian Höring

@jcrist I continued investigating this issue. Application logs to NM_REMOTE_APP_LOG_DIR are written at the end of the application run. So there is indeed a timeframe where LogClient.getLogs will find files...

@jcrist Any news ? I added it to our [code base](https://github.com/criteo/tf-yarn/blob/master/tf_yarn/__init__.py#L818) for the moment. When I look at the logs (warn + info) they happen all the time in this...

OK. Thanks for the answer. On our side we don't currently use dynamic workloads. That's why the workaround is good enough for us. I like solution 3 as it seems...

@jcrist Any news on that ? What about making it configurable ? ``` grpc_max_receive_message_length = os.environ.get("SKEIN_MAX_MSG_SIZE", 50 * 1024 * 1024) .. ('grpc.max_receive_message_length', grpc_max_receive_message_length ) .. ```

Why not but I think one response by container could still easily go above 4 MB. It turns out even 50 MB seems not enough in some cases. We have...

OK. I agree it would be nice not having to specify this setting. Having a quick look we could change the proto like this : ``` rpc getLogs (LogsRequest) returns...

OK. I also had a look at LogClient and it could also just produce a stream with Java8 stream API. I will work on it and propose something in another...

I tested PR #108. I works with minor changes. It would be cool to merge it. As it is closed I can also create a new PR request if needed.

From reading the code you need to provide those options to the spark context. ``` spark.hadoop.mapreduce.output.fileoutputformat.compress:true spark.hadoop.mapreduce.output.fileoutputformat.compress.codec: org.apache.hadoop.io.compress.GzipCodec ``` So they will apply to the whole spark job. If it...

Hello @palenica, Any news on that ? I had a look a the repo but the AWS lambda functions are only provided as [JARs](([https://storage.googleapis.com/trusted-execution-aggregation-service-public-artifacts](https://storage.googleapis.com/trusted-execution-aggregation-service-public-artifacts/%7Bversion%7D/%7Bjar_file%7D)). I would expect this to be...