apache
FALCON-2112 Set property value to set map memory for replication and retention
@sandeepSamudrala Can't parameters for map task memory be set in the Feed entity properties section.
@peeyushb : Yes. They can be set. This way gives it to define globally for all feeds and not each feed has to specify it.
Each feed entity must have it's own calculated requirement of memory for retention and replication job, so I am more of the opinion let each feed entity provide memory requirements by specifying in properties rather setting it globally across for all feeds.
Thoughts please.
In addition, "oozie.launcher.mapreduce.map.memory.mb" and "mapreduce.map.memory.mb" is not just one memory parameter for Hadoop jobs. There are bunch of other memory parameters as well and those parameters values are set in accordance with each other.
@peeyushb : Even with this patch, user can specify his own requirements in the feed properties and they have precedence over global settings. In an organization running large clusters, it would be difficult to update all the feeds and so this provides leisure to just touch the feeds and the new settings will get kicked.
@peeyushb : On coming to other parameters. Yes I agree, there are bunch of other parameters which work in accordance with the above settings. But coming to feed replication and retention there is no MR job, they are simple java action to delete data and distcp job to copy the data, which won't require many other settings.
These properties are set only to feeds and not processes.
Please also update documentation.
The build has failed, but, I guess the ITs that have failed are un-related.
@sandeepSamudrala Just setting this particular memory parameter alone and that too globally for all the Falcon feed entity scheduled MR jobs, many of the times it will not help, if other memory parameters are not in accordance and user has not defined anything in feed entity to override the global settings.
Also, why you think that update functionality for feed entity can't be leveraged to set the relevant memory through properties for the required and particular feed entity MR jobs. As definitely not all the feed entities in Falcon may require this.
If we do comparision with Oozie, if any particular workflow has different memory requirement they just specify the value to the required memory parameters for that particular workflow only.
Moreover, feed replication and retention run as MR jobs.
Hi @sandeepSamudrala - setting map memory mb alone is not enough. Since the minimum allocation will be determined by yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb you may indeed get more than 512mb.
Also, even if you changed the minimum allocation to 512m (which should not be the default as you have), then you need to make sure that the mapreduce.map.java.opts has -Xmx set less than that with allowance for the JVM requirements.
@vrangan : Thanks for pointing the same. I had realized what other effects it could introduce. I had made the changes accordingly for the same to accommodate yarn.app.mapreduce.am.resource.mb yarn.app.mapreduce.am.command-opts
mapreduce.map.memory.mb mapreduce.map.java.opts
I will test the updated patch and update the pull request.
@vrangan @peeyushb : I have made changes accordingly. Please review.