GoogleCloudPlatform
Exceeded soft private memory limit in python map reduce job
I am running mapreduce on python app engine that throws alot of exceptions like: "Exceeded soft private memory limit of 128 MB with 131 MB after servicing 17 requests total".
The job uses a GoogleCloudStorageRecordInputReader for reading data and GoogleCloudStorageConsistentRecordOutputWriter for writing.
The input file sizes range from 1 MB to 200 MB. The mapper/combiner/reducer functions are very simple string processing with no calls to external services like datastore. Each input record is a string of less than a couple hundred bytes.
The job finishes correctly in the end though I suspect it runs longer than necessary due to all of the instance shutdowns from exceeding memory available. In a job running right now, I've had 12 soft memory limit exceptions that shut down an instance and the job is not yet done.
I'm not sure what is driving all of the memory use. Could it result from buffering on the readers or writers? Other likely causes?
It could be. Python memory is very hard to deal with cleanly. We don't have a memory limiter, like we do in Java: https://github.com/GoogleCloudPlatform/appengine-mapreduce/blob/master/java/src/main/java/com/google/appengine/tools/mapreduce/impl/handlers/MemoryLimiter.java so aside from buffer sizes there are only a few nobs to turn: https://github.com/GoogleCloudPlatform/appengine-mapreduce/blob/2a4e2431607a8cb1ddd821de44c2d2d6991b7a5a/python/src/mapreduce/util.py#L377 https://github.com/GoogleCloudPlatform/appengine-mapreduce/blob/9f0d9a15cc3fc559a6296bbb3bf1676be942c7de/python/src/mapreduce/model.py#L925 https://github.com/GoogleCloudPlatform/appengine-mapreduce/blob/2a4e2431607a8cb1ddd821de44c2d2d6991b7a5a/python/src/mapreduce/context.py#L54
Try playing with the buffer sizes first and see if it helps. What stage did the soft memory limit hit during? It if was during shuffle that might have nothing to do with the buffer, and instead be related to how much data it is reading in a slice.
Had a similar experience trying to do the stock word count on 60MB of Project Gutenberg text. Perhaps because of all the shutdowns (not sure), this module proved too slow and costly to use in production.
How many reduce shards are you running? Each mapper is going to write data out for each reduce shard, and each output file is going to have an associated buffer. So if you have 20 reduce shards, and the buffer size is set to 2MB (I believe that is the default) you could end up with theoretically up to 80MB in just buffers (It does double buffering for performance). That could be really hard to fit in a 128MB VM. (Particularly if the scheduler decides to run multiple requests in the same instance.)
I apologize for not responding to the earlier suggestions. Since I posted the issue, I haven't run a new map reduce process so haven't been able to test the nobs.
I believe my job used around ten shards on the last run but can't remember for sure (could be more; i enforce a 20 shard max limit). Anyway, you might be right as I've noticed that the number of instances running my job rarely exceeds 5 (I use b4 class basic scaling for the mr job) and can drop as low as 1 or 2.
I'm pretty sure the exceptions occur in the mapper stage as well, so I don't think it's exclusively an issue with write buffering during the reduce stage. When I run the process again, I'll keep a closer watch so I can provide more details. I'd run it again now, but it takes a bit of time and money to complete.
B4s should have more than 128mb... Also I should clarify, I was referring to the map stage. Because the files are split out by the mapper according to the reduce shard the output will be going to before sorting, there will be a buffer for each reduce shard that the mapper is observing data for.