appengine-mapreduce
appengine-mapreduce copied to clipboard
MapreducePipeline behavior is inconsistent
In the below snippet from mapreduce_pipeline.py (MapreducePipeline.run()), the mapper_params default to None, showing that they are not always required, but the first line of the constructor immediately de-references it.
def run(self,
job_name,
mapper_spec,
reducer_spec,
input_reader_spec,
output_writer_spec=None,
mapper_params=None,
reducer_params=None,
shards=None,
combiner_spec=None):
# Check that you have a bucket_name set in the mapper_params and set it
# to the default if not.
if mapper_params.get("bucket_name") is None:
try:
mapper_params["bucket_name"] = (
app_identity.get_default_gcs_bucket_name())
Yeah, this should be fixed by simply making mapper_params an empty dict if it comes in as None.